[ { "path": ".asf.yaml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n# This file controls the settings of this repository\n#\n# See more details at\n# https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features\n\nnotifications:\n commits: commits@datafusion.apache.org\n issues: github@datafusion.apache.org\n pullrequests: github@datafusion.apache.org\n discussions: github@datafusion.apache.org\n jira_options: link label worklog\ngithub:\n description: \"Apache DataFusion Comet Spark Accelerator\"\n homepage: https://datafusion.apache.org/comet\n labels:\n - arrow\n - datafusion\n - rust\n - spark\n enabled_merge_buttons:\n squash: true\n merge: false\n rebase: false\n features:\n issues: true\n discussions: true\n protected_branches:\n main:\n required_pull_request_reviews:\n required_approving_review_count: 1\n pull_requests:\n allow_update_branch: true\n# publishes the content of the `asf-site` branch to\n# https://datafusion.apache.org/comet/\npublish:\n whoami: asf-site\n subdir: comet\n" }, { "path": ".claude/skills/audit-comet-expression/SKILL.md", "content": "---\nname: audit-comet-expression\ndescription: Audit an existing Comet expression for correctness and test coverage. Studies the Spark implementation across versions 3.4.3, 3.5.8, and 4.0.1, reviews the Comet and DataFusion implementations, identifies missing test coverage, and offers to implement additional tests.\nargument-hint: \n---\n\nAudit the Comet implementation of the `$ARGUMENTS` expression for correctness and test coverage.\n\n## Overview\n\nThis audit covers:\n\n1. Spark implementation across versions 3.4.3, 3.5.8, and 4.0.1\n2. Comet Scala serde implementation\n3. Comet Rust / DataFusion implementation\n4. Existing test coverage (SQL file tests and Scala tests)\n5. Gap analysis and test recommendations\n\n---\n\n## Step 1: Locate the Spark Implementations\n\nClone specific Spark version tags (use shallow clones to avoid polluting the workspace). Only clone a version if it is not already present.\n\n```bash\nset -eu -o pipefail\nfor tag in v3.4.3 v3.5.8 v4.0.1; do\n dir=\"/tmp/spark-${tag}\"\n if [ ! -d \"$dir\" ]; then\n git clone --depth 1 --branch \"$tag\" https://github.com/apache/spark.git \"$dir\"\n fi\ndone\n```\n\n### Find the expression class in each Spark version\n\nSearch the Catalyst SQL expressions source:\n\n```bash\nfor tag in v3.4.3 v3.5.8 v4.0.1; do\n dir=\"/tmp/spark-${tag}\"\n echo \"=== $tag ===\"\n find \"$dir/sql/catalyst/src/main/scala\" -name \"*.scala\" | \\\n xargs grep -l \"case class $ARGUMENTS\\b\\|object $ARGUMENTS\\b\" 2>/dev/null\ndone\n```\n\nIf the expression is not found in catalyst, also check core:\n\n```bash\nfor tag in v3.4.3 v3.5.8 v4.0.1; do\n dir=\"/tmp/spark-${tag}\"\n echo \"=== $tag ===\"\n find \"$dir/sql\" -name \"*.scala\" | \\\n xargs grep -l \"case class $ARGUMENTS\\b\\|object $ARGUMENTS\\b\" 2>/dev/null\ndone\n```\n\n### Read the Spark source for each version\n\nFor each Spark version, read the expression file and note:\n\n- The `eval`, `nullSafeEval`, and `doGenCode` / `doGenCodeSafe` methods\n- The `inputTypes` and `dataType` fields (accepted input types, return type)\n- Null handling strategy (`nullable`, `nullSafeEval`)\n- ANSI mode behavior (`ansiEnabled`, `failOnError`)\n- Special cases, guards, `require` assertions, and runtime exceptions\n- Any constants or configuration the expression reads\n\n### Compare across Spark versions\n\nProduce a concise diff summary of what changed between:\n\n- 3.4.3 → 3.5.8\n- 3.5.8 → 4.0.1\n\nPay attention to:\n\n- New input types added or removed\n- Behavior changes for edge cases (null, overflow, empty, boundary)\n- New ANSI mode branches\n- New parameters or configuration\n- Breaking API changes that Comet must shim\n\n---\n\n## Step 2: Locate the Spark Tests\n\n```bash\nfor tag in v3.4.3 v3.5.8 v4.0.1; do\n dir=\"/tmp/spark-${tag}\"\n echo \"=== $tag ===\"\n find \"$dir/sql\" -name \"*.scala\" -path \"*/test/*\" | \\\n xargs grep -l \"$ARGUMENTS\" 2>/dev/null\ndone\n```\n\nRead the relevant Spark test files and produce a list of:\n\n- Input types covered\n- Edge cases exercised (null, empty, overflow, negative, boundary values, special characters, etc.)\n- ANSI mode tests\n- Error cases\n\nThis list will be the reference for the coverage gap analysis in Step 5.\n\n---\n\n## Step 3: Locate the Comet Implementation\n\n### Scala serde\n\n```bash\n# Find the serde object\ngrep -r \"$ARGUMENTS\" spark/src/main/scala/org/apache/comet/serde/ --include=\"*.scala\" -l\ngrep -r \"$ARGUMENTS\" spark/src/main/scala/org/apache/comet/ --include=\"*.scala\" -l\n```\n\nRead the serde implementation and check:\n\n- Which Spark versions the serde handles\n- Whether `getSupportLevel` is implemented and accurate\n- Whether all input types are handled\n- Whether any types are explicitly marked `Unsupported`\n\n### Shims\n\n```bash\nfind spark/src/main -name \"CometExprShim.scala\" | xargs grep -l \"$ARGUMENTS\" 2>/dev/null\n```\n\nIf shims exist, read them and note any version-specific handling.\n\n### Rust / DataFusion implementation\n\n```bash\n# Search for the function in native/spark-expr\ngrep -r \"$ARGUMENTS\" native/spark-expr/src/ --include=\"*.rs\" -l\ngrep -r \"$ARGUMENTS\" native/core/src/ --include=\"*.rs\" -l\n```\n\nIf the expression delegates to DataFusion, find it there too. Set `$DATAFUSION_SRC` to a local DataFusion checkout, or fall back to searching the cargo registry:\n\n```bash\nif [ -n \"${DATAFUSION_SRC:-}\" ]; then\n grep -r \"$ARGUMENTS\" \"$DATAFUSION_SRC\" --include=\"*.rs\" -l 2>/dev/null | head -10\nelse\n # Fall back to cargo registry (may include unrelated crates)\n grep -r \"$ARGUMENTS\" ~/.cargo/registry/src/*/datafusion* --include=\"*.rs\" -l 2>/dev/null | head -10\nfi\n```\n\nRead the Rust implementation and check:\n\n- Null handling (does it propagate nulls correctly?)\n- Overflow and underflow handling (returns `Err` vs panics)\n- Type dispatch (does it handle all types that Spark supports?)\n- ANSI / fail-on-error mode\n\n---\n\n## Step 4: Locate Existing Comet Tests\n\n### SQL file tests\n\n```bash\n# Find SQL test files for this expression\nfind spark/src/test/resources/sql-tests/expressions/ -name \"*.sql\" | \\\n xargs grep -l \"$ARGUMENTS\" 2>/dev/null\n\n# Also check if there's a dedicated file\nfind spark/src/test/resources/sql-tests/expressions/ -name \"*$(echo $ARGUMENTS | tr '[:upper:]' '[:lower:]')*\"\n```\n\nRead every SQL test file found and list:\n\n- Table schemas and data values used\n- Queries exercised\n- Query modes used (`query`, `spark_answer_only`, `tolerance`, `ignore`, `expect_error`)\n- Any ConfigMatrix directives\n\n### Scala tests\n\n```bash\ngrep -r \"$ARGUMENTS\" spark/src/test/scala/ --include=\"*.scala\" -l\n```\n\nRead the relevant Scala test files and list:\n\n- Input types covered\n- Edge cases exercised\n- Whether constant folding is disabled for literal tests\n\n---\n\n## Step 5: Gap Analysis\n\nCompare the Spark test coverage (Step 2) against the Comet test coverage (Step 4). Produce a structured gap report:\n\n### Coverage matrix\n\nFor each of the following dimensions, note whether it is covered in Comet tests or missing:\n\n| Dimension | Spark tests it | Comet SQL test | Comet Scala test | Gap? |\n| ------------------------------------------------------------------------------------------------------ | -------------- | -------------- | ---------------- | ---- |\n| Column reference argument(s) | | | | |\n| Literal argument(s) | | | | |\n| NULL input | | | | |\n| Empty string / empty array / empty map | | | | |\n| Array/map with NULL elements | | | | |\n| Zero, negative zero, negative values (numeric) | | | | |\n| Underflow, overflow | | | | |\n| Boundary values (INT_MIN, INT_MAX, Long.MinValue, minimum positive, etc.) | | | | |\n| NaN, Infinity, -Infinity, subnormal (float/double) | | | | |\n| Multibyte / special UTF-8 (composed vs decomposed, e.g. `é` U+00E9 vs `e` + U+0301, non-Latin scripts) | | | | |\n| ANSI mode (failOnError=true) | | | | |\n| Non-ANSI mode (failOnError=false) | | | | |\n| All supported input types | | | | |\n| Parquet dictionary encoding (ConfigMatrix) | | | | |\n| Cross-version behavior differences | | | | |\n\n### Implementation gaps\n\nAlso review the Comet implementation (Step 3) against the Spark behavior (Step 1):\n\n- Are there input types that Spark supports but `getSupportLevel` returns `Unsupported` without comment?\n- Are there behavioral differences that are NOT marked `Incompatible` but should be?\n- Are there behavioral differences between Spark versions that the Comet implementation does not account for (missing shim)?\n- Does the Rust implementation match the Spark behavior for all edge cases?\n\n---\n\n## Step 6: Recommendations\n\nSummarize findings as a prioritized list.\n\n### High priority\n\nIssues where Comet may silently produce wrong results compared to Spark.\n\n### Medium priority\n\nMissing test coverage for edge cases that could expose bugs.\n\n### Low priority\n\nMinor gaps, cosmetic improvements, or nice-to-have tests.\n\n---\n\n## Step 7: Offer to Implement Missing Tests\n\nAfter presenting the gap analysis, ask the user:\n\n> I found the following missing test cases. Would you like me to implement them?\n>\n> - [list each missing test case]\n>\n> I can add them as SQL file tests in `spark/src/test/resources/sql-tests/expressions//$ARGUMENTS.sql`\n> (or as Scala tests in `CometExpressionSuite` for cases that require programmatic setup).\n\nIf the user says yes, implement the missing tests following the SQL file test format described in\n`docs/source/contributor-guide/sql-file-tests.md`. Prefer SQL file tests over Scala tests.\n\n### SQL file test template\n\n```sql\n-- Licensed to the Apache Software Foundation (ASF) under one\n-- or more contributor license agreements. See the NOTICE file\n-- distributed with this work for additional information\n-- regarding copyright ownership. The ASF licenses this file\n-- to you under the Apache License, Version 2.0 (the\n-- \"License\"); you may not use this file except in compliance\n-- with the License. You may obtain a copy of the License at\n--\n-- http://www.apache.org/licenses/LICENSE-2.0\n--\n-- Unless required by applicable law or agreed to in writing,\n-- software distributed under the License is distributed on an\n-- \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n-- KIND, either express or implied. See the License for the\n-- specific language governing permissions and limitations\n-- under the License.\n\n-- ConfigMatrix: parquet.enable.dictionary=false,true\n\nstatement\nCREATE TABLE test_$ARGUMENTS(...) USING parquet\n\nstatement\nINSERT INTO test_$ARGUMENTS VALUES\n (...),\n (NULL)\n\n-- column argument\nquery\nSELECT $ARGUMENTS(col) FROM test_$ARGUMENTS\n\n-- literal arguments\nquery\nSELECT $ARGUMENTS('value'), $ARGUMENTS(''), $ARGUMENTS(NULL)\n```\n\n### Verify the tests pass\n\nAfter implementing tests, tell the user how to run them:\n\n```bash\n./mvnw test -DwildcardSuites=\"CometSqlFileTestSuite\" -Dsuites=\"org.apache.comet.CometSqlFileTestSuite $ARGUMENTS\" -Dtest=none\n```\n\n---\n\n## Step 8: Update the Expression Audit Log\n\nAfter completing the audit (whether or not tests were added), append a row to the audit log at\n`docs/source/contributor-guide/expression-audit-log.md`.\n\nThe row should include:\n\n- Expression name\n- Spark versions checked (e.g. 3.4.3, 3.5.8, 4.0.1)\n- Today's date\n- A brief summary of findings (behavioral differences, bugs found/fixed, tests added, known incompatibilities)\n\n---\n\n## Output Format\n\nPresent the audit as:\n\n1. **Expression Summary** - Brief description of what `$ARGUMENTS` does, its input/output types, and null behavior\n2. **Spark Version Differences** - Summary of any behavioral or API differences across Spark 3.4.3, 3.5.8, and 4.0.1\n3. **Comet Implementation Notes** - Summary of how Comet implements this expression and any concerns\n4. **Coverage Gap Analysis** - The gap table from Step 5, plus implementation gaps\n5. **Recommendations** - Prioritized list from Step 6\n6. **Offer to add tests** - The prompt from Step 7\n\n## Tone and Style\n\n- Write in clear, concise prose\n- Use backticks around code references (function names, file paths, class names, types, config keys)\n- Avoid robotic or formulaic language\n- Be constructive and acknowledge what is already well-covered before raising gaps\n- Avoid em dashes and semicolons; use separate sentences instead\n" }, { "path": ".claude/skills/review-comet-pr/SKILL.md", "content": "---\nname: review-comet-pr\ndescription: Review a DataFusion Comet pull request for Spark compatibility and implementation correctness. Provides guidance to a reviewer rather than posting comments directly.\nargument-hint: \n---\n\nReview Comet PR #$ARGUMENTS\n\n## Before You Start\n\n### Gather PR Metadata\n\nFetch the PR details to understand the scope:\n\n```bash\ngh pr view $ARGUMENTS --repo apache/datafusion-comet --json title,body,author,isDraft,state,files\n```\n\n### Review Existing Comments First\n\nBefore forming your review:\n\n1. **Read all existing review comments** on the PR\n2. **Check the conversation tab** for any discussion\n3. **Avoid duplicating feedback** that others have already provided\n4. **Build on existing discussions** rather than starting new threads on the same topic\n5. **If you have no additional concerns beyond what's already discussed, say so**\n6. **Ignore Copilot reviews** - do not reference or build upon comments from GitHub Copilot\n\n```bash\n# View existing comments on a PR\ngh pr view $ARGUMENTS --repo apache/datafusion-comet --comments\n```\n\n---\n\n## Review Workflow\n\n### 1. Gather Context\n\nRead the changed files and understand the area of the codebase being modified:\n\n```bash\n# View the diff\ngh pr diff $ARGUMENTS --repo apache/datafusion-comet\n```\n\nFor expression PRs, check how similar expressions are implemented in the codebase. Look at the serde files in `spark/src/main/scala/org/apache/comet/serde/` and Rust implementations in `native/spark-expr/src/`.\n\n### 2. Read Spark Source (Expression PRs)\n\n**For any PR that adds or modifies an expression, you must read the Spark source code to understand the canonical behavior.** This is the authoritative reference for what Comet must match.\n\n1. **Clone or update the Spark repo:**\n\n ```bash\n # Clone if not already present (use /tmp to avoid polluting the workspace)\n if [ ! -d /tmp/spark ]; then\n git clone --depth 1 https://github.com/apache/spark.git /tmp/spark\n fi\n ```\n\n2. **Find the expression implementation in Spark:**\n\n ```bash\n # Search for the expression class (e.g., for \"Conv\", \"Hex\", \"Substring\")\n find /tmp/spark/sql/catalyst/src/main/scala -name \"*.scala\" | xargs grep -l \"case class \"\n ```\n\n3. **Read the Spark implementation carefully.** Pay attention to:\n - The `eval` and `doGenEval`/`nullSafeEval` methods. These define the exact behavior.\n - The `inputTypes` and `dataType` fields. These define which types Spark accepts and what it returns.\n - Null handling. Does it use `nullable = true`? Does `nullSafeEval` handle nulls implicitly?\n - Special cases, guards, and `require` assertions.\n - ANSI mode branches (look for `SQLConf.get.ansiEnabled` or `failOnError`).\n\n4. **Read the Spark tests for the expression:**\n\n ```bash\n # Find test files\n find /tmp/spark/sql -name \"*.scala\" -path \"*/test/*\" | xargs grep -l \"\"\n ```\n\n5. **Compare the Spark behavior against the Comet implementation in the PR.** Identify:\n - Edge cases tested in Spark but not in the PR\n - Data types supported in Spark but not handled in the PR\n - Behavioral differences that should be marked `Incompatible`\n\n6. **Suggest additional tests** for any edge cases or type combinations covered in Spark's tests that are missing from the PR's tests.\n\n### 3. Spark Compatibility Check\n\n**This is the most critical aspect of Comet reviews.** Comet must produce identical results to Spark.\n\nFor expression PRs, verify against the Spark source you read in step 2:\n\n1. **Check edge cases**\n - Null handling\n - Overflow behavior\n - Empty input behavior\n - Type-specific behavior\n\n2. **Verify all data types are handled**\n - Does Spark support this type? (Check `inputTypes` in Spark source)\n - Does the PR handle all Spark-supported types?\n\n3. **Check for ANSI mode differences**\n - Spark behavior may differ between legacy and ANSI modes\n - PR should handle both or mark as `Incompatible`\n\n### 4. Check Against Implementation Guidelines\n\n**Always verify PRs follow the implementation guidelines.**\n\n#### Scala Serde (`spark/src/main/scala/org/apache/comet/serde/`)\n\n- [ ] Expression class correctly identified\n- [ ] All child expressions converted via `exprToProtoInternal`\n- [ ] Return type correctly serialized\n- [ ] `getSupportLevel` reflects true compatibility:\n - `Compatible()` - matches Spark exactly\n - `Incompatible(Some(\"reason\"))` - differs in documented ways\n - `Unsupported(Some(\"reason\"))` - cannot be implemented\n- [ ] Serde in appropriate file (`datetime.scala`, `strings.scala`, `arithmetic.scala`, etc.)\n\n#### Registration (`QueryPlanSerde.scala`)\n\n- [ ] Added to correct map (temporal, string, arithmetic, etc.)\n- [ ] No duplicate registrations\n- [ ] Import statement added\n\n#### Rust Implementation (if applicable)\n\nLocation: `native/spark-expr/src/`\n\n- [ ] Matches DataFusion and Arrow conventions\n- [ ] Null handling is correct\n- [ ] No panics. Use `Result` types.\n- [ ] Efficient array operations (avoid row-by-row)\n\n#### Tests - Prefer SQL File-Based Framework\n\n**Expression tests should use the SQL file-based framework (`CometSqlFileTestSuite`) where possible.** This framework automatically runs each query through both Spark and Comet and compares results. No Scala code is needed. Only fall back to Scala tests in `CometExpressionSuite` when the SQL framework cannot express the test. Examples include complex `DataFrame` setup, programmatic data generation, or non-expression tests.\n\n**SQL file test location:** `spark/src/test/resources/sql-tests/expressions//`\n\nCategories include: `aggregate/`, `array/`, `string/`, `math/`, `struct/`, `map/`, `datetime/`, `hash/`, etc.\n\n**SQL file structure:**\n\n```sql\n-- Create test data\nstatement\nCREATE TABLE test_crc32(col string, a int, b float) USING parquet\n\nstatement\nINSERT INTO test_crc32 VALUES ('Spark', 10, 1.5), (NULL, NULL, NULL), ('', 0, 0.0)\n\n-- Default mode: verifies native Comet execution + result matches Spark\nquery\nSELECT crc32(col) FROM test_crc32\n\n-- spark_answer_only: compares results without requiring native execution\nquery spark_answer_only\nSELECT crc32(cast(a as string)) FROM test_crc32\n\n-- tolerance: allows numeric variance for floating-point results\nquery tolerance=0.0001\nSELECT cos(v) FROM test_trig\n\n-- expect_fallback: asserts fallback to Spark occurs\nquery expect_fallback(unsupported expression)\nSELECT unsupported_func(v) FROM test_table\n\n-- expect_error: verifies both engines throw matching exceptions\nquery expect_error(ARITHMETIC_OVERFLOW)\nSELECT 2147483647 + 1\n\n-- ignore: skip queries with known bugs (include GitHub issue link)\nquery ignore(https://github.com/apache/datafusion-comet/issues/NNNN)\nSELECT known_buggy_expr(v) FROM test_table\n```\n\n**Running SQL file tests:**\n\n```bash\n# All SQL file tests\n./mvnw test -Dsuites=\"org.apache.comet.CometSqlFileTestSuite\" -Dtest=none\n\n# Specific test file (substring match)\n./mvnw test -Dsuites=\"org.apache.comet.CometSqlFileTestSuite crc32\" -Dtest=none\n```\n\n**CRITICAL: Verify all test requirements (regardless of framework):**\n\n- [ ] Basic functionality tested (column data, not just literals)\n- [ ] Null handling tested (`SELECT expression(NULL)`)\n- [ ] Edge cases tested (empty input, overflow, boundary values)\n- [ ] Both literal values and column references tested (they use different code paths)\n- [ ] For timestamp/datetime expressions, timezone handling is tested (e.g., UTC, non-UTC session timezone, timestamps with and without timezone)\n- [ ] One expression per SQL file for easier debugging\n- [ ] If using Scala tests instead, literal tests MUST disable constant folding:\n ```scala\n withSQLConf(SQLConf.OPTIMIZER_EXCLUDED_RULES.key ->\n \"org.apache.spark.sql.catalyst.optimizer.ConstantFolding\") {\n checkSparkAnswerAndOperator(\"SELECT func(literal)\")\n }\n ```\n\n### 5. Performance Review (Expression PRs)\n\n**For PRs that add new expressions, performance is not optional.** The whole point of Comet is to be faster than Spark. If a new expression is not faster, it may not be worth adding.\n\n1. **Check that the PR includes microbenchmark results.** The PR description should contain benchmark numbers comparing Comet vs Spark for the new expression. If benchmark results are missing, flag this as a required addition.\n\n2. **Look for a microbenchmark implementation.** Expression benchmarks live in `spark/src/test/scala/org/apache/spark/sql/benchmark/`. Check whether the PR adds a benchmark for the new expression.\n\n3. **Review the benchmark results if provided:**\n - Is Comet actually faster than Spark for this expression?\n - Are the benchmarks representative? They should test with realistic data sizes, not just trivial inputs.\n - Are different data types benchmarked if the expression supports multiple types?\n\n4. **Review the Rust implementation for performance concerns:**\n - Unnecessary allocations or copies\n - Row-by-row processing where batch/array operations are possible\n - Redundant type conversions\n - Inefficient string handling (e.g., repeated UTF-8 validation)\n - Missing use of Arrow compute kernels where they exist\n\n5. **If benchmark results show Comet is slower than Spark**, flag this clearly. The PR should explain why the regression is acceptable or include a plan to optimize.\n\n### 6. Check CI Test Failures\n\n**Always check the CI status and summarize any test failures in your review.**\n\n```bash\n# View CI check status\ngh pr checks $ARGUMENTS --repo apache/datafusion-comet\n\n# View failed check details\ngh pr checks $ARGUMENTS --repo apache/datafusion-comet --failed\n```\n\n### 7. Documentation Check\n\nCheck whether the PR requires updates to user-facing documentation in `docs/`:\n\n- **Compatibility guide** (`docs/source/user-guide/compatibility.md`): New expressions or operators should be listed. Incompatible behaviors should be documented.\n- **Configuration guide** (`docs/source/user-guide/configs.md`): New config options should be documented.\n- **Expressions list** (`docs/source/user-guide/expressions.md`): New expressions should be added.\n\nIf the PR adds a new expression or operator but does not update the relevant docs, flag this as something that needs to be addressed.\n\n### 8. Common Comet Review Issues\n\n1. **Incomplete type support**: Spark expression supports types not handled in PR\n2. **Missing edge cases**: Null, overflow, empty string, negative values\n3. **Wrong return type**: Return type must match Spark exactly\n4. **Tests in wrong framework**: Expression tests should use the SQL file-based framework (`CometSqlFileTestSuite`) rather than adding to Scala test suites like `CometExpressionSuite`. Suggest migration if the PR adds Scala tests for expressions that could use SQL files instead.\n5. **Stale native code**: PR might need `./mvnw install -pl common -DskipTests`\n6. **Missing `getSupportLevel`**: Edge cases should be marked as `Incompatible`\n\n---\n\n## Output Format\n\nPresent your review as guidance for the reviewer. Structure your output as:\n\n1. **PR Summary** - Brief description of what the PR does\n2. **CI Status** - Summary of CI check results\n3. **Findings** - Your analysis organized by area (Spark compatibility, implementation, tests, etc.)\n4. **Suggested Review Comments** - Specific comments the reviewer could leave on the PR, with file and line references where applicable\n\n## Review Tone and Style\n\nWrite reviews that sound human and conversational. Avoid:\n\n- Robotic or formulaic language\n- Em dashes. Use separate sentences instead.\n- Semicolons. Use separate sentences instead.\n\nInstead:\n\n- Write in flowing paragraphs using simple grammar\n- Keep sentences short and separate rather than joining them with punctuation\n- Be kind and constructive, even when raising concerns\n- Use backticks around any code references (function names, file paths, class names, types, config keys, etc.)\n- **Suggest** adding tests rather than stating tests are missing (e.g., \"It might be worth adding a test for X\" not \"Tests are missing for X\")\n- **Ask questions** about edge cases rather than asserting they aren't handled (e.g., \"Does this handle the case where X is null?\" not \"This doesn't handle null\")\n- Frame concerns as questions or suggestions when possible\n- Acknowledge what the PR does well before raising concerns\n\n## Do Not Post Comments\n\n**IMPORTANT: Never post comments or reviews on the PR directly.** This skill is for providing guidance to a human reviewer. Present all findings and suggested comments to the user. The user will decide what to post.\n" }, { "path": ".dockerignore", "content": ".git\n.github\n.idea\nbin\nconf\ndocs/build\ndocs/temp\ndocs/venv\nmetastore_db\ntarget\ncommon/target\nspark-integration/target\nfuzz-testing/target\nspark/target\nnative/target\ncore/target\nspark-warehouse\nvenv\n" }, { "path": ".github/ISSUE_TEMPLATE/bug_report.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: Bug report\ndescription: Create a bug report\nlabels: bug\nbody:\n - type: textarea\n attributes:\n label: Describe the bug\n description: Describe the bug.\n placeholder: >\n A clear and concise description of what the bug is.\n validations:\n required: true\n - type: textarea\n attributes:\n label: Steps to reproduce\n placeholder: >\n Describe steps to reproduce the bug:\n - type: textarea\n attributes:\n label: Expected behavior\n placeholder: >\n A clear and concise description of what you expected to happen.\n - type: textarea\n attributes:\n label: Additional context\n placeholder: >\n Add any other context about the problem here.\n" }, { "path": ".github/ISSUE_TEMPLATE/feature_request.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: Feature request\ndescription: Suggest an idea for this project\nlabels: enhancement\nbody:\n - type: textarea\n attributes:\n label: What is the problem the feature request solves?\n description: Please describe how feature request improves the Comet.\n placeholder: >\n A clear and concise description of what the improvement is. Ex. I'm always frustrated when [...]\n (This section helps Comet developers understand the context and *why* for this feature, in addition to the *what*)\n - type: textarea\n attributes:\n label: Describe the potential solution\n placeholder: >\n A clear and concise description of what you want to happen.\n - type: textarea\n attributes:\n label: Additional context\n placeholder: >\n Add any other context or screenshots about the feature request here.\n" }, { "path": ".github/actions/java-test/action.yaml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: \"Java Test\"\ndescription: \"Run Java tests\"\ninputs:\n artifact_name:\n description: \"Unique name for uploaded artifacts for this run\"\n required: true\n suites:\n description: 'Which Scalatest test suites to run'\n required: false\n default: ''\n maven_opts:\n description: 'Maven options passed to the mvn command'\n required: false\n default: ''\n scan_impl:\n description: 'The default Parquet scan implementation'\n required: false\n default: 'auto'\n upload-test-reports:\n description: 'Whether to upload test results including coverage to GitHub'\n required: false\n default: 'false'\n skip-native-build:\n description: 'Skip native build (when using pre-built artifact)'\n required: false\n default: 'false'\n\nruns:\n using: \"composite\"\n steps:\n - name: Run Cargo release build\n if: ${{ inputs.skip-native-build != 'true' }}\n shell: bash\n # it is important that we run the Scala tests against a release build rather than a debug build\n # to make sure that no tests are relying on overflow checks that are present only in debug builds\n run: |\n cd native\n cargo build --release\n\n - name: Cache Maven dependencies\n # TODO: remove next line after working again\n # temporarily work around https://github.com/actions/runner-images/issues/13341\n # by disabling caching for macOS\n if: ${{ runner.os != 'macOS' }}\n uses: actions/cache@v5\n with:\n path: |\n ~/.m2/repository\n /root/.m2/repository\n key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}\n restore-keys: |\n ${{ runner.os }}-java-maven-\n\n - name: Run all tests\n shell: bash\n if: ${{ inputs.suites == '' }}\n env:\n COMET_PARQUET_SCAN_IMPL: ${{ inputs.scan_impl }}\n SPARK_LOCAL_HOSTNAME: \"localhost\"\n SPARK_LOCAL_IP: \"127.0.0.1\"\n run: |\n MAVEN_OPTS=\"-Xmx4G -Xms2G -XX:+UnlockDiagnosticVMOptions -XX:+ShowMessageBoxOnError -XX:+HeapDumpOnOutOfMemoryError -XX:ErrorFile=./hs_err_pid%p.log\" SPARK_HOME=`pwd` ./mvnw -B -Prelease install ${{ inputs.maven_opts }}\n - name: Run specified tests\n shell: bash\n if: ${{ inputs.suites != '' }}\n env:\n COMET_PARQUET_SCAN_IMPL: ${{ inputs.scan_impl }}\n SPARK_LOCAL_HOSTNAME: \"localhost\"\n SPARK_LOCAL_IP: \"127.0.0.1\"\n run: |\n MAVEN_SUITES=\"$(echo \"${{ inputs.suites }}\" | paste -sd, -)\"\n echo \"Running with MAVEN_SUITES=$MAVEN_SUITES\"\n MAVEN_OPTS=\"-Xmx4G -Xms2G -DwildcardSuites=$MAVEN_SUITES -XX:+UnlockDiagnosticVMOptions -XX:+ShowMessageBoxOnError -XX:+HeapDumpOnOutOfMemoryError -XX:ErrorFile=./hs_err_pid%p.log\" SPARK_HOME=`pwd` ./mvnw -B -Prelease install ${{ inputs.maven_opts }}\n - name: Upload crash logs\n if: failure()\n uses: actions/upload-artifact@v6\n with:\n name: crash-logs-${{ inputs.artifact_name }}\n path: \"**/hs_err_pid*.log\"\n - name: Debug listing\n if: failure()\n shell: bash\n run: | \n echo \"CWD: $(pwd)\"\n ls -lah .\n ls -lah target\n find . -name 'unit-tests.log'\n - name: Upload unit-tests.log\n if: failure()\n uses: actions/upload-artifact@v6\n with:\n name: unit-tests-${{ inputs.artifact_name }}\n path: \"**/target/unit-tests.log\"\n - name: Upload test results\n if: ${{ inputs.upload-test-reports == 'true' }}\n uses: actions/upload-artifact@v6\n with:\n name: java-test-reports-${{ inputs.artifact_name }}\n path: \"**/target/surefire-reports/*.txt\"\n retention-days: 7 # 1 week for test reports\n overwrite: true\n" }, { "path": ".github/actions/rust-test/action.yaml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: \"Rust Test\"\ndescription: \"Run Rust tests\"\n\nruns:\n using: \"composite\"\n steps:\n # Note: cargo fmt check is now handled by the lint job that gates this workflow\n\n - name: Check Cargo clippy\n shell: bash\n run: |\n cd native\n cargo clippy --color=never --all-targets --workspace -- -D warnings\n\n - name: Check compilation\n shell: bash\n run: |\n cd native\n cargo check --benches\n\n - name: Check unused dependencies\n shell: bash\n run: |\n cd native\n cargo install cargo-machete --version 0.7.0 && cargo machete\n\n - name: Cache Maven dependencies\n uses: actions/cache@v4\n with:\n path: |\n ~/.m2/repository\n /root/.m2/repository\n key: ${{ runner.os }}-rust-maven-${{ hashFiles('**/pom.xml') }}\n restore-keys: |\n ${{ runner.os }}-rust-maven-\n\n - name: Build common module (pre-requisite for Rust tests)\n shell: bash\n run: |\n cd common\n ../mvnw -B clean compile -DskipTests\n\n - name: Install nextest\n shell: bash\n run: |\n cargo install cargo-nextest --locked\n\n - name: Run Cargo test\n shell: bash\n run: |\n cd native\n # Set LD_LIBRARY_PATH to include JVM library path for tests that use JNI\n export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH}\n RUST_BACKTRACE=1 cargo nextest run\n\n" }, { "path": ".github/actions/setup-builder/action.yaml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: Prepare Builder\ndescription: 'Prepare Build Environment'\ninputs:\n rust-version:\n description: 'version of rust to install (e.g. nightly)'\n required: true\n default: 'stable'\n jdk-version:\n description: 'jdk version to install (e.g., 17)'\n required: true\n default: '17'\nruns:\n using: \"composite\"\n steps:\n - name: Install Build Dependencies\n shell: bash\n run: |\n apt-get update\n apt-get install -y protobuf-compiler\n apt-get install -y clang\n\n - name: Install JDK ${{inputs.jdk-version}}\n uses: actions/setup-java@v4\n with:\n # distribution is chosen to be zulu as it still offers JDK 8 with Silicon support, which\n # is not available in the adopt distribution\n distribution: 'zulu'\n java-version: ${{inputs.jdk-version}}\n\n - name: Set JAVA_HOME\n shell: bash\n run: echo \"JAVA_HOME=$(echo ${JAVA_HOME})\" >> $GITHUB_ENV\n\n - name: Setup Rust toolchain\n shell: bash\n # rustfmt is needed for the substrait build script\n run: |\n echo \"Installing ${{inputs.rust-version}}\"\n rustup toolchain install ${{inputs.rust-version}}\n rustup default ${{inputs.rust-version}}\n rustup component add rustfmt clippy\n" }, { "path": ".github/actions/setup-iceberg-builder/action.yaml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: Setup Iceberg Builder\ndescription: 'Setup Apache Iceberg to run Spark SQL tests'\ninputs:\n iceberg-version:\n description: 'The Apache Iceberg version (e.g., 1.8.1) to build'\n required: true\nruns:\n using: \"composite\"\n steps:\n - name: Clone Iceberg repo\n uses: actions/checkout@v6\n with:\n repository: apache/iceberg\n path: apache-iceberg\n ref: apache-iceberg-${{inputs.iceberg-version}}\n fetch-depth: 1\n\n - name: Setup Iceberg for Comet\n shell: bash\n run: |\n cd apache-iceberg\n git apply ../dev/diffs/iceberg/${{inputs.iceberg-version}}.diff\n" }, { "path": ".github/actions/setup-macos-builder/action.yaml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: Prepare Builder for MacOS\ndescription: 'Prepare Build Environment'\ninputs:\n rust-version:\n description: 'version of rust to install (e.g. nightly)'\n required: true\n default: 'stable'\n jdk-version:\n description: 'jdk version to install (e.g., 17)'\n required: true\n default: '17'\n jdk-architecture:\n description: 'OS architecture for the JDK'\n required: true\n default: 'x64'\n protoc-architecture:\n description: 'OS architecture for protobuf compiler'\n required: true\n default: 'x86_64'\n\nruns:\n using: \"composite\"\n steps:\n - name: Install Build Dependencies\n shell: bash\n run: |\n # Install protobuf\n mkdir -p $HOME/d/protoc\n cd $HOME/d/protoc\n export PROTO_ZIP=\"protoc-21.4-osx-${{inputs.protoc-architecture}}.zip\"\n curl -LO https://github.com/protocolbuffers/protobuf/releases/download/v21.4/$PROTO_ZIP\n unzip $PROTO_ZIP\n echo \"$HOME/d/protoc/bin\" >> $GITHUB_PATH\n export PATH=$PATH:$HOME/d/protoc/bin\n # install openssl and setup DYLD_LIBRARY_PATH\n brew install openssl\n OPENSSL_LIB_PATH=`brew --prefix openssl`/lib\n echo \"openssl lib path is: ${OPENSSL_LIB_PATH}\"\n echo \"DYLD_LIBRARY_PATH=$OPENSSL_LIB_PATH:$DYLD_LIBRARY_PATH\" >> $GITHUB_ENV\n # output the current status of SIP for later debugging\n csrutil status || true\n\n - name: Install JDK ${{inputs.jdk-version}}\n uses: actions/setup-java@v4\n with:\n # distribution is chosen to be zulu as it still offers JDK 8 with Silicon support, which\n # is not available in the adopt distribution\n distribution: 'zulu'\n java-version: ${{inputs.jdk-version}}\n architecture: ${{inputs.jdk-architecture}}\n\n - name: Set JAVA_HOME\n shell: bash\n run: echo \"JAVA_HOME=$(echo ${JAVA_HOME})\" >> $GITHUB_ENV\n\n - name: Setup Rust toolchain\n shell: bash\n # rustfmt is needed for the substrait build script\n run: |\n echo \"Installing ${{inputs.rust-version}}\"\n rustup toolchain install ${{inputs.rust-version}}\n rustup default ${{inputs.rust-version}}\n rustup component add rustfmt clippy\n" }, { "path": ".github/actions/setup-spark-builder/action.yaml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: Setup Spark Builder\ndescription: 'Setup Apache Spark to run SQL tests'\ninputs:\n spark-short-version:\n description: 'The Apache Spark short version (e.g., 3.5) to build'\n required: true\n spark-version:\n description: 'The Apache Spark version (e.g., 3.5.8) to build'\n required: true\n skip-native-build:\n description: 'Skip native build (when using pre-built artifact)'\n required: false\n default: 'false'\nruns:\n using: \"composite\"\n steps:\n - name: Clone Spark repo\n uses: actions/checkout@v6\n with:\n repository: apache/spark\n path: apache-spark\n ref: v${{inputs.spark-version}}\n fetch-depth: 1\n\n - name: Setup Spark for Comet\n shell: bash\n run: |\n cd apache-spark\n git apply ../dev/diffs/${{inputs.spark-version}}.diff\n\n - name: Cache Maven dependencies\n uses: actions/cache@v4\n with:\n path: |\n ~/.m2/repository\n /root/.m2/repository\n key: ${{ runner.os }}-spark-sql-${{ hashFiles('spark/**/pom.xml', 'common/**/pom.xml') }}\n restore-keys: |\n ${{ runner.os }}-spark-sql-\n\n - name: Build Comet (with native)\n if: ${{ inputs.skip-native-build != 'true' }}\n shell: bash\n run: |\n PROFILES=\"-Pspark-${{inputs.spark-short-version}}\" make release\n\n - name: Build Comet (Maven only, skip native)\n if: ${{ inputs.skip-native-build == 'true' }}\n shell: bash\n run: |\n # Native library should already be in native/target/release/\n ./mvnw install -Prelease -DskipTests -Pspark-${{inputs.spark-short-version}}\n" }, { "path": ".github/dependabot.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nversion: 2\nupdates:\n - package-ecosystem: cargo\n directory: \"/native\"\n schedule:\n interval: weekly\n target-branch: main\n labels: [dependencies]\n ignore:\n # major version bumps of datafusion*, arrow*, parquet are handled manually\n - dependency-name: \"arrow*\"\n update-types: [\"version-update:semver-major\"]\n - dependency-name: \"parquet\"\n update-types: [\"version-update:semver-major\"]\n - dependency-name: \"datafusion*\"\n update-types: [\"version-update:semver-major\"]\n groups:\n proto:\n applies-to: version-updates\n patterns:\n - \"prost*\"\n - \"pbjson*\"\n # Catch-all: group only minor/patch into a single PR,\n # excluding deps we want always separate (and excluding arrow/parquet which have their own group)\n all-other-cargo-deps:\n applies-to: version-updates\n patterns:\n - \"*\"\n exclude-patterns:\n - \"arrow*\"\n - \"parquet\"\n - \"object_store\"\n - \"sqlparser\"\n - \"prost*\"\n - \"pbjson*\"\n update-types:\n - \"minor\"\n - \"patch\"\n - package-ecosystem: \"github-actions\"\n directory: \"/\"\n schedule:\n interval: \"weekly\"\n open-pull-requests-limit: 10\n labels: [dependencies]\n" }, { "path": ".github/pull_request_template.md", "content": "## Which issue does this PR close?\n\n\n\nCloses #.\n\n## Rationale for this change\n\n\n\n## What changes are included in this PR?\n\n\n\n## How are these changes tested?\n\n\n" }, { "path": ".github/workflows/codeql.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\nname: \"CodeQL\"\n\non:\n push:\n branches: [ \"main\" ]\n pull_request:\n branches: [ \"main\" ]\n schedule:\n - cron: '16 4 * * 1'\n\npermissions:\n contents: read\n\njobs:\n analyze:\n name: Analyze Actions\n runs-on: ubuntu-latest\n permissions:\n contents: read\n security-events: write\n packages: read\n\n steps:\n - name: Checkout repository\n uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6\n with:\n persist-credentials: false\n\n - name: Initialize CodeQL\n uses: github/codeql-action/init@95e58e9a2cdfd71adc6e0353d5c52f41a045d225 # v4\n with:\n languages: actions\n\n - name: Perform CodeQL Analysis\n uses: github/codeql-action/analyze@95e58e9a2cdfd71adc6e0353d5c52f41a045d225 # v4\n with:\n category: \"/language:actions\"\n" }, { "path": ".github/workflows/docker-publish.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: Publish Docker images\n\nconcurrency:\n group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}\n cancel-in-progress: true\n\non:\n push:\n tags:\n - '*.*.*'\n - '*.*.*-rc*'\n - 'test-docker-publish-*'\n\njobs:\n docker:\n name: Docker\n if: ${{ startsWith(github.repository, 'apache/') }}\n runs-on: ubuntu-22.04\n permissions:\n contents: read\n packages: write\n steps:\n - name: Remove unnecessary files\n run: |\n echo \"Disk space before cleanup:\"\n df -h\n docker system prune -af\n sudo rm -rf /tmp/*\n sudo rm -rf /opt/hostedtoolcache\n sudo rm -rf \"$AGENT_TOOLSDIRECTORY\"\n sudo apt-get clean\n echo \"Disk space after cleanup:\"\n df -h\n - name: Set up Java\n uses: actions/setup-java@v5\n with:\n java-version: '17'\n distribution: 'temurin'\n cache: 'maven'\n - name: Extract Comet version\n id: extract_version\n run: |\n # use the tag that triggered this workflow as the Comet version e.g. 0.4.0-rc1\n echo \"COMET_VERSION=${GITHUB_REF##*/}\" >> $GITHUB_ENV\n - name: Echo Comet version\n run: echo \"The current Comet version is ${{ env.COMET_VERSION }}\"\n - name: Set up Docker Buildx\n uses: docker/setup-buildx-action@v4\n - name: Login to GitHub Container Registry\n uses: docker/login-action@v4\n with:\n registry: ghcr.io\n username: ${{ github.actor }}\n password: ${{ secrets.GITHUB_TOKEN }}\n - name: Build and push\n uses: docker/build-push-action@v7\n with:\n platforms: linux/amd64,linux/arm64\n push: true\n tags: ghcr.io/apache/datafusion-comet:spark-3.5-scala-2.12-${{ env.COMET_VERSION }}\n file: kube/Dockerfile\n no-cache: true\n" }, { "path": ".github/workflows/docs.yaml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\non:\n push:\n branches:\n - main\n paths:\n - .asf.yaml\n - .github/workflows/docs.yaml\n - docs/**\n\nname: Deploy DataFusion Comet site\n\njobs:\n build-docs:\n name: Build docs\n if: ${{ startsWith(github.repository, 'apache/') }}\n runs-on: ubuntu-latest\n steps:\n - name: Checkout docs sources\n uses: actions/checkout@v6\n\n - name: Checkout asf-site branch\n uses: actions/checkout@v6\n with:\n ref: asf-site\n path: asf-site\n\n - name: Setup Python\n uses: actions/setup-python@v6\n with:\n python-version: \"3.10\"\n\n - name: Setup Java\n uses: actions/setup-java@v5\n with:\n distribution: 'temurin'\n java-version: '17'\n cache: 'maven'\n\n - name: Install dependencies\n run: |\n set -x\n python3 -m venv venv\n source venv/bin/activate\n pip install -r docs/requirements.txt\n\n - name: Build docs\n run: |\n set -x\n source venv/bin/activate\n cd docs\n ./build.sh\n\n - name: Copy & push the generated HTML\n run: |\n set -x\n cd asf-site/\n rsync \\\n -a \\\n --delete \\\n --exclude '/.git/' \\\n ../docs/build/html/ \\\n ./\n cp ../.asf.yaml .\n touch .nojekyll\n git status --porcelain\n if [ \"$(git status --porcelain)\" != \"\" ]; then\n git config user.name \"github-actions[bot]\"\n git config user.email \"github-actions[bot]@users.noreply.github.com\"\n git add --all\n git commit -m 'Publish built docs triggered by ${{ github.sha }}'\n git push || git push --force\n fi" }, { "path": ".github/workflows/iceberg_spark_test.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: Iceberg Spark SQL Tests\n\nconcurrency:\n group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}\n cancel-in-progress: true\n\non:\n push:\n branches:\n - main\n paths-ignore:\n - \"benchmarks/**\"\n - \"doc/**\"\n - \"docs/**\"\n - \"**.md\"\n - \"native/core/benches/**\"\n - \"native/spark-expr/benches/**\"\n - \"spark/src/test/scala/org/apache/spark/sql/benchmark/**\"\n pull_request:\n paths-ignore:\n - \"benchmarks/**\"\n - \"doc/**\"\n - \"docs/**\"\n - \"**.md\"\n - \"native/core/benches/**\"\n - \"native/spark-expr/benches/**\"\n - \"spark/src/test/scala/org/apache/spark/sql/benchmark/**\"\n # manual trigger\n # https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow\n workflow_dispatch:\n\nenv:\n RUST_VERSION: stable\n RUST_BACKTRACE: 1\n\njobs:\n # Build native library once and share with all test jobs\n build-native:\n name: Build Native Library\n runs-on: ubuntu-24.04\n container:\n image: amd64/rust\n steps:\n - uses: actions/checkout@v6\n\n - name: Setup Rust & Java toolchain\n uses: ./.github/actions/setup-builder\n with:\n rust-version: ${{ env.RUST_VERSION }}\n jdk-version: 17\n\n - name: Restore Cargo cache\n uses: actions/cache/restore@v5\n with:\n path: |\n ~/.cargo/registry\n ~/.cargo/git\n native/target\n key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}\n restore-keys: |\n ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-\n\n - name: Build native library\n # Use CI profile for faster builds (no LTO) and to share cache with pr_build_linux.yml.\n run: |\n cd native && cargo build --profile ci\n env:\n RUSTFLAGS: \"-Ctarget-cpu=x86-64-v3\"\n\n - name: Save Cargo cache\n uses: actions/cache/save@v5\n if: github.ref == 'refs/heads/main'\n with:\n path: |\n ~/.cargo/registry\n ~/.cargo/git\n native/target\n key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}\n\n - name: Upload native library\n uses: actions/upload-artifact@v7\n with:\n name: native-lib-iceberg\n path: native/target/ci/libcomet.so\n retention-days: 1\n\n iceberg-spark:\n needs: build-native\n strategy:\n matrix:\n os: [ubuntu-24.04]\n java-version: [11, 17]\n iceberg-version: [{short: '1.8', full: '1.8.1'}, {short: '1.9', full: '1.9.1'}, {short: '1.10', full: '1.10.0'}]\n spark-version: [{short: '3.4', full: '3.4.3'}, {short: '3.5', full: '3.5.8'}]\n scala-version: ['2.13']\n fail-fast: false\n name: iceberg-spark/${{ matrix.os }}/iceberg-${{ matrix.iceberg-version.full }}/spark-${{ matrix.spark-version.full }}/scala-${{ matrix.scala-version }}/java-${{ matrix.java-version }}\n runs-on: ${{ matrix.os }}\n container:\n image: amd64/rust\n env:\n SPARK_LOCAL_IP: localhost\n steps:\n - uses: actions/checkout@v6\n - name: Setup Rust & Java toolchain\n uses: ./.github/actions/setup-builder\n with:\n rust-version: ${{env.RUST_VERSION}}\n jdk-version: ${{ matrix.java-version }}\n - name: Download native library\n uses: actions/download-artifact@v8\n with:\n name: native-lib-iceberg\n path: native/target/release/\n - name: Build Comet\n run: |\n ./mvnw install -Prelease -DskipTests -Pspark-${{ matrix.spark-version.short }} -Pscala-${{ matrix.scala-version }}\n - name: Setup Iceberg\n uses: ./.github/actions/setup-iceberg-builder\n with:\n iceberg-version: ${{ matrix.iceberg-version.full }}\n - name: Run Iceberg Spark tests\n run: |\n cd apache-iceberg\n rm -rf /root/.m2/repository/org/apache/parquet # somehow parquet cache requires cleanups\n ENABLE_COMET=true ENABLE_COMET_ONHEAP=true ./gradlew -DsparkVersions=${{ matrix.spark-version.short }} -DscalaVersion=${{ matrix.scala-version }} -DflinkVersions= -DkafkaVersions= \\\n :iceberg-spark:iceberg-spark-${{ matrix.spark-version.short }}_${{ matrix.scala-version }}:test \\\n -Pquick=true -x javadoc\n\n iceberg-spark-extensions:\n needs: build-native\n strategy:\n matrix:\n os: [ubuntu-24.04]\n java-version: [11, 17]\n iceberg-version: [{short: '1.8', full: '1.8.1'}, {short: '1.9', full: '1.9.1'}, {short: '1.10', full: '1.10.0'}]\n spark-version: [{short: '3.4', full: '3.4.3'}, {short: '3.5', full: '3.5.8'}]\n scala-version: ['2.13']\n fail-fast: false\n name: iceberg-spark-extensions/${{ matrix.os }}/iceberg-${{ matrix.iceberg-version.full }}/spark-${{ matrix.spark-version.full }}/scala-${{ matrix.scala-version }}/java-${{ matrix.java-version }}\n runs-on: ${{ matrix.os }}\n container:\n image: amd64/rust\n env:\n SPARK_LOCAL_IP: localhost\n steps:\n - uses: actions/checkout@v6\n - name: Setup Rust & Java toolchain\n uses: ./.github/actions/setup-builder\n with:\n rust-version: ${{env.RUST_VERSION}}\n jdk-version: ${{ matrix.java-version }}\n - name: Download native library\n uses: actions/download-artifact@v8\n with:\n name: native-lib-iceberg\n path: native/target/release/\n - name: Build Comet\n run: |\n ./mvnw install -Prelease -DskipTests -Pspark-${{ matrix.spark-version.short }} -Pscala-${{ matrix.scala-version }}\n - name: Setup Iceberg\n uses: ./.github/actions/setup-iceberg-builder\n with:\n iceberg-version: ${{ matrix.iceberg-version.full }}\n - name: Run Iceberg Spark extensions tests\n run: |\n cd apache-iceberg\n rm -rf /root/.m2/repository/org/apache/parquet # somehow parquet cache requires cleanups\n ENABLE_COMET=true ENABLE_COMET_ONHEAP=true ./gradlew -DsparkVersions=${{ matrix.spark-version.short }} -DscalaVersion=${{ matrix.scala-version }} -DflinkVersions= -DkafkaVersions= \\\n :iceberg-spark:iceberg-spark-extensions-${{ matrix.spark-version.short }}_${{ matrix.scala-version }}:test \\\n -Pquick=true -x javadoc\n\n iceberg-spark-runtime:\n needs: build-native\n strategy:\n matrix:\n os: [ubuntu-24.04]\n java-version: [11, 17]\n iceberg-version: [{short: '1.8', full: '1.8.1'}, {short: '1.9', full: '1.9.1'}, {short: '1.10', full: '1.10.0'}]\n spark-version: [{short: '3.4', full: '3.4.3'}, {short: '3.5', full: '3.5.8'}]\n scala-version: ['2.13']\n fail-fast: false\n name: iceberg-spark-runtime/${{ matrix.os }}/iceberg-${{ matrix.iceberg-version.full }}/spark-${{ matrix.spark-version.full }}/scala-${{ matrix.scala-version }}/java-${{ matrix.java-version }}\n runs-on: ${{ matrix.os }}\n container:\n image: amd64/rust\n env:\n SPARK_LOCAL_IP: localhost\n steps:\n - uses: actions/checkout@v6\n - name: Setup Rust & Java toolchain\n uses: ./.github/actions/setup-builder\n with:\n rust-version: ${{env.RUST_VERSION}}\n jdk-version: ${{ matrix.java-version }}\n - name: Download native library\n uses: actions/download-artifact@v8\n with:\n name: native-lib-iceberg\n path: native/target/release/\n - name: Build Comet\n run: |\n ./mvnw install -Prelease -DskipTests -Pspark-${{ matrix.spark-version.short }} -Pscala-${{ matrix.scala-version }}\n - name: Setup Iceberg\n uses: ./.github/actions/setup-iceberg-builder\n with:\n iceberg-version: ${{ matrix.iceberg-version.full }}\n - name: Run Iceberg Spark runtime tests\n run: |\n cd apache-iceberg\n rm -rf /root/.m2/repository/org/apache/parquet # somehow parquet cache requires cleanups\n ENABLE_COMET=true ENABLE_COMET_ONHEAP=true ./gradlew -DsparkVersions=${{ matrix.spark-version.short }} -DscalaVersion=${{ matrix.scala-version }} -DflinkVersions= -DkafkaVersions= \\\n :iceberg-spark:iceberg-spark-runtime-${{ matrix.spark-version.short }}_${{ matrix.scala-version }}:integrationTest \\\n -Pquick=true -x javadoc\n" }, { "path": ".github/workflows/label_new_issues.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: Label new issues with requires-triage\n\non:\n issues:\n types: [opened]\n\npermissions:\n issues: write\n\njobs:\n add-triage-label:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/github-script@v9\n with:\n script: |\n await github.rest.issues.addLabels({\n owner: context.repo.owner,\n repo: context.repo.repo,\n issue_number: context.issue.number,\n labels: ['requires-triage']\n })\n" }, { "path": ".github/workflows/miri.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: Run Miri Safety Checks\n\non:\n push:\n branches:\n - main\n paths-ignore:\n - \"doc/**\"\n - \"docs/**\"\n - \"**.md\"\n - \"native/core/benches/**\"\n - \"native/spark-expr/benches/**\"\n - \"spark/src/test/scala/org/apache/spark/sql/benchmark/**\"\n pull_request:\n paths-ignore:\n - \"doc/**\"\n - \"docs/**\"\n - \"**.md\"\n - \"native/core/benches/**\"\n - \"native/spark-expr/benches/**\"\n - \"spark/src/test/scala/org/apache/spark/sql/benchmark/**\"\n # manual trigger\n # https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow\n workflow_dispatch:\n\njobs:\n miri:\n name: \"Miri\"\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v6\n - name: Install Build Dependencies\n shell: bash\n run: |\n sudo apt-get update\n sudo apt-get install -y protobuf-compiler\n sudo apt-get install -y clang\n - name: Install Miri\n run: |\n rustup toolchain install nightly --component miri\n rustup override set nightly\n cargo miri setup\n - name: Test with Miri\n run: |\n cd native\n MIRIFLAGS=\"-Zmiri-disable-isolation\" cargo miri test --lib --bins --tests --examples\n" }, { "path": ".github/workflows/pr_benchmark_check.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n# Lightweight CI for benchmark-only changes - verifies compilation and linting\n# without running full test suites\n\nname: PR Benchmark Check\n\nconcurrency:\n group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}\n cancel-in-progress: true\n\non:\n push:\n branches:\n - main\n paths:\n - \"native/core/benches/**\"\n - \"native/spark-expr/benches/**\"\n - \"spark/src/test/scala/org/apache/spark/sql/benchmark/**\"\n pull_request:\n paths:\n - \"native/core/benches/**\"\n - \"native/spark-expr/benches/**\"\n - \"spark/src/test/scala/org/apache/spark/sql/benchmark/**\"\n workflow_dispatch:\n\nenv:\n RUST_VERSION: stable\n RUST_BACKTRACE: 1\n\njobs:\n benchmark-check:\n name: Benchmark Compile & Lint Check\n runs-on: ubuntu-latest\n container:\n image: amd64/rust\n steps:\n - uses: actions/checkout@v6\n\n - name: Setup Rust & Java toolchain\n uses: ./.github/actions/setup-builder\n with:\n rust-version: ${{ env.RUST_VERSION }}\n jdk-version: 17\n\n - name: Check Cargo fmt\n run: |\n cd native\n cargo fmt --all -- --check --color=never\n\n - name: Check Cargo clippy\n run: |\n cd native\n cargo clippy --color=never --all-targets --workspace -- -D warnings\n\n - name: Check benchmark compilation\n run: |\n cd native\n cargo check --benches\n\n - name: Cache Maven dependencies\n uses: actions/cache@v5\n with:\n path: |\n ~/.m2/repository\n /root/.m2/repository\n key: ${{ runner.os }}-benchmark-maven-${{ hashFiles('**/pom.xml') }}\n restore-keys: |\n ${{ runner.os }}-benchmark-maven-\n\n - name: Check Scala compilation and linting\n run: |\n ./mvnw -B compile test-compile scalafix:scalafix -Dscalafix.mode=CHECK -Psemanticdb -DskipTests\n" }, { "path": ".github/workflows/pr_build_linux.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: PR Build (Linux)\n\nconcurrency:\n group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}\n cancel-in-progress: true\n\non:\n push:\n branches:\n - main\n paths-ignore:\n - \"benchmarks/**\"\n - \"doc/**\"\n - \"docs/**\"\n - \"**.md\"\n - \"native/core/benches/**\"\n - \"native/spark-expr/benches/**\"\n - \"spark/src/test/scala/org/apache/spark/sql/benchmark/**\"\n pull_request:\n paths-ignore:\n - \"benchmarks/**\"\n - \"doc/**\"\n - \"docs/**\"\n - \"**.md\"\n - \"native/core/benches/**\"\n - \"native/spark-expr/benches/**\"\n - \"spark/src/test/scala/org/apache/spark/sql/benchmark/**\"\n # manual trigger\n # https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow\n workflow_dispatch:\n\nenv:\n RUST_VERSION: stable\n RUST_BACKTRACE: 1\n\njobs:\n\n # Fast lint check - gates all other jobs\n lint:\n name: Lint\n runs-on: ubuntu-latest\n container:\n image: amd64/rust\n steps:\n - uses: actions/checkout@v6\n\n - name: Check Rust formatting\n run: |\n rustup component add rustfmt\n cd native && cargo fmt --all -- --check\n\n lint-java:\n needs: lint\n name: Lint Java (${{ matrix.profile.name }})\n runs-on: ubuntu-latest\n container:\n image: amd64/rust\n env:\n JAVA_TOOL_OPTIONS: ${{ matrix.profile.java_version == '17' && '--add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED' || '' }}\n strategy:\n matrix:\n profile:\n - name: \"Spark 3.4, JDK 11, Scala 2.12\"\n java_version: \"11\"\n maven_opts: \"-Pspark-3.4 -Pscala-2.12\"\n - name: \"Spark 3.5, JDK 17, Scala 2.12\"\n java_version: \"17\"\n maven_opts: \"-Pspark-3.5 -Pscala-2.12\"\n - name: \"Spark 4.0, JDK 17\"\n java_version: \"17\"\n maven_opts: \"-Pspark-4.0\"\n fail-fast: false\n steps:\n - uses: runs-on/action@742bf56072eb4845a0f94b3394673e4903c90ff0 # v2.1.0\n - uses: actions/checkout@v6\n\n - name: Setup Rust & Java toolchain\n uses: ./.github/actions/setup-builder\n with:\n rust-version: ${{ env.RUST_VERSION }}\n jdk-version: ${{ matrix.profile.java_version }}\n\n - name: Cache Maven dependencies\n uses: actions/cache@v5\n with:\n path: |\n ~/.m2/repository\n /root/.m2/repository\n key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-lint\n restore-keys: |\n ${{ runner.os }}-java-maven-\n\n - name: Run scalafix check\n run: |\n ./mvnw -B package -DskipTests scalafix:scalafix -Dscalafix.mode=CHECK -Psemanticdb ${{ matrix.profile.maven_opts }}\n\n - name: Setup Node.js\n uses: actions/setup-node@v6\n with:\n node-version: '24'\n\n - name: Install prettier\n run: |\n npm install -g prettier\n\n - name: Run prettier\n run: |\n npx prettier \"**/*.md\" --write\n\n - name: Mark workspace as safe for git\n run: |\n git config --global --add safe.directory \"$GITHUB_WORKSPACE\"\n\n - name: Check for any local git changes (such as generated docs)\n run: |\n ./dev/ci/check-working-tree-clean.sh\n\n # Build native library once and share with all test jobs\n build-native:\n needs: lint\n name: Build Native Library\n runs-on: ${{ github.repository_owner == 'apache' && format('runs-on={0},family=m8a+m7a+c8a,cpu=8,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion-comet', github.run_id) || 'ubuntu-latest' }}\n container:\n image: amd64/rust\n steps:\n - uses: runs-on/action@742bf56072eb4845a0f94b3394673e4903c90ff0 # v2.1.0\n - uses: actions/checkout@v6\n - name: Setup Rust toolchain\n uses: ./.github/actions/setup-builder\n with:\n rust-version: ${{ env.RUST_VERSION }}\n jdk-version: 17 # JDK only needed for common module proto generation\n\n - name: Restore Cargo cache\n uses: actions/cache/restore@v5\n with:\n path: |\n ~/.cargo/registry\n ~/.cargo/git\n native/target\n key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}\n restore-keys: |\n ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-\n\n - name: Build native library (CI profile)\n run: |\n cd native\n # CI profile: same overflow behavior as release, but faster compilation\n # (no LTO, parallel codegen)\n cargo build --profile ci\n env:\n RUSTFLAGS: \"-Ctarget-cpu=x86-64-v3\"\n\n - name: Upload native library\n uses: actions/upload-artifact@v7\n with:\n name: native-lib-linux\n path: native/target/ci/libcomet.so\n retention-days: 1\n\n - name: Save Cargo cache\n uses: actions/cache/save@v5\n if: github.ref == 'refs/heads/main'\n with:\n path: |\n ~/.cargo/registry\n ~/.cargo/git\n native/target\n key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}\n\n # Run Rust tests (runs in parallel with build-native, uses debug builds)\n linux-test-rust:\n needs: lint\n name: ubuntu-latest/rust-test\n runs-on: ${{ github.repository_owner == 'apache' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion-comet', github.run_id) || 'ubuntu-latest' }}\n container:\n image: amd64/rust\n steps:\n - uses: runs-on/action@742bf56072eb4845a0f94b3394673e4903c90ff0 # v2.1.0\n\n - uses: actions/checkout@v6\n\n - name: Setup Rust & Java toolchain\n uses: ./.github/actions/setup-builder\n with:\n rust-version: ${{ env.RUST_VERSION }}\n jdk-version: 17\n\n - name: Restore Cargo cache\n uses: actions/cache/restore@v5\n with:\n path: |\n ~/.cargo/registry\n ~/.cargo/git\n native/target\n # Note: Java version intentionally excluded - Rust target is JDK-independent\n key: ${{ runner.os }}-cargo-debug-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}\n restore-keys: |\n ${{ runner.os }}-cargo-debug-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-\n\n - name: Rust test steps\n uses: ./.github/actions/rust-test\n\n - name: Save Cargo cache\n uses: actions/cache/save@v5\n if: github.ref == 'refs/heads/main'\n with:\n path: |\n ~/.cargo/registry\n ~/.cargo/git\n native/target\n key: ${{ runner.os }}-cargo-debug-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}\n\n linux-test:\n needs: build-native\n strategy:\n matrix:\n # the goal with these profiles is to get coverage of all Java, Scala, and Spark\n # versions without testing all possible combinations, which would be overkill\n profile:\n - name: \"Spark 3.4, JDK 11, Scala 2.12\"\n java_version: \"11\"\n maven_opts: \"-Pspark-3.4 -Pscala-2.12\"\n scan_impl: \"auto\"\n\n - name: \"Spark 3.5.5, JDK 17, Scala 2.13\"\n java_version: \"17\"\n maven_opts: \"-Pspark-3.5 -Dspark.version=3.5.5 -Pscala-2.13\"\n scan_impl: \"auto\"\n\n - name: \"Spark 3.5.6, JDK 17, Scala 2.13\"\n java_version: \"17\"\n maven_opts: \"-Pspark-3.5 -Dspark.version=3.5.6 -Pscala-2.13\"\n scan_impl: \"auto\"\n\n - name: \"Spark 3.5, JDK 17, Scala 2.12\"\n java_version: \"17\"\n maven_opts: \"-Pspark-3.5 -Pscala-2.12\"\n scan_impl: \"native_iceberg_compat\"\n\n - name: \"Spark 4.0, JDK 17\"\n java_version: \"17\"\n maven_opts: \"-Pspark-4.0\"\n scan_impl: \"auto\"\n suite:\n - name: \"fuzz\"\n value: |\n org.apache.comet.CometFuzzTestSuite\n org.apache.comet.CometFuzzAggregateSuite\n org.apache.comet.CometFuzzIcebergSuite\n org.apache.comet.CometFuzzMathSuite\n org.apache.comet.DataGeneratorSuite\n - name: \"shuffle\"\n value: |\n org.apache.comet.exec.CometShuffleSuite\n org.apache.comet.exec.CometShuffle4_0Suite\n org.apache.comet.exec.CometNativeColumnarToRowSuite\n org.apache.comet.exec.CometNativeShuffleSuite\n org.apache.comet.exec.CometShuffleEncryptionSuite\n org.apache.comet.exec.CometShuffleManagerSuite\n org.apache.comet.exec.CometAsyncShuffleSuite\n org.apache.comet.exec.DisableAQECometShuffleSuite\n org.apache.comet.exec.DisableAQECometAsyncShuffleSuite\n org.apache.spark.shuffle.sort.SpillSorterSuite\n - name: \"parquet\"\n value: |\n org.apache.comet.parquet.CometParquetWriterSuite\n org.apache.comet.parquet.ParquetReadV1Suite\n org.apache.comet.parquet.ParquetReadV2Suite\n org.apache.comet.parquet.ParquetReadFromFakeHadoopFsSuite\n org.apache.spark.sql.comet.ParquetDatetimeRebaseV1Suite\n org.apache.spark.sql.comet.ParquetDatetimeRebaseV2Suite\n org.apache.spark.sql.comet.ParquetEncryptionITCase\n org.apache.comet.exec.CometNativeReaderSuite\n org.apache.comet.CometIcebergNativeSuite\n - name: \"csv\"\n value: |\n org.apache.comet.csv.CometCsvNativeReadSuite\n - name: \"exec\"\n value: |\n org.apache.comet.exec.CometAggregateSuite\n org.apache.comet.exec.CometExec3_4PlusSuite\n org.apache.comet.exec.CometExecSuite\n org.apache.comet.exec.CometGenerateExecSuite\n org.apache.comet.exec.CometWindowExecSuite\n org.apache.comet.exec.CometJoinSuite\n org.apache.comet.CometNativeSuite\n org.apache.comet.CometSparkSessionExtensionsSuite\n org.apache.spark.CometPluginsSuite\n org.apache.spark.CometPluginsDefaultSuite\n org.apache.spark.CometPluginsNonOverrideSuite\n org.apache.spark.CometPluginsUnifiedModeOverrideSuite\n org.apache.comet.rules.CometScanRuleSuite\n org.apache.comet.rules.CometExecRuleSuite\n org.apache.spark.sql.CometTPCDSQuerySuite\n org.apache.spark.sql.CometTPCDSQueryTestSuite\n org.apache.spark.sql.CometTPCHQuerySuite\n org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite\n org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite\n org.apache.spark.sql.comet.CometTaskMetricsSuite\n org.apache.spark.sql.comet.CometDppFallbackRepro3949Suite\n org.apache.spark.sql.comet.CometShuffleFallbackStickinessSuite\n org.apache.comet.objectstore.NativeConfigSuite\n - name: \"expressions\"\n value: |\n org.apache.comet.CometExpressionSuite\n org.apache.comet.CometSqlFileTestSuite\n org.apache.comet.CometExpressionCoverageSuite\n org.apache.comet.CometHashExpressionSuite\n org.apache.comet.CometTemporalExpressionSuite\n org.apache.comet.CometArrayExpressionSuite\n org.apache.comet.CometCastSuite\n org.apache.comet.CometDateTimeUtilsSuite\n org.apache.comet.CometMathExpressionSuite\n org.apache.comet.CometStringExpressionSuite\n org.apache.comet.CometBitwiseExpressionSuite\n org.apache.comet.CometMapExpressionSuite\n org.apache.comet.CometCsvExpressionSuite\n org.apache.comet.CometJsonExpressionSuite\n org.apache.comet.CometDateTimeUtilsSuite\n org.apache.comet.SparkErrorConverterSuite\n org.apache.comet.expressions.conditional.CometIfSuite\n org.apache.comet.expressions.conditional.CometCoalesceSuite\n org.apache.comet.expressions.conditional.CometCaseWhenSuite\n - name: \"sql\"\n value: |\n org.apache.spark.sql.CometToPrettyStringSuite\n fail-fast: false\n name: ${{ matrix.profile.name }}/${{ matrix.profile.scan_impl }} [${{ matrix.suite.name }}]\n runs-on: ${{ github.repository_owner == 'apache' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion-comet', github.run_id) || 'ubuntu-latest' }}\n container:\n image: amd64/rust\n env:\n JAVA_TOOL_OPTIONS: ${{ matrix.profile.java_version == '17' && '--add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED' || '' }}\n\n steps:\n - uses: runs-on/action@742bf56072eb4845a0f94b3394673e4903c90ff0 # v2.1.0\n - uses: actions/checkout@v6\n\n - name: Setup Rust & Java toolchain\n uses: ./.github/actions/setup-builder\n with:\n rust-version: ${{ env.RUST_VERSION }}\n jdk-version: ${{ matrix.profile.java_version }}\n\n - name: Download native library\n uses: actions/download-artifact@v8\n with:\n name: native-lib-linux\n # Download to release/ since Maven's -Prelease expects libcomet.so there\n path: native/target/release/\n\n # Restore cargo registry cache (for any cargo commands that might run)\n - name: Cache Cargo registry\n uses: actions/cache@v5\n with:\n path: |\n ~/.cargo/registry\n ~/.cargo/git\n key: ${{ runner.os }}-cargo-registry-${{ hashFiles('native/**/Cargo.lock') }}\n restore-keys: |\n ${{ runner.os }}-cargo-registry-\n\n - name: Java test steps\n uses: ./.github/actions/java-test\n with:\n artifact_name: ${{ matrix.profile.name }}-${{ matrix.suite.name }}-${{ github.run_id }}-${{ github.run_number }}-${{ github.run_attempt }}-${{ matrix.profile.scan_impl }}\n suites: ${{ matrix.suite.name == 'sql' && matrix.profile.name == 'Spark 3.4, JDK 11, Scala 2.12' && '' || matrix.suite.value }}\n maven_opts: ${{ matrix.profile.maven_opts }}\n scan_impl: ${{ matrix.profile.scan_impl }}\n upload-test-reports: true\n skip-native-build: true\n\n # TPC-H correctness test - verifies benchmark queries produce correct results\n verify-benchmark-results-tpch:\n needs: build-native\n name: Verify TPC-H Results\n runs-on: ${{ github.repository_owner == 'apache' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion-comet', github.run_id) || 'ubuntu-latest' }}\n container:\n image: amd64/rust\n steps:\n - uses: runs-on/action@742bf56072eb4845a0f94b3394673e4903c90ff0 # v2.1.0\n\n - uses: actions/checkout@v6\n\n - name: Setup Rust & Java toolchain\n uses: ./.github/actions/setup-builder\n with:\n rust-version: ${{ env.RUST_VERSION }}\n jdk-version: 11\n\n - name: Download native library\n uses: actions/download-artifact@v8\n with:\n name: native-lib-linux\n path: native/target/release/\n\n - name: Cache Maven dependencies\n uses: actions/cache@v5\n with:\n path: |\n ~/.m2/repository\n /root/.m2/repository\n key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}\n restore-keys: |\n ${{ runner.os }}-java-maven-\n\n - name: Cache TPC-H data\n id: cache-tpch\n uses: actions/cache@v5\n with:\n path: ./tpch\n key: tpch-${{ hashFiles('.github/workflows/pr_build_linux.yml') }}\n\n - name: Build project\n run: |\n ./mvnw -B -Prelease install -DskipTests\n\n - name: Generate TPC-H data (SF=1)\n if: steps.cache-tpch.outputs.cache-hit != 'true'\n run: |\n cd spark && MAVEN_OPTS='-Xmx20g' ../mvnw -B -Prelease exec:java -Dexec.mainClass=\"org.apache.spark.sql.GenTPCHData\" -Dexec.classpathScope=\"test\" -Dexec.cleanupDaemonThreads=\"false\" -Dexec.args=\"--location `pwd`/.. --scaleFactor 1 --numPartitions 1 --overwrite\"\n\n - name: Run TPC-H queries\n run: |\n SPARK_HOME=`pwd` SPARK_TPCH_DATA=`pwd`/tpch/sf1_parquet ./mvnw -B -Prelease -Dsuites=org.apache.spark.sql.CometTPCHQuerySuite test\n\n # TPC-DS correctness tests - verifies benchmark queries produce correct results\n verify-benchmark-results-tpcds:\n needs: build-native\n name: Verify TPC-DS Results (${{ matrix.join }})\n runs-on: ${{ github.repository_owner == 'apache' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion-comet', github.run_id) || 'ubuntu-latest' }}\n container:\n image: amd64/rust\n strategy:\n matrix:\n join: [sort_merge, broadcast, hash]\n fail-fast: false\n steps:\n - uses: runs-on/action@742bf56072eb4845a0f94b3394673e4903c90ff0 # v2.1.0\n\n - uses: actions/checkout@v6\n\n - name: Setup Rust & Java toolchain\n uses: ./.github/actions/setup-builder\n with:\n rust-version: ${{ env.RUST_VERSION }}\n jdk-version: 11\n\n - name: Download native library\n uses: actions/download-artifact@v8\n with:\n name: native-lib-linux\n path: native/target/release/\n\n - name: Cache Maven dependencies\n uses: actions/cache@v5\n with:\n path: |\n ~/.m2/repository\n /root/.m2/repository\n key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}\n restore-keys: |\n ${{ runner.os }}-java-maven-\n\n - name: Cache TPC-DS data\n id: cache-tpcds\n uses: actions/cache@v5\n with:\n path: ./tpcds-sf-1\n key: tpcds-${{ hashFiles('.github/workflows/pr_build_linux.yml') }}\n\n - name: Build project\n run: |\n ./mvnw -B -Prelease install -DskipTests\n\n - name: Checkout tpcds-kit\n if: steps.cache-tpcds.outputs.cache-hit != 'true'\n uses: actions/checkout@v6\n with:\n repository: databricks/tpcds-kit\n path: ./tpcds-kit\n\n - name: Build tpcds-kit\n if: steps.cache-tpcds.outputs.cache-hit != 'true'\n run: |\n apt-get update && apt-get install -y yacc bison flex gcc-12 g++-12\n update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 120 --slave /usr/bin/g++ g++ /usr/bin/g++-12\n cd tpcds-kit/tools && make OS=LINUX\n\n - name: Generate TPC-DS data (SF=1)\n if: steps.cache-tpcds.outputs.cache-hit != 'true'\n run: |\n cd spark && MAVEN_OPTS='-Xmx20g' ../mvnw -B -Prelease exec:java -Dexec.mainClass=\"org.apache.spark.sql.GenTPCDSData\" -Dexec.classpathScope=\"test\" -Dexec.cleanupDaemonThreads=\"false\" -Dexec.args=\"--dsdgenDir `pwd`/../tpcds-kit/tools --location `pwd`/../tpcds-sf-1 --scaleFactor 1 --numPartitions 1\"\n\n - name: Run TPC-DS queries (Sort merge join)\n if: matrix.join == 'sort_merge'\n run: |\n SPARK_HOME=`pwd` SPARK_TPCDS_DATA=`pwd`/tpcds-sf-1 ./mvnw -B -Prelease -Dsuites=org.apache.spark.sql.CometTPCDSQuerySuite test\n env:\n SPARK_TPCDS_JOIN_CONF: |\n spark.sql.autoBroadcastJoinThreshold=-1\n spark.sql.join.preferSortMergeJoin=true\n\n - name: Run TPC-DS queries (Broadcast hash join)\n if: matrix.join == 'broadcast'\n run: |\n SPARK_HOME=`pwd` SPARK_TPCDS_DATA=`pwd`/tpcds-sf-1 ./mvnw -B -Prelease -Dsuites=org.apache.spark.sql.CometTPCDSQuerySuite test\n env:\n SPARK_TPCDS_JOIN_CONF: |\n spark.sql.autoBroadcastJoinThreshold=10485760\n\n - name: Run TPC-DS queries (Shuffled hash join)\n if: matrix.join == 'hash'\n run: |\n SPARK_HOME=`pwd` SPARK_TPCDS_DATA=`pwd`/tpcds-sf-1 ./mvnw -B -Prelease -Dsuites=org.apache.spark.sql.CometTPCDSQuerySuite test\n env:\n SPARK_TPCDS_JOIN_CONF: |\n spark.sql.autoBroadcastJoinThreshold=-1\n spark.sql.join.forceApplyShuffledHashJoin=true\n" }, { "path": ".github/workflows/pr_build_macos.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: PR Build (macOS)\n\nconcurrency:\n group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}\n cancel-in-progress: true\n\non:\n push:\n branches:\n - main\n paths-ignore:\n - \"benchmarks/**\"\n - \"doc/**\"\n - \"docs/**\"\n - \"**.md\"\n - \"native/core/benches/**\"\n - \"native/spark-expr/benches/**\"\n - \"spark/src/test/scala/org/apache/spark/sql/benchmark/**\"\n pull_request:\n paths-ignore:\n - \"benchmarks/**\"\n - \"doc/**\"\n - \"docs/**\"\n - \"**.md\"\n - \"native/core/benches/**\"\n - \"native/spark-expr/benches/**\"\n - \"spark/src/test/scala/org/apache/spark/sql/benchmark/**\"\n # manual trigger\n # https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow\n workflow_dispatch:\n\nenv:\n RUST_VERSION: stable\n RUST_BACKTRACE: 1\n\njobs:\n\n # Fast lint check - gates all other jobs (runs on Linux for cost efficiency)\n lint:\n name: Lint\n runs-on: ubuntu-latest\n container:\n image: amd64/rust\n steps:\n - uses: actions/checkout@v6\n\n - name: Check Rust formatting\n run: |\n rustup component add rustfmt\n cd native && cargo fmt --all -- --check\n\n # Build native library once and share with all test jobs\n build-native:\n needs: lint\n name: Build Native Library (macOS)\n runs-on: macos-14\n steps:\n - uses: actions/checkout@v6\n\n - name: Setup Rust & Java toolchain\n uses: ./.github/actions/setup-macos-builder\n with:\n rust-version: ${{ env.RUST_VERSION }}\n jdk-version: 17\n jdk-architecture: aarch64\n protoc-architecture: aarch_64\n\n - name: Restore Cargo cache\n uses: actions/cache/restore@v5\n with:\n path: |\n ~/.cargo/registry\n ~/.cargo/git\n native/target\n key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}\n restore-keys: |\n ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-\n\n - name: Build native library (CI profile)\n run: |\n cd native\n # CI profile: same overflow behavior as release, but faster compilation\n # (no LTO, parallel codegen)\n cargo build --profile ci\n env:\n RUSTFLAGS: \"-Ctarget-cpu=apple-m1\"\n\n - name: Upload native library\n uses: actions/upload-artifact@v7\n with:\n name: native-lib-macos\n path: native/target/ci/libcomet.dylib\n retention-days: 1\n\n - name: Save Cargo cache\n uses: actions/cache/save@v5\n if: github.ref == 'refs/heads/main'\n with:\n path: |\n ~/.cargo/registry\n ~/.cargo/git\n native/target\n key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}\n\n macos-aarch64-test:\n needs: build-native\n strategy:\n matrix:\n os: [macos-14]\n\n # the goal with these profiles is to get coverage of all Java, Scala, and Spark\n # versions without testing all possible combinations, which would be overkill\n profile:\n - name: \"Spark 3.4, JDK 11, Scala 2.12\"\n java_version: \"11\"\n maven_opts: \"-Pspark-3.4 -Pscala-2.12\"\n\n - name: \"Spark 3.5, JDK 17, Scala 2.13\"\n java_version: \"17\"\n maven_opts: \"-Pspark-3.5 -Pscala-2.13\"\n\n - name: \"Spark 4.0, JDK 17, Scala 2.13\"\n java_version: \"17\"\n maven_opts: \"-Pspark-4.0 -Pscala-2.13\"\n\n suite:\n - name: \"fuzz\"\n value: |\n org.apache.comet.CometFuzzTestSuite\n org.apache.comet.CometFuzzAggregateSuite\n org.apache.comet.CometFuzzIcebergSuite\n org.apache.comet.CometFuzzMathSuite\n org.apache.comet.DataGeneratorSuite\n - name: \"shuffle\"\n value: |\n org.apache.comet.exec.CometShuffleSuite\n org.apache.comet.exec.CometShuffle4_0Suite\n org.apache.comet.exec.CometNativeColumnarToRowSuite\n org.apache.comet.exec.CometNativeShuffleSuite\n org.apache.comet.exec.CometShuffleEncryptionSuite\n org.apache.comet.exec.CometShuffleManagerSuite\n org.apache.comet.exec.CometAsyncShuffleSuite\n org.apache.comet.exec.DisableAQECometShuffleSuite\n org.apache.comet.exec.DisableAQECometAsyncShuffleSuite\n org.apache.spark.shuffle.sort.SpillSorterSuite\n - name: \"parquet\"\n value: |\n org.apache.comet.parquet.CometParquetWriterSuite\n org.apache.comet.parquet.ParquetReadV1Suite\n org.apache.comet.parquet.ParquetReadV2Suite\n org.apache.comet.parquet.ParquetReadFromFakeHadoopFsSuite\n org.apache.spark.sql.comet.ParquetDatetimeRebaseV1Suite\n org.apache.spark.sql.comet.ParquetDatetimeRebaseV2Suite\n org.apache.spark.sql.comet.ParquetEncryptionITCase\n org.apache.comet.exec.CometNativeReaderSuite\n org.apache.comet.CometIcebergNativeSuite\n - name: \"csv\"\n value: |\n org.apache.comet.csv.CometCsvNativeReadSuite\n - name: \"exec\"\n value: |\n org.apache.comet.exec.CometAggregateSuite\n org.apache.comet.exec.CometExec3_4PlusSuite\n org.apache.comet.exec.CometExecSuite\n org.apache.comet.exec.CometGenerateExecSuite\n org.apache.comet.exec.CometWindowExecSuite\n org.apache.comet.exec.CometJoinSuite\n org.apache.comet.CometNativeSuite\n org.apache.comet.CometSparkSessionExtensionsSuite\n org.apache.spark.CometPluginsSuite\n org.apache.spark.CometPluginsDefaultSuite\n org.apache.spark.CometPluginsNonOverrideSuite\n org.apache.spark.CometPluginsUnifiedModeOverrideSuite\n org.apache.comet.rules.CometScanRuleSuite\n org.apache.comet.rules.CometExecRuleSuite\n org.apache.spark.sql.CometTPCDSQuerySuite\n org.apache.spark.sql.CometTPCDSQueryTestSuite\n org.apache.spark.sql.CometTPCHQuerySuite\n org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite\n org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite\n org.apache.spark.sql.comet.CometTaskMetricsSuite\n org.apache.spark.sql.comet.CometDppFallbackRepro3949Suite\n org.apache.spark.sql.comet.CometShuffleFallbackStickinessSuite\n org.apache.comet.objectstore.NativeConfigSuite\n - name: \"expressions\"\n value: |\n org.apache.comet.CometExpressionSuite\n org.apache.comet.CometSqlFileTestSuite\n org.apache.comet.CometExpressionCoverageSuite\n org.apache.comet.CometHashExpressionSuite\n org.apache.comet.CometTemporalExpressionSuite\n org.apache.comet.CometArrayExpressionSuite\n org.apache.comet.CometCastSuite\n org.apache.comet.CometMathExpressionSuite\n org.apache.comet.CometStringExpressionSuite\n org.apache.comet.CometBitwiseExpressionSuite\n org.apache.comet.CometMapExpressionSuite\n org.apache.comet.CometJsonExpressionSuite\n org.apache.comet.CometCsvExpressionSuite\n org.apache.comet.CometDateTimeUtilsSuite\n org.apache.comet.SparkErrorConverterSuite\n org.apache.comet.expressions.conditional.CometIfSuite\n org.apache.comet.expressions.conditional.CometCoalesceSuite\n org.apache.comet.expressions.conditional.CometCaseWhenSuite\n - name: \"sql\"\n value: |\n org.apache.spark.sql.CometToPrettyStringSuite\n\n fail-fast: false\n name: ${{ matrix.os }}/${{ matrix.profile.name }} [${{ matrix.suite.name }}]\n runs-on: ${{ matrix.os }}\n steps:\n - uses: actions/checkout@v6\n\n - name: Setup Rust & Java toolchain\n uses: ./.github/actions/setup-macos-builder\n with:\n rust-version: ${{ env.RUST_VERSION }}\n jdk-version: ${{ matrix.profile.java_version }}\n jdk-architecture: aarch64\n protoc-architecture: aarch_64\n\n - name: Download native library\n uses: actions/download-artifact@v8\n with:\n name: native-lib-macos\n # Download to release/ since Maven's -Prelease expects libcomet.dylib there\n path: native/target/release/\n\n # Restore cargo registry cache (for any cargo commands that might run)\n - name: Cache Cargo registry\n uses: actions/cache@v5\n with:\n path: |\n ~/.cargo/registry\n ~/.cargo/git\n key: ${{ runner.os }}-cargo-registry-${{ hashFiles('native/**/Cargo.lock') }}\n restore-keys: |\n ${{ runner.os }}-cargo-registry-\n\n - name: Set thread thresholds envs for spark test on macOS\n # see: https://github.com/apache/datafusion-comet/issues/2965\n shell: bash\n run: |\n echo \"SPARK_TEST_SQL_SHUFFLE_EXCHANGE_MAX_THREAD_THRESHOLD=256\" >> $GITHUB_ENV\n echo \"SPARK_TEST_SQL_RESULT_QUERY_STAGE_MAX_THREAD_THRESHOLD=256\" >> $GITHUB_ENV\n echo \"SPARK_TEST_HIVE_SHUFFLE_EXCHANGE_MAX_THREAD_THRESHOLD=48\" >> $GITHUB_ENV\n echo \"SPARK_TEST_HIVE_RESULT_QUERY_STAGE_MAX_THREAD_THRESHOLD=48\" >> $GITHUB_ENV\n\n - name: Java test steps\n uses: ./.github/actions/java-test\n with:\n artifact_name: ${{ matrix.os }}-${{ matrix.profile.name }}-${{ matrix.suite.name }}-${{ github.run_id }}-${{ github.run_number }}-${{ github.run_attempt }}\n suites: ${{ matrix.suite.name == 'sql' && matrix.profile.name == 'Spark 3.4, JDK 11, Scala 2.12' && '' || matrix.suite.value }}\n maven_opts: ${{ matrix.profile.maven_opts }}\n skip-native-build: true\n" }, { "path": ".github/workflows/pr_markdown_format.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: Check Markdown Formatting\n\nconcurrency:\n group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}\n cancel-in-progress: true\n\non:\n pull_request:\n paths:\n - '**.md'\n\njobs:\n prettier-check:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v6\n\n - name: Setup Node.js\n uses: actions/setup-node@v6\n with:\n node-version: '24'\n\n - name: Install prettier\n run: npm install -g prettier\n\n - name: Check markdown formatting\n run: |\n # if you encounter error, run prettier locally and commit changes using instructions at:\n #\n # https://datafusion.apache.org/comet/contributor-guide/development.html#how-to-format-md-document\n #\n prettier --check \"**/*.md\"\n" }, { "path": ".github/workflows/pr_missing_suites.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: Check that all test suites are added to PR workflows\n\non:\n push:\n branches:\n - main\n pull_request:\n types: [opened, synchronize, reopened]\n\njobs:\n check-missing-suites:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v6\n - name: Check Missing Suites\n run: python3 dev/ci/check-suites.py\n" }, { "path": ".github/workflows/pr_rat_check.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: RAT License Check\n\nconcurrency:\n group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}\n cancel-in-progress: true\n\npermissions:\n contents: read\n\n# No paths-ignore: this workflow must run for ALL changes including docs\non:\n push:\n branches:\n - main\n pull_request:\n workflow_dispatch:\n\njobs:\n rat-check:\n name: RAT License Check\n runs-on: ubuntu-slim\n steps:\n - uses: actions/checkout@v6\n - name: Set up Java\n uses: actions/setup-java@v5\n with:\n distribution: temurin\n java-version: 11\n - name: Run RAT check\n run: ./mvnw -B -N apache-rat:check\n" }, { "path": ".github/workflows/pr_title_check.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: Check PR Title\n\nconcurrency:\n group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}\n cancel-in-progress: true\n\non:\n pull_request:\n types: [opened, edited, reopened]\n\njobs:\n check-pr-title:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v6\n - name: Check PR title\n env:\n PR_TITLE: ${{ github.event.pull_request.title }}\n run: |\n if ! echo $PR_TITLE | grep -Eq '^(\\w+)(\\(.+\\))?: .+$'; then\n echo \"PR title does not follow conventional commit style.\"\n echo \"Please use a title in the format: type: message, or type(scope): message\"\n echo \"Example: feat: Add support for sort-merge join\"\n exit 1\n fi\n\n" }, { "path": ".github/workflows/spark_sql_test.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: Spark SQL Tests\n\nconcurrency:\n group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}\n cancel-in-progress: true\n\non:\n push:\n branches:\n - main\n paths-ignore:\n - \"benchmarks/**\"\n - \"doc/**\"\n - \"docs/**\"\n - \"**.md\"\n - \"native/core/benches/**\"\n - \"native/spark-expr/benches/**\"\n - \"spark/src/test/scala/org/apache/spark/sql/benchmark/**\"\n pull_request:\n paths-ignore:\n - \"benchmarks/**\"\n - \"doc/**\"\n - \"docs/**\"\n - \"**.md\"\n - \"native/core/benches/**\"\n - \"native/spark-expr/benches/**\"\n - \"spark/src/test/scala/org/apache/spark/sql/benchmark/**\"\n # manual trigger\n # https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow\n workflow_dispatch:\n inputs:\n collect-fallback-logs:\n description: 'Whether to collect Comet fallback reasons from spark sql unit test logs'\n required: false\n default: 'false'\n type: boolean\n\nenv:\n RUST_VERSION: stable\n RUST_BACKTRACE: 1\n\njobs:\n\n # Build native library once and share with all test jobs\n build-native:\n name: Build Native Library\n runs-on: ubuntu-24.04\n container:\n image: amd64/rust\n steps:\n - uses: actions/checkout@v6\n\n - name: Setup Rust toolchain\n uses: ./.github/actions/setup-builder\n with:\n rust-version: ${{ env.RUST_VERSION }}\n jdk-version: 17\n\n - name: Restore Cargo cache\n uses: actions/cache/restore@v5\n with:\n path: |\n ~/.cargo/registry\n ~/.cargo/git\n native/target\n key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}\n restore-keys: |\n ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-\n\n - name: Build native library (CI profile)\n run: |\n cd native\n cargo build --profile ci\n env:\n RUSTFLAGS: \"-Ctarget-cpu=x86-64-v3\"\n\n - name: Upload native library\n uses: actions/upload-artifact@v7\n with:\n name: native-lib-linux\n path: native/target/ci/libcomet.so\n retention-days: 1\n\n - name: Save Cargo cache\n uses: actions/cache/save@v5\n if: github.ref == 'refs/heads/main'\n with:\n path: |\n ~/.cargo/registry\n ~/.cargo/git\n native/target\n key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}\n\n spark-sql-test:\n needs: build-native\n strategy:\n matrix:\n os: [ubuntu-24.04]\n module:\n - {name: \"catalyst\", args1: \"catalyst/test\", args2: \"\"}\n - {name: \"sql_core-1\", args1: \"\", args2: sql/testOnly * -- -l org.apache.spark.tags.ExtendedSQLTest -l org.apache.spark.tags.SlowSQLTest}\n - {name: \"sql_core-2\", args1: \"\", args2: \"sql/testOnly * -- -n org.apache.spark.tags.ExtendedSQLTest\"}\n - {name: \"sql_core-3\", args1: \"\", args2: \"sql/testOnly * -- -n org.apache.spark.tags.SlowSQLTest\"}\n - {name: \"sql_hive-1\", args1: \"\", args2: \"hive/testOnly * -- -l org.apache.spark.tags.ExtendedHiveTest -l org.apache.spark.tags.SlowHiveTest\"}\n - {name: \"sql_hive-2\", args1: \"\", args2: \"hive/testOnly * -- -n org.apache.spark.tags.ExtendedHiveTest\"}\n - {name: \"sql_hive-3\", args1: \"\", args2: \"hive/testOnly * -- -n org.apache.spark.tags.SlowHiveTest\"}\n # Since 4f5eaf0, auto mode uses native_datafusion for V1 scans,\n # so we only need to test with auto.\n config:\n - {spark-short: '3.4', spark-full: '3.4.3', java: 11, scan-impl: 'auto'}\n - {spark-short: '3.5', spark-full: '3.5.8', java: 11, scan-impl: 'auto'}\n - {spark-short: '4.0', spark-full: '4.0.1', java: 17, scan-impl: 'auto'}\n # Skip sql_hive-1 for Spark 4.0 due to https://github.com/apache/datafusion-comet/issues/2946\n exclude:\n - config: {spark-short: '4.0', spark-full: '4.0.1', java: 17, scan-impl: 'auto'}\n module: {name: \"sql_hive-1\", args1: \"\", args2: \"hive/testOnly * -- -l org.apache.spark.tags.ExtendedHiveTest -l org.apache.spark.tags.SlowHiveTest\"}\n fail-fast: false\n name: spark-sql-${{ matrix.config.scan-impl }}-${{ matrix.module.name }}/spark-${{ matrix.config.spark-full }}\n runs-on: ${{ matrix.os }}\n container:\n image: amd64/rust\n steps:\n - uses: actions/checkout@v6\n - name: Setup Rust & Java toolchain\n uses: ./.github/actions/setup-builder\n with:\n rust-version: ${{env.RUST_VERSION}}\n jdk-version: ${{ matrix.config.java }}\n - name: Download native library\n uses: actions/download-artifact@v8\n with:\n name: native-lib-linux\n path: native/target/release/\n - name: Setup Spark\n uses: ./.github/actions/setup-spark-builder\n with:\n spark-version: ${{ matrix.config.spark-full }}\n spark-short-version: ${{ matrix.config.spark-short }}\n skip-native-build: true\n - name: Run Spark tests\n run: |\n cd apache-spark\n rm -rf /root/.m2/repository/org/apache/parquet # somehow parquet cache requires cleanups\n NOLINT_ON_COMPILE=true ENABLE_COMET=true ENABLE_COMET_ONHEAP=true COMET_PARQUET_SCAN_IMPL=${{ matrix.config.scan-impl }} ENABLE_COMET_LOG_FALLBACK_REASONS=${{ github.event.inputs.collect-fallback-logs || 'false' }} \\\n build/sbt -Dsbt.log.noformat=true ${{ matrix.module.args1 }} \"${{ matrix.module.args2 }}\"\n if [ \"${{ github.event.inputs.collect-fallback-logs }}\" = \"true\" ]; then\n find . -type f -name \"unit-tests.log\" -print0 | xargs -0 grep -h \"Comet cannot accelerate\" | sed 's/.*Comet cannot accelerate/Comet cannot accelerate/' | sort -u > fallback.log\n fi\n env:\n LC_ALL: \"C.UTF-8\"\n - name: Upload fallback log\n if: ${{ github.event.inputs.collect-fallback-logs == 'true' }}\n uses: actions/upload-artifact@v7\n with:\n name: fallback-log-spark-sql-${{ matrix.config.scan-impl }}-${{ matrix.module.name }}-spark-${{ matrix.config.spark-full }}\n path: \"**/fallback.log\"\n\n merge-fallback-logs:\n if: ${{ github.event.inputs.collect-fallback-logs == 'true' }}\n name: merge-fallback-logs\n needs: [spark-sql-test]\n runs-on: ubuntu-24.04\n steps:\n - name: Download fallback log artifacts\n uses: actions/download-artifact@v8\n with:\n path: fallback-logs/\n - name: Merge fallback logs\n run: |\n find ./fallback-logs/ -type f -name \"fallback.log\" -print0 | xargs -0 cat | sort -u > all_fallback.log\n - name: Upload merged fallback log\n uses: actions/upload-artifact@v7\n with:\n name: all-fallback-log\n path: all_fallback.log\n" }, { "path": ".github/workflows/spark_sql_test_native_iceberg_compat.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: Spark SQL Tests (native_iceberg_compat)\n\nconcurrency:\n group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}\n cancel-in-progress: true\n\non:\n # manual trigger\n # https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow\n workflow_dispatch:\n\nenv:\n RUST_VERSION: stable\n RUST_BACKTRACE: 1\n\njobs:\n spark-sql-catalyst-native-iceberg-compat:\n strategy:\n matrix:\n os: [ubuntu-24.04]\n java-version: [11]\n spark-version: [{short: '3.4', full: '3.4.3'}, {short: '3.5', full: '3.5.8'}]\n module:\n - {name: \"catalyst\", args1: \"catalyst/test\", args2: \"\"}\n - {name: \"sql/core-1\", args1: \"\", args2: sql/testOnly * -- -l org.apache.spark.tags.ExtendedSQLTest -l org.apache.spark.tags.SlowSQLTest}\n - {name: \"sql/core-2\", args1: \"\", args2: \"sql/testOnly * -- -n org.apache.spark.tags.ExtendedSQLTest\"}\n - {name: \"sql/core-3\", args1: \"\", args2: \"sql/testOnly * -- -n org.apache.spark.tags.SlowSQLTest\"}\n - {name: \"sql/hive-1\", args1: \"\", args2: \"hive/testOnly * -- -l org.apache.spark.tags.ExtendedHiveTest -l org.apache.spark.tags.SlowHiveTest\"}\n - {name: \"sql/hive-2\", args1: \"\", args2: \"hive/testOnly * -- -n org.apache.spark.tags.ExtendedHiveTest\"}\n - {name: \"sql/hive-3\", args1: \"\", args2: \"hive/testOnly * -- -n org.apache.spark.tags.SlowHiveTest\"}\n fail-fast: false\n name: spark-sql-native-iceberg-compat-${{ matrix.module.name }}/${{ matrix.os }}/spark-${{ matrix.spark-version.full }}/java-${{ matrix.java-version }}\n runs-on: ${{ matrix.os }}\n container:\n image: amd64/rust\n steps:\n - uses: actions/checkout@v6\n - name: Setup Rust & Java toolchain\n uses: ./.github/actions/setup-builder\n with:\n rust-version: ${{env.RUST_VERSION}}\n jdk-version: ${{ matrix.java-version }}\n - name: Setup Spark\n uses: ./.github/actions/setup-spark-builder\n with:\n spark-version: ${{ matrix.spark-version.full }}\n spark-short-version: ${{ matrix.spark-version.short }}\n - name: Run Spark tests\n run: |\n cd apache-spark\n rm -rf /root/.m2/repository/org/apache/parquet # somehow parquet cache requires cleanups\n ENABLE_COMET=true ENABLE_COMET_ONHEAP=true COMET_PARQUET_SCAN_IMPL=native_iceberg_compat build/sbt -Dsbt.log.noformat=true ${{ matrix.module.args1 }} \"${{ matrix.module.args2 }}\"\n env:\n LC_ALL: \"C.UTF-8\"\n\n" }, { "path": ".github/workflows/stale.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: \"Close stale PRs\"\non:\n schedule:\n - cron: \"30 1 * * *\"\n\njobs:\n close-stale-prs:\n runs-on: ubuntu-latest\n permissions:\n issues: write\n pull-requests: write\n steps:\n - uses: actions/stale@b5d41d4e1d5dceea10e7104786b73624c18a190f # v10.2.0\n with:\n stale-pr-message: \"Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.\"\n days-before-pr-stale: 60\n days-before-pr-close: 7\n # do not close stale issues\n days-before-issue-stale: -1\n days-before-issue-close: -1\n repo-token: ${{ secrets.GITHUB_TOKEN }}" }, { "path": ".github/workflows/take.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: Assign/unassign the issue via `take` or `untake` comment\non:\n issue_comment:\n types: created\n\npermissions:\n issues: write\n\njobs:\n issue_assign:\n runs-on: ubuntu-latest\n if: (!github.event.issue.pull_request) && (github.event.comment.body == 'take' || github.event.comment.body == 'untake')\n concurrency:\n group: ${{ github.actor }}-issue-assign\n steps:\n - name: Take or untake issue\n env:\n COMMENT_BODY: ${{ github.event.comment.body }}\n ISSUE_NUMBER: ${{ github.event.issue.number }}\n USER_LOGIN: ${{ github.event.comment.user.login }}\n REPO: ${{ github.repository }}\n TOKEN: ${{ secrets.GITHUB_TOKEN }}\n run: |\n if [ \"$COMMENT_BODY\" == \"take\" ]\n then\n CODE=$(curl -H \"Authorization: token $TOKEN\" -LI https://api.github.com/repos/$REPO/issues/$ISSUE_NUMBER/assignees/$USER_LOGIN -o /dev/null -w '%{http_code}\\n' -s)\n if [ \"$CODE\" -eq \"204\" ]\n then\n echo \"Assigning issue $ISSUE_NUMBER to $USER_LOGIN\"\n curl -X POST -H \"Authorization: token $TOKEN\" -H \"Content-Type: application/json\" -d \"{\\\"assignees\\\": [\\\"$USER_LOGIN\\\"]}\" https://api.github.com/repos/$REPO/issues/$ISSUE_NUMBER/assignees\n else\n echo \"Cannot assign issue $ISSUE_NUMBER to $USER_LOGIN\"\n fi\n elif [ \"$COMMENT_BODY\" == \"untake\" ]\n then\n echo \"Unassigning issue $ISSUE_NUMBER from $USER_LOGIN\"\n curl -X DELETE -H \"Authorization: token $TOKEN\" -H \"Content-Type: application/json\" -d \"{\\\"assignees\\\": [\\\"$USER_LOGIN\\\"]}\" https://api.github.com/repos/$REPO/issues/$ISSUE_NUMBER/assignees\n fi\n" }, { "path": ".github/workflows/validate_workflows.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: Validate Github Workflows\n\non:\n pull_request:\n paths:\n - \".github/workflows/*.yml\"\n - \".github/workflows/*.yaml\"\n push:\n branches:\n - main\n\njobs:\n validate:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v6\n\n - name: Install actionlint\n run: |\n curl -sSfL https://raw.githubusercontent.com/rhysd/actionlint/main/scripts/download-actionlint.bash | bash\n echo \"$PWD\" >> $GITHUB_PATH\n\n - name: Lint GitHub Actions workflows\n run: actionlint -color --shellcheck=off\n" }, { "path": ".gitignore", "content": "CLAUDE.md\ntarget\n.idea\n*.iml\n.vscode/\n.bloop/\n.metals/\nderby.log\nmetastore_db/\nspark-warehouse/\ndependency-reduced-pom.xml\nnative/proto/src/generated\nprebuild\n.flattened-pom.xml\nrat.txt\nfiltered_rat.txt\ndev/dist\napache-rat-*.jar\nvenv\n.venv\ndev/release/comet-rm/workdir\nspark/benchmarks\n.DS_Store\ncomet-event-trace.json\n__pycache__\noutput\n" }, { "path": ".mvn/wrapper/maven-wrapper.properties", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\ndistributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.9.6/apache-maven-3.9.6-bin.zip\nwrapperUrl=https://repo.maven.apache.org/maven2/org/apache/maven/wrapper/maven-wrapper/3.2.0/maven-wrapper-3.2.0.jar\n" }, { "path": ".scalafix.conf", "content": "// Licensed to the Apache Software Foundation (ASF) under one\n// or more contributor license agreements. See the NOTICE file\n// distributed with this work for additional information\n// regarding copyright ownership. The ASF licenses this file\n// to you under the Apache License, Version 2.0 (the\n// \"License\"); you may not use this file except in compliance\n// with the License. You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing,\n// software distributed under the License is distributed on an\n// \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n// KIND, either express or implied. See the License for the\n// specific language governing permissions and limitations\n// under the License.\nrules = [\n ExplicitResultTypes,\n NoAutoTupling,\n RemoveUnused,\n\n DisableSyntax,\n LeakingImplicitClassVal,\n NoValInForComprehension,\n ProcedureSyntax,\n RedundantSyntax\n]\n" }, { "path": "CHANGELOG.md", "content": "\n\n# Apache DataFusion Comet Changelog\n\nComprehensive changelogs for each release are available [here](dev/changelog).\n" }, { "path": "LICENSE.txt", "content": "\n Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n END OF TERMS AND CONDITIONS\n\n APPENDIX: How to apply the Apache License to your work.\n\n To apply the Apache License to your work, attach the following\n boilerplate notice, with the fields enclosed by brackets \"[]\"\n replaced with your own identifying information. (Don't include\n the brackets!) The text should be enclosed in the appropriate\n comment syntax for the file format. We also recommend that a\n file or class name and description of purpose be included on the\n same \"printed page\" as the copyright notice for easier\n identification within third-party archives.\n\n Copyright [yyyy] [name of copyright owner]\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n\n--------------------------------------------------------------------------------\n\nThis project includes code from Apache Aurora.\n\n* dev/release/{release,changelog,release-candidate} are based on the scripts from\n Apache Aurora\n\nCopyright: 2016 The Apache Software Foundation.\nHome page: https://aurora.apache.org/\nLicense: http://www.apache.org/licenses/LICENSE-2.0" }, { "path": "Makefile", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n.PHONY: all core jvm test clean release-linux release bench\n\ndefine spark_jvm_17_extra_args\n$(shell ./mvnw help:evaluate -q -DforceStdout -Dexpression=extraJavaTestArgs)\nendef\n\n# Build optional Comet native features (like hdfs e.g)\nFEATURES_ARG := $(shell ! [ -z $(COMET_FEATURES) ] && echo '--features=$(COMET_FEATURES)')\n\nall: core jvm\n\ncore:\n\tcd native && cargo build $(FEATURES_ARG)\ntest-rust:\n\t# We need to compile CometException so that the cargo test can pass\n\t./mvnw compile -pl common -DskipTests $(PROFILES)\n\tcd native && cargo build $(FEATURES_ARG) && \\\n\tRUST_BACKTRACE=1 cargo test $(FEATURES_ARG)\njvm:\n\t./mvnw clean package -DskipTests $(PROFILES)\ntest-jvm: core\n\tSPARK_HOME=`pwd` COMET_CONF_DIR=$(shell pwd)/conf RUST_BACKTRACE=1 ./mvnw verify $(PROFILES)\ntest: test-rust test-jvm\nclean:\n\tcd native && cargo clean\n\t./mvnw clean $(PROFILES)\n\trm -rf .dist\nbench:\n\tcd native && RUSTFLAGS=\"-Ctarget-cpu=native\" cargo bench $(FEATURES_ARG) $(filter-out $@,$(MAKECMDGOALS))\nformat:\n\tcd native && cargo fmt\n\t./mvnw compile test-compile scalafix:scalafix -Psemanticdb $(PROFILES)\n\t./mvnw spotless:apply $(PROFILES)\n\n# build native libs for amd64 architecture Linux/MacOS on a Linux/amd64 machine/container\ncore-amd64-libs:\n\tcd native && RUSTFLAGS=\"-Ctarget-cpu=x86-64-v3\" cargo build -j 2 --release $(FEATURES_ARG)\nifdef HAS_OSXCROSS\n\trustup target add x86_64-apple-darwin\n\tcd native && cargo build -j 2 --target x86_64-apple-darwin --release $(FEATURES_ARG)\nendif\n\n# build native libs for arm64 architecture Linux/MacOS on a Linux/arm64 machine/container\ncore-arm64-libs:\n\tcd native && RUSTFLAGS=\"-Ctarget-cpu=neoverse-n1\" cargo build -j 2 --release $(FEATURES_ARG)\nifdef HAS_OSXCROSS\n\trustup target add aarch64-apple-darwin\n\tcd native && cargo build -j 2 --target aarch64-apple-darwin --release $(FEATURES_ARG)\nendif\n\ncore-amd64:\n\trustup target add x86_64-apple-darwin\n\tcd native && RUSTFLAGS=\"-Ctarget-cpu=skylake\" CC=o64-clang CXX=o64-clang++ cargo build --target x86_64-apple-darwin --release $(FEATURES_ARG)\n\tmkdir -p common/target/classes/org/apache/comet/darwin/x86_64\n\tcp native/target/x86_64-apple-darwin/release/libcomet.dylib common/target/classes/org/apache/comet/darwin/x86_64\n\tcd native && RUSTFLAGS=\"-Ctarget-cpu=x86-64-v3\" cargo build --release $(FEATURES_ARG)\n\tmkdir -p common/target/classes/org/apache/comet/linux/amd64\n\tcp native/target/release/libcomet.so common/target/classes/org/apache/comet/linux/amd64\n\tjar -cf common/target/comet-native-x86_64.jar \\\n\t\t-C common/target/classes/org/apache/comet darwin \\\n\t\t-C common/target/classes/org/apache/comet linux\n\t./dev/deploy-file common/target/comet-native-x86_64.jar comet-native-x86_64${COMET_CLASSIFIER} jar\n\ncore-arm64:\n\trustup target add aarch64-apple-darwin\n\tcd native && RUSTFLAGS=\"-Ctarget-cpu=apple-m1\" CC=arm64-apple-darwin21.4-clang CXX=arm64-apple-darwin21.4-clang++ CARGO_FEATURE_NEON=1 cargo build --target aarch64-apple-darwin --release $(FEATURES_ARG)\n\tmkdir -p common/target/classes/org/apache/comet/darwin/aarch64\n\tcp native/target/aarch64-apple-darwin/release/libcomet.dylib common/target/classes/org/apache/comet/darwin/aarch64\n\tcd native && RUSTFLAGS=\"-Ctarget-cpu=neoverse-n1\" cargo build --release $(FEATURES_ARG)\n\tmkdir -p common/target/classes/org/apache/comet/linux/aarch64\n\tcp native/target/release/libcomet.so common/target/classes/org/apache/comet/linux/aarch64\n\tjar -cf common/target/comet-native-aarch64.jar \\\n\t\t-C common/target/classes/org/apache/comet darwin \\\n\t\t-C common/target/classes/org/apache/comet linux\n\t./dev/deploy-file common/target/comet-native-aarch64.jar comet-native-aarch64${COMET_CLASSIFIER} jar\n\nrelease-linux: clean\n\trustup target add aarch64-apple-darwin x86_64-apple-darwin\n\tcd native && RUSTFLAGS=\"-Ctarget-cpu=apple-m1\" CC=arm64-apple-darwin21.4-clang CXX=arm64-apple-darwin21.4-clang++ CARGO_FEATURE_NEON=1 cargo build --target aarch64-apple-darwin --release $(FEATURES_ARG)\n\tcd native && RUSTFLAGS=\"-Ctarget-cpu=skylake\" CC=o64-clang CXX=o64-clang++ cargo build --target x86_64-apple-darwin --release $(FEATURES_ARG)\n\tcd native && RUSTFLAGS=\"-Ctarget-cpu=native\" cargo build --release $(FEATURES_ARG)\n\t./mvnw install -Prelease -DskipTests $(PROFILES)\nrelease:\n\tcd native && RUSTFLAGS=\"$(RUSTFLAGS) -Ctarget-cpu=native\" cargo build --release $(FEATURES_ARG)\n\t./mvnw install -Prelease -DskipTests $(PROFILES)\nrelease-nogit:\n\tcd native && RUSTFLAGS=\"-Ctarget-cpu=native\" cargo build --release\n\t./mvnw install -Prelease -DskipTests $(PROFILES) -Dmaven.gitcommitid.skip=true\nbenchmark-%: release\n\tcd spark && COMET_CONF_DIR=$(shell pwd)/conf MAVEN_OPTS='-Xmx20g ${call spark_jvm_17_extra_args}' ../mvnw exec:java -Dexec.mainClass=\"$*\" -Dexec.classpathScope=\"test\" -Dexec.cleanupDaemonThreads=\"false\" -Dexec.args=\"$(filter-out $@,$(MAKECMDGOALS))\" $(PROFILES)\n.DEFAULT:\n\t@: # ignore arguments provided to benchmarks e.g. \"make benchmark-foo -- --bar\", we do not want to treat \"--bar\" as target\n" }, { "path": "NOTICE.txt", "content": "Apache DataFusion Comet\nCopyright 2024 The Apache Software Foundation\n\nThis product includes software developed at\nThe Apache Software Foundation (http://www.apache.org/).\n\nThis product includes software developed at\nApache Gluten (https://github.com/apache/incubator-gluten/)\nSpecifically:\n- Optimizer rule to replace SortMergeJoin with ShuffleHashJoin\n\nThis product includes software developed at\nDataFusion HDFS ObjectStore Contrib Package(https://github.com/datafusion-contrib/datafusion-objectstore-hdfs)\n\nThis product includes software developed at\nDataFusion fs-hdfs3 Contrib Package(https://github.com/datafusion-contrib/fs-hdfs)\n" }, { "path": "README.md", "content": "\n\n# Apache DataFusion Comet\n\n[![Apache licensed][license-badge]][license-url]\n[![Discord chat][discord-badge]][discord-url]\n[![Pending PRs][pending-pr-badge]][pending-pr-url]\n[![Maven Central][maven-badge]][maven-url]\n\n[license-badge]: https://img.shields.io/badge/license-Apache%20v2-blue.svg\n[license-url]: https://github.com/apache/datafusion-comet/blob/main/LICENSE.txt\n[discord-badge]: https://img.shields.io/discord/885562378132000778.svg?logo=discord&style=flat-square\n[discord-url]: https://discord.gg/3EAr4ZX6JK\n[pending-pr-badge]: https://img.shields.io/github/issues-search/apache/datafusion-comet?query=is%3Apr+is%3Aopen+draft%3Afalse+review%3Arequired+status%3Asuccess&label=Pending%20PRs&logo=github\n[pending-pr-url]: https://github.com/apache/datafusion-comet/pulls?q=is%3Apr+is%3Aopen+draft%3Afalse+review%3Arequired+status%3Asuccess+sort%3Aupdated-desc\n[maven-badge]: https://img.shields.io/maven-central/v/org.apache.datafusion/comet-spark-spark4.0_2.13\n[maven-url]: https://search.maven.org/search?q=g:org.apache.datafusion%20AND%20comet-spark\n\n\"logo\"/\n\nApache DataFusion Comet is a high-performance accelerator for Apache Spark, built on top of the powerful\n[Apache DataFusion] query engine. Comet is designed to significantly enhance the\nperformance of Apache Spark workloads while leveraging commodity hardware and seamlessly integrating with the\nSpark ecosystem without requiring any code changes.\n\nComet also accelerates Apache Iceberg, when performing Parquet scans from Spark.\n\n[Apache DataFusion]: https://datafusion.apache.org\n\n# Benefits of Using Comet\n\n## Run Spark Queries at DataFusion Speeds\n\nComet delivers a performance speedup for many queries, enabling faster data processing and shorter time-to-insights.\n\nThe following chart shows the time it takes to run the 22 TPC-H queries against 100 GB of data in Parquet format\nusing a single executor with 8 cores. See the [Comet Benchmarking Guide](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html)\nfor details of the environment used for these benchmarks.\n\nWhen using Comet, the overall run time is reduced from 687 seconds to 302 seconds, a 2.2x speedup.\n\n![](docs/source/_static/images/benchmark-results/0.11.0/tpch_allqueries.png)\n\nHere is a breakdown showing relative performance of Spark and Comet for each TPC-H query.\n\n![](docs/source/_static/images/benchmark-results/0.11.0/tpch_queries_compare.png)\n\nThe following charts shows how much Comet currently accelerates each query from the benchmark.\n\n### Relative speedup\n\n![](docs/source/_static/images/benchmark-results/0.11.0/tpch_queries_speedup_rel.png)\n\n### Absolute speedup\n\n![](docs/source/_static/images/benchmark-results/0.11.0/tpch_queries_speedup_abs.png)\n\nThese benchmarks can be reproduced in any environment using the documentation in the\n[Comet Benchmarking Guide](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html). We encourage\nyou to run your own benchmarks.\n\nResults for our benchmark derived from TPC-DS are available in the [benchmarking guide](https://datafusion.apache.org/comet/contributor-guide/benchmark-results/tpc-ds.html).\n\n## Use Commodity Hardware\n\nComet leverages commodity hardware, eliminating the need for costly hardware upgrades or\nspecialized hardware accelerators, such as GPUs or FPGA. By maximizing the utilization of commodity hardware, Comet\nensures cost-effectiveness and scalability for your Spark deployments.\n\n## Spark Compatibility\n\nComet aims for 100% compatibility with all supported versions of Apache Spark, allowing you to integrate Comet into\nyour existing Spark deployments and workflows seamlessly. With no code changes required, you can immediately harness\nthe benefits of Comet's acceleration capabilities without disrupting your Spark applications.\n\n## Tight Integration with Apache DataFusion\n\nComet tightly integrates with the core Apache DataFusion project, leveraging its powerful execution engine. With\nseamless interoperability between Comet and DataFusion, you can achieve optimal performance and efficiency in your\nSpark workloads.\n\n## Active Community\n\nComet boasts a vibrant and active community of developers, contributors, and users dedicated to advancing the\ncapabilities of Apache DataFusion and accelerating the performance of Apache Spark.\n\n## Getting Started\n\nTo get started with Apache DataFusion Comet, follow the\n[installation instructions](https://datafusion.apache.org/comet/user-guide/installation.html). Join the\n[DataFusion Slack and Discord channels](https://datafusion.apache.org/contributor-guide/communication.html) to connect\nwith other users, ask questions, and share your experiences with Comet.\n\nFollow [Apache DataFusion Comet Overview](https://datafusion.apache.org/comet/about/index.html#comet-overview) to get more detailed information\n\n## Contributing\n\nWe welcome contributions from the community to help improve and enhance Apache DataFusion Comet. Whether it's fixing\nbugs, adding new features, writing documentation, or optimizing performance, your contributions are invaluable in\nshaping the future of Comet. Check out our\n[contributor guide](https://datafusion.apache.org/comet/contributor-guide/contributing.html) to get started.\n\n## License\n\nApache DataFusion Comet is licensed under the Apache License 2.0. See the [LICENSE.txt](LICENSE.txt) file for details.\n\n## Acknowledgments\n\nWe would like to express our gratitude to the Apache DataFusion community for their support and contributions to\nComet. Together, we're building a faster, more efficient future for big data processing with Apache Spark.\n" }, { "path": "benchmarks/Dockerfile", "content": "# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nFROM apache/datafusion-comet:0.7.0-spark3.5.5-scala2.12-java11\n\nRUN apt update \\\n && apt install -y git python3 python3-pip \\\n && apt clean\n\nRUN cd /opt \\\n && git clone https://github.com/apache/datafusion-benchmarks.git\n" }, { "path": "benchmarks/README.md", "content": "\n\n# Running Comet Benchmarks in Microk8s\n\nThis guide explains how to run benchmarks derived from TPC-H and TPC-DS in Apache DataFusion Comet deployed in a\nlocal Microk8s cluster.\n\n## Use Microk8s locally\n\nInstall Micro8s following the instructions at https://microk8s.io/docs/getting-started and then perform these\nadditional steps, ensuring that any existing kube config is backed up first.\n\n```shell\nmkdir -p ~/.kube\nmicrok8s config > ~/.kube/config\n\nmicrok8s enable dns\nmicrok8s enable registry\n\nmicrok8s kubectl create serviceaccount spark\n```\n\n## Build Comet Docker Image\n\nRun the following command from the root of this repository to build the Comet Docker image, or use a published\nDocker image from https://github.com/orgs/apache/packages?repo_name=datafusion-comet\n\n```shell\ndocker build -t apache/datafusion-comet -f kube/Dockerfile .\n```\n\n## Build Comet Benchmark Docker Image\n\nBuild the benchmark Docker image and push to the Microk8s Docker registry.\n\n```shell\ndocker build -t apache/datafusion-comet-tpcbench .\ndocker tag apache/datafusion-comet-tpcbench localhost:32000/apache/datafusion-comet-tpcbench:latest\ndocker push localhost:32000/apache/datafusion-comet-tpcbench:latest\n```\n\n## Run benchmarks\n\n```shell\nexport SPARK_MASTER=k8s://https://127.0.0.1:16443\nexport COMET_DOCKER_IMAGE=localhost:32000/apache/datafusion-comet-tpcbench:latest\n# Location of Comet JAR within the Docker image\nexport COMET_JAR=/opt/spark/jars/comet-spark-spark3.4_2.12-0.5.0-SNAPSHOT.jar\n\n$SPARK_HOME/bin/spark-submit \\\n --master $SPARK_MASTER \\\n --deploy-mode cluster \\\n --name comet-tpcbench \\\n --driver-memory 8G \\\n --conf spark.driver.memory=8G \\\n --conf spark.executor.instances=1 \\\n --conf spark.executor.memory=32G \\\n --conf spark.executor.cores=8 \\\n --conf spark.cores.max=8 \\\n --conf spark.task.cpus=1 \\\n --conf spark.executor.memoryOverhead=3G \\\n --jars local://$COMET_JAR \\\n --conf spark.executor.extraClassPath=$COMET_JAR \\\n --conf spark.driver.extraClassPath=$COMET_JAR \\\n --conf spark.plugins=org.apache.spark.CometPlugin \\\n --conf spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions \\\n --conf spark.comet.enabled=true \\\n --conf spark.comet.exec.enabled=true \\\n --conf spark.comet.exec.all.enabled=true \\\n --conf spark.comet.cast.allowIncompatible=true \\\n --conf spark.comet.exec.shuffle.enabled=true \\\n --conf spark.comet.exec.shuffle.mode=auto \\\n --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \\\n --conf spark.kubernetes.namespace=default \\\n --conf spark.kubernetes.driver.pod.name=tpcbench \\\n --conf spark.kubernetes.container.image=$COMET_DOCKER_IMAGE \\\n --conf spark.kubernetes.driver.volumes.hostPath.tpcdata.mount.path=/mnt/bigdata/tpcds/sf100/ \\\n --conf spark.kubernetes.driver.volumes.hostPath.tpcdata.options.path=/mnt/bigdata/tpcds/sf100/ \\\n --conf spark.kubernetes.executor.volumes.hostPath.tpcdata.mount.path=/mnt/bigdata/tpcds/sf100/ \\\n --conf spark.kubernetes.executor.volumes.hostPath.tpcdata.options.path=/mnt/bigdata/tpcds/sf100/ \\\n --conf spark.kubernetes.authenticate.caCertFile=/var/snap/microk8s/current/certs/ca.crt \\\n local:///opt/datafusion-benchmarks/runners/datafusion-comet/tpcbench.py \\\n --benchmark tpcds \\\n --data /mnt/bigdata/tpcds/sf100/ \\\n --queries /opt/datafusion-benchmarks/tpcds/queries-spark \\\n --iterations 1\n```\n" }, { "path": "benchmarks/pyspark/README.md", "content": "\n\n# PySpark Benchmarks\n\nA suite of PySpark benchmarks for comparing performance between Spark, Comet JVM, and Comet Native implementations.\n\n## Available Benchmarks\n\nRun `python run_benchmark.py --list-benchmarks` to see all available benchmarks:\n\n- **shuffle-hash** - Shuffle all columns using hash partitioning on group_key\n- **shuffle-roundrobin** - Shuffle all columns using round-robin partitioning\n\n## Prerequisites\n\n- Apache Spark cluster (standalone, YARN, or Kubernetes)\n- PySpark installed\n- Comet JAR built\n\n## Build Comet JAR\n\n```bash\ncd /path/to/datafusion-comet\nmake release\n```\n\n## Step 1: Generate Test Data\n\nGenerate test data with realistic 50-column schema (nested structs, arrays, maps):\n\n```bash\nspark-submit \\\n --master spark://master:7077 \\\n --executor-memory 16g \\\n generate_data.py \\\n --output /tmp/shuffle-benchmark-data \\\n --rows 10000000 \\\n --partitions 200\n```\n\n### Data Generation Options\n\n| Option | Default | Description |\n| -------------------- | ---------- | ---------------------------- |\n| `--output`, `-o` | (required) | Output path for parquet data |\n| `--rows`, `-r` | 10000000 | Number of rows |\n| `--partitions`, `-p` | 200 | Number of output partitions |\n\n## Step 2: Run Benchmarks\n\n### List Available Benchmarks\n\n```bash\npython run_benchmark.py --list-benchmarks\n```\n\n### Run Individual Benchmarks\n\nYou can run specific benchmarks by name:\n\n```bash\n# Hash partitioning shuffle - Spark baseline\nspark-submit --master spark://master:7077 \\\n run_benchmark.py --data /tmp/shuffle-benchmark-data --mode spark --benchmark shuffle-hash\n\n# Round-robin shuffle - Spark baseline\nspark-submit --master spark://master:7077 \\\n run_benchmark.py --data /tmp/shuffle-benchmark-data --mode spark --benchmark shuffle-roundrobin\n\n# Hash partitioning - Comet JVM shuffle\nspark-submit --master spark://master:7077 \\\n --jars /path/to/comet.jar \\\n --conf spark.comet.enabled=true \\\n --conf spark.comet.exec.shuffle.enabled=true \\\n --conf spark.comet.shuffle.mode=jvm \\\n --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \\\n run_benchmark.py --data /tmp/shuffle-benchmark-data --mode jvm --benchmark shuffle-hash\n\n# Round-robin - Comet Native shuffle\nspark-submit --master spark://master:7077 \\\n --jars /path/to/comet.jar \\\n --conf spark.comet.enabled=true \\\n --conf spark.comet.exec.shuffle.enabled=true \\\n --conf spark.comet.exec.shuffle.mode=native \\\n --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \\\n run_benchmark.py --data /tmp/shuffle-benchmark-data --mode native --benchmark shuffle-roundrobin\n```\n\n### Run All Benchmarks\n\nUse the provided script to run all benchmarks across all modes:\n\n```bash\nSPARK_MASTER=spark://master:7077 \\\nEXECUTOR_MEMORY=16g \\\n./run_all_benchmarks.sh /tmp/shuffle-benchmark-data\n```\n\n## Checking Results\n\nOpen the Spark UI (default: http://localhost:4040) during each benchmark run to compare shuffle write sizes in the Stages tab.\n\n## Adding New Benchmarks\n\nThe benchmark framework makes it easy to add new benchmarks:\n\n1. **Create a benchmark class** in `benchmarks/` directory (or add to existing file):\n\n```python\nfrom benchmarks.base import Benchmark\n\nclass MyBenchmark(Benchmark):\n @classmethod\n def name(cls) -> str:\n return \"my-benchmark\"\n\n @classmethod\n def description(cls) -> str:\n return \"Description of what this benchmark does\"\n\n def run(self) -> Dict[str, Any]:\n # Read data\n df = self.spark.read.parquet(self.data_path)\n\n # Run your benchmark operation\n def benchmark_operation():\n result = df.filter(...).groupBy(...).agg(...)\n result.write.mode(\"overwrite\").parquet(\"/tmp/output\")\n\n # Time it\n duration_ms = self._time_operation(benchmark_operation)\n\n return {\n 'duration_ms': duration_ms,\n # Add any other metrics you want to track\n }\n```\n\n2. **Register the benchmark** in `benchmarks/__init__.py`:\n\n```python\nfrom .my_module import MyBenchmark\n\n_BENCHMARK_REGISTRY = {\n # ... existing benchmarks\n MyBenchmark.name(): MyBenchmark,\n}\n```\n\n3. **Run your new benchmark**:\n\n```bash\npython run_benchmark.py --data /path/to/data --mode spark --benchmark my-benchmark\n```\n\nThe base `Benchmark` class provides:\n\n- Automatic timing via `_time_operation()`\n- Standard output formatting via `execute_timed()`\n- Access to SparkSession, data path, and mode\n- Spark configuration printing\n" }, { "path": "benchmarks/pyspark/benchmarks/__init__.py", "content": "#!/usr/bin/env python3\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n\"\"\"\nBenchmark registry for PySpark benchmarks.\n\nThis module provides a central registry for discovering and running benchmarks.\n\"\"\"\n\nfrom typing import Dict, Type, List\n\nfrom .base import Benchmark\nfrom .shuffle import ShuffleHashBenchmark, ShuffleRoundRobinBenchmark\n\n\n# Registry of all available benchmarks\n_BENCHMARK_REGISTRY: Dict[str, Type[Benchmark]] = {\n ShuffleHashBenchmark.name(): ShuffleHashBenchmark,\n ShuffleRoundRobinBenchmark.name(): ShuffleRoundRobinBenchmark,\n}\n\n\ndef get_benchmark(name: str) -> Type[Benchmark]:\n \"\"\"\n Get a benchmark class by name.\n\n Args:\n name: Benchmark name\n\n Returns:\n Benchmark class\n\n Raises:\n KeyError: If benchmark name is not found\n \"\"\"\n if name not in _BENCHMARK_REGISTRY:\n available = \", \".join(sorted(_BENCHMARK_REGISTRY.keys()))\n raise KeyError(\n f\"Unknown benchmark: {name}. Available benchmarks: {available}\"\n )\n return _BENCHMARK_REGISTRY[name]\n\n\ndef list_benchmarks() -> List[tuple[str, str]]:\n \"\"\"\n List all available benchmarks.\n\n Returns:\n List of (name, description) tuples\n \"\"\"\n benchmarks = []\n for name in sorted(_BENCHMARK_REGISTRY.keys()):\n benchmark_cls = _BENCHMARK_REGISTRY[name]\n benchmarks.append((name, benchmark_cls.description()))\n return benchmarks\n\n\n__all__ = [\n 'Benchmark',\n 'get_benchmark',\n 'list_benchmarks',\n 'ShuffleHashBenchmark',\n 'ShuffleRoundRobinBenchmark',\n]\n" }, { "path": "benchmarks/pyspark/benchmarks/base.py", "content": "#!/usr/bin/env python3\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n\"\"\"\nBase benchmark class providing common functionality for all benchmarks.\n\"\"\"\n\nimport time\nfrom abc import ABC, abstractmethod\nfrom typing import Dict, Any\n\nfrom pyspark.sql import SparkSession\n\n\nclass Benchmark(ABC):\n \"\"\"Base class for all PySpark benchmarks.\"\"\"\n\n def __init__(self, spark: SparkSession, data_path: str, mode: str):\n \"\"\"\n Initialize benchmark.\n\n Args:\n spark: SparkSession instance\n data_path: Path to input data\n mode: Execution mode (spark, jvm, native)\n \"\"\"\n self.spark = spark\n self.data_path = data_path\n self.mode = mode\n\n @classmethod\n @abstractmethod\n def name(cls) -> str:\n \"\"\"Return the benchmark name (used for CLI).\"\"\"\n pass\n\n @classmethod\n @abstractmethod\n def description(cls) -> str:\n \"\"\"Return a short description of the benchmark.\"\"\"\n pass\n\n @abstractmethod\n def run(self) -> Dict[str, Any]:\n \"\"\"\n Run the benchmark and return results.\n\n Returns:\n Dictionary containing benchmark results (must include 'duration_ms')\n \"\"\"\n pass\n\n def execute_timed(self) -> Dict[str, Any]:\n \"\"\"\n Execute the benchmark with timing and standard output.\n\n Returns:\n Dictionary containing benchmark results\n \"\"\"\n print(f\"\\n{'=' * 80}\")\n print(f\"Benchmark: {self.name()}\")\n print(f\"Mode: {self.mode.upper()}\")\n print(f\"{'=' * 80}\")\n print(f\"Data path: {self.data_path}\")\n\n # Print relevant Spark configuration\n self._print_spark_config()\n\n # Clear cache before running\n self.spark.catalog.clearCache()\n\n # Run the benchmark\n print(f\"\\nRunning benchmark...\")\n results = self.run()\n\n # Print results\n print(f\"\\nDuration: {results['duration_ms']:,} ms\")\n if 'row_count' in results:\n print(f\"Rows processed: {results['row_count']:,}\")\n\n # Print any additional metrics\n for key, value in results.items():\n if key not in ['duration_ms', 'row_count']:\n print(f\"{key}: {value}\")\n\n print(f\"{'=' * 80}\\n\")\n\n return results\n\n def _print_spark_config(self):\n \"\"\"Print relevant Spark configuration.\"\"\"\n conf = self.spark.sparkContext.getConf()\n print(f\"Shuffle manager: {conf.get('spark.shuffle.manager', 'default')}\")\n print(f\"Comet enabled: {conf.get('spark.comet.enabled', 'false')}\")\n print(f\"Comet shuffle enabled: {conf.get('spark.comet.exec.shuffle.enabled', 'false')}\")\n print(f\"Comet shuffle mode: {conf.get('spark.comet.shuffle.mode', 'not set')}\")\n print(f\"Spark UI: {self.spark.sparkContext.uiWebUrl}\")\n\n def _time_operation(self, operation_fn):\n \"\"\"\n Time an operation and return duration in milliseconds.\n\n Args:\n operation_fn: Function to time (takes no arguments)\n\n Returns:\n Duration in milliseconds\n \"\"\"\n start_time = time.time()\n operation_fn()\n duration_ms = int((time.time() - start_time) * 1000)\n return duration_ms\n" }, { "path": "benchmarks/pyspark/benchmarks/shuffle.py", "content": "#!/usr/bin/env python3\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n\"\"\"\nShuffle benchmarks for comparing shuffle file sizes and performance.\n\nThese benchmarks test different partitioning strategies (hash, round-robin)\nacross Spark, Comet JVM, and Comet Native shuffle implementations.\n\"\"\"\n\nfrom typing import Dict, Any\nfrom pyspark.sql import DataFrame\n\nfrom .base import Benchmark\n\n\nclass ShuffleBenchmark(Benchmark):\n \"\"\"Base class for shuffle benchmarks with common repartitioning logic.\"\"\"\n\n def __init__(self, spark, data_path: str, mode: str, num_partitions: int = 200):\n \"\"\"\n Initialize shuffle benchmark.\n\n Args:\n spark: SparkSession instance\n data_path: Path to input parquet data\n mode: Execution mode (spark, jvm, native)\n num_partitions: Number of partitions to shuffle to\n \"\"\"\n super().__init__(spark, data_path, mode)\n self.num_partitions = num_partitions\n\n def _read_and_count(self) -> tuple[DataFrame, int]:\n \"\"\"Read input data and count rows.\"\"\"\n df = self.spark.read.parquet(self.data_path)\n row_count = df.count()\n return df, row_count\n\n def _repartition(self, df: DataFrame) -> DataFrame:\n \"\"\"\n Repartition dataframe using the strategy defined by subclass.\n\n Args:\n df: Input dataframe\n\n Returns:\n Repartitioned dataframe\n \"\"\"\n raise NotImplementedError(\"Subclasses must implement _repartition\")\n\n def _write_output(self, df: DataFrame, output_path: str):\n \"\"\"Write repartitioned data to parquet.\"\"\"\n df.write.mode(\"overwrite\").parquet(output_path)\n\n def run(self) -> Dict[str, Any]:\n \"\"\"\n Run the shuffle benchmark.\n\n Returns:\n Dictionary with duration_ms and row_count\n \"\"\"\n # Read input data\n df, row_count = self._read_and_count()\n print(f\"Number of rows: {row_count:,}\")\n\n # Define the benchmark operation\n def benchmark_operation():\n # Repartition using the specific strategy\n repartitioned = self._repartition(df)\n\n # Write to parquet to force materialization\n output_path = f\"/tmp/shuffle-benchmark-output-{self.mode}-{self.name()}\"\n self._write_output(repartitioned, output_path)\n print(f\"Wrote repartitioned data to: {output_path}\")\n\n # Time the operation\n duration_ms = self._time_operation(benchmark_operation)\n\n return {\n 'duration_ms': duration_ms,\n 'row_count': row_count,\n 'num_partitions': self.num_partitions,\n }\n\n\nclass ShuffleHashBenchmark(ShuffleBenchmark):\n \"\"\"Shuffle benchmark using hash partitioning on a key column.\"\"\"\n\n @classmethod\n def name(cls) -> str:\n return \"shuffle-hash\"\n\n @classmethod\n def description(cls) -> str:\n return \"Shuffle all columns using hash partitioning on group_key\"\n\n def _repartition(self, df: DataFrame) -> DataFrame:\n \"\"\"Repartition using hash partitioning on group_key.\"\"\"\n return df.repartition(self.num_partitions, \"group_key\")\n\n\nclass ShuffleRoundRobinBenchmark(ShuffleBenchmark):\n \"\"\"Shuffle benchmark using round-robin partitioning.\"\"\"\n\n @classmethod\n def name(cls) -> str:\n return \"shuffle-roundrobin\"\n\n @classmethod\n def description(cls) -> str:\n return \"Shuffle all columns using round-robin partitioning\"\n\n def _repartition(self, df: DataFrame) -> DataFrame:\n \"\"\"Repartition using round-robin (no partition columns specified).\"\"\"\n return df.repartition(self.num_partitions)\n" }, { "path": "benchmarks/pyspark/generate_data.py", "content": "#!/usr/bin/env python3\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n\"\"\"\nGenerate test data for shuffle size comparison benchmark.\n\nThis script generates a parquet dataset with a realistic schema (100 columns\nincluding deeply nested structs, arrays, and maps) for benchmarking shuffle\noperations across Spark, Comet JVM, and Comet Native shuffle modes.\n\"\"\"\n\nimport argparse\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql import functions as F\nfrom pyspark.sql.types import (\n StructType, StructField, IntegerType, LongType, DoubleType,\n StringType, BooleanType, DateType, TimestampType, ArrayType,\n MapType, DecimalType\n)\n\n\ndef generate_data(output_path: str, num_rows: int, num_partitions: int):\n \"\"\"Generate test data with realistic schema and write to parquet.\"\"\"\n\n spark = SparkSession.builder \\\n .appName(\"ShuffleBenchmark-DataGen\") \\\n .getOrCreate()\n\n print(f\"Generating {num_rows:,} rows with {num_partitions} partitions\")\n print(f\"Output path: {output_path}\")\n print(\"Schema: 100 columns including deeply nested structs, arrays, and maps\")\n\n # Start with a range and build up the columns\n df = spark.range(0, num_rows, numPartitions=num_partitions)\n\n # Add columns using selectExpr for better performance\n df = df.selectExpr(\n # Key columns for grouping/partitioning (1-3)\n \"cast(id % 1000 as int) as partition_key\",\n \"cast(id % 100 as int) as group_key\",\n \"id as row_id\",\n\n # Integer columns (4-15)\n \"cast(id % 10000 as int) as category_id\",\n \"cast(id % 500 as int) as region_id\",\n \"cast(id % 50 as int) as department_id\",\n \"cast((id * 7) % 1000000 as int) as customer_id\",\n \"cast((id * 13) % 100000 as int) as product_id\",\n \"cast(id % 12 + 1 as int) as month\",\n \"cast(id % 28 + 1 as int) as day\",\n \"cast(2020 + (id % 5) as int) as year\",\n \"cast((id * 17) % 256 as int) as priority\",\n \"cast((id * 19) % 1000 as int) as rank\",\n \"cast((id * 23) % 10000 as int) as score_int\",\n \"cast((id * 29) % 500 as int) as level\",\n\n # Long columns (16-22)\n \"id * 1000 as transaction_id\",\n \"(id * 17) % 10000000000 as account_number\",\n \"(id * 31) % 1000000000 as reference_id\",\n \"(id * 37) % 10000000000 as external_id\",\n \"(id * 41) % 1000000000 as correlation_id\",\n \"(id * 43) % 10000000000 as trace_id\",\n \"(id * 47) % 1000000000 as span_id\",\n\n # Double columns (23-35)\n \"cast(id % 10000 as double) / 100.0 as amount\",\n \"cast((id * 3) % 10000 as double) / 100.0 as price\",\n \"cast(id % 100 as double) / 100.0 as discount\",\n \"cast((id * 7) % 500 as double) / 10.0 as weight\",\n \"cast((id * 11) % 1000 as double) / 10.0 as height\",\n \"cast(id % 360 as double) as latitude\",\n \"cast((id * 2) % 360 as double) as longitude\",\n \"cast((id * 13) % 10000 as double) / 1000.0 as rate\",\n \"cast((id * 17) % 100 as double) / 100.0 as percentage\",\n \"cast((id * 19) % 1000 as double) as velocity\",\n \"cast((id * 23) % 500 as double) / 10.0 as acceleration\",\n \"cast((id * 29) % 10000 as double) / 100.0 as temperature\",\n \"cast((id * 31) % 1000 as double) / 10.0 as pressure\",\n\n # String columns (36-50)\n \"concat('user_', cast(id % 100000 as string)) as user_name\",\n \"concat('email_', cast(id % 50000 as string), '@example.com') as email\",\n \"concat('SKU-', lpad(cast(id % 10000 as string), 6, '0')) as sku\",\n \"concat('ORD-', cast(id as string)) as order_id\",\n \"array('pending', 'processing', 'shipped', 'delivered', 'cancelled')[cast(id % 5 as int)] as status\",\n \"array('USD', 'EUR', 'GBP', 'JPY', 'CAD')[cast(id % 5 as int)] as currency\",\n \"concat('Description for item ', cast(id % 1000 as string), ' with additional details') as description\",\n \"concat('REF-', lpad(cast(id % 100000 as string), 8, '0')) as reference_code\",\n \"concat('TXN-', cast(id as string), '-', cast(id % 1000 as string)) as transaction_code\",\n \"array('A', 'B', 'C', 'D', 'E')[cast(id % 5 as int)] as grade\",\n \"concat('Note: Record ', cast(id as string), ' processed successfully') as notes\",\n \"concat('Session-', lpad(cast(id % 10000 as string), 6, '0')) as session_id\",\n \"concat('Device-', cast(id % 1000 as string)) as device_id\",\n \"array('chrome', 'firefox', 'safari', 'edge')[cast(id % 4 as int)] as browser\",\n \"array('windows', 'macos', 'linux', 'ios', 'android')[cast(id % 5 as int)] as os\",\n\n # Boolean columns (51-56)\n \"id % 2 = 0 as is_active\",\n \"id % 3 = 0 as is_verified\",\n \"id % 7 = 0 as is_premium\",\n \"id % 5 = 0 as is_deleted\",\n \"id % 11 = 0 as is_featured\",\n \"id % 13 = 0 as is_archived\",\n\n # Date and timestamp columns (57-60)\n \"date_add(to_date('2020-01-01'), cast(id % 1500 as int)) as created_date\",\n \"date_add(to_date('2020-01-01'), cast((id + 30) % 1500 as int)) as updated_date\",\n \"date_add(to_date('2020-01-01'), cast((id + 60) % 1500 as int)) as expires_date\",\n \"to_timestamp(concat('2020-01-01 ', lpad(cast(id % 24 as string), 2, '0'), ':00:00')) as created_at\",\n\n # Simple arrays (61-65)\n \"array(cast(id % 100 as int), cast((id + 1) % 100 as int), cast((id + 2) % 100 as int), cast((id + 3) % 100 as int), cast((id + 4) % 100 as int)) as tag_ids\",\n \"array(cast(id % 1000 as double) / 10.0, cast((id * 2) % 1000 as double) / 10.0, cast((id * 3) % 1000 as double) / 10.0) as scores\",\n \"array(concat('tag_', cast(id % 20 as string)), concat('tag_', cast((id + 5) % 20 as string)), concat('tag_', cast((id + 10) % 20 as string))) as tags\",\n \"array(id % 2 = 0, id % 3 = 0, id % 5 = 0, id % 7 = 0) as flag_array\",\n \"array(id * 1000, id * 2000, id * 3000) as long_array\",\n\n # Simple maps (66-68)\n \"map('key1', cast(id % 100 as string), 'key2', cast((id * 2) % 100 as string), 'key3', cast((id * 3) % 100 as string)) as str_attributes\",\n \"map('score1', cast(id % 100 as double), 'score2', cast((id * 2) % 100 as double)) as double_attributes\",\n \"map(cast(id % 10 as int), concat('val_', cast(id % 100 as string)), cast((id + 1) % 10 as int), concat('val_', cast((id + 1) % 100 as string))) as int_key_map\",\n\n # Level 2 nested struct: address with nested geo (69)\n \"named_struct(\"\n \" 'street', concat(cast(id % 9999 as string), ' Main St'),\"\n \" 'city', array('New York', 'Los Angeles', 'Chicago', 'Houston', 'Phoenix')[cast(id % 5 as int)],\"\n \" 'state', array('NY', 'CA', 'IL', 'TX', 'AZ')[cast(id % 5 as int)],\"\n \" 'zip', lpad(cast(id % 99999 as string), 5, '0'),\"\n \" 'country', 'USA',\"\n \" 'geo', named_struct(\"\n \" 'lat', cast(id % 180 as double) - 90.0,\"\n \" 'lng', cast(id % 360 as double) - 180.0,\"\n \" 'accuracy', cast(id % 100 as double)\"\n \" )\"\n \") as address\",\n\n # Level 3 nested struct: organization hierarchy (70)\n \"named_struct(\"\n \" 'company', named_struct(\"\n \" 'name', concat('Company_', cast(id % 1000 as string)),\"\n \" 'industry', array('tech', 'finance', 'healthcare', 'retail')[cast(id % 4 as int)],\"\n \" 'headquarters', named_struct(\"\n \" 'city', array('NYC', 'SF', 'LA', 'CHI')[cast(id % 4 as int)],\"\n \" 'country', 'USA',\"\n \" 'timezone', array('EST', 'PST', 'PST', 'CST')[cast(id % 4 as int)]\"\n \" )\"\n \" ),\"\n \" 'department', named_struct(\"\n \" 'name', array('Engineering', 'Sales', 'Marketing', 'HR')[cast(id % 4 as int)],\"\n \" 'code', concat('DEPT-', cast(id % 100 as string)),\"\n \" 'budget', cast(id % 1000000 as double)\"\n \" )\"\n \") as organization\",\n\n # Level 4 nested struct: deep config (71)\n \"named_struct(\"\n \" 'level1', named_struct(\"\n \" 'level2a', named_struct(\"\n \" 'level3a', named_struct(\"\n \" 'value_int', cast(id % 1000 as int),\"\n \" 'value_str', concat('deep_', cast(id % 100 as string)),\"\n \" 'value_bool', id % 2 = 0\"\n \" ),\"\n \" 'level3b', named_struct(\"\n \" 'metric1', cast(id % 100 as double),\"\n \" 'metric2', cast((id * 2) % 100 as double)\"\n \" )\"\n \" ),\"\n \" 'level2b', named_struct(\"\n \" 'setting1', concat('setting_', cast(id % 50 as string)),\"\n \" 'setting2', id % 3 = 0,\"\n \" 'values', array(cast(id % 10 as int), cast((id + 1) % 10 as int), cast((id + 2) % 10 as int))\"\n \" )\"\n \" ),\"\n \" 'metadata', named_struct(\"\n \" 'version', concat('v', cast(id % 10 as string)),\"\n \" 'timestamp', id * 1000\"\n \" )\"\n \") as deep_config\",\n\n # Array of structs with nested structs (72)\n \"array(\"\n \" named_struct(\"\n \" 'item_id', cast(id % 1000 as int),\"\n \" 'details', named_struct(\"\n \" 'name', concat('Item_', cast(id % 100 as string)),\"\n \" 'category', array('electronics', 'clothing', 'food', 'books')[cast(id % 4 as int)],\"\n \" 'pricing', named_struct(\"\n \" 'base', cast(id % 100 as double) + 0.99,\"\n \" 'discount', cast(id % 20 as double) / 100.0,\"\n \" 'tax_rate', 0.08\"\n \" )\"\n \" ),\"\n \" 'quantity', cast(id % 10 + 1 as int)\"\n \" ),\"\n \" named_struct(\"\n \" 'item_id', cast((id + 100) % 1000 as int),\"\n \" 'details', named_struct(\"\n \" 'name', concat('Item_', cast((id + 100) % 100 as string)),\"\n \" 'category', array('electronics', 'clothing', 'food', 'books')[cast((id + 1) % 4 as int)],\"\n \" 'pricing', named_struct(\"\n \" 'base', cast((id + 50) % 100 as double) + 0.99,\"\n \" 'discount', cast((id + 5) % 20 as double) / 100.0,\"\n \" 'tax_rate', 0.08\"\n \" )\"\n \" ),\"\n \" 'quantity', cast((id + 1) % 10 + 1 as int)\"\n \" )\"\n \") as line_items\",\n\n # Map with struct values (73)\n \"map(\"\n \" 'primary', named_struct('name', concat('Primary_', cast(id % 100 as string)), 'score', cast(id % 100 as double), 'active', true),\"\n \" 'secondary', named_struct('name', concat('Secondary_', cast(id % 100 as string)), 'score', cast((id * 2) % 100 as double), 'active', id % 2 = 0)\"\n \") as contact_map\",\n\n # Struct with map containing arrays (74)\n \"named_struct(\"\n \" 'config_name', concat('Config_', cast(id % 100 as string)),\"\n \" 'settings', map(\"\n \" 'integers', array(cast(id % 10 as int), cast((id + 1) % 10 as int), cast((id + 2) % 10 as int)),\"\n \" 'strings', array(concat('s1_', cast(id % 10 as string)), concat('s2_', cast(id % 10 as string)))\"\n \" ),\"\n \" 'enabled', id % 2 = 0\"\n \") as config_with_map\",\n\n # Array of arrays (75)\n \"array(\"\n \" array(cast(id % 10 as int), cast((id + 1) % 10 as int), cast((id + 2) % 10 as int)),\"\n \" array(cast((id * 2) % 10 as int), cast((id * 2 + 1) % 10 as int)),\"\n \" array(cast((id * 3) % 10 as int), cast((id * 3 + 1) % 10 as int), cast((id * 3 + 2) % 10 as int), cast((id * 3 + 3) % 10 as int))\"\n \") as nested_int_arrays\",\n\n # Array of maps (76)\n \"array(\"\n \" map('a', cast(id % 100 as string), 'b', cast((id + 1) % 100 as string)),\"\n \" map('x', cast((id * 2) % 100 as string), 'y', cast((id * 2 + 1) % 100 as string), 'z', cast((id * 2 + 2) % 100 as string))\"\n \") as array_of_maps\",\n\n # Map with array values (77)\n \"map(\"\n \" 'scores', array(cast(id % 100 as double), cast((id * 2) % 100 as double), cast((id * 3) % 100 as double)),\"\n \" 'ranks', array(cast(id % 10 as double), cast((id + 1) % 10 as double))\"\n \") as map_with_arrays\",\n\n # Complex event structure (78)\n \"named_struct(\"\n \" 'event_id', concat('EVT-', cast(id as string)),\"\n \" 'event_type', array('click', 'view', 'purchase', 'signup')[cast(id % 4 as int)],\"\n \" 'timestamp', id * 1000,\"\n \" 'properties', map(\"\n \" 'source', array('web', 'mobile', 'api')[cast(id % 3 as int)],\"\n \" 'campaign', concat('camp_', cast(id % 50 as string))\"\n \" ),\"\n \" 'user', named_struct(\"\n \" 'id', cast(id % 100000 as int),\"\n \" 'segment', array('new', 'returning', 'premium')[cast(id % 3 as int)],\"\n \" 'attributes', named_struct(\"\n \" 'age_group', array('18-24', '25-34', '35-44', '45+')[cast(id % 4 as int)],\"\n \" 'interests', array(concat('int_', cast(id % 10 as string)), concat('int_', cast((id + 1) % 10 as string)))\"\n \" )\"\n \" )\"\n \") as event_data\",\n\n # Financial transaction with deep nesting (79)\n \"named_struct(\"\n \" 'txn_id', concat('TXN-', cast(id as string)),\"\n \" 'amount', named_struct(\"\n \" 'value', cast(id % 10000 as double) / 100.0,\"\n \" 'currency', array('USD', 'EUR', 'GBP')[cast(id % 3 as int)],\"\n \" 'exchange', named_struct(\"\n \" 'rate', 1.0 + cast(id % 100 as double) / 1000.0,\"\n \" 'source', 'market',\"\n \" 'timestamp', id * 1000\"\n \" )\"\n \" ),\"\n \" 'parties', named_struct(\"\n \" 'sender', named_struct(\"\n \" 'account', concat('ACC-', lpad(cast(id % 100000 as string), 8, '0')),\"\n \" 'bank', named_struct(\"\n \" 'code', concat('BNK-', cast(id % 100 as string)),\"\n \" 'country', array('US', 'UK', 'DE', 'JP')[cast(id % 4 as int)]\"\n \" )\"\n \" ),\"\n \" 'receiver', named_struct(\"\n \" 'account', concat('ACC-', lpad(cast((id + 50000) % 100000 as string), 8, '0')),\"\n \" 'bank', named_struct(\"\n \" 'code', concat('BNK-', cast((id + 50) % 100 as string)),\"\n \" 'country', array('US', 'UK', 'DE', 'JP')[cast((id + 1) % 4 as int)]\"\n \" )\"\n \" )\"\n \" )\"\n \") as financial_txn\",\n\n # Product catalog entry (80)\n \"named_struct(\"\n \" 'product_id', concat('PROD-', lpad(cast(id % 10000 as string), 6, '0')),\"\n \" 'variants', array(\"\n \" named_struct(\"\n \" 'sku', concat('VAR-', cast(id % 1000 as string), '-A'),\"\n \" 'attributes', map('color', 'red', 'size', 'S'),\"\n \" 'inventory', named_struct('quantity', cast(id % 100 as int), 'warehouse', 'WH-1')\"\n \" ),\"\n \" named_struct(\"\n \" 'sku', concat('VAR-', cast(id % 1000 as string), '-B'),\"\n \" 'attributes', map('color', 'blue', 'size', 'M'),\"\n \" 'inventory', named_struct('quantity', cast((id + 10) % 100 as int), 'warehouse', 'WH-2')\"\n \" )\"\n \" ),\"\n \" 'pricing', named_struct(\"\n \" 'list_price', cast(id % 1000 as double) + 0.99,\"\n \" 'tiers', array(\"\n \" named_struct('min_qty', 1, 'price', cast(id % 1000 as double) + 0.99),\"\n \" named_struct('min_qty', 10, 'price', cast(id % 1000 as double) * 0.9 + 0.99),\"\n \" named_struct('min_qty', 100, 'price', cast(id % 1000 as double) * 0.8 + 0.99)\"\n \" )\"\n \" )\"\n \") as product_catalog\",\n\n # Additional scalar columns (81-90)\n \"cast((id * 53) % 10000 as int) as metric_1\",\n \"cast((id * 59) % 10000 as int) as metric_2\",\n \"cast((id * 61) % 10000 as int) as metric_3\",\n \"cast((id * 67) % 1000000 as long) as counter_1\",\n \"cast((id * 71) % 1000000 as long) as counter_2\",\n \"cast((id * 73) % 10000 as double) / 100.0 as measure_1\",\n \"cast((id * 79) % 10000 as double) / 100.0 as measure_2\",\n \"concat('label_', cast(id % 500 as string)) as label_1\",\n \"concat('category_', cast(id % 200 as string)) as label_2\",\n \"id % 17 = 0 as flag_1\",\n\n # Additional complex columns (91-95)\n \"array(\"\n \" named_struct('ts', id * 1000, 'value', cast(id % 100 as double)),\"\n \" named_struct('ts', id * 1000 + 1000, 'value', cast((id + 1) % 100 as double)),\"\n \" named_struct('ts', id * 1000 + 2000, 'value', cast((id + 2) % 100 as double))\"\n \") as time_series\",\n\n \"map(\"\n \" 'en', concat('English text ', cast(id % 100 as string)),\"\n \" 'es', concat('Spanish texto ', cast(id % 100 as string)),\"\n \" 'fr', concat('French texte ', cast(id % 100 as string))\"\n \") as translations\",\n\n \"named_struct(\"\n \" 'rules', array(\"\n \" named_struct('id', cast(id % 100 as int), 'condition', concat('cond_', cast(id % 10 as string)), 'action', concat('act_', cast(id % 5 as string))),\"\n \" named_struct('id', cast((id + 1) % 100 as int), 'condition', concat('cond_', cast((id + 1) % 10 as string)), 'action', concat('act_', cast((id + 1) % 5 as string)))\"\n \" ),\"\n \" 'default_action', 'none',\"\n \" 'priority', cast(id % 10 as int)\"\n \") as rule_engine\",\n\n \"array(\"\n \" map('metric', 'cpu', 'value', cast(id % 100 as double), 'unit', 'percent'),\"\n \" map('metric', 'memory', 'value', cast((id * 2) % 100 as double), 'unit', 'percent'),\"\n \" map('metric', 'disk', 'value', cast((id * 3) % 100 as double), 'unit', 'percent')\"\n \") as system_metrics\",\n\n \"named_struct(\"\n \" 'permissions', map(\"\n \" 'read', array('user', 'admin'),\"\n \" 'write', array('admin'),\"\n \" 'delete', array('admin')\"\n \" ),\"\n \" 'roles', array(\"\n \" named_struct('name', 'viewer', 'level', 1),\"\n \" named_struct('name', 'editor', 'level', 2),\"\n \" named_struct('name', 'admin', 'level', 3)\"\n \" )\"\n \") as access_control\",\n\n # Final columns (96-100)\n \"cast((id * 83) % 1000 as int) as final_metric_1\",\n \"cast((id * 89) % 10000 as double) / 100.0 as final_measure_1\",\n \"concat('final_', cast(id % 1000 as string)) as final_label\",\n \"array(cast(id % 5 as int), cast((id + 1) % 5 as int), cast((id + 2) % 5 as int), cast((id + 3) % 5 as int), cast((id + 4) % 5 as int)) as final_array\",\n \"named_struct(\"\n \" 'summary', concat('Summary for record ', cast(id as string)),\"\n \" 'checksum', concat(cast(id % 256 as string), '-', cast((id * 7) % 256 as string), '-', cast((id * 13) % 256 as string)),\"\n \" 'version', cast(id % 100 as int)\"\n \") as final_metadata\"\n )\n\n print(f\"Generated schema with {len(df.columns)} columns\")\n df.printSchema()\n\n # Write as parquet\n df.write.mode(\"overwrite\").parquet(output_path)\n\n # Verify the data\n written_df = spark.read.parquet(output_path)\n actual_count = written_df.count()\n print(f\"Wrote {actual_count:,} rows to {output_path}\")\n\n spark.stop()\n\n\ndef main():\n parser = argparse.ArgumentParser(\n description=\"Generate test data for shuffle benchmark\"\n )\n parser.add_argument(\n \"--output\", \"-o\",\n required=True,\n help=\"Output path for parquet data (local path or hdfs://...)\"\n )\n parser.add_argument(\n \"--rows\", \"-r\",\n type=int,\n default=10_000_000,\n help=\"Number of rows to generate (default: 10000000 for ~1GB with wide schema)\"\n )\n parser.add_argument(\n \"--partitions\", \"-p\",\n type=int,\n default=None,\n help=\"Number of output partitions (default: auto based on cluster)\"\n )\n\n args = parser.parse_args()\n\n # Default partitions to a reasonable number if not specified\n num_partitions = args.partitions if args.partitions else 200\n\n generate_data(args.output, args.rows, num_partitions)\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "benchmarks/pyspark/run_all_benchmarks.sh", "content": "#!/bin/bash\n#\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n# Run all shuffle benchmarks (Spark, Comet JVM, Comet Native)\n# Check the Spark UI during each run to compare shuffle sizes\n\nset -e\n\nSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\nDATA_PATH=\"${1:-/tmp/shuffle-benchmark-data}\"\nCOMET_JAR=\"${COMET_JAR:-$SCRIPT_DIR/../../spark/target/comet-spark-spark3.5_2.12-0.14.0-SNAPSHOT.jar}\"\nSPARK_MASTER=\"${SPARK_MASTER:-local[*]}\"\nEXECUTOR_MEMORY=\"${EXECUTOR_MEMORY:-16g}\"\nEVENT_LOG_DIR=\"${EVENT_LOG_DIR:-/tmp/spark-events}\"\n\n# Create event log directory\nmkdir -p \"$EVENT_LOG_DIR\"\n\necho \"========================================\"\necho \"Shuffle Size Comparison Benchmark\"\necho \"========================================\"\necho \"Data path: $DATA_PATH\"\necho \"Comet JAR: $COMET_JAR\"\necho \"Spark master: $SPARK_MASTER\"\necho \"Executor memory: $EXECUTOR_MEMORY\"\necho \"Event log dir: $EVENT_LOG_DIR\"\necho \"========================================\"\n\n# Run Spark baseline (no Comet)\necho \"\"\necho \">>> Running SPARK shuffle benchmark...\"\n$SPARK_HOME/bin/spark-submit \\\n --master \"$SPARK_MASTER\" \\\n --executor-memory \"$EXECUTOR_MEMORY\" \\\n --conf spark.eventLog.enabled=true \\\n --conf spark.eventLog.dir=\"$EVENT_LOG_DIR\" \\\n --conf spark.comet.enabled=false \\\n --conf spark.comet.exec.shuffle.enabled=false \\\n \"$SCRIPT_DIR/run_benchmark.py\" \\\n --data \"$DATA_PATH\" \\\n --mode spark\n\n# Run Comet JVM shuffle\necho \"\"\necho \">>> Running COMET JVM shuffle benchmark...\"\n$SPARK_HOME/bin/spark-submit \\\n --master \"$SPARK_MASTER\" \\\n --executor-memory \"$EXECUTOR_MEMORY\" \\\n --jars \"$COMET_JAR\" \\\n --driver-class-path \"$COMET_JAR\" \\\n --conf spark.executor.extraClassPath=\"$COMET_JAR\" \\\n --conf spark.eventLog.enabled=true \\\n --conf spark.eventLog.dir=\"$EVENT_LOG_DIR\" \\\n --conf spark.memory.offHeap.enabled=true \\\n --conf spark.memory.offHeap.size=16g \\\n --conf spark.comet.enabled=true \\\n --conf spark.comet.operator.DataWritingCommandExec.allowIncompatible=true \\\n --conf spark.comet.parquet.write.enabled=true \\\n --conf spark.comet.logFallbackReasons.enabled=true \\\n --conf spark.comet.explainFallback.enabled=true \\\n --conf spark.comet.shuffle.mode=jvm \\\n --conf spark.comet.exec.shuffle.mode=jvm \\\n --conf spark.comet.exec.replaceSortMergeJoin=true \\\n --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \\\n --conf spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions \\\n --conf spark.comet.cast.allowIncompatible=true \\\n \"$SCRIPT_DIR/run_benchmark.py\" \\\n --data \"$DATA_PATH\" \\\n --mode jvm\n\n# Run Comet Native shuffle\necho \"\"\necho \">>> Running COMET NATIVE shuffle benchmark...\"\n$SPARK_HOME/bin/spark-submit \\\n --master \"$SPARK_MASTER\" \\\n --executor-memory \"$EXECUTOR_MEMORY\" \\\n --jars \"$COMET_JAR\" \\\n --driver-class-path \"$COMET_JAR\" \\\n --conf spark.executor.extraClassPath=\"$COMET_JAR\" \\\n --conf spark.eventLog.enabled=true \\\n --conf spark.eventLog.dir=\"$EVENT_LOG_DIR\" \\\n --conf spark.memory.offHeap.enabled=true \\\n --conf spark.memory.offHeap.size=16g \\\n --conf spark.comet.enabled=true \\\n --conf spark.comet.operator.DataWritingCommandExec.allowIncompatible=true \\\n --conf spark.comet.parquet.write.enabled=true \\\n --conf spark.comet.logFallbackReasons.enabled=true \\\n --conf spark.comet.explainFallback.enabled=true \\\n --conf spark.comet.exec.shuffle.mode=native \\\n --conf spark.comet.exec.replaceSortMergeJoin=true \\\n --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \\\n --conf spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions \\\n --conf spark.comet.cast.allowIncompatible=true \\\n \"$SCRIPT_DIR/run_benchmark.py\" \\\n --data \"$DATA_PATH\" \\\n --mode native\n\necho \"\"\necho \"========================================\"\necho \"BENCHMARK COMPLETE\"\necho \"========================================\"\necho \"Event logs written to: $EVENT_LOG_DIR\"\necho \"\"\n" }, { "path": "benchmarks/pyspark/run_benchmark.py", "content": "#!/usr/bin/env python3\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n\"\"\"\nRun PySpark benchmarks.\n\nRun benchmarks by name with appropriate spark-submit configs for different modes\n(spark, jvm, native). Check the Spark UI to compare results between modes.\n\"\"\"\n\nimport argparse\nimport sys\n\nfrom pyspark.sql import SparkSession\n\nfrom benchmarks import get_benchmark, list_benchmarks\n\n\ndef main():\n parser = argparse.ArgumentParser(\n description=\"Run PySpark benchmarks\",\n formatter_class=argparse.RawDescriptionHelpFormatter,\n epilog=\"\"\"\nExamples:\n # Run hash partitioning shuffle benchmark in Spark mode\n python run_benchmark.py --data /path/to/data --mode spark --benchmark shuffle-hash\n\n # Run round-robin shuffle benchmark in Comet native mode\n python run_benchmark.py --data /path/to/data --mode native --benchmark shuffle-roundrobin\n\n # List all available benchmarks\n python run_benchmark.py --list-benchmarks\n \"\"\"\n )\n parser.add_argument(\n \"--data\", \"-d\",\n help=\"Path to input parquet data\"\n )\n parser.add_argument(\n \"--mode\", \"-m\",\n choices=[\"spark\", \"jvm\", \"native\"],\n help=\"Shuffle mode being tested\"\n )\n parser.add_argument(\n \"--benchmark\", \"-b\",\n default=\"shuffle-hash\",\n help=\"Benchmark to run (default: shuffle-hash)\"\n )\n parser.add_argument(\n \"--list-benchmarks\",\n action=\"store_true\",\n help=\"List all available benchmarks and exit\"\n )\n\n args = parser.parse_args()\n\n # Handle --list-benchmarks\n if args.list_benchmarks:\n print(\"Available benchmarks:\\n\")\n for name, description in list_benchmarks():\n print(f\" {name:25s} - {description}\")\n return 0\n\n # Validate required arguments\n if not args.data:\n parser.error(\"--data is required when running a benchmark\")\n if not args.mode:\n parser.error(\"--mode is required when running a benchmark\")\n\n # Get the benchmark class\n try:\n benchmark_cls = get_benchmark(args.benchmark)\n except KeyError as e:\n print(f\"Error: {e}\", file=sys.stderr)\n print(\"\\nUse --list-benchmarks to see available benchmarks\", file=sys.stderr)\n return 1\n\n # Create Spark session\n spark = SparkSession.builder \\\n .appName(f\"{benchmark_cls.name()}-{args.mode.upper()}\") \\\n .getOrCreate()\n\n try:\n # Create and run the benchmark\n benchmark = benchmark_cls(spark, args.data, args.mode)\n results = benchmark.execute_timed()\n\n print(\"\\nCheck Spark UI for shuffle sizes and detailed metrics\")\n return 0\n\n finally:\n spark.stop()\n\n\nif __name__ == \"__main__\":\n sys.exit(main())\n" }, { "path": "benchmarks/tpc/.gitignore", "content": "*.json\n*.png" }, { "path": "benchmarks/tpc/README.md", "content": "\n\n# Comet Benchmarking Scripts\n\nThis directory contains scripts used for generating benchmark results that are published in this repository and in\nthe Comet documentation.\n\nFor full instructions on running these benchmarks on an EC2 instance, see the [Comet Benchmarking on EC2 Guide].\n\n[Comet Benchmarking on EC2 Guide]: https://datafusion.apache.org/comet/contributor-guide/benchmarking_aws_ec2.html\n\n## Setup\n\nTPC queries are bundled in `benchmarks/tpc/queries/` (derived from TPC-H/DS under the TPC Fair Use Policy).\n\n## Usage\n\nAll benchmarks are run via `run.py`:\n\n```\npython3 run.py --engine --benchmark [options]\n```\n\n| Option | Description |\n| ------------------------- | ------------------------------------------------------------------------------- |\n| `--engine` | Engine name (matches a TOML file in `engines/`) |\n| `--benchmark` | `tpch` or `tpcds` |\n| `--iterations` | Number of iterations (default: 1) |\n| `--output` | Output directory (default: `.`) |\n| `--query` | Run a single query number |\n| `--no-restart` | Skip Spark master/worker restart |\n| `--dry-run` | Print the spark-submit command without executing |\n| `--jfr` | Enable Java Flight Recorder profiling |\n| `--jfr-dir` | Directory for JFR output files (default: `/results/jfr`) |\n| `--async-profiler` | Enable async-profiler (profiles Java + native code) |\n| `--async-profiler-dir` | Directory for async-profiler output (default: `/results/async-profiler`) |\n| `--async-profiler-event` | Event type: `cpu`, `wall`, `alloc`, `lock`, etc. (default: `cpu`) |\n| `--async-profiler-format` | Output format: `flamegraph`, `jfr`, `collapsed`, `text` (default: `flamegraph`) |\n\nAvailable engines: `spark`, `comet`, `comet-iceberg`, `gluten`\n\n## Example usage\n\nSet Spark environment variables:\n\n```shell\nexport SPARK_HOME=/opt/spark-3.5.3-bin-hadoop3/\nexport SPARK_MASTER=spark://yourhostname:7077\n```\n\nSet path to data (TPC queries are bundled in `benchmarks/tpc/queries/`):\n\n```shell\nexport TPCH_DATA=/mnt/bigdata/tpch/sf100/\n```\n\nRun Spark benchmark:\n\n```shell\nexport JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64\nsudo ./drop-caches.sh\npython3 run.py --engine spark --benchmark tpch\n```\n\nRun Comet benchmark:\n\n```shell\nexport JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64\nexport COMET_JAR=/opt/comet/comet-spark-spark3.5_2.12-0.10.0.jar\nsudo ./drop-caches.sh\npython3 run.py --engine comet --benchmark tpch\n```\n\nRun Gluten benchmark:\n\n```shell\nexport JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64\nexport GLUTEN_JAR=/opt/gluten/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar\nsudo ./drop-caches.sh\npython3 run.py --engine gluten --benchmark tpch\n```\n\nPreview a command without running it:\n\n```shell\npython3 run.py --engine comet --benchmark tpch --dry-run\n```\n\nGenerating charts:\n\n```shell\npython3 generate-comparison.py --benchmark tpch --labels \"Spark 3.5.3\" \"Comet 0.9.0\" \"Gluten 1.4.0\" --title \"TPC-H @ 100 GB (single executor, 8 cores, local Parquet files)\" spark-tpch-1752338506381.json comet-tpch-1752337818039.json gluten-tpch-1752337474344.json\n```\n\n## Engine Configuration\n\nEach engine is defined by a TOML file in `engines/`. The config specifies JARs, Spark conf overrides,\nrequired environment variables, and optional defaults/exports. See existing files for examples.\n\n## Iceberg Benchmarking\n\nComet includes native Iceberg support via iceberg-rust integration. This enables benchmarking TPC-H queries\nagainst Iceberg tables with native scan acceleration.\n\n### Prerequisites\n\nDownload the Iceberg Spark runtime JAR (required for running the benchmark):\n\n```shell\nwget https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/1.8.1/iceberg-spark-runtime-3.5_2.12-1.8.1.jar\nexport ICEBERG_JAR=/path/to/iceberg-spark-runtime-3.5_2.12-1.8.1.jar\n```\n\nNote: Table creation uses `--packages` which auto-downloads the dependency.\n\n### Create Iceberg tables\n\nConvert existing Parquet data to Iceberg format using `create-iceberg-tables.py`.\nThe script configures the Iceberg catalog automatically -- no `--conf` flags needed.\n\n```shell\nexport ICEBERG_WAREHOUSE=/mnt/bigdata/iceberg-warehouse\nmkdir -p $ICEBERG_WAREHOUSE\n\n# TPC-H\n$SPARK_HOME/bin/spark-submit \\\n --master $SPARK_MASTER \\\n --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1 \\\n --conf spark.driver.memory=8G \\\n --conf spark.executor.instances=2 \\\n --conf spark.executor.cores=8 \\\n --conf spark.cores.max=16 \\\n --conf spark.executor.memory=16g \\\n create-iceberg-tables.py \\\n --benchmark tpch \\\n --parquet-path $TPCH_DATA \\\n --warehouse $ICEBERG_WAREHOUSE\n\n# TPC-DS\n$SPARK_HOME/bin/spark-submit \\\n --master $SPARK_MASTER \\\n --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1 \\\n --conf spark.driver.memory=8G \\\n --conf spark.executor.instances=2 \\\n --conf spark.executor.cores=8 \\\n --conf spark.cores.max=16 \\\n --conf spark.executor.memory=16g \\\n create-iceberg-tables.py \\\n --benchmark tpcds \\\n --parquet-path $TPCDS_DATA \\\n --warehouse $ICEBERG_WAREHOUSE\n```\n\n### Run Iceberg benchmark\n\n```shell\nexport JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64\nexport COMET_JAR=/opt/comet/comet-spark-spark3.5_2.12-0.10.0.jar\nexport ICEBERG_JAR=/path/to/iceberg-spark-runtime-3.5_2.12-1.8.1.jar\nexport ICEBERG_WAREHOUSE=/mnt/bigdata/iceberg-warehouse\nsudo ./drop-caches.sh\npython3 run.py --engine comet-iceberg --benchmark tpch\n```\n\nThe benchmark uses Comet's native iceberg-rust integration, which is enabled by default.\nVerify native scanning is active by checking for `CometIcebergNativeScanExec` in the\nphysical plan output.\n\n### create-iceberg-tables.py options\n\n| Option | Required | Default | Description |\n| ---------------- | -------- | -------------- | ----------------------------------- |\n| `--benchmark` | Yes | | `tpch` or `tpcds` |\n| `--parquet-path` | Yes | | Path to source Parquet data |\n| `--warehouse` | Yes | | Path to Iceberg warehouse directory |\n| `--catalog` | No | `local` | Iceberg catalog name |\n| `--database` | No | benchmark name | Database name for the tables |\n\n## Running with Docker\n\nA Docker Compose setup is provided in `infra/docker/` for running benchmarks in an isolated\nSpark standalone cluster. The Docker image supports both **Linux (amd64)** and **macOS (arm64)**\nvia architecture-agnostic Java symlinks created at build time.\n\n### Build the image\n\nThe image must be built for the correct platform to match the native libraries in the\nengine JARs (e.g. Comet bundles `libcomet.so` for a specific OS/arch).\n\n```shell\ndocker build -t comet-bench -f benchmarks/tpc/infra/docker/Dockerfile .\n```\n\n### Building a compatible Comet JAR\n\nThe Comet JAR contains platform-specific native libraries (`libcomet.so` / `libcomet.dylib`).\nA JAR built on the host may not work inside the Docker container due to OS, architecture,\nor glibc version mismatches. Use `Dockerfile.build-comet` to build a JAR with compatible\nnative libraries:\n\n- **macOS (Apple Silicon):** The host JAR contains `darwin/aarch64` libraries which\n won't work in Linux containers. You **must** use the build Dockerfile.\n- **Linux:** If your host glibc version differs from the container's, the native library\n will fail to load with a `GLIBC_x.xx not found` error. The build Dockerfile uses\n Ubuntu 20.04 (glibc 2.31) for broad compatibility. Use it if you see\n `UnsatisfiedLinkError` mentioning glibc when running benchmarks.\n\n```shell\nmkdir -p output\ndocker build -t comet-builder \\\n -f benchmarks/tpc/infra/docker/Dockerfile.build-comet .\ndocker run --rm -v $(pwd)/output:/output comet-builder\nexport COMET_JAR=$(pwd)/output/comet-spark-spark3.5_2.12-*.jar\n```\n\n### Platform notes\n\n**macOS (Apple Silicon):** Docker Desktop is required.\n\n- **Memory:** Docker Desktop defaults to a small memory allocation (often 8 GB) which\n is not enough for Spark benchmarks. Go to **Docker Desktop > Settings > Resources >\n Memory** and increase it to at least 48 GB (each worker requests 16 GB for its executor\n plus overhead, and the driver needs 8 GB). Without enough memory, executors will be\n OOM-killed (exit code 137).\n- **File Sharing:** You may need to add your data directory (e.g. `/opt`) to\n **Docker Desktop > Settings > Resources > File Sharing** before mounting host volumes.\n\n**Linux (amd64):** Docker uses cgroup memory limits directly without a VM layer. No\nspecial Docker configuration is needed, but you may still need to build the Comet JAR\nusing `Dockerfile.build-comet` (see above) if your host glibc version doesn't match\nthe container's.\n\nThe Docker image auto-detects the container architecture (amd64/arm64) and sets up\narch-agnostic Java symlinks. The compose file uses `BENCH_JAVA_HOME` (not `JAVA_HOME`)\nto avoid inheriting the host's Java path into the container.\n\n### Start the cluster\n\nSet environment variables pointing to your host paths, then start the Spark master and\ntwo workers:\n\n```shell\nexport DATA_DIR=/mnt/bigdata/tpch/sf100\nexport RESULTS_DIR=/tmp/bench-results\nexport COMET_JAR=/opt/comet/comet-spark-spark3.5_2.12-0.10.0.jar\n\nmkdir -p $RESULTS_DIR/spark-events\ndocker compose -f benchmarks/tpc/infra/docker/docker-compose.yml up -d\n```\n\nSet `COMET_JAR`, `GLUTEN_JAR`, or `ICEBERG_JAR` to the host path of the engine JAR you\nwant to use. Each JAR is mounted individually into the container, so you can easily switch\nbetween versions by changing the path and restarting.\n\n### Run benchmarks\n\nUse `docker compose run --rm` to execute benchmarks. The `--rm` flag removes the\ncontainer when it exits, preventing port conflicts on subsequent runs. Pass\n`--no-restart` since the cluster is already managed by Compose, and `--output /results`\nso that output files land in the mounted results directory:\n\n```shell\ndocker compose -f benchmarks/tpc/infra/docker/docker-compose.yml \\\n run --rm -p 4040:4040 bench \\\n python3 /opt/benchmarks/run.py \\\n --engine comet --benchmark tpch --output /results --no-restart\n```\n\nThe `-p 4040:4040` flag exposes the Spark Application UI on the host. The following\nUIs are available during a benchmark run:\n\n| UI | URL |\n| ----------------- | ---------------------- |\n| Spark Master | http://localhost:8080 |\n| Worker 1 | http://localhost:8081 |\n| Worker 2 | http://localhost:8082 |\n| Spark Application | http://localhost:4040 |\n| History Server | http://localhost:18080 |\n\n> **Note:** The Master UI links to the Application UI using the container's internal\n> hostname, which is not reachable from the host. Use `http://localhost:4040` directly\n> to access the Application UI.\n\nThe Spark Application UI is only available while a benchmark is running. To inspect\ncompleted runs, uncomment the `history-server` service in `docker-compose.yml` and\nrestart the cluster. The History Server reads event logs from `$RESULTS_DIR/spark-events`.\n\nFor Gluten (requires Java 8), you must restart the **entire cluster** with `JAVA_HOME`\nset so that all services (master, workers, and bench) use Java 8:\n\n```shell\nexport BENCH_JAVA_HOME=/usr/lib/jvm/java-8-openjdk\ndocker compose -f benchmarks/tpc/infra/docker/docker-compose.yml down\ndocker compose -f benchmarks/tpc/infra/docker/docker-compose.yml up -d\n\ndocker compose -f benchmarks/tpc/infra/docker/docker-compose.yml \\\n run --rm bench \\\n python3 /opt/benchmarks/run.py \\\n --engine gluten --benchmark tpch --output /results --no-restart\n```\n\n> **Important:** Only passing `-e JAVA_HOME=...` to the `bench` container is not\n> sufficient -- the workers also need Java 8 or Gluten will fail at runtime with\n> `sun.misc.Unsafe` errors. Unset `BENCH_JAVA_HOME` (or switch it back to Java 17)\n> and restart the cluster before running Comet or Spark benchmarks.\n\n### Memory limits\n\nTwo compose files are provided for different hardware profiles:\n\n| File | Workers | Total memory | Use case |\n| --------------------------- | ------- | ------------ | ------------------------------ |\n| `docker-compose.yml` | 2 | ~74 GB | SF100+ on a workstation/server |\n| `docker-compose-laptop.yml` | 1 | ~12 GB | SF1–SF10 on a laptop |\n\n**`docker-compose.yml`** (workstation default):\n\n| Container | Container limit (`mem_limit`) | Spark JVM allocation |\n| -------------- | ----------------------------- | ------------------------- |\n| spark-worker-1 | 32 GB | 16 GB executor + overhead |\n| spark-worker-2 | 32 GB | 16 GB executor + overhead |\n| bench (driver) | 10 GB | 8 GB driver |\n| **Total** | **74 GB** | |\n\nConfigure via environment variables: `WORKER_MEM_LIMIT` (default: 32g per worker),\n`BENCH_MEM_LIMIT` (default: 10g), `WORKER_MEMORY` (default: 16g, Spark executor memory),\n`WORKER_CORES` (default: 8).\n\n### Running on a laptop with small scale factors\n\nFor local development or testing with small scale factors (e.g. SF1 or SF10), use the\nlaptop compose file which runs a single worker with reduced memory:\n\n```shell\ndocker compose -f benchmarks/tpc/infra/docker/docker-compose-laptop.yml up -d\n```\n\nThis starts one worker (4 GB executor inside an 8 GB container) and a 4 GB bench\ncontainer, totaling approximately **12 GB** of memory.\n\nThe benchmark scripts request 2 executor instances and 16 max cores by default\n(`run.py`). Spark will simply use whatever resources are available on the single worker,\nso no script changes are needed.\n\n### Comparing Parquet vs Iceberg performance\n\nRun both benchmarks and compare:\n\n```shell\npython3 generate-comparison.py --benchmark tpch \\\n --labels \"Comet (Parquet)\" \"Comet (Iceberg)\" \\\n --title \"TPC-H @ 100 GB: Parquet vs Iceberg\" \\\n comet-tpch-*.json comet-iceberg-tpch-*.json\n```\n\n## Java Flight Recorder Profiling\n\nUse the `--jfr` flag to capture JFR profiles from the Spark driver and executors.\nJFR is built into JDK 11+ so no additional dependencies are needed.\n\n```shell\npython3 run.py --engine comet --benchmark tpch --jfr\n```\n\nJFR recordings are written to `/results/jfr/` by default (configurable with\n`--jfr-dir`). The driver writes `driver.jfr` and each executor writes\n`executor.jfr` (JFR appends the PID when multiple executors share a path).\n\nWith Docker Compose, the `/results` volume is shared across all containers,\nso JFR files from both driver and executors are collected in\n`$RESULTS_DIR/jfr/` on the host:\n\n```shell\ndocker compose -f benchmarks/tpc/infra/docker/docker-compose.yml \\\n run --rm bench \\\n python3 /opt/benchmarks/run.py \\\n --engine comet --benchmark tpch --output /results --no-restart --jfr\n```\n\nOpen the `.jfr` files with [JDK Mission Control](https://jdk.java.net/jmc/),\nIntelliJ IDEA's profiler, or `jfr` CLI tool (`jfr summary driver.jfr`).\n\n## async-profiler Profiling\n\nUse the `--async-profiler` flag to capture profiles with\n[async-profiler](https://github.com/async-profiler/async-profiler). Unlike JFR,\nasync-profiler can profile **both Java and native (Rust/C++) code** in the same\nflame graph, making it especially useful for profiling Comet workloads.\n\n### Prerequisites\n\nasync-profiler must be installed on every node where the driver or executors run.\nSet `ASYNC_PROFILER_HOME` to the installation directory:\n\n```shell\n# Download and extract (Linux x64 example)\nwget https://github.com/async-profiler/async-profiler/releases/download/v3.0/async-profiler-3.0-linux-x64.tar.gz\ntar xzf async-profiler-3.0-linux-x64.tar.gz -C /opt/async-profiler --strip-components=1\nexport ASYNC_PROFILER_HOME=/opt/async-profiler\n```\n\nOn Linux, `perf_event_paranoid` must be set to allow profiling:\n\n```shell\nsudo sysctl kernel.perf_event_paranoid=1 # or 0 / -1 for full access\nsudo sysctl kernel.kptr_restrict=0 # optional: enable kernel symbols\n```\n\n### Basic usage\n\n```shell\npython3 run.py --engine comet --benchmark tpch --async-profiler\n```\n\nThis produces HTML flame graphs in `/results/async-profiler/` by default\n(`driver.html` and `executor.html`).\n\n### Choosing events and output format\n\n```shell\n# Wall-clock profiling (includes time spent waiting/sleeping)\npython3 run.py --engine comet --benchmark tpch \\\n --async-profiler --async-profiler-event wall\n\n# Allocation profiling with JFR output\npython3 run.py --engine comet --benchmark tpch \\\n --async-profiler --async-profiler-event alloc --async-profiler-format jfr\n\n# Lock contention profiling\npython3 run.py --engine comet --benchmark tpch \\\n --async-profiler --async-profiler-event lock\n```\n\n| Event | Description |\n| ------- | --------------------------------------------------- |\n| `cpu` | On-CPU time (default). Shows where CPU cycles go. |\n| `wall` | Wall-clock time. Includes threads that are blocked. |\n| `alloc` | Heap allocation profiling. |\n| `lock` | Lock contention profiling. |\n\n| Format | Extension | Description |\n| ------------ | --------- | ---------------------------------------- |\n| `flamegraph` | `.html` | Interactive HTML flame graph (default). |\n| `jfr` | `.jfr` | JFR format, viewable in JMC or IntelliJ. |\n| `collapsed` | `.txt` | Collapsed stacks for FlameGraph scripts. |\n| `text` | `.txt` | Flat text summary of hot methods. |\n\n### Docker usage\n\nThe Docker image includes async-profiler pre-installed at\n`/opt/async-profiler`. The `ASYNC_PROFILER_HOME` environment variable is\nalready set in the compose files, so no extra configuration is needed:\n\n```shell\ndocker compose -f benchmarks/tpc/infra/docker/docker-compose.yml \\\n run --rm bench \\\n python3 /opt/benchmarks/run.py \\\n --engine comet --benchmark tpch --output /results --no-restart --async-profiler\n```\n\nOutput files are collected in `$RESULTS_DIR/async-profiler/` on the host.\n\n**Note:** On Linux, the Docker container needs `--privileged` or\n`SYS_PTRACE` capability and `perf_event_paranoid <= 1` on the host for\n`cpu`/`wall` events. Allocation (`alloc`) and lock (`lock`) events work\nwithout special privileges.\n" }, { "path": "benchmarks/tpc/create-iceberg-tables.py", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n\"\"\"\nConvert TPC-H or TPC-DS Parquet data to Iceberg tables.\n\nUsage:\n spark-submit \\\n --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1 \\\n create-iceberg-tables.py \\\n --benchmark tpch \\\n --parquet-path /path/to/tpch/parquet \\\n --warehouse /path/to/iceberg-warehouse\n\n spark-submit \\\n --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1 \\\n create-iceberg-tables.py \\\n --benchmark tpcds \\\n --parquet-path /path/to/tpcds/parquet \\\n --warehouse /path/to/iceberg-warehouse\n\"\"\"\n\nimport argparse\nimport os\nimport sys\nfrom pyspark.sql import SparkSession\nimport time\n\nTPCH_TABLES = [\n \"customer\",\n \"lineitem\",\n \"nation\",\n \"orders\",\n \"part\",\n \"partsupp\",\n \"region\",\n \"supplier\",\n]\n\nTPCDS_TABLES = [\n \"call_center\",\n \"catalog_page\",\n \"catalog_returns\",\n \"catalog_sales\",\n \"customer\",\n \"customer_address\",\n \"customer_demographics\",\n \"date_dim\",\n \"time_dim\",\n \"household_demographics\",\n \"income_band\",\n \"inventory\",\n \"item\",\n \"promotion\",\n \"reason\",\n \"ship_mode\",\n \"store\",\n \"store_returns\",\n \"store_sales\",\n \"warehouse\",\n \"web_page\",\n \"web_returns\",\n \"web_sales\",\n \"web_site\",\n]\n\nBENCHMARK_TABLES = {\n \"tpch\": TPCH_TABLES,\n \"tpcds\": TPCDS_TABLES,\n}\n\n\ndef main(benchmark: str, parquet_path: str, warehouse: str, catalog: str, database: str):\n table_names = BENCHMARK_TABLES[benchmark]\n\n # Validate paths before starting Spark\n errors = []\n if not os.path.isdir(parquet_path):\n errors.append(f\"Error: --parquet-path '{parquet_path}' does not exist or is not a directory\")\n if not os.path.isdir(warehouse):\n errors.append(f\"Error: --warehouse '{warehouse}' does not exist or is not a directory. \"\n \"Create it with: mkdir -p \" + warehouse)\n if errors:\n for e in errors:\n print(e, file=sys.stderr)\n sys.exit(1)\n\n spark = SparkSession.builder \\\n .appName(f\"Create Iceberg {benchmark.upper()} Tables\") \\\n .config(f\"spark.sql.catalog.{catalog}\", \"org.apache.iceberg.spark.SparkCatalog\") \\\n .config(f\"spark.sql.catalog.{catalog}.type\", \"hadoop\") \\\n .config(f\"spark.sql.catalog.{catalog}.warehouse\", warehouse) \\\n .getOrCreate()\n\n # Set the Iceberg catalog as the current catalog so that\n # namespace operations are routed correctly\n spark.sql(f\"USE {catalog}\")\n\n # Create namespace if it doesn't exist\n try:\n spark.sql(f\"CREATE NAMESPACE IF NOT EXISTS {database}\")\n except Exception:\n # Namespace may already exist\n pass\n\n for table in table_names:\n parquet_table_path = f\"{parquet_path}/{table}.parquet\"\n iceberg_table = f\"{catalog}.{database}.{table}\"\n\n print(f\"Converting {parquet_table_path} -> {iceberg_table}\")\n start_time = time.time()\n\n # Drop table if exists to allow re-running\n spark.sql(f\"DROP TABLE IF EXISTS {iceberg_table}\")\n\n # Read parquet and write as Iceberg\n df = spark.read.parquet(parquet_table_path)\n df.writeTo(iceberg_table).using(\"iceberg\").create()\n\n row_count = spark.table(iceberg_table).count()\n elapsed = time.time() - start_time\n print(f\" Created {iceberg_table} with {row_count} rows in {elapsed:.2f}s\")\n\n print(f\"\\nAll {benchmark.upper()} tables created successfully!\")\n print(f\"Tables available at: {catalog}.{database}.*\")\n\n spark.stop()\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(\n description=\"Convert TPC-H or TPC-DS Parquet data to Iceberg tables\"\n )\n parser.add_argument(\n \"--benchmark\", required=True, choices=[\"tpch\", \"tpcds\"],\n help=\"Benchmark whose tables to convert (tpch or tpcds)\"\n )\n parser.add_argument(\n \"--parquet-path\", required=True,\n help=\"Path to Parquet data directory\"\n )\n parser.add_argument(\n \"--warehouse\", required=True,\n help=\"Path to Iceberg warehouse directory\"\n )\n parser.add_argument(\n \"--catalog\", default=\"local\",\n help=\"Iceberg catalog name (default: 'local')\"\n )\n parser.add_argument(\n \"--database\", default=None,\n help=\"Database name to create tables in (defaults to benchmark name)\"\n )\n args = parser.parse_args()\n\n database = args.database if args.database else args.benchmark\n main(args.benchmark, args.parquet_path, args.warehouse, args.catalog, database)\n" }, { "path": "benchmarks/tpc/drop-caches.sh", "content": "#!/bin/bash\n#\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\necho 1 > /proc/sys/vm/drop_caches\n" }, { "path": "benchmarks/tpc/engines/comet-hashjoin.toml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n[engine]\nname = \"comet-hashjoin\"\n\n[env]\nrequired = [\"COMET_JAR\"]\n\n[spark_submit]\njars = [\"$COMET_JAR\"]\ndriver_class_path = [\"$COMET_JAR\"]\n\n[spark_conf]\n\"spark.driver.extraClassPath\" = \"$COMET_JAR\"\n\"spark.executor.extraClassPath\" = \"$COMET_JAR\"\n\"spark.plugins\" = \"org.apache.spark.CometPlugin\"\n\"spark.shuffle.manager\" = \"org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager\"\n\"spark.comet.scan.impl\" = \"native_datafusion\"\n\"spark.comet.exec.replaceSortMergeJoin\" = \"true\"\n\"spark.comet.expression.Cast.allowIncompatible\" = \"true\"\n" }, { "path": "benchmarks/tpc/engines/comet-iceberg-hashjoin.toml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n[engine]\nname = \"comet-iceberg-hashjoin\"\n\n[env]\nrequired = [\"COMET_JAR\", \"ICEBERG_JAR\", \"ICEBERG_WAREHOUSE\"]\n\n[env.defaults]\nICEBERG_CATALOG = \"local\"\n\n[spark_submit]\njars = [\"$COMET_JAR\", \"$ICEBERG_JAR\"]\ndriver_class_path = [\"$COMET_JAR\", \"$ICEBERG_JAR\"]\n\n[spark_conf]\n\"spark.driver.extraClassPath\" = \"$COMET_JAR:$ICEBERG_JAR\"\n\"spark.executor.extraClassPath\" = \"$COMET_JAR:$ICEBERG_JAR\"\n\"spark.plugins\" = \"org.apache.spark.CometPlugin\"\n\"spark.shuffle.manager\" = \"org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager\"\n\"spark.comet.exec.replaceSortMergeJoin\" = \"true\"\n\"spark.comet.expression.Cast.allowIncompatible\" = \"true\"\n\"spark.comet.enabled\" = \"true\"\n\"spark.comet.exec.enabled\" = \"true\"\n\"spark.comet.scan.icebergNative.enabled\" = \"true\"\n\"spark.comet.explainFallback.enabled\" = \"true\"\n\"spark.sql.catalog.${ICEBERG_CATALOG}\" = \"org.apache.iceberg.spark.SparkCatalog\"\n\"spark.sql.catalog.${ICEBERG_CATALOG}.type\" = \"hadoop\"\n\"spark.sql.catalog.${ICEBERG_CATALOG}.warehouse\" = \"$ICEBERG_WAREHOUSE\"\n\"spark.sql.defaultCatalog\" = \"${ICEBERG_CATALOG}\"\n\n[tpcbench_args]\nuse_iceberg = true\n" }, { "path": "benchmarks/tpc/engines/comet-iceberg.toml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n[engine]\nname = \"comet-iceberg\"\n\n[env]\nrequired = [\"COMET_JAR\", \"ICEBERG_JAR\", \"ICEBERG_WAREHOUSE\"]\n\n[env.defaults]\nICEBERG_CATALOG = \"local\"\n\n[spark_submit]\njars = [\"$COMET_JAR\", \"$ICEBERG_JAR\"]\ndriver_class_path = [\"$COMET_JAR\", \"$ICEBERG_JAR\"]\n\n[spark_conf]\n\"spark.driver.extraClassPath\" = \"$COMET_JAR:$ICEBERG_JAR\"\n\"spark.executor.extraClassPath\" = \"$COMET_JAR:$ICEBERG_JAR\"\n\"spark.plugins\" = \"org.apache.spark.CometPlugin\"\n\"spark.shuffle.manager\" = \"org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager\"\n\"spark.comet.expression.Cast.allowIncompatible\" = \"true\"\n\"spark.comet.enabled\" = \"true\"\n\"spark.comet.exec.enabled\" = \"true\"\n\"spark.comet.scan.icebergNative.enabled\" = \"true\"\n\"spark.comet.explainFallback.enabled\" = \"true\"\n\"spark.sql.catalog.${ICEBERG_CATALOG}\" = \"org.apache.iceberg.spark.SparkCatalog\"\n\"spark.sql.catalog.${ICEBERG_CATALOG}.type\" = \"hadoop\"\n\"spark.sql.catalog.${ICEBERG_CATALOG}.warehouse\" = \"$ICEBERG_WAREHOUSE\"\n\"spark.sql.defaultCatalog\" = \"${ICEBERG_CATALOG}\"\n\n[tpcbench_args]\nuse_iceberg = true\n" }, { "path": "benchmarks/tpc/engines/comet.toml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n[engine]\nname = \"comet\"\n\n[env]\nrequired = [\"COMET_JAR\"]\n\n[spark_submit]\njars = [\"$COMET_JAR\"]\ndriver_class_path = [\"$COMET_JAR\"]\n\n[spark_conf]\n\"spark.driver.extraClassPath\" = \"$COMET_JAR\"\n\"spark.executor.extraClassPath\" = \"$COMET_JAR\"\n\"spark.plugins\" = \"org.apache.spark.CometPlugin\"\n\"spark.shuffle.manager\" = \"org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager\"\n\"spark.comet.scan.impl\" = \"native_datafusion\"\n\"spark.comet.expression.Cast.allowIncompatible\" = \"true\"\n" }, { "path": "benchmarks/tpc/engines/gluten.toml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n[engine]\nname = \"gluten\"\n\n[env]\nrequired = [\"GLUTEN_JAR\"]\nexports = { TZ = \"UTC\" }\n\n[spark_submit]\njars = [\"$GLUTEN_JAR\"]\n\n[spark_conf]\n\"spark.plugins\" = \"org.apache.gluten.GlutenPlugin\"\n\"spark.driver.extraClassPath\" = \"${GLUTEN_JAR}\"\n\"spark.executor.extraClassPath\" = \"${GLUTEN_JAR}\"\n\"spark.gluten.sql.columnar.forceShuffledHashJoin\" = \"true\"\n\"spark.shuffle.manager\" = \"org.apache.spark.shuffle.sort.ColumnarShuffleManager\"\n\"spark.sql.session.timeZone\" = \"UTC\"\n" }, { "path": "benchmarks/tpc/engines/spark.toml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n[engine]\nname = \"spark\"\n" }, { "path": "benchmarks/tpc/generate-comparison.py", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nimport argparse\nimport json\nimport logging\nimport matplotlib.pyplot as plt\nimport numpy as np\n\nlogging.basicConfig(level=logging.INFO)\nlogger = logging.getLogger(__name__)\n\ndef geomean(data):\n return np.prod(data) ** (1 / len(data))\n\ndef get_durations(result, query_key):\n \"\"\"Extract durations from a query result, supporting both old (list) and new (dict) formats.\"\"\"\n value = result[query_key]\n if isinstance(value, dict):\n return value[\"durations\"]\n return value\n\ndef get_all_queries(results):\n \"\"\"Return the sorted union of all query keys across all result sets.\"\"\"\n all_keys = set()\n for result in results:\n all_keys.update(result.keys())\n # Filter to numeric query keys and sort numerically\n numeric_keys = []\n for k in all_keys:\n try:\n numeric_keys.append(int(k))\n except ValueError:\n pass\n return sorted(numeric_keys)\n\ndef get_common_queries(results, labels):\n \"\"\"Return queries present in ALL result sets, warning about queries missing from some files.\"\"\"\n all_queries = get_all_queries(results)\n common = []\n for query in all_queries:\n key = str(query)\n present = [labels[i] for i, r in enumerate(results) if key in r]\n missing = [labels[i] for i, r in enumerate(results) if key not in r]\n if missing:\n logger.warning(f\"Query {query}: present in [{', '.join(present)}] but missing from [{', '.join(missing)}]\")\n if not missing:\n common.append(query)\n return common\n\ndef check_result_consistency(results, labels, benchmark):\n \"\"\"Log warnings if row counts or result hashes differ across result sets.\"\"\"\n all_queries = get_all_queries(results)\n for query in all_queries:\n key = str(query)\n row_counts = []\n hashes = []\n for i, result in enumerate(results):\n if key not in result:\n continue\n value = result[key]\n if not isinstance(value, dict):\n continue\n if \"row_count\" in value:\n row_counts.append((labels[i], value[\"row_count\"]))\n if \"result_hash\" in value:\n hashes.append((labels[i], value[\"result_hash\"]))\n\n if len(row_counts) > 1:\n counts = set(rc for _, rc in row_counts)\n if len(counts) > 1:\n details = \", \".join(f\"{label}={rc}\" for label, rc in row_counts)\n logger.warning(f\"Query {query}: row count mismatch: {details}\")\n\n if len(hashes) > 1:\n hash_values = set(h for _, h in hashes)\n if len(hash_values) > 1:\n details = \", \".join(f\"{label}={h}\" for label, h in hashes)\n logger.warning(f\"Query {query}: result hash mismatch: {details}\")\n\ndef generate_query_rel_speedup_chart(baseline, comparison, label1: str, label2: str, benchmark: str, title: str, common_queries=None):\n if common_queries is None:\n common_queries = range(1, query_count(benchmark)+1)\n results = []\n for query in common_queries:\n a = np.median(np.array(get_durations(baseline, str(query))))\n b = np.median(np.array(get_durations(comparison, str(query))))\n if a > b:\n speedup = a/b-1\n else:\n speedup = -(1/(a/b)-1)\n results.append((\"q\" + str(query), round(speedup*100, 0)))\n\n results = sorted(results, key=lambda x: -x[1])\n\n queries, speedups = zip(*results)\n\n # Create figure and axis\n if benchmark == \"tpch\":\n fig, ax = plt.subplots(figsize=(10, 6))\n else:\n fig, ax = plt.subplots(figsize=(35, 10))\n\n # Create bar chart\n bars = ax.bar(queries, speedups, color='skyblue')\n\n # Add text annotations\n for bar, speedup in zip(bars, speedups):\n yval = bar.get_height()\n if yval >= 0:\n ax.text(bar.get_x() + bar.get_width() / 2.0, min(800, yval+5), f'{yval:.0f}%', va='bottom', ha='center', fontsize=8,\n color='blue', rotation=90)\n else:\n ax.text(bar.get_x() + bar.get_width() / 2.0, yval, f'{yval:.0f}%', va='top', ha='center', fontsize=8,\n color='blue', rotation=90)\n\n # Add title and labels\n ax.set_title(label2 + \" speedup over \" + label1 + \" (\" + title + \")\")\n ax.set_ylabel('Speedup Percentage (100% speedup = 2x faster)')\n ax.set_xlabel('Query')\n\n # Customize the y-axis to handle both positive and negative values better\n ax.axhline(0, color='black', linewidth=0.8)\n min_value = (min(speedups) // 100) * 100\n max_value = ((max(speedups) // 100) + 1) * 100 + 50\n if benchmark == \"tpch\":\n ax.set_ylim(min_value, max_value)\n else:\n # TODO improve this\n ax.set_ylim(-250, 300)\n\n # Show grid for better readability\n ax.yaxis.grid(True)\n\n # Save the plot as an image file\n plt.savefig(f'{benchmark}_queries_speedup_rel.png', format='png')\n\ndef generate_query_abs_speedup_chart(baseline, comparison, label1: str, label2: str, benchmark: str, title: str, common_queries=None):\n if common_queries is None:\n common_queries = range(1, query_count(benchmark)+1)\n results = []\n for query in common_queries:\n a = np.median(np.array(get_durations(baseline, str(query))))\n b = np.median(np.array(get_durations(comparison, str(query))))\n speedup = a-b\n results.append((\"q\" + str(query), round(speedup, 1)))\n\n results = sorted(results, key=lambda x: -x[1])\n\n queries, speedups = zip(*results)\n\n # Create figure and axis\n if benchmark == \"tpch\":\n fig, ax = plt.subplots(figsize=(10, 6))\n else:\n fig, ax = plt.subplots(figsize=(35, 10))\n\n # Create bar chart\n bars = ax.bar(queries, speedups, color='skyblue')\n\n # Add text annotations\n for bar, speedup in zip(bars, speedups):\n yval = bar.get_height()\n if yval >= 0:\n ax.text(bar.get_x() + bar.get_width() / 2.0, min(800, yval+5), f'{yval:.1f}', va='bottom', ha='center', fontsize=8,\n color='blue', rotation=90)\n else:\n ax.text(bar.get_x() + bar.get_width() / 2.0, yval, f'{yval:.1f}', va='top', ha='center', fontsize=8,\n color='blue', rotation=90)\n\n # Add title and labels\n ax.set_title(label2 + \" speedup over \" + label1 + \" (\" + title + \")\")\n ax.set_ylabel('Speedup (in seconds)')\n ax.set_xlabel('Query')\n\n # Customize the y-axis to handle both positive and negative values better\n ax.axhline(0, color='black', linewidth=0.8)\n min_value = min(speedups) * 2 - 20\n max_value = max(speedups) * 1.5\n ax.set_ylim(min_value, max_value)\n\n # Show grid for better readability\n ax.yaxis.grid(True)\n\n # Save the plot as an image file\n plt.savefig(f'{benchmark}_queries_speedup_abs.png', format='png')\n\ndef generate_query_comparison_chart(results, labels, benchmark: str, title: str, common_queries=None):\n if common_queries is None:\n common_queries = range(1, query_count(benchmark)+1)\n queries = []\n benches = []\n for _ in results:\n benches.append([])\n for query in common_queries:\n queries.append(\"q\" + str(query))\n for i in range(0, len(results)):\n benches[i].append(np.median(np.array(get_durations(results[i], str(query)))))\n\n # Define the width of the bars\n bar_width = 0.3\n\n # Define the positions of the bars on the x-axis\n index = np.arange(len(queries)) * 1.5\n\n # Create a bar chart\n if benchmark == \"tpch\":\n fig, ax = plt.subplots(figsize=(15, 6))\n else:\n fig, ax = plt.subplots(figsize=(35, 6))\n\n for i in range(0, len(results)):\n bar = ax.bar(index + i * bar_width, benches[i], bar_width, label=labels[i])\n\n # Add labels, title, and legend\n ax.set_title(title)\n ax.set_xlabel('Queries')\n ax.set_ylabel('Query Time (seconds)')\n ax.set_xticks(index + bar_width / 2)\n ax.set_xticklabels(queries)\n ax.legend()\n\n # Save the plot as an image file\n plt.savefig(f'{benchmark}_queries_compare.png', format='png')\n\ndef generate_summary(results, labels, benchmark: str, title: str, common_queries=None):\n if common_queries is None:\n common_queries = range(1, query_count(benchmark)+1)\n timings = []\n for _ in results:\n timings.append(0)\n\n num_queries = len([q for q in common_queries])\n for query in common_queries:\n for i in range(0, len(results)):\n timings[i] += np.median(np.array(get_durations(results[i], str(query))))\n\n # Create figure and axis\n fig, ax = plt.subplots()\n fig.set_size_inches(10, 6)\n\n # Add title and labels\n ax.set_title(title)\n ax.set_ylabel(f'Time in seconds to run {num_queries} {benchmark} queries (lower is better)')\n\n times = [round(x,0) for x in timings]\n\n # Create bar chart\n bars = ax.bar(labels, times, color='skyblue', width=0.8)\n\n # Add text annotations\n for bar in bars:\n yval = bar.get_height()\n ax.text(bar.get_x() + bar.get_width() / 2.0, yval, f'{yval}', va='bottom') # va: vertical alignment\n\n plt.savefig(f'{benchmark}_allqueries.png', format='png')\n\ndef query_count(benchmark: str):\n if benchmark == \"tpch\":\n return 22\n elif benchmark == \"tpcds\":\n return 99\n else:\n raise \"invalid benchmark name\"\n\ndef main(files, labels, benchmark: str, title: str):\n results = []\n for filename in files:\n with open(filename) as f:\n results.append(json.load(f))\n check_result_consistency(results, labels, benchmark)\n common_queries = get_common_queries(results, labels)\n if not common_queries:\n logger.error(\"No queries found in common across all result files\")\n return\n generate_summary(results, labels, benchmark, title, common_queries)\n generate_query_comparison_chart(results, labels, benchmark, title, common_queries)\n if len(files) == 2:\n generate_query_abs_speedup_chart(results[0], results[1], labels[0], labels[1], benchmark, title, common_queries)\n generate_query_rel_speedup_chart(results[0], results[1], labels[0], labels[1], benchmark, title, common_queries)\n\nif __name__ == '__main__':\n argparse = argparse.ArgumentParser(description='Generate comparison')\n argparse.add_argument('filenames', nargs='+', type=str, help='JSON result files')\n argparse.add_argument('--labels', nargs='+', type=str, help='Labels')\n argparse.add_argument('--benchmark', type=str, help='Benchmark name (tpch or tpcds)')\n argparse.add_argument('--title', type=str, help='Chart title')\n args = argparse.parse_args()\n main(args.filenames, args.labels, args.benchmark, args.title)\n" }, { "path": "benchmarks/tpc/infra/docker/Dockerfile", "content": "# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# Benchmark image for running TPC-H and TPC-DS benchmarks across engines\n# (Spark, Comet, Gluten).\n#\n# Build (from repository root):\n# docker build -t comet-bench -f benchmarks/tpc/infra/docker/Dockerfile .\n\nARG SPARK_IMAGE=apache/spark:3.5.2-python3\nFROM ${SPARK_IMAGE}\n\nUSER root\n\n# Install Java 8 (Gluten) and Java 17 (Comet) plus Python 3.\nRUN apt-get update \\\n && apt-get install -y --no-install-recommends \\\n openjdk-8-jdk-headless \\\n openjdk-17-jdk-headless \\\n python3 python3-pip procps wget \\\n && apt-get clean \\\n && rm -rf /var/lib/apt/lists/*\n\n# Install async-profiler for profiling Java + native (Rust/C++) code.\nARG ASYNC_PROFILER_VERSION=3.0\nRUN ARCH=$(uname -m) && \\\n if [ \"$ARCH\" = \"x86_64\" ]; then AP_ARCH=\"linux-x64\"; \\\n elif [ \"$ARCH\" = \"aarch64\" ]; then AP_ARCH=\"linux-aarch64\"; \\\n else echo \"Unsupported architecture: $ARCH\" && exit 1; fi && \\\n wget -q \"https://github.com/async-profiler/async-profiler/releases/download/v${ASYNC_PROFILER_VERSION}/async-profiler-${ASYNC_PROFILER_VERSION}-${AP_ARCH}.tar.gz\" \\\n -O /tmp/async-profiler.tar.gz && \\\n mkdir -p /opt/async-profiler && \\\n tar xzf /tmp/async-profiler.tar.gz -C /opt/async-profiler --strip-components=1 && \\\n rm /tmp/async-profiler.tar.gz\nENV ASYNC_PROFILER_HOME=/opt/async-profiler\n\n# Default to Java 17 (override with JAVA_HOME at runtime for Gluten).\n# Detect architecture (amd64 or arm64) so the image works on both Linux and macOS.\nARG TARGETARCH\nRUN ln -s /usr/lib/jvm/java-17-openjdk-${TARGETARCH} /usr/lib/jvm/java-17-openjdk && \\\n ln -s /usr/lib/jvm/java-8-openjdk-${TARGETARCH} /usr/lib/jvm/java-8-openjdk\nENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk\n\n# Copy the benchmark scripts into the image.\nCOPY benchmarks/tpc/run.py /opt/benchmarks/run.py\nCOPY benchmarks/tpc/tpcbench.py /opt/benchmarks/tpcbench.py\nCOPY benchmarks/tpc/engines /opt/benchmarks/engines\nCOPY benchmarks/tpc/queries /opt/benchmarks/queries\nCOPY benchmarks/tpc/create-iceberg-tables.py /opt/benchmarks/create-iceberg-tables.py\nCOPY benchmarks/tpc/generate-comparison.py /opt/benchmarks/generate-comparison.py\n\n# Engine JARs are bind-mounted or copied in at runtime via --jars.\n# Data and query paths are also bind-mounted.\n\nWORKDIR /opt/benchmarks\n\n# Defined in the base apache/spark image.\nARG spark_uid\nUSER ${spark_uid}\n" }, { "path": "benchmarks/tpc/infra/docker/Dockerfile.build-comet", "content": "# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# Build a Comet JAR with native libraries for the current platform.\n#\n# This is useful on macOS (Apple Silicon) where the host-built JAR contains\n# darwin/aarch64 native libraries but Docker containers need linux/aarch64.\n#\n# Usage (from repository root):\n# docker build -t comet-builder -f benchmarks/tpc/infra/docker/Dockerfile.build-comet .\n# docker run --rm -v $(pwd)/output:/output comet-builder\n#\n# The JAR is copied to ./output/ on the host.\n\n# Use Ubuntu 20.04 to match the GLIBC version (2.31) in apache/spark images.\nFROM ubuntu:20.04 AS builder\n\nARG TARGETARCH\nENV DEBIAN_FRONTEND=noninteractive\n\n# Install build dependencies: Java 17, Maven wrapper prerequisites, GCC 11.\n# Ubuntu 20.04's default GCC 9 has a memcmp bug (GCC #95189) that breaks aws-lc-sys.\nRUN apt-get update && apt-get install -y --no-install-recommends \\\n openjdk-17-jdk-headless \\\n curl ca-certificates git pkg-config \\\n libssl-dev unzip software-properties-common \\\n && add-apt-repository -y ppa:ubuntu-toolchain-r/test \\\n && apt-get update \\\n && apt-get install -y --no-install-recommends gcc-11 g++-11 make \\\n && update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 110 \\\n && update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-11 110 \\\n && update-alternatives --install /usr/bin/cc cc /usr/bin/gcc-11 110 \\\n && apt-get clean && rm -rf /var/lib/apt/lists/*\n\n# Install protoc 25.x (Ubuntu 22.04's protoc is too old for proto3 optional fields).\nARG PROTOC_VERSION=25.6\nRUN ARCH=$(uname -m) && \\\n if [ \"$ARCH\" = \"aarch64\" ]; then PROTOC_ARCH=\"linux-aarch_64\"; \\\n else PROTOC_ARCH=\"linux-x86_64\"; fi && \\\n curl -sLO \"https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/protoc-${PROTOC_VERSION}-${PROTOC_ARCH}.zip\" && \\\n unzip -o \"protoc-${PROTOC_VERSION}-${PROTOC_ARCH}.zip\" -d /usr/local bin/protoc && \\\n rm \"protoc-${PROTOC_VERSION}-${PROTOC_ARCH}.zip\" && \\\n protoc --version\n\n# Set JAVA_HOME and LD_LIBRARY_PATH so the Rust build can find libjvm.\nRUN ln -s /usr/lib/jvm/java-17-openjdk-${TARGETARCH} /usr/lib/jvm/java-17-openjdk\nENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk\nENV LD_LIBRARY_PATH=${JAVA_HOME}/lib/server\n\n# Install Rust.\nRUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y\nENV PATH=\"/root/.cargo/bin:${PATH}\"\n\nWORKDIR /build\n\n# Copy the full source tree.\nCOPY . .\n\n# Build native code + package the JAR (skip tests).\nRUN make release-nogit\n\n# The entrypoint copies the built JAR to /output (bind-mounted from host).\nRUN mkdir -p /output\nCMD [\"sh\", \"-c\", \"cp spark/target/comet-spark-spark3.5_2.12-*-SNAPSHOT.jar /output/ && echo 'Comet JAR copied to /output/' && ls -lh /output/*.jar\"]\n" }, { "path": "benchmarks/tpc/infra/docker/docker-compose-laptop.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# Lightweight Spark standalone cluster for TPC benchmarks on a laptop.\n#\n# Single worker, ~12 GB total memory. Suitable for SF1-SF10 testing.\n#\n# Usage:\n# export COMET_JAR=/path/to/comet-spark-0.10.0.jar\n# docker compose -f benchmarks/tpc/infra/docker/docker-compose-laptop.yml up -d\n#\n# Environment variables (set in .env or export before running):\n# BENCH_IMAGE - Docker image to use (default: comet-bench)\n# DATA_DIR - Host path to TPC data (default: /tmp/tpc-data)\n# RESULTS_DIR - Host path for results output (default: /tmp/bench-results)\n# COMET_JAR - Host path to Comet JAR\n# GLUTEN_JAR - Host path to Gluten JAR\n# ICEBERG_JAR - Host path to Iceberg Spark runtime JAR\n# BENCH_JAVA_HOME - Java home inside container (default: /usr/lib/jvm/java-17-openjdk)\n# Set to /usr/lib/jvm/java-8-openjdk for Gluten\n# ASYNC_PROFILER_HOME - async-profiler install path (default: /opt/async-profiler)\n\nx-volumes: &volumes\n - ${DATA_DIR:-/tmp/tpc-data}:/data:ro\n - ${RESULTS_DIR:-/tmp/bench-results}:/results\n - ${COMET_JAR:-/dev/null}:/jars/comet.jar:ro\n - ${GLUTEN_JAR:-/dev/null}:/jars/gluten.jar:ro\n - ${ICEBERG_JAR:-/dev/null}:/jars/iceberg.jar:ro\n - ${RESULTS_DIR:-/tmp/bench-results}/logs:/opt/spark/logs\n - ${RESULTS_DIR:-/tmp/bench-results}/work:/opt/spark/work\n\nservices:\n spark-master:\n image: ${BENCH_IMAGE:-comet-bench}\n container_name: spark-master\n hostname: spark-master\n command: /opt/spark/sbin/start-master.sh --host spark-master\n ports:\n - \"7077:7077\"\n - \"8080:8080\"\n volumes: *volumes\n environment:\n - JAVA_HOME=${BENCH_JAVA_HOME:-/usr/lib/jvm/java-17-openjdk}\n - SPARK_MASTER_HOST=spark-master\n - SPARK_NO_DAEMONIZE=true\n\n spark-worker-1:\n image: ${BENCH_IMAGE:-comet-bench}\n container_name: spark-worker-1\n hostname: spark-worker-1\n depends_on:\n - spark-master\n command: /opt/spark/sbin/start-worker.sh spark://spark-master:7077\n ports:\n - \"8081:8081\"\n volumes: *volumes\n environment:\n - JAVA_HOME=${BENCH_JAVA_HOME:-/usr/lib/jvm/java-17-openjdk}\n - SPARK_WORKER_CORES=4\n - SPARK_WORKER_MEMORY=4g\n - SPARK_NO_DAEMONIZE=true\n mem_limit: 8g\n memswap_limit: 8g\n stop_grace_period: 30s\n\n bench:\n image: ${BENCH_IMAGE:-comet-bench}\n container_name: bench-runner\n depends_on:\n - spark-master\n - spark-worker-1\n # Override 'command' to run a specific benchmark, e.g.:\n # docker compose run bench python3 /opt/benchmarks/run.py \\\n # --engine comet --benchmark tpch --no-restart\n command: [\"echo\", \"Use 'docker compose run bench python3 /opt/benchmarks/run.py ...' to run benchmarks\"]\n volumes: *volumes\n environment:\n - JAVA_HOME=${BENCH_JAVA_HOME:-/usr/lib/jvm/java-17-openjdk}\n - SPARK_HOME=/opt/spark\n - SPARK_MASTER=spark://spark-master:7077\n - COMET_JAR=/jars/comet.jar\n - GLUTEN_JAR=/jars/gluten.jar\n - ICEBERG_JAR=/jars/iceberg.jar\n - TPCH_DATA=/data\n - TPCDS_DATA=/data\n - SPARK_EVENT_LOG_DIR=/results/spark-events\n - ASYNC_PROFILER_HOME=/opt/async-profiler\n mem_limit: 4g\n memswap_limit: 4g\n" }, { "path": "benchmarks/tpc/infra/docker/docker-compose.yml", "content": "# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# Spark standalone cluster for TPC benchmarks.\n#\n# Two workers are used so that shuffles go through the network stack,\n# which better reflects real cluster behavior.\n#\n# Usage:\n# export COMET_JAR=/path/to/comet-spark-0.10.0.jar\n# docker compose -f benchmarks/tpc/infra/docker/docker-compose.yml up -d\n#\n# Environment variables (set in .env or export before running):\n# BENCH_IMAGE - Docker image to use (default: comet-bench)\n# DATA_DIR - Host path to TPC data (default: /tmp/tpc-data)\n# RESULTS_DIR - Host path for results output (default: /tmp/bench-results)\n# COMET_JAR - Host path to Comet JAR\n# GLUTEN_JAR - Host path to Gluten JAR\n# ICEBERG_JAR - Host path to Iceberg Spark runtime JAR\n# WORKER_MEM_LIMIT - Hard memory limit per worker container (default: 32g)\n# BENCH_MEM_LIMIT - Hard memory limit for the bench runner (default: 10g)\n# BENCH_JAVA_HOME - Java home inside container (default: /usr/lib/jvm/java-17-openjdk)\n# Set to /usr/lib/jvm/java-8-openjdk for Gluten\n# ASYNC_PROFILER_HOME - async-profiler install path (default: /opt/async-profiler)\n\nx-volumes: &volumes\n - ${DATA_DIR:-/tmp/tpc-data}:/data:ro\n - ${RESULTS_DIR:-/tmp/bench-results}:/results\n - ${COMET_JAR:-/dev/null}:/jars/comet.jar:ro\n - ${GLUTEN_JAR:-/dev/null}:/jars/gluten.jar:ro\n - ${ICEBERG_JAR:-/dev/null}:/jars/iceberg.jar:ro\n - ${RESULTS_DIR:-/tmp/bench-results}/logs:/opt/spark/logs\n - ${RESULTS_DIR:-/tmp/bench-results}/work:/opt/spark/work\n\nx-worker: &worker\n image: ${BENCH_IMAGE:-comet-bench}\n depends_on:\n - spark-master\n command: /opt/spark/sbin/start-worker.sh spark://spark-master:7077\n volumes: *volumes\n environment:\n - JAVA_HOME=${BENCH_JAVA_HOME:-/usr/lib/jvm/java-17-openjdk}\n - SPARK_WORKER_CORES=${WORKER_CORES:-8}\n - SPARK_WORKER_MEMORY=${WORKER_MEMORY:-16g}\n - SPARK_NO_DAEMONIZE=true\n mem_limit: ${WORKER_MEM_LIMIT:-32g}\n memswap_limit: ${WORKER_MEM_LIMIT:-32g}\n stop_grace_period: 30s\n\nservices:\n spark-master:\n image: ${BENCH_IMAGE:-comet-bench}\n container_name: spark-master\n hostname: spark-master\n command: /opt/spark/sbin/start-master.sh --host spark-master\n ports:\n - \"7077:7077\"\n - \"8080:8080\"\n volumes: *volumes\n environment:\n - JAVA_HOME=${BENCH_JAVA_HOME:-/usr/lib/jvm/java-17-openjdk}\n - SPARK_MASTER_HOST=spark-master\n - SPARK_NO_DAEMONIZE=true\n\n spark-worker-1:\n <<: *worker\n container_name: spark-worker-1\n hostname: spark-worker-1\n ports:\n - \"8081:8081\"\n\n spark-worker-2:\n <<: *worker\n container_name: spark-worker-2\n hostname: spark-worker-2\n ports:\n - \"8082:8081\"\n\n bench:\n image: ${BENCH_IMAGE:-comet-bench}\n container_name: bench-runner\n depends_on:\n - spark-master\n - spark-worker-1\n - spark-worker-2\n # Override 'command' to run a specific benchmark, e.g.:\n # docker compose run bench python3 /opt/benchmarks/run.py \\\n # --engine comet --benchmark tpch --no-restart\n command: [\"echo\", \"Use 'docker compose run bench python3 /opt/benchmarks/run.py ...' to run benchmarks\"]\n volumes: *volumes\n environment:\n - JAVA_HOME=${BENCH_JAVA_HOME:-/usr/lib/jvm/java-17-openjdk}\n - SPARK_HOME=/opt/spark\n - SPARK_MASTER=spark://spark-master:7077\n - COMET_JAR=/jars/comet.jar\n - GLUTEN_JAR=/jars/gluten.jar\n - ICEBERG_JAR=/jars/iceberg.jar\n - TPCH_DATA=/data\n - TPCDS_DATA=/data\n - SPARK_EVENT_LOG_DIR=/results/spark-events\n - ASYNC_PROFILER_HOME=/opt/async-profiler\n mem_limit: ${BENCH_MEM_LIMIT:-10g}\n memswap_limit: ${BENCH_MEM_LIMIT:-10g}\n\n # Uncomment to enable the Spark History Server for inspecting completed\n # benchmark runs at http://localhost:18080. Requires event logs in\n # $RESULTS_DIR/spark-events (created by `mkdir -p $RESULTS_DIR/spark-events`\n # before starting the cluster).\n #\n # history-server:\n # image: ${BENCH_IMAGE:-comet-bench}\n # container_name: spark-history\n # hostname: spark-history\n # command: /opt/spark/sbin/start-history-server.sh\n # ports:\n # - \"18080:18080\"\n # volumes:\n # - ${RESULTS_DIR:-/tmp/bench-results}:/results:ro\n # environment:\n # - JAVA_HOME=${BENCH_JAVA_HOME:-/usr/lib/jvm/java-17-openjdk}\n # - SPARK_HISTORY_OPTS=-Dspark.history.fs.logDirectory=/results/spark-events\n # - SPARK_NO_DAEMONIZE=true\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q1.sql", "content": "-- CometBench-DS query 1 derived from TPC-DS query 1 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith customer_total_return as\n(select sr_customer_sk as ctr_customer_sk\n,sr_store_sk as ctr_store_sk\n,sum(SR_RETURN_AMT_INC_TAX) as ctr_total_return\nfrom store_returns\n,date_dim\nwhere sr_returned_date_sk = d_date_sk\nand d_year =1999\ngroup by sr_customer_sk\n,sr_store_sk)\n select c_customer_id\nfrom customer_total_return ctr1\n,store\n,customer\nwhere ctr1.ctr_total_return > (select avg(ctr_total_return)*1.2\nfrom customer_total_return ctr2\nwhere ctr1.ctr_store_sk = ctr2.ctr_store_sk)\nand s_store_sk = ctr1.ctr_store_sk\nand s_state = 'TN'\nand ctr1.ctr_customer_sk = c_customer_sk\norder by c_customer_id\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q10.sql", "content": "-- CometBench-DS query 10 derived from TPC-DS query 10 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect \n cd_gender,\n cd_marital_status,\n cd_education_status,\n count(*) cnt1,\n cd_purchase_estimate,\n count(*) cnt2,\n cd_credit_rating,\n count(*) cnt3,\n cd_dep_count,\n count(*) cnt4,\n cd_dep_employed_count,\n count(*) cnt5,\n cd_dep_college_count,\n count(*) cnt6\n from\n customer c,customer_address ca,customer_demographics\n where\n c.c_current_addr_sk = ca.ca_address_sk and\n ca_county in ('Clinton County','Platte County','Franklin County','Louisa County','Harmon County') and\n cd_demo_sk = c.c_current_cdemo_sk and \n exists (select *\n from store_sales,date_dim\n where c.c_customer_sk = ss_customer_sk and\n ss_sold_date_sk = d_date_sk and\n d_year = 2002 and\n d_moy between 3 and 3+3) and\n (exists (select *\n from web_sales,date_dim\n where c.c_customer_sk = ws_bill_customer_sk and\n ws_sold_date_sk = d_date_sk and\n d_year = 2002 and\n d_moy between 3 ANd 3+3) or \n exists (select * \n from catalog_sales,date_dim\n where c.c_customer_sk = cs_ship_customer_sk and\n cs_sold_date_sk = d_date_sk and\n d_year = 2002 and\n d_moy between 3 and 3+3))\n group by cd_gender,\n cd_marital_status,\n cd_education_status,\n cd_purchase_estimate,\n cd_credit_rating,\n cd_dep_count,\n cd_dep_employed_count,\n cd_dep_college_count\n order by cd_gender,\n cd_marital_status,\n cd_education_status,\n cd_purchase_estimate,\n cd_credit_rating,\n cd_dep_count,\n cd_dep_employed_count,\n cd_dep_college_count\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q11.sql", "content": "-- CometBench-DS query 11 derived from TPC-DS query 11 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith year_total as (\n select c_customer_id customer_id\n ,c_first_name customer_first_name\n ,c_last_name customer_last_name\n ,c_preferred_cust_flag customer_preferred_cust_flag\n ,c_birth_country customer_birth_country\n ,c_login customer_login\n ,c_email_address customer_email_address\n ,d_year dyear\n ,sum(ss_ext_list_price-ss_ext_discount_amt) year_total\n ,'s' sale_type\n from customer\n ,store_sales\n ,date_dim\n where c_customer_sk = ss_customer_sk\n and ss_sold_date_sk = d_date_sk\n group by c_customer_id\n ,c_first_name\n ,c_last_name\n ,c_preferred_cust_flag \n ,c_birth_country\n ,c_login\n ,c_email_address\n ,d_year \n union all\n select c_customer_id customer_id\n ,c_first_name customer_first_name\n ,c_last_name customer_last_name\n ,c_preferred_cust_flag customer_preferred_cust_flag\n ,c_birth_country customer_birth_country\n ,c_login customer_login\n ,c_email_address customer_email_address\n ,d_year dyear\n ,sum(ws_ext_list_price-ws_ext_discount_amt) year_total\n ,'w' sale_type\n from customer\n ,web_sales\n ,date_dim\n where c_customer_sk = ws_bill_customer_sk\n and ws_sold_date_sk = d_date_sk\n group by c_customer_id\n ,c_first_name\n ,c_last_name\n ,c_preferred_cust_flag \n ,c_birth_country\n ,c_login\n ,c_email_address\n ,d_year\n )\n select \n t_s_secyear.customer_id\n ,t_s_secyear.customer_first_name\n ,t_s_secyear.customer_last_name\n ,t_s_secyear.customer_email_address\n from year_total t_s_firstyear\n ,year_total t_s_secyear\n ,year_total t_w_firstyear\n ,year_total t_w_secyear\n where t_s_secyear.customer_id = t_s_firstyear.customer_id\n and t_s_firstyear.customer_id = t_w_secyear.customer_id\n and t_s_firstyear.customer_id = t_w_firstyear.customer_id\n and t_s_firstyear.sale_type = 's'\n and t_w_firstyear.sale_type = 'w'\n and t_s_secyear.sale_type = 's'\n and t_w_secyear.sale_type = 'w'\n and t_s_firstyear.dyear = 1999\n and t_s_secyear.dyear = 1999+1\n and t_w_firstyear.dyear = 1999\n and t_w_secyear.dyear = 1999+1\n and t_s_firstyear.year_total > 0\n and t_w_firstyear.year_total > 0\n and case when t_w_firstyear.year_total > 0 then t_w_secyear.year_total / t_w_firstyear.year_total else 0.0 end\n > case when t_s_firstyear.year_total > 0 then t_s_secyear.year_total / t_s_firstyear.year_total else 0.0 end\n order by t_s_secyear.customer_id\n ,t_s_secyear.customer_first_name\n ,t_s_secyear.customer_last_name\n ,t_s_secyear.customer_email_address\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q12.sql", "content": "-- CometBench-DS query 12 derived from TPC-DS query 12 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect i_item_id\n ,i_item_desc \n ,i_category \n ,i_class \n ,i_current_price\n ,sum(ws_ext_sales_price) as itemrevenue \n ,sum(ws_ext_sales_price)*100/sum(sum(ws_ext_sales_price)) over\n (partition by i_class) as revenueratio\nfrom\t\n\tweb_sales\n \t,item \n \t,date_dim\nwhere \n\tws_item_sk = i_item_sk \n \tand i_category in ('Jewelry', 'Books', 'Women')\n \tand ws_sold_date_sk = d_date_sk\n\tand d_date between cast('2002-03-22' as date) \n\t\t\t\tand (cast('2002-03-22' as date) + INTERVAL '30 DAYS')\ngroup by \n\ti_item_id\n ,i_item_desc \n ,i_category\n ,i_class\n ,i_current_price\norder by \n\ti_category\n ,i_class\n ,i_item_id\n ,i_item_desc\n ,revenueratio\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q13.sql", "content": "-- CometBench-DS query 13 derived from TPC-DS query 13 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect avg(ss_quantity)\n ,avg(ss_ext_sales_price)\n ,avg(ss_ext_wholesale_cost)\n ,sum(ss_ext_wholesale_cost)\n from store_sales\n ,store\n ,customer_demographics\n ,household_demographics\n ,customer_address\n ,date_dim\n where s_store_sk = ss_store_sk\n and ss_sold_date_sk = d_date_sk and d_year = 2001\n and((ss_hdemo_sk=hd_demo_sk\n and cd_demo_sk = ss_cdemo_sk\n and cd_marital_status = 'U'\n and cd_education_status = '4 yr Degree'\n and ss_sales_price between 100.00 and 150.00\n and hd_dep_count = 3 \n )or\n (ss_hdemo_sk=hd_demo_sk\n and cd_demo_sk = ss_cdemo_sk\n and cd_marital_status = 'S'\n and cd_education_status = 'Unknown'\n and ss_sales_price between 50.00 and 100.00 \n and hd_dep_count = 1\n ) or \n (ss_hdemo_sk=hd_demo_sk\n and cd_demo_sk = ss_cdemo_sk\n and cd_marital_status = 'D'\n and cd_education_status = '2 yr Degree'\n and ss_sales_price between 150.00 and 200.00 \n and hd_dep_count = 1 \n ))\n and((ss_addr_sk = ca_address_sk\n and ca_country = 'United States'\n and ca_state in ('CO', 'MI', 'MN')\n and ss_net_profit between 100 and 200 \n ) or\n (ss_addr_sk = ca_address_sk\n and ca_country = 'United States'\n and ca_state in ('NC', 'NY', 'TX')\n and ss_net_profit between 150 and 300 \n ) or\n (ss_addr_sk = ca_address_sk\n and ca_country = 'United States'\n and ca_state in ('CA', 'NE', 'TN')\n and ss_net_profit between 50 and 250 \n ))\n;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q14.sql", "content": "-- CometBench-DS query 14 derived from TPC-DS query 14 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith cross_items as\n (select i_item_sk ss_item_sk\n from item,\n (select iss.i_brand_id brand_id\n ,iss.i_class_id class_id\n ,iss.i_category_id category_id\n from store_sales\n ,item iss\n ,date_dim d1\n where ss_item_sk = iss.i_item_sk\n and ss_sold_date_sk = d1.d_date_sk\n and d1.d_year between 1999 AND 1999 + 2\n intersect \n select ics.i_brand_id\n ,ics.i_class_id\n ,ics.i_category_id\n from catalog_sales\n ,item ics\n ,date_dim d2\n where cs_item_sk = ics.i_item_sk\n and cs_sold_date_sk = d2.d_date_sk\n and d2.d_year between 1999 AND 1999 + 2\n intersect\n select iws.i_brand_id\n ,iws.i_class_id\n ,iws.i_category_id\n from web_sales\n ,item iws\n ,date_dim d3\n where ws_item_sk = iws.i_item_sk\n and ws_sold_date_sk = d3.d_date_sk\n and d3.d_year between 1999 AND 1999 + 2)\n where i_brand_id = brand_id\n and i_class_id = class_id\n and i_category_id = category_id\n),\n avg_sales as\n (select avg(quantity*list_price) average_sales\n from (select ss_quantity quantity\n ,ss_list_price list_price\n from store_sales\n ,date_dim\n where ss_sold_date_sk = d_date_sk\n and d_year between 1999 and 1999 + 2\n union all \n select cs_quantity quantity \n ,cs_list_price list_price\n from catalog_sales\n ,date_dim\n where cs_sold_date_sk = d_date_sk\n and d_year between 1999 and 1999 + 2 \n union all\n select ws_quantity quantity\n ,ws_list_price list_price\n from web_sales\n ,date_dim\n where ws_sold_date_sk = d_date_sk\n and d_year between 1999 and 1999 + 2) x)\n select channel, i_brand_id,i_class_id,i_category_id,sum(sales), sum(number_sales)\n from(\n select 'store' channel, i_brand_id,i_class_id\n ,i_category_id,sum(ss_quantity*ss_list_price) sales\n , count(*) number_sales\n from store_sales\n ,item\n ,date_dim\n where ss_item_sk in (select ss_item_sk from cross_items)\n and ss_item_sk = i_item_sk\n and ss_sold_date_sk = d_date_sk\n and d_year = 1999+2 \n and d_moy = 11\n group by i_brand_id,i_class_id,i_category_id\n having sum(ss_quantity*ss_list_price) > (select average_sales from avg_sales)\n union all\n select 'catalog' channel, i_brand_id,i_class_id,i_category_id, sum(cs_quantity*cs_list_price) sales, count(*) number_sales\n from catalog_sales\n ,item\n ,date_dim\n where cs_item_sk in (select ss_item_sk from cross_items)\n and cs_item_sk = i_item_sk\n and cs_sold_date_sk = d_date_sk\n and d_year = 1999+2 \n and d_moy = 11\n group by i_brand_id,i_class_id,i_category_id\n having sum(cs_quantity*cs_list_price) > (select average_sales from avg_sales)\n union all\n select 'web' channel, i_brand_id,i_class_id,i_category_id, sum(ws_quantity*ws_list_price) sales , count(*) number_sales\n from web_sales\n ,item\n ,date_dim\n where ws_item_sk in (select ss_item_sk from cross_items)\n and ws_item_sk = i_item_sk\n and ws_sold_date_sk = d_date_sk\n and d_year = 1999+2\n and d_moy = 11\n group by i_brand_id,i_class_id,i_category_id\n having sum(ws_quantity*ws_list_price) > (select average_sales from avg_sales)\n ) y\n group by rollup (channel, i_brand_id,i_class_id,i_category_id)\n order by channel,i_brand_id,i_class_id,i_category_id\n LIMIT 100;\nwith cross_items as\n (select i_item_sk ss_item_sk\n from item,\n (select iss.i_brand_id brand_id\n ,iss.i_class_id class_id\n ,iss.i_category_id category_id\n from store_sales\n ,item iss\n ,date_dim d1\n where ss_item_sk = iss.i_item_sk\n and ss_sold_date_sk = d1.d_date_sk\n and d1.d_year between 1999 AND 1999 + 2\n intersect\n select ics.i_brand_id\n ,ics.i_class_id\n ,ics.i_category_id\n from catalog_sales\n ,item ics\n ,date_dim d2\n where cs_item_sk = ics.i_item_sk\n and cs_sold_date_sk = d2.d_date_sk\n and d2.d_year between 1999 AND 1999 + 2\n intersect\n select iws.i_brand_id\n ,iws.i_class_id\n ,iws.i_category_id\n from web_sales\n ,item iws\n ,date_dim d3\n where ws_item_sk = iws.i_item_sk\n and ws_sold_date_sk = d3.d_date_sk\n and d3.d_year between 1999 AND 1999 + 2) x\n where i_brand_id = brand_id\n and i_class_id = class_id\n and i_category_id = category_id\n),\n avg_sales as\n(select avg(quantity*list_price) average_sales\n from (select ss_quantity quantity\n ,ss_list_price list_price\n from store_sales\n ,date_dim\n where ss_sold_date_sk = d_date_sk\n and d_year between 1999 and 1999 + 2\n union all\n select cs_quantity quantity\n ,cs_list_price list_price\n from catalog_sales\n ,date_dim\n where cs_sold_date_sk = d_date_sk\n and d_year between 1999 and 1999 + 2\n union all\n select ws_quantity quantity\n ,ws_list_price list_price\n from web_sales\n ,date_dim\n where ws_sold_date_sk = d_date_sk\n and d_year between 1999 and 1999 + 2) x)\n select this_year.channel ty_channel\n ,this_year.i_brand_id ty_brand\n ,this_year.i_class_id ty_class\n ,this_year.i_category_id ty_category\n ,this_year.sales ty_sales\n ,this_year.number_sales ty_number_sales\n ,last_year.channel ly_channel\n ,last_year.i_brand_id ly_brand\n ,last_year.i_class_id ly_class\n ,last_year.i_category_id ly_category\n ,last_year.sales ly_sales\n ,last_year.number_sales ly_number_sales \n from\n (select 'store' channel, i_brand_id,i_class_id,i_category_id\n ,sum(ss_quantity*ss_list_price) sales, count(*) number_sales\n from store_sales \n ,item\n ,date_dim\n where ss_item_sk in (select ss_item_sk from cross_items)\n and ss_item_sk = i_item_sk\n and ss_sold_date_sk = d_date_sk\n and d_week_seq = (select d_week_seq\n from date_dim\n where d_year = 1999 + 1\n and d_moy = 12\n and d_dom = 14)\n group by i_brand_id,i_class_id,i_category_id\n having sum(ss_quantity*ss_list_price) > (select average_sales from avg_sales)) this_year,\n (select 'store' channel, i_brand_id,i_class_id\n ,i_category_id, sum(ss_quantity*ss_list_price) sales, count(*) number_sales\n from store_sales\n ,item\n ,date_dim\n where ss_item_sk in (select ss_item_sk from cross_items)\n and ss_item_sk = i_item_sk\n and ss_sold_date_sk = d_date_sk\n and d_week_seq = (select d_week_seq\n from date_dim\n where d_year = 1999\n and d_moy = 12\n and d_dom = 14)\n group by i_brand_id,i_class_id,i_category_id\n having sum(ss_quantity*ss_list_price) > (select average_sales from avg_sales)) last_year\n where this_year.i_brand_id= last_year.i_brand_id\n and this_year.i_class_id = last_year.i_class_id\n and this_year.i_category_id = last_year.i_category_id\n order by this_year.channel, this_year.i_brand_id, this_year.i_class_id, this_year.i_category_id\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q15.sql", "content": "-- CometBench-DS query 15 derived from TPC-DS query 15 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect ca_zip\n ,sum(cs_sales_price)\n from catalog_sales\n ,customer\n ,customer_address\n ,date_dim\n where cs_bill_customer_sk = c_customer_sk\n \tand c_current_addr_sk = ca_address_sk \n \tand ( substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475',\n '85392', '85460', '80348', '81792')\n \t or ca_state in ('CA','WA','GA')\n \t or cs_sales_price > 500)\n \tand cs_sold_date_sk = d_date_sk\n \tand d_qoy = 2 and d_year = 2002\n group by ca_zip\n order by ca_zip\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q16.sql", "content": "-- CometBench-DS query 16 derived from TPC-DS query 16 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect \n count(distinct cs_order_number) as `order count`\n ,sum(cs_ext_ship_cost) as `total shipping cost`\n ,sum(cs_net_profit) as `total net profit`\nfrom\n catalog_sales cs1\n ,date_dim\n ,customer_address\n ,call_center\nwhere\n d_date between '1999-5-01' and \n (cast('1999-5-01' as date) + INTERVAL '60 DAYS')\nand cs1.cs_ship_date_sk = d_date_sk\nand cs1.cs_ship_addr_sk = ca_address_sk\nand ca_state = 'ID'\nand cs1.cs_call_center_sk = cc_call_center_sk\nand cc_county in ('Williamson County','Williamson County','Williamson County','Williamson County',\n 'Williamson County'\n)\nand exists (select *\n from catalog_sales cs2\n where cs1.cs_order_number = cs2.cs_order_number\n and cs1.cs_warehouse_sk <> cs2.cs_warehouse_sk)\nand not exists(select *\n from catalog_returns cr1\n where cs1.cs_order_number = cr1.cr_order_number)\norder by count(distinct cs_order_number)\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q17.sql", "content": "-- CometBench-DS query 17 derived from TPC-DS query 17 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect i_item_id\n ,i_item_desc\n ,s_state\n ,count(ss_quantity) as store_sales_quantitycount\n ,avg(ss_quantity) as store_sales_quantityave\n ,stddev_samp(ss_quantity) as store_sales_quantitystdev\n ,stddev_samp(ss_quantity)/avg(ss_quantity) as store_sales_quantitycov\n ,count(sr_return_quantity) as store_returns_quantitycount\n ,avg(sr_return_quantity) as store_returns_quantityave\n ,stddev_samp(sr_return_quantity) as store_returns_quantitystdev\n ,stddev_samp(sr_return_quantity)/avg(sr_return_quantity) as store_returns_quantitycov\n ,count(cs_quantity) as catalog_sales_quantitycount ,avg(cs_quantity) as catalog_sales_quantityave\n ,stddev_samp(cs_quantity) as catalog_sales_quantitystdev\n ,stddev_samp(cs_quantity)/avg(cs_quantity) as catalog_sales_quantitycov\n from store_sales\n ,store_returns\n ,catalog_sales\n ,date_dim d1\n ,date_dim d2\n ,date_dim d3\n ,store\n ,item\n where d1.d_quarter_name = '1999Q1'\n and d1.d_date_sk = ss_sold_date_sk\n and i_item_sk = ss_item_sk\n and s_store_sk = ss_store_sk\n and ss_customer_sk = sr_customer_sk\n and ss_item_sk = sr_item_sk\n and ss_ticket_number = sr_ticket_number\n and sr_returned_date_sk = d2.d_date_sk\n and d2.d_quarter_name in ('1999Q1','1999Q2','1999Q3')\n and sr_customer_sk = cs_bill_customer_sk\n and sr_item_sk = cs_item_sk\n and cs_sold_date_sk = d3.d_date_sk\n and d3.d_quarter_name in ('1999Q1','1999Q2','1999Q3')\n group by i_item_id\n ,i_item_desc\n ,s_state\n order by i_item_id\n ,i_item_desc\n ,s_state\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q18.sql", "content": "-- CometBench-DS query 18 derived from TPC-DS query 18 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect i_item_id,\n ca_country,\n ca_state, \n ca_county,\n avg( cast(cs_quantity as decimal(12,2))) agg1,\n avg( cast(cs_list_price as decimal(12,2))) agg2,\n avg( cast(cs_coupon_amt as decimal(12,2))) agg3,\n avg( cast(cs_sales_price as decimal(12,2))) agg4,\n avg( cast(cs_net_profit as decimal(12,2))) agg5,\n avg( cast(c_birth_year as decimal(12,2))) agg6,\n avg( cast(cd1.cd_dep_count as decimal(12,2))) agg7\n from catalog_sales, customer_demographics cd1, \n customer_demographics cd2, customer, customer_address, date_dim, item\n where cs_sold_date_sk = d_date_sk and\n cs_item_sk = i_item_sk and\n cs_bill_cdemo_sk = cd1.cd_demo_sk and\n cs_bill_customer_sk = c_customer_sk and\n cd1.cd_gender = 'M' and \n cd1.cd_education_status = 'Primary' and\n c_current_cdemo_sk = cd2.cd_demo_sk and\n c_current_addr_sk = ca_address_sk and\n c_birth_month in (1,2,9,5,11,3) and\n d_year = 1998 and\n ca_state in ('MS','NE','IA'\n ,'MI','GA','NY','CO')\n group by rollup (i_item_id, ca_country, ca_state, ca_county)\n order by ca_country,\n ca_state, \n ca_county,\n\ti_item_id\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q19.sql", "content": "-- CometBench-DS query 19 derived from TPC-DS query 19 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect i_brand_id brand_id, i_brand brand, i_manufact_id, i_manufact,\n \tsum(ss_ext_sales_price) ext_price\n from date_dim, store_sales, item,customer,customer_address,store\n where d_date_sk = ss_sold_date_sk\n and ss_item_sk = i_item_sk\n and i_manager_id=8\n and d_moy=11\n and d_year=1999\n and ss_customer_sk = c_customer_sk \n and c_current_addr_sk = ca_address_sk\n and substr(ca_zip,1,5) <> substr(s_zip,1,5) \n and ss_store_sk = s_store_sk \n group by i_brand\n ,i_brand_id\n ,i_manufact_id\n ,i_manufact\n order by ext_price desc\n ,i_brand\n ,i_brand_id\n ,i_manufact_id\n ,i_manufact\n LIMIT 100 ;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q2.sql", "content": "-- CometBench-DS query 2 derived from TPC-DS query 2 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith wscs as\n (select sold_date_sk\n ,sales_price\n from (select ws_sold_date_sk sold_date_sk\n ,ws_ext_sales_price sales_price\n from web_sales \n union all\n select cs_sold_date_sk sold_date_sk\n ,cs_ext_sales_price sales_price\n from catalog_sales)),\n wswscs as \n (select d_week_seq,\n sum(case when (d_day_name='Sunday') then sales_price else null end) sun_sales,\n sum(case when (d_day_name='Monday') then sales_price else null end) mon_sales,\n sum(case when (d_day_name='Tuesday') then sales_price else null end) tue_sales,\n sum(case when (d_day_name='Wednesday') then sales_price else null end) wed_sales,\n sum(case when (d_day_name='Thursday') then sales_price else null end) thu_sales,\n sum(case when (d_day_name='Friday') then sales_price else null end) fri_sales,\n sum(case when (d_day_name='Saturday') then sales_price else null end) sat_sales\n from wscs\n ,date_dim\n where d_date_sk = sold_date_sk\n group by d_week_seq)\n select d_week_seq1\n ,round(sun_sales1/sun_sales2,2)\n ,round(mon_sales1/mon_sales2,2)\n ,round(tue_sales1/tue_sales2,2)\n ,round(wed_sales1/wed_sales2,2)\n ,round(thu_sales1/thu_sales2,2)\n ,round(fri_sales1/fri_sales2,2)\n ,round(sat_sales1/sat_sales2,2)\n from\n (select wswscs.d_week_seq d_week_seq1\n ,sun_sales sun_sales1\n ,mon_sales mon_sales1\n ,tue_sales tue_sales1\n ,wed_sales wed_sales1\n ,thu_sales thu_sales1\n ,fri_sales fri_sales1\n ,sat_sales sat_sales1\n from wswscs,date_dim \n where date_dim.d_week_seq = wswscs.d_week_seq and\n d_year = 2000) y,\n (select wswscs.d_week_seq d_week_seq2\n ,sun_sales sun_sales2\n ,mon_sales mon_sales2\n ,tue_sales tue_sales2\n ,wed_sales wed_sales2\n ,thu_sales thu_sales2\n ,fri_sales fri_sales2\n ,sat_sales sat_sales2\n from wswscs\n ,date_dim \n where date_dim.d_week_seq = wswscs.d_week_seq and\n d_year = 2000+1) z\n where d_week_seq1=d_week_seq2-53\n order by d_week_seq1;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q20.sql", "content": "-- CometBench-DS query 20 derived from TPC-DS query 20 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect i_item_id\n ,i_item_desc \n ,i_category \n ,i_class \n ,i_current_price\n ,sum(cs_ext_sales_price) as itemrevenue \n ,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over\n (partition by i_class) as revenueratio\n from\tcatalog_sales\n ,item \n ,date_dim\n where cs_item_sk = i_item_sk \n and i_category in ('Children', 'Sports', 'Music')\n and cs_sold_date_sk = d_date_sk\n and d_date between cast('2002-04-01' as date) \n \t\t\t\tand (cast('2002-04-01' as date) + INTERVAL '30 DAYS')\n group by i_item_id\n ,i_item_desc \n ,i_category\n ,i_class\n ,i_current_price\n order by i_category\n ,i_class\n ,i_item_id\n ,i_item_desc\n ,revenueratio\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q21.sql", "content": "-- CometBench-DS query 21 derived from TPC-DS query 21 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect *\n from(select w_warehouse_name\n ,i_item_id\n ,sum(case when (cast(d_date as date) < cast ('2000-05-19' as date))\n\t then inv_quantity_on_hand \n else 0 end) as inv_before\n ,sum(case when (cast(d_date as date) >= cast ('2000-05-19' as date))\n then inv_quantity_on_hand \n else 0 end) as inv_after\n from inventory\n ,warehouse\n ,item\n ,date_dim\n where i_current_price between 0.99 and 1.49\n and i_item_sk = inv_item_sk\n and inv_warehouse_sk = w_warehouse_sk\n and inv_date_sk = d_date_sk\n and d_date between (cast ('2000-05-19' as date) - INTERVAL '30 DAYS')\n and (cast ('2000-05-19' as date) + INTERVAL '30 DAYS')\n group by w_warehouse_name, i_item_id) x\n where (case when inv_before > 0 \n then inv_after / inv_before \n else null\n end) between 2.0/3.0 and 3.0/2.0\n order by w_warehouse_name\n ,i_item_id\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q22.sql", "content": "-- CometBench-DS query 22 derived from TPC-DS query 22 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect i_product_name\n ,i_brand\n ,i_class\n ,i_category\n ,avg(inv_quantity_on_hand) qoh\n from inventory\n ,date_dim\n ,item\n where inv_date_sk=d_date_sk\n and inv_item_sk=i_item_sk\n and d_month_seq between 1201 and 1201 + 11\n group by rollup(i_product_name\n ,i_brand\n ,i_class\n ,i_category)\norder by qoh, i_product_name, i_brand, i_class, i_category\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q23.sql", "content": "-- CometBench-DS query 23 derived from TPC-DS query 23 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith frequent_ss_items as \n (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date solddate,count(*) cnt\n from store_sales\n ,date_dim \n ,item\n where ss_sold_date_sk = d_date_sk\n and ss_item_sk = i_item_sk \n and d_year in (2000,2000+1,2000+2,2000+3)\n group by substr(i_item_desc,1,30),i_item_sk,d_date\n having count(*) >4),\n max_store_sales as\n (select max(csales) tpcds_cmax \n from (select c_customer_sk,sum(ss_quantity*ss_sales_price) csales\n from store_sales\n ,customer\n ,date_dim \n where ss_customer_sk = c_customer_sk\n and ss_sold_date_sk = d_date_sk\n and d_year in (2000,2000+1,2000+2,2000+3) \n group by c_customer_sk)),\n best_ss_customer as\n (select c_customer_sk,sum(ss_quantity*ss_sales_price) ssales\n from store_sales\n ,customer\n where ss_customer_sk = c_customer_sk\n group by c_customer_sk\n having sum(ss_quantity*ss_sales_price) > (95/100.0) * (select\n *\nfrom\n max_store_sales))\n select sum(sales)\n from (select cs_quantity*cs_list_price sales\n from catalog_sales\n ,date_dim \n where d_year = 2000 \n and d_moy = 3 \n and cs_sold_date_sk = d_date_sk \n and cs_item_sk in (select item_sk from frequent_ss_items)\n and cs_bill_customer_sk in (select c_customer_sk from best_ss_customer)\n union all\n select ws_quantity*ws_list_price sales\n from web_sales \n ,date_dim \n where d_year = 2000 \n and d_moy = 3 \n and ws_sold_date_sk = d_date_sk \n and ws_item_sk in (select item_sk from frequent_ss_items)\n and ws_bill_customer_sk in (select c_customer_sk from best_ss_customer)) \n LIMIT 100;\nwith frequent_ss_items as\n (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date solddate,count(*) cnt\n from store_sales\n ,date_dim\n ,item\n where ss_sold_date_sk = d_date_sk\n and ss_item_sk = i_item_sk\n and d_year in (2000,2000 + 1,2000 + 2,2000 + 3)\n group by substr(i_item_desc,1,30),i_item_sk,d_date\n having count(*) >4),\n max_store_sales as\n (select max(csales) tpcds_cmax\n from (select c_customer_sk,sum(ss_quantity*ss_sales_price) csales\n from store_sales\n ,customer\n ,date_dim \n where ss_customer_sk = c_customer_sk\n and ss_sold_date_sk = d_date_sk\n and d_year in (2000,2000+1,2000+2,2000+3)\n group by c_customer_sk)),\n best_ss_customer as\n (select c_customer_sk,sum(ss_quantity*ss_sales_price) ssales\n from store_sales\n ,customer\n where ss_customer_sk = c_customer_sk\n group by c_customer_sk\n having sum(ss_quantity*ss_sales_price) > (95/100.0) * (select\n *\n from max_store_sales))\n select c_last_name,c_first_name,sales\n from (select c_last_name,c_first_name,sum(cs_quantity*cs_list_price) sales\n from catalog_sales\n ,customer\n ,date_dim \n where d_year = 2000 \n and d_moy = 3 \n and cs_sold_date_sk = d_date_sk \n and cs_item_sk in (select item_sk from frequent_ss_items)\n and cs_bill_customer_sk in (select c_customer_sk from best_ss_customer)\n and cs_bill_customer_sk = c_customer_sk \n group by c_last_name,c_first_name\n union all\n select c_last_name,c_first_name,sum(ws_quantity*ws_list_price) sales\n from web_sales\n ,customer\n ,date_dim \n where d_year = 2000 \n and d_moy = 3 \n and ws_sold_date_sk = d_date_sk \n and ws_item_sk in (select item_sk from frequent_ss_items)\n and ws_bill_customer_sk in (select c_customer_sk from best_ss_customer)\n and ws_bill_customer_sk = c_customer_sk\n group by c_last_name,c_first_name) \n order by c_last_name,c_first_name,sales\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q24.sql", "content": "-- CometBench-DS query 24 derived from TPC-DS query 24 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith ssales as\n(select c_last_name\n ,c_first_name\n ,s_store_name\n ,ca_state\n ,s_state\n ,i_color\n ,i_current_price\n ,i_manager_id\n ,i_units\n ,i_size\n ,sum(ss_net_profit) netpaid\nfrom store_sales\n ,store_returns\n ,store\n ,item\n ,customer\n ,customer_address\nwhere ss_ticket_number = sr_ticket_number\n and ss_item_sk = sr_item_sk\n and ss_customer_sk = c_customer_sk\n and ss_item_sk = i_item_sk\n and ss_store_sk = s_store_sk\n and c_current_addr_sk = ca_address_sk\n and c_birth_country <> upper(ca_country)\n and s_zip = ca_zip\nand s_market_id=10\ngroup by c_last_name\n ,c_first_name\n ,s_store_name\n ,ca_state\n ,s_state\n ,i_color\n ,i_current_price\n ,i_manager_id\n ,i_units\n ,i_size)\nselect c_last_name\n ,c_first_name\n ,s_store_name\n ,sum(netpaid) paid\nfrom ssales\nwhere i_color = 'orchid'\ngroup by c_last_name\n ,c_first_name\n ,s_store_name\nhaving sum(netpaid) > (select 0.05*avg(netpaid)\n from ssales)\norder by c_last_name\n ,c_first_name\n ,s_store_name\n;\nwith ssales as\n(select c_last_name\n ,c_first_name\n ,s_store_name\n ,ca_state\n ,s_state\n ,i_color\n ,i_current_price\n ,i_manager_id\n ,i_units\n ,i_size\n ,sum(ss_net_profit) netpaid\nfrom store_sales\n ,store_returns\n ,store\n ,item\n ,customer\n ,customer_address\nwhere ss_ticket_number = sr_ticket_number\n and ss_item_sk = sr_item_sk\n and ss_customer_sk = c_customer_sk\n and ss_item_sk = i_item_sk\n and ss_store_sk = s_store_sk\n and c_current_addr_sk = ca_address_sk\n and c_birth_country <> upper(ca_country)\n and s_zip = ca_zip\n and s_market_id = 10\ngroup by c_last_name\n ,c_first_name\n ,s_store_name\n ,ca_state\n ,s_state\n ,i_color\n ,i_current_price\n ,i_manager_id\n ,i_units\n ,i_size)\nselect c_last_name\n ,c_first_name\n ,s_store_name\n ,sum(netpaid) paid\nfrom ssales\nwhere i_color = 'green'\ngroup by c_last_name\n ,c_first_name\n ,s_store_name\nhaving sum(netpaid) > (select 0.05*avg(netpaid)\n from ssales)\norder by c_last_name\n ,c_first_name\n ,s_store_name\n;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q25.sql", "content": "-- CometBench-DS query 25 derived from TPC-DS query 25 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect \n i_item_id\n ,i_item_desc\n ,s_store_id\n ,s_store_name\n ,min(ss_net_profit) as store_sales_profit\n ,min(sr_net_loss) as store_returns_loss\n ,min(cs_net_profit) as catalog_sales_profit\n from\n store_sales\n ,store_returns\n ,catalog_sales\n ,date_dim d1\n ,date_dim d2\n ,date_dim d3\n ,store\n ,item\n where\n d1.d_moy = 4\n and d1.d_year = 2002\n and d1.d_date_sk = ss_sold_date_sk\n and i_item_sk = ss_item_sk\n and s_store_sk = ss_store_sk\n and ss_customer_sk = sr_customer_sk\n and ss_item_sk = sr_item_sk\n and ss_ticket_number = sr_ticket_number\n and sr_returned_date_sk = d2.d_date_sk\n and d2.d_moy between 4 and 10\n and d2.d_year = 2002\n and sr_customer_sk = cs_bill_customer_sk\n and sr_item_sk = cs_item_sk\n and cs_sold_date_sk = d3.d_date_sk\n and d3.d_moy between 4 and 10 \n and d3.d_year = 2002\n group by\n i_item_id\n ,i_item_desc\n ,s_store_id\n ,s_store_name\n order by\n i_item_id\n ,i_item_desc\n ,s_store_id\n ,s_store_name\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q26.sql", "content": "-- CometBench-DS query 26 derived from TPC-DS query 26 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect i_item_id, \n avg(cs_quantity) agg1,\n avg(cs_list_price) agg2,\n avg(cs_coupon_amt) agg3,\n avg(cs_sales_price) agg4 \n from catalog_sales, customer_demographics, date_dim, item, promotion\n where cs_sold_date_sk = d_date_sk and\n cs_item_sk = i_item_sk and\n cs_bill_cdemo_sk = cd_demo_sk and\n cs_promo_sk = p_promo_sk and\n cd_gender = 'F' and \n cd_marital_status = 'M' and\n cd_education_status = '4 yr Degree' and\n (p_channel_email = 'N' or p_channel_event = 'N') and\n d_year = 2000 \n group by i_item_id\n order by i_item_id\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q27.sql", "content": "-- CometBench-DS query 27 derived from TPC-DS query 27 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect i_item_id,\n s_state, grouping(s_state) g_state,\n avg(ss_quantity) agg1,\n avg(ss_list_price) agg2,\n avg(ss_coupon_amt) agg3,\n avg(ss_sales_price) agg4\n from store_sales, customer_demographics, date_dim, store, item\n where ss_sold_date_sk = d_date_sk and\n ss_item_sk = i_item_sk and\n ss_store_sk = s_store_sk and\n ss_cdemo_sk = cd_demo_sk and\n cd_gender = 'M' and\n cd_marital_status = 'U' and\n cd_education_status = 'Secondary' and\n d_year = 2000 and\n s_state in ('TN','TN', 'TN', 'TN', 'TN', 'TN')\n group by rollup (i_item_id, s_state)\n order by i_item_id\n ,s_state\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q28.sql", "content": "-- CometBench-DS query 28 derived from TPC-DS query 28 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect *\nfrom (select avg(ss_list_price) B1_LP\n ,count(ss_list_price) B1_CNT\n ,count(distinct ss_list_price) B1_CNTD\n from store_sales\n where ss_quantity between 0 and 5\n and (ss_list_price between 28 and 28+10 \n or ss_coupon_amt between 12573 and 12573+1000\n or ss_wholesale_cost between 33 and 33+20)) B1,\n (select avg(ss_list_price) B2_LP\n ,count(ss_list_price) B2_CNT\n ,count(distinct ss_list_price) B2_CNTD\n from store_sales\n where ss_quantity between 6 and 10\n and (ss_list_price between 143 and 143+10\n or ss_coupon_amt between 5562 and 5562+1000\n or ss_wholesale_cost between 45 and 45+20)) B2,\n (select avg(ss_list_price) B3_LP\n ,count(ss_list_price) B3_CNT\n ,count(distinct ss_list_price) B3_CNTD\n from store_sales\n where ss_quantity between 11 and 15\n and (ss_list_price between 159 and 159+10\n or ss_coupon_amt between 2807 and 2807+1000\n or ss_wholesale_cost between 24 and 24+20)) B3,\n (select avg(ss_list_price) B4_LP\n ,count(ss_list_price) B4_CNT\n ,count(distinct ss_list_price) B4_CNTD\n from store_sales\n where ss_quantity between 16 and 20\n and (ss_list_price between 24 and 24+10\n or ss_coupon_amt between 3706 and 3706+1000\n or ss_wholesale_cost between 46 and 46+20)) B4,\n (select avg(ss_list_price) B5_LP\n ,count(ss_list_price) B5_CNT\n ,count(distinct ss_list_price) B5_CNTD\n from store_sales\n where ss_quantity between 21 and 25\n and (ss_list_price between 76 and 76+10\n or ss_coupon_amt between 2096 and 2096+1000\n or ss_wholesale_cost between 50 and 50+20)) B5,\n (select avg(ss_list_price) B6_LP\n ,count(ss_list_price) B6_CNT\n ,count(distinct ss_list_price) B6_CNTD\n from store_sales\n where ss_quantity between 26 and 30\n and (ss_list_price between 169 and 169+10\n or ss_coupon_amt between 10672 and 10672+1000\n or ss_wholesale_cost between 58 and 58+20)) B6\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q29.sql", "content": "-- CometBench-DS query 29 derived from TPC-DS query 29 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect \n i_item_id\n ,i_item_desc\n ,s_store_id\n ,s_store_name\n ,stddev_samp(ss_quantity) as store_sales_quantity\n ,stddev_samp(sr_return_quantity) as store_returns_quantity\n ,stddev_samp(cs_quantity) as catalog_sales_quantity\n from\n store_sales\n ,store_returns\n ,catalog_sales\n ,date_dim d1\n ,date_dim d2\n ,date_dim d3\n ,store\n ,item\n where\n d1.d_moy = 4 \n and d1.d_year = 1999\n and d1.d_date_sk = ss_sold_date_sk\n and i_item_sk = ss_item_sk\n and s_store_sk = ss_store_sk\n and ss_customer_sk = sr_customer_sk\n and ss_item_sk = sr_item_sk\n and ss_ticket_number = sr_ticket_number\n and sr_returned_date_sk = d2.d_date_sk\n and d2.d_moy between 4 and 4 + 3 \n and d2.d_year = 1999\n and sr_customer_sk = cs_bill_customer_sk\n and sr_item_sk = cs_item_sk\n and cs_sold_date_sk = d3.d_date_sk \n and d3.d_year in (1999,1999+1,1999+2)\n group by\n i_item_id\n ,i_item_desc\n ,s_store_id\n ,s_store_name\n order by\n i_item_id \n ,i_item_desc\n ,s_store_id\n ,s_store_name\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q3.sql", "content": "-- CometBench-DS query 3 derived from TPC-DS query 3 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect dt.d_year \n ,item.i_brand_id brand_id \n ,item.i_brand brand\n ,sum(ss_net_profit) sum_agg\n from date_dim dt \n ,store_sales\n ,item\n where dt.d_date_sk = store_sales.ss_sold_date_sk\n and store_sales.ss_item_sk = item.i_item_sk\n and item.i_manufact_id = 445\n and dt.d_moy=12\n group by dt.d_year\n ,item.i_brand\n ,item.i_brand_id\n order by dt.d_year\n ,sum_agg desc\n ,brand_id\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q30.sql", "content": "-- CometBench-DS query 30 derived from TPC-DS query 30 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith customer_total_return as\n (select wr_returning_customer_sk as ctr_customer_sk\n ,ca_state as ctr_state, \n \tsum(wr_return_amt) as ctr_total_return\n from web_returns\n ,date_dim\n ,customer_address\n where wr_returned_date_sk = d_date_sk \n and d_year =2000\n and wr_returning_addr_sk = ca_address_sk \n group by wr_returning_customer_sk\n ,ca_state)\n select c_customer_id,c_salutation,c_first_name,c_last_name,c_preferred_cust_flag\n ,c_birth_day,c_birth_month,c_birth_year,c_birth_country,c_login,c_email_address\n ,c_last_review_date_sk,ctr_total_return\n from customer_total_return ctr1\n ,customer_address\n ,customer\n where ctr1.ctr_total_return > (select avg(ctr_total_return)*1.2\n \t\t\t from customer_total_return ctr2 \n \t where ctr1.ctr_state = ctr2.ctr_state)\n and ca_address_sk = c_current_addr_sk\n and ca_state = 'KS'\n and ctr1.ctr_customer_sk = c_customer_sk\n order by c_customer_id,c_salutation,c_first_name,c_last_name,c_preferred_cust_flag\n ,c_birth_day,c_birth_month,c_birth_year,c_birth_country,c_login,c_email_address\n ,c_last_review_date_sk,ctr_total_return\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q31.sql", "content": "-- CometBench-DS query 31 derived from TPC-DS query 31 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith ss as\n (select ca_county,d_qoy, d_year,sum(ss_ext_sales_price) as store_sales\n from store_sales,date_dim,customer_address\n where ss_sold_date_sk = d_date_sk\n and ss_addr_sk=ca_address_sk\n group by ca_county,d_qoy, d_year),\n ws as\n (select ca_county,d_qoy, d_year,sum(ws_ext_sales_price) as web_sales\n from web_sales,date_dim,customer_address\n where ws_sold_date_sk = d_date_sk\n and ws_bill_addr_sk=ca_address_sk\n group by ca_county,d_qoy, d_year)\n select \n ss1.ca_county\n ,ss1.d_year\n ,ws2.web_sales/ws1.web_sales web_q1_q2_increase\n ,ss2.store_sales/ss1.store_sales store_q1_q2_increase\n ,ws3.web_sales/ws2.web_sales web_q2_q3_increase\n ,ss3.store_sales/ss2.store_sales store_q2_q3_increase\n from\n ss ss1\n ,ss ss2\n ,ss ss3\n ,ws ws1\n ,ws ws2\n ,ws ws3\n where\n ss1.d_qoy = 1\n and ss1.d_year = 1999\n and ss1.ca_county = ss2.ca_county\n and ss2.d_qoy = 2\n and ss2.d_year = 1999\n and ss2.ca_county = ss3.ca_county\n and ss3.d_qoy = 3\n and ss3.d_year = 1999\n and ss1.ca_county = ws1.ca_county\n and ws1.d_qoy = 1\n and ws1.d_year = 1999\n and ws1.ca_county = ws2.ca_county\n and ws2.d_qoy = 2\n and ws2.d_year = 1999\n and ws1.ca_county = ws3.ca_county\n and ws3.d_qoy = 3\n and ws3.d_year =1999\n and case when ws1.web_sales > 0 then ws2.web_sales/ws1.web_sales else null end \n > case when ss1.store_sales > 0 then ss2.store_sales/ss1.store_sales else null end\n and case when ws2.web_sales > 0 then ws3.web_sales/ws2.web_sales else null end\n > case when ss2.store_sales > 0 then ss3.store_sales/ss2.store_sales else null end\n order by ss1.ca_county;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q32.sql", "content": "-- CometBench-DS query 32 derived from TPC-DS query 32 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect sum(cs_ext_discount_amt) as `excess discount amount` \nfrom \n catalog_sales \n ,item \n ,date_dim\nwhere\ni_manufact_id = 283\nand i_item_sk = cs_item_sk \nand d_date between '1999-02-22' and \n (cast('1999-02-22' as date) + INTERVAL '90 DAYS')\nand d_date_sk = cs_sold_date_sk \nand cs_ext_discount_amt \n > ( \n select \n 1.3 * avg(cs_ext_discount_amt) \n from \n catalog_sales \n ,date_dim\n where \n cs_item_sk = i_item_sk \n and d_date between '1999-02-22' and\n (cast('1999-02-22' as date) + INTERVAL '90 DAYS')\n and d_date_sk = cs_sold_date_sk \n ) \n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q33.sql", "content": "-- CometBench-DS query 33 derived from TPC-DS query 33 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith ss as (\n select\n i_manufact_id,sum(ss_ext_sales_price) total_sales\n from\n \tstore_sales,\n \tdate_dim,\n customer_address,\n item\n where\n i_manufact_id in (select\n i_manufact_id\nfrom\n item\nwhere i_category in ('Books'))\n and ss_item_sk = i_item_sk\n and ss_sold_date_sk = d_date_sk\n and d_year = 1999\n and d_moy = 4\n and ss_addr_sk = ca_address_sk\n and ca_gmt_offset = -5 \n group by i_manufact_id),\n cs as (\n select\n i_manufact_id,sum(cs_ext_sales_price) total_sales\n from\n \tcatalog_sales,\n \tdate_dim,\n customer_address,\n item\n where\n i_manufact_id in (select\n i_manufact_id\nfrom\n item\nwhere i_category in ('Books'))\n and cs_item_sk = i_item_sk\n and cs_sold_date_sk = d_date_sk\n and d_year = 1999\n and d_moy = 4\n and cs_bill_addr_sk = ca_address_sk\n and ca_gmt_offset = -5 \n group by i_manufact_id),\n ws as (\n select\n i_manufact_id,sum(ws_ext_sales_price) total_sales\n from\n \tweb_sales,\n \tdate_dim,\n customer_address,\n item\n where\n i_manufact_id in (select\n i_manufact_id\nfrom\n item\nwhere i_category in ('Books'))\n and ws_item_sk = i_item_sk\n and ws_sold_date_sk = d_date_sk\n and d_year = 1999\n and d_moy = 4\n and ws_bill_addr_sk = ca_address_sk\n and ca_gmt_offset = -5\n group by i_manufact_id)\n select i_manufact_id ,sum(total_sales) total_sales\n from (select * from ss \n union all\n select * from cs \n union all\n select * from ws) tmp1\n group by i_manufact_id\n order by total_sales\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q34.sql", "content": "-- CometBench-DS query 34 derived from TPC-DS query 34 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect c_last_name\n ,c_first_name\n ,c_salutation\n ,c_preferred_cust_flag\n ,ss_ticket_number\n ,cnt from\n (select ss_ticket_number\n ,ss_customer_sk\n ,count(*) cnt\n from store_sales,date_dim,store,household_demographics\n where store_sales.ss_sold_date_sk = date_dim.d_date_sk\n and store_sales.ss_store_sk = store.s_store_sk \n and store_sales.ss_hdemo_sk = household_demographics.hd_demo_sk\n and (date_dim.d_dom between 1 and 3 or date_dim.d_dom between 25 and 28)\n and (household_demographics.hd_buy_potential = '501-1000' or\n household_demographics.hd_buy_potential = 'Unknown')\n and household_demographics.hd_vehicle_count > 0\n and (case when household_demographics.hd_vehicle_count > 0 \n\tthen household_demographics.hd_dep_count/ household_demographics.hd_vehicle_count \n\telse null \n\tend) > 1.2\n and date_dim.d_year in (2000,2000+1,2000+2)\n and store.s_county in ('Williamson County','Williamson County','Williamson County','Williamson County',\n 'Williamson County','Williamson County','Williamson County','Williamson County')\n group by ss_ticket_number,ss_customer_sk) dn,customer\n where ss_customer_sk = c_customer_sk\n and cnt between 15 and 20\n order by c_last_name,c_first_name,c_salutation,c_preferred_cust_flag desc, ss_ticket_number;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q35.sql", "content": "-- CometBench-DS query 35 derived from TPC-DS query 35 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect \n ca_state,\n cd_gender,\n cd_marital_status,\n cd_dep_count,\n count(*) cnt1,\n max(cd_dep_count),\n stddev_samp(cd_dep_count),\n stddev_samp(cd_dep_count),\n cd_dep_employed_count,\n count(*) cnt2,\n max(cd_dep_employed_count),\n stddev_samp(cd_dep_employed_count),\n stddev_samp(cd_dep_employed_count),\n cd_dep_college_count,\n count(*) cnt3,\n max(cd_dep_college_count),\n stddev_samp(cd_dep_college_count),\n stddev_samp(cd_dep_college_count)\n from\n customer c,customer_address ca,customer_demographics\n where\n c.c_current_addr_sk = ca.ca_address_sk and\n cd_demo_sk = c.c_current_cdemo_sk and \n exists (select *\n from store_sales,date_dim\n where c.c_customer_sk = ss_customer_sk and\n ss_sold_date_sk = d_date_sk and\n d_year = 2000 and\n d_qoy < 4) and\n (exists (select *\n from web_sales,date_dim\n where c.c_customer_sk = ws_bill_customer_sk and\n ws_sold_date_sk = d_date_sk and\n d_year = 2000 and\n d_qoy < 4) or \n exists (select * \n from catalog_sales,date_dim\n where c.c_customer_sk = cs_ship_customer_sk and\n cs_sold_date_sk = d_date_sk and\n d_year = 2000 and\n d_qoy < 4))\n group by ca_state,\n cd_gender,\n cd_marital_status,\n cd_dep_count,\n cd_dep_employed_count,\n cd_dep_college_count\n order by ca_state,\n cd_gender,\n cd_marital_status,\n cd_dep_count,\n cd_dep_employed_count,\n cd_dep_college_count\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q36.sql", "content": "-- CometBench-DS query 36 derived from TPC-DS query 36 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect \n sum(ss_net_profit)/sum(ss_ext_sales_price) as gross_margin\n ,i_category\n ,i_class\n ,grouping(i_category)+grouping(i_class) as lochierarchy\n ,rank() over (\n \tpartition by grouping(i_category)+grouping(i_class),\n \tcase when grouping(i_class) = 0 then i_category end \n \torder by sum(ss_net_profit)/sum(ss_ext_sales_price) asc) as rank_within_parent\n from\n store_sales\n ,date_dim d1\n ,item\n ,store\n where\n d1.d_year = 2001 \n and d1.d_date_sk = ss_sold_date_sk\n and i_item_sk = ss_item_sk \n and s_store_sk = ss_store_sk\n and s_state in ('TN','TN','TN','TN',\n 'TN','TN','TN','TN')\n group by rollup(i_category,i_class)\n order by\n lochierarchy desc\n ,case when lochierarchy = 0 then i_category end\n ,rank_within_parent\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q37.sql", "content": "-- CometBench-DS query 37 derived from TPC-DS query 37 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect i_item_id\n ,i_item_desc\n ,i_current_price\n from item, inventory, date_dim, catalog_sales\n where i_current_price between 26 and 26 + 30\n and inv_item_sk = i_item_sk\n and d_date_sk=inv_date_sk\n and d_date between cast('2001-06-09' as date) and (cast('2001-06-09' as date) + INTERVAL '60 DAYS')\n and i_manufact_id in (744,884,722,693)\n and inv_quantity_on_hand between 100 and 500\n and cs_item_sk = i_item_sk\n group by i_item_id,i_item_desc,i_current_price\n order by i_item_id\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q38.sql", "content": "-- CometBench-DS query 38 derived from TPC-DS query 38 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect count(*) from (\n select distinct c_last_name, c_first_name, d_date\n from store_sales, date_dim, customer\n where store_sales.ss_sold_date_sk = date_dim.d_date_sk\n and store_sales.ss_customer_sk = customer.c_customer_sk\n and d_month_seq between 1190 and 1190 + 11\n intersect\n select distinct c_last_name, c_first_name, d_date\n from catalog_sales, date_dim, customer\n where catalog_sales.cs_sold_date_sk = date_dim.d_date_sk\n and catalog_sales.cs_bill_customer_sk = customer.c_customer_sk\n and d_month_seq between 1190 and 1190 + 11\n intersect\n select distinct c_last_name, c_first_name, d_date\n from web_sales, date_dim, customer\n where web_sales.ws_sold_date_sk = date_dim.d_date_sk\n and web_sales.ws_bill_customer_sk = customer.c_customer_sk\n and d_month_seq between 1190 and 1190 + 11\n) hot_cust\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q39.sql", "content": "-- CometBench-DS query 39 derived from TPC-DS query 39 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith inv as\n(select w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy\n ,stdev,mean, case mean when 0 then null else stdev/mean end cov\n from(select w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy\n ,stddev_samp(inv_quantity_on_hand) stdev,avg(inv_quantity_on_hand) mean\n from inventory\n ,item\n ,warehouse\n ,date_dim\n where inv_item_sk = i_item_sk\n and inv_warehouse_sk = w_warehouse_sk\n and inv_date_sk = d_date_sk\n and d_year =2001\n group by w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy) foo\n where case mean when 0 then 0 else stdev/mean end > 1)\nselect inv1.w_warehouse_sk,inv1.i_item_sk,inv1.d_moy,inv1.mean, inv1.cov\n ,inv2.w_warehouse_sk,inv2.i_item_sk,inv2.d_moy,inv2.mean, inv2.cov\nfrom inv inv1,inv inv2\nwhere inv1.i_item_sk = inv2.i_item_sk\n and inv1.w_warehouse_sk = inv2.w_warehouse_sk\n and inv1.d_moy=1\n and inv2.d_moy=1+1\norder by inv1.w_warehouse_sk,inv1.i_item_sk,inv1.d_moy,inv1.mean,inv1.cov\n ,inv2.d_moy,inv2.mean, inv2.cov\n;\nwith inv as\n(select w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy\n ,stdev,mean, case mean when 0 then null else stdev/mean end cov\n from(select w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy\n ,stddev_samp(inv_quantity_on_hand) stdev,avg(inv_quantity_on_hand) mean\n from inventory\n ,item\n ,warehouse\n ,date_dim\n where inv_item_sk = i_item_sk\n and inv_warehouse_sk = w_warehouse_sk\n and inv_date_sk = d_date_sk\n and d_year =2001\n group by w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy) foo\n where case mean when 0 then 0 else stdev/mean end > 1)\nselect inv1.w_warehouse_sk,inv1.i_item_sk,inv1.d_moy,inv1.mean, inv1.cov\n ,inv2.w_warehouse_sk,inv2.i_item_sk,inv2.d_moy,inv2.mean, inv2.cov\nfrom inv inv1,inv inv2\nwhere inv1.i_item_sk = inv2.i_item_sk\n and inv1.w_warehouse_sk = inv2.w_warehouse_sk\n and inv1.d_moy=1\n and inv2.d_moy=1+1\n and inv1.cov > 1.5\norder by inv1.w_warehouse_sk,inv1.i_item_sk,inv1.d_moy,inv1.mean,inv1.cov\n ,inv2.d_moy,inv2.mean, inv2.cov\n;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q4.sql", "content": "-- CometBench-DS query 4 derived from TPC-DS query 4 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith year_total as (\n select c_customer_id customer_id\n ,c_first_name customer_first_name\n ,c_last_name customer_last_name\n ,c_preferred_cust_flag customer_preferred_cust_flag\n ,c_birth_country customer_birth_country\n ,c_login customer_login\n ,c_email_address customer_email_address\n ,d_year dyear\n ,sum(((ss_ext_list_price-ss_ext_wholesale_cost-ss_ext_discount_amt)+ss_ext_sales_price)/2) year_total\n ,'s' sale_type\n from customer\n ,store_sales\n ,date_dim\n where c_customer_sk = ss_customer_sk\n and ss_sold_date_sk = d_date_sk\n group by c_customer_id\n ,c_first_name\n ,c_last_name\n ,c_preferred_cust_flag\n ,c_birth_country\n ,c_login\n ,c_email_address\n ,d_year\n union all\n select c_customer_id customer_id\n ,c_first_name customer_first_name\n ,c_last_name customer_last_name\n ,c_preferred_cust_flag customer_preferred_cust_flag\n ,c_birth_country customer_birth_country\n ,c_login customer_login\n ,c_email_address customer_email_address\n ,d_year dyear\n ,sum((((cs_ext_list_price-cs_ext_wholesale_cost-cs_ext_discount_amt)+cs_ext_sales_price)/2) ) year_total\n ,'c' sale_type\n from customer\n ,catalog_sales\n ,date_dim\n where c_customer_sk = cs_bill_customer_sk\n and cs_sold_date_sk = d_date_sk\n group by c_customer_id\n ,c_first_name\n ,c_last_name\n ,c_preferred_cust_flag\n ,c_birth_country\n ,c_login\n ,c_email_address\n ,d_year\nunion all\n select c_customer_id customer_id\n ,c_first_name customer_first_name\n ,c_last_name customer_last_name\n ,c_preferred_cust_flag customer_preferred_cust_flag\n ,c_birth_country customer_birth_country\n ,c_login customer_login\n ,c_email_address customer_email_address\n ,d_year dyear\n ,sum((((ws_ext_list_price-ws_ext_wholesale_cost-ws_ext_discount_amt)+ws_ext_sales_price)/2) ) year_total\n ,'w' sale_type\n from customer\n ,web_sales\n ,date_dim\n where c_customer_sk = ws_bill_customer_sk\n and ws_sold_date_sk = d_date_sk\n group by c_customer_id\n ,c_first_name\n ,c_last_name\n ,c_preferred_cust_flag\n ,c_birth_country\n ,c_login\n ,c_email_address\n ,d_year\n )\n select \n t_s_secyear.customer_id\n ,t_s_secyear.customer_first_name\n ,t_s_secyear.customer_last_name\n ,t_s_secyear.customer_email_address\n from year_total t_s_firstyear\n ,year_total t_s_secyear\n ,year_total t_c_firstyear\n ,year_total t_c_secyear\n ,year_total t_w_firstyear\n ,year_total t_w_secyear\n where t_s_secyear.customer_id = t_s_firstyear.customer_id\n and t_s_firstyear.customer_id = t_c_secyear.customer_id\n and t_s_firstyear.customer_id = t_c_firstyear.customer_id\n and t_s_firstyear.customer_id = t_w_firstyear.customer_id\n and t_s_firstyear.customer_id = t_w_secyear.customer_id\n and t_s_firstyear.sale_type = 's'\n and t_c_firstyear.sale_type = 'c'\n and t_w_firstyear.sale_type = 'w'\n and t_s_secyear.sale_type = 's'\n and t_c_secyear.sale_type = 'c'\n and t_w_secyear.sale_type = 'w'\n and t_s_firstyear.dyear = 2001\n and t_s_secyear.dyear = 2001+1\n and t_c_firstyear.dyear = 2001\n and t_c_secyear.dyear = 2001+1\n and t_w_firstyear.dyear = 2001\n and t_w_secyear.dyear = 2001+1\n and t_s_firstyear.year_total > 0\n and t_c_firstyear.year_total > 0\n and t_w_firstyear.year_total > 0\n and case when t_c_firstyear.year_total > 0 then t_c_secyear.year_total / t_c_firstyear.year_total else null end\n > case when t_s_firstyear.year_total > 0 then t_s_secyear.year_total / t_s_firstyear.year_total else null end\n and case when t_c_firstyear.year_total > 0 then t_c_secyear.year_total / t_c_firstyear.year_total else null end\n > case when t_w_firstyear.year_total > 0 then t_w_secyear.year_total / t_w_firstyear.year_total else null end\n order by t_s_secyear.customer_id\n ,t_s_secyear.customer_first_name\n ,t_s_secyear.customer_last_name\n ,t_s_secyear.customer_email_address\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q40.sql", "content": "-- CometBench-DS query 40 derived from TPC-DS query 40 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect \n w_state\n ,i_item_id\n ,sum(case when (cast(d_date as date) < cast ('2002-05-18' as date)) \n \t\tthen cs_sales_price - coalesce(cr_refunded_cash,0) else 0 end) as sales_before\n ,sum(case when (cast(d_date as date) >= cast ('2002-05-18' as date)) \n \t\tthen cs_sales_price - coalesce(cr_refunded_cash,0) else 0 end) as sales_after\n from\n catalog_sales left outer join catalog_returns on\n (cs_order_number = cr_order_number \n and cs_item_sk = cr_item_sk)\n ,warehouse \n ,item\n ,date_dim\n where\n i_current_price between 0.99 and 1.49\n and i_item_sk = cs_item_sk\n and cs_warehouse_sk = w_warehouse_sk \n and cs_sold_date_sk = d_date_sk\n and d_date between (cast ('2002-05-18' as date) - INTERVAL '30 DAYS')\n and (cast ('2002-05-18' as date) + INTERVAL '30 DAYS') \n group by\n w_state,i_item_id\n order by w_state,i_item_id\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q41.sql", "content": "-- CometBench-DS query 41 derived from TPC-DS query 41 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect distinct(i_product_name)\n from item i1\n where i_manufact_id between 668 and 668+40 \n and (select count(*) as item_cnt\n from item\n where (i_manufact = i1.i_manufact and\n ((i_category = 'Women' and \n (i_color = 'cream' or i_color = 'ghost') and \n (i_units = 'Ton' or i_units = 'Gross') and\n (i_size = 'economy' or i_size = 'small')\n ) or\n (i_category = 'Women' and\n (i_color = 'midnight' or i_color = 'burlywood') and\n (i_units = 'Tsp' or i_units = 'Bundle') and\n (i_size = 'medium' or i_size = 'extra large')\n ) or\n (i_category = 'Men' and\n (i_color = 'lavender' or i_color = 'azure') and\n (i_units = 'Each' or i_units = 'Lb') and\n (i_size = 'large' or i_size = 'N/A')\n ) or\n (i_category = 'Men' and\n (i_color = 'chocolate' or i_color = 'steel') and\n (i_units = 'N/A' or i_units = 'Dozen') and\n (i_size = 'economy' or i_size = 'small')\n ))) or\n (i_manufact = i1.i_manufact and\n ((i_category = 'Women' and \n (i_color = 'floral' or i_color = 'royal') and \n (i_units = 'Unknown' or i_units = 'Tbl') and\n (i_size = 'economy' or i_size = 'small')\n ) or\n (i_category = 'Women' and\n (i_color = 'navy' or i_color = 'forest') and\n (i_units = 'Bunch' or i_units = 'Dram') and\n (i_size = 'medium' or i_size = 'extra large')\n ) or\n (i_category = 'Men' and\n (i_color = 'cyan' or i_color = 'indian') and\n (i_units = 'Carton' or i_units = 'Cup') and\n (i_size = 'large' or i_size = 'N/A')\n ) or\n (i_category = 'Men' and\n (i_color = 'coral' or i_color = 'pale') and\n (i_units = 'Pallet' or i_units = 'Gram') and\n (i_size = 'economy' or i_size = 'small')\n )))) > 0\n order by i_product_name\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q42.sql", "content": "-- CometBench-DS query 42 derived from TPC-DS query 42 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect dt.d_year\n \t,item.i_category_id\n \t,item.i_category\n \t,sum(ss_ext_sales_price)\n from \tdate_dim dt\n \t,store_sales\n \t,item\n where dt.d_date_sk = store_sales.ss_sold_date_sk\n \tand store_sales.ss_item_sk = item.i_item_sk\n \tand item.i_manager_id = 1 \t\n \tand dt.d_moy=11\n \tand dt.d_year=1998\n group by \tdt.d_year\n \t\t,item.i_category_id\n \t\t,item.i_category\n order by sum(ss_ext_sales_price) desc,dt.d_year\n \t\t,item.i_category_id\n \t\t,item.i_category\n LIMIT 100 ;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q43.sql", "content": "-- CometBench-DS query 43 derived from TPC-DS query 43 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect s_store_name, s_store_id,\n sum(case when (d_day_name='Sunday') then ss_sales_price else null end) sun_sales,\n sum(case when (d_day_name='Monday') then ss_sales_price else null end) mon_sales,\n sum(case when (d_day_name='Tuesday') then ss_sales_price else null end) tue_sales,\n sum(case when (d_day_name='Wednesday') then ss_sales_price else null end) wed_sales,\n sum(case when (d_day_name='Thursday') then ss_sales_price else null end) thu_sales,\n sum(case when (d_day_name='Friday') then ss_sales_price else null end) fri_sales,\n sum(case when (d_day_name='Saturday') then ss_sales_price else null end) sat_sales\n from date_dim, store_sales, store\n where d_date_sk = ss_sold_date_sk and\n s_store_sk = ss_store_sk and\n s_gmt_offset = -5 and\n d_year = 2000 \n group by s_store_name, s_store_id\n order by s_store_name, s_store_id,sun_sales,mon_sales,tue_sales,wed_sales,thu_sales,fri_sales,sat_sales\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q44.sql", "content": "-- CometBench-DS query 44 derived from TPC-DS query 44 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect asceding.rnk, i1.i_product_name best_performing, i2.i_product_name worst_performing\nfrom(select *\n from (select item_sk,rank() over (order by rank_col asc) rnk\n from (select ss_item_sk item_sk,avg(ss_net_profit) rank_col \n from store_sales ss1\n where ss_store_sk = 6\n group by ss_item_sk\n having avg(ss_net_profit) > 0.9*(select avg(ss_net_profit) rank_col\n from store_sales\n where ss_store_sk = 6\n and ss_hdemo_sk is null\n group by ss_store_sk))V1)V11\n where rnk < 11) asceding,\n (select *\n from (select item_sk,rank() over (order by rank_col desc) rnk\n from (select ss_item_sk item_sk,avg(ss_net_profit) rank_col\n from store_sales ss1\n where ss_store_sk = 6\n group by ss_item_sk\n having avg(ss_net_profit) > 0.9*(select avg(ss_net_profit) rank_col\n from store_sales\n where ss_store_sk = 6\n and ss_hdemo_sk is null\n group by ss_store_sk))V2)V21\n where rnk < 11) descending,\nitem i1,\nitem i2\nwhere asceding.rnk = descending.rnk \n and i1.i_item_sk=asceding.item_sk\n and i2.i_item_sk=descending.item_sk\norder by asceding.rnk\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q45.sql", "content": "-- CometBench-DS query 45 derived from TPC-DS query 45 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect ca_zip, ca_city, sum(ws_sales_price)\n from web_sales, customer, customer_address, date_dim, item\n where ws_bill_customer_sk = c_customer_sk\n \tand c_current_addr_sk = ca_address_sk \n \tand ws_item_sk = i_item_sk \n \tand ( substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475', '85392', '85460', '80348', '81792')\n \t or \n \t i_item_id in (select i_item_id\n from item\n where i_item_sk in (2, 3, 5, 7, 11, 13, 17, 19, 23, 29)\n )\n \t )\n \tand ws_sold_date_sk = d_date_sk\n \tand d_qoy = 2 and d_year = 2000\n group by ca_zip, ca_city\n order by ca_zip, ca_city\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q46.sql", "content": "-- CometBench-DS query 46 derived from TPC-DS query 46 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect c_last_name\n ,c_first_name\n ,ca_city\n ,bought_city\n ,ss_ticket_number\n ,amt,profit \n from\n (select ss_ticket_number\n ,ss_customer_sk\n ,ca_city bought_city\n ,sum(ss_coupon_amt) amt\n ,sum(ss_net_profit) profit\n from store_sales,date_dim,store,household_demographics,customer_address \n where store_sales.ss_sold_date_sk = date_dim.d_date_sk\n and store_sales.ss_store_sk = store.s_store_sk \n and store_sales.ss_hdemo_sk = household_demographics.hd_demo_sk\n and store_sales.ss_addr_sk = customer_address.ca_address_sk\n and (household_demographics.hd_dep_count = 3 or\n household_demographics.hd_vehicle_count= 1)\n and date_dim.d_dow in (6,0)\n and date_dim.d_year in (1999,1999+1,1999+2) \n and store.s_city in ('Midway','Fairview','Fairview','Midway','Fairview') \n group by ss_ticket_number,ss_customer_sk,ss_addr_sk,ca_city) dn,customer,customer_address current_addr\n where ss_customer_sk = c_customer_sk\n and customer.c_current_addr_sk = current_addr.ca_address_sk\n and current_addr.ca_city <> bought_city\n order by c_last_name\n ,c_first_name\n ,ca_city\n ,bought_city\n ,ss_ticket_number\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q47.sql", "content": "-- CometBench-DS query 47 derived from TPC-DS query 47 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith v1 as(\n select i_category, i_brand,\n s_store_name, s_company_name,\n d_year, d_moy,\n sum(ss_sales_price) sum_sales,\n avg(sum(ss_sales_price)) over\n (partition by i_category, i_brand,\n s_store_name, s_company_name, d_year)\n avg_monthly_sales,\n rank() over\n (partition by i_category, i_brand,\n s_store_name, s_company_name\n order by d_year, d_moy) rn\n from item, store_sales, date_dim, store\n where ss_item_sk = i_item_sk and\n ss_sold_date_sk = d_date_sk and\n ss_store_sk = s_store_sk and\n (\n d_year = 2001 or\n ( d_year = 2001-1 and d_moy =12) or\n ( d_year = 2001+1 and d_moy =1)\n )\n group by i_category, i_brand,\n s_store_name, s_company_name,\n d_year, d_moy),\n v2 as(\n select v1.i_category, v1.i_brand, v1.s_store_name, v1.s_company_name\n ,v1.d_year\n ,v1.avg_monthly_sales\n ,v1.sum_sales, v1_lag.sum_sales psum, v1_lead.sum_sales nsum\n from v1, v1 v1_lag, v1 v1_lead\n where v1.i_category = v1_lag.i_category and\n v1.i_category = v1_lead.i_category and\n v1.i_brand = v1_lag.i_brand and\n v1.i_brand = v1_lead.i_brand and\n v1.s_store_name = v1_lag.s_store_name and\n v1.s_store_name = v1_lead.s_store_name and\n v1.s_company_name = v1_lag.s_company_name and\n v1.s_company_name = v1_lead.s_company_name and\n v1.rn = v1_lag.rn + 1 and\n v1.rn = v1_lead.rn - 1)\n select *\n from v2\n where d_year = 2001 and \n avg_monthly_sales > 0 and\n case when avg_monthly_sales > 0 then abs(sum_sales - avg_monthly_sales) / avg_monthly_sales else null end > 0.1\n order by sum_sales - avg_monthly_sales, nsum\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q48.sql", "content": "-- CometBench-DS query 48 derived from TPC-DS query 48 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect sum (ss_quantity)\n from store_sales, store, customer_demographics, customer_address, date_dim\n where s_store_sk = ss_store_sk\n and ss_sold_date_sk = d_date_sk and d_year = 2001\n and \n (\n (\n cd_demo_sk = ss_cdemo_sk\n and \n cd_marital_status = 'W'\n and \n cd_education_status = '2 yr Degree'\n and \n ss_sales_price between 100.00 and 150.00 \n )\n or\n (\n cd_demo_sk = ss_cdemo_sk\n and \n cd_marital_status = 'S'\n and \n cd_education_status = 'Advanced Degree'\n and \n ss_sales_price between 50.00 and 100.00 \n )\n or \n (\n cd_demo_sk = ss_cdemo_sk\n and \n cd_marital_status = 'D'\n and \n cd_education_status = 'Primary'\n and \n ss_sales_price between 150.00 and 200.00 \n )\n )\n and\n (\n (\n ss_addr_sk = ca_address_sk\n and\n ca_country = 'United States'\n and\n ca_state in ('IL', 'KY', 'OR')\n and ss_net_profit between 0 and 2000 \n )\n or\n (ss_addr_sk = ca_address_sk\n and\n ca_country = 'United States'\n and\n ca_state in ('VA', 'FL', 'AL')\n and ss_net_profit between 150 and 3000 \n )\n or\n (ss_addr_sk = ca_address_sk\n and\n ca_country = 'United States'\n and\n ca_state in ('OK', 'IA', 'TX')\n and ss_net_profit between 50 and 25000 \n )\n )\n;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q49.sql", "content": "-- CometBench-DS query 49 derived from TPC-DS query 49 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect channel, item, return_ratio, return_rank, currency_rank from\n (select\n 'web' as channel\n ,web.item\n ,web.return_ratio\n ,web.return_rank\n ,web.currency_rank\n from (\n \tselect \n \t item\n \t,return_ratio\n \t,currency_ratio\n \t,rank() over (order by return_ratio) as return_rank\n \t,rank() over (order by currency_ratio) as currency_rank\n \tfrom\n \t(\tselect ws.ws_item_sk as item\n \t\t,(cast(sum(coalesce(wr.wr_return_quantity,0)) as decimal(15,4))/\n \t\tcast(sum(coalesce(ws.ws_quantity,0)) as decimal(15,4) )) as return_ratio\n \t\t,(cast(sum(coalesce(wr.wr_return_amt,0)) as decimal(15,4))/\n \t\tcast(sum(coalesce(ws.ws_net_paid,0)) as decimal(15,4) )) as currency_ratio\n \t\tfrom \n \t\t web_sales ws left outer join web_returns wr \n \t\t\ton (ws.ws_order_number = wr.wr_order_number and \n \t\t\tws.ws_item_sk = wr.wr_item_sk)\n ,date_dim\n \t\twhere \n \t\t\twr.wr_return_amt > 10000 \n \t\t\tand ws.ws_net_profit > 1\n and ws.ws_net_paid > 0\n and ws.ws_quantity > 0\n and ws_sold_date_sk = d_date_sk\n and d_year = 2000\n and d_moy = 12\n \t\tgroup by ws.ws_item_sk\n \t) in_web\n ) web\n where \n (\n web.return_rank <= 10\n or\n web.currency_rank <= 10\n )\n union\n select \n 'catalog' as channel\n ,catalog.item\n ,catalog.return_ratio\n ,catalog.return_rank\n ,catalog.currency_rank\n from (\n \tselect \n \t item\n \t,return_ratio\n \t,currency_ratio\n \t,rank() over (order by return_ratio) as return_rank\n \t,rank() over (order by currency_ratio) as currency_rank\n \tfrom\n \t(\tselect \n \t\tcs.cs_item_sk as item\n \t\t,(cast(sum(coalesce(cr.cr_return_quantity,0)) as decimal(15,4))/\n \t\tcast(sum(coalesce(cs.cs_quantity,0)) as decimal(15,4) )) as return_ratio\n \t\t,(cast(sum(coalesce(cr.cr_return_amount,0)) as decimal(15,4))/\n \t\tcast(sum(coalesce(cs.cs_net_paid,0)) as decimal(15,4) )) as currency_ratio\n \t\tfrom \n \t\tcatalog_sales cs left outer join catalog_returns cr\n \t\t\ton (cs.cs_order_number = cr.cr_order_number and \n \t\t\tcs.cs_item_sk = cr.cr_item_sk)\n ,date_dim\n \t\twhere \n \t\t\tcr.cr_return_amount > 10000 \n \t\t\tand cs.cs_net_profit > 1\n and cs.cs_net_paid > 0\n and cs.cs_quantity > 0\n and cs_sold_date_sk = d_date_sk\n and d_year = 2000\n and d_moy = 12\n group by cs.cs_item_sk\n \t) in_cat\n ) catalog\n where \n (\n catalog.return_rank <= 10\n or\n catalog.currency_rank <=10\n )\n union\n select \n 'store' as channel\n ,store.item\n ,store.return_ratio\n ,store.return_rank\n ,store.currency_rank\n from (\n \tselect \n \t item\n \t,return_ratio\n \t,currency_ratio\n \t,rank() over (order by return_ratio) as return_rank\n \t,rank() over (order by currency_ratio) as currency_rank\n \tfrom\n \t(\tselect sts.ss_item_sk as item\n \t\t,(cast(sum(coalesce(sr.sr_return_quantity,0)) as decimal(15,4))/cast(sum(coalesce(sts.ss_quantity,0)) as decimal(15,4) )) as return_ratio\n \t\t,(cast(sum(coalesce(sr.sr_return_amt,0)) as decimal(15,4))/cast(sum(coalesce(sts.ss_net_paid,0)) as decimal(15,4) )) as currency_ratio\n \t\tfrom \n \t\tstore_sales sts left outer join store_returns sr\n \t\t\ton (sts.ss_ticket_number = sr.sr_ticket_number and sts.ss_item_sk = sr.sr_item_sk)\n ,date_dim\n \t\twhere \n \t\t\tsr.sr_return_amt > 10000 \n \t\t\tand sts.ss_net_profit > 1\n and sts.ss_net_paid > 0 \n and sts.ss_quantity > 0\n and ss_sold_date_sk = d_date_sk\n and d_year = 2000\n and d_moy = 12\n \t\tgroup by sts.ss_item_sk\n \t) in_store\n ) store\n where (\n store.return_rank <= 10\n or \n store.currency_rank <= 10\n )\n )\n order by 1,4,5,2\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q5.sql", "content": "-- CometBench-DS query 5 derived from TPC-DS query 5 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith ssr as\n (select s_store_id,\n sum(sales_price) as sales,\n sum(profit) as profit,\n sum(return_amt) as returns,\n sum(net_loss) as profit_loss\n from\n ( select ss_store_sk as store_sk,\n ss_sold_date_sk as date_sk,\n ss_ext_sales_price as sales_price,\n ss_net_profit as profit,\n cast(0 as decimal(7,2)) as return_amt,\n cast(0 as decimal(7,2)) as net_loss\n from store_sales\n union all\n select sr_store_sk as store_sk,\n sr_returned_date_sk as date_sk,\n cast(0 as decimal(7,2)) as sales_price,\n cast(0 as decimal(7,2)) as profit,\n sr_return_amt as return_amt,\n sr_net_loss as net_loss\n from store_returns\n ) salesreturns,\n date_dim,\n store\n where date_sk = d_date_sk\n and d_date between cast('2001-08-04' as date) \n and (cast('2001-08-04' as date) + INTERVAL '14 DAYS')\n and store_sk = s_store_sk\n group by s_store_id)\n ,\n csr as\n (select cp_catalog_page_id,\n sum(sales_price) as sales,\n sum(profit) as profit,\n sum(return_amt) as returns,\n sum(net_loss) as profit_loss\n from\n ( select cs_catalog_page_sk as page_sk,\n cs_sold_date_sk as date_sk,\n cs_ext_sales_price as sales_price,\n cs_net_profit as profit,\n cast(0 as decimal(7,2)) as return_amt,\n cast(0 as decimal(7,2)) as net_loss\n from catalog_sales\n union all\n select cr_catalog_page_sk as page_sk,\n cr_returned_date_sk as date_sk,\n cast(0 as decimal(7,2)) as sales_price,\n cast(0 as decimal(7,2)) as profit,\n cr_return_amount as return_amt,\n cr_net_loss as net_loss\n from catalog_returns\n ) salesreturns,\n date_dim,\n catalog_page\n where date_sk = d_date_sk\n and d_date between cast('2001-08-04' as date)\n and (cast('2001-08-04' as date) + INTERVAL '14 DAYS')\n and page_sk = cp_catalog_page_sk\n group by cp_catalog_page_id)\n ,\n wsr as\n (select web_site_id,\n sum(sales_price) as sales,\n sum(profit) as profit,\n sum(return_amt) as returns,\n sum(net_loss) as profit_loss\n from\n ( select ws_web_site_sk as wsr_web_site_sk,\n ws_sold_date_sk as date_sk,\n ws_ext_sales_price as sales_price,\n ws_net_profit as profit,\n cast(0 as decimal(7,2)) as return_amt,\n cast(0 as decimal(7,2)) as net_loss\n from web_sales\n union all\n select ws_web_site_sk as wsr_web_site_sk,\n wr_returned_date_sk as date_sk,\n cast(0 as decimal(7,2)) as sales_price,\n cast(0 as decimal(7,2)) as profit,\n wr_return_amt as return_amt,\n wr_net_loss as net_loss\n from web_returns left outer join web_sales on\n ( wr_item_sk = ws_item_sk\n and wr_order_number = ws_order_number)\n ) salesreturns,\n date_dim,\n web_site\n where date_sk = d_date_sk\n and d_date between cast('2001-08-04' as date)\n and (cast('2001-08-04' as date) + INTERVAL '14 DAYS')\n and wsr_web_site_sk = web_site_sk\n group by web_site_id)\n select channel\n , id\n , sum(sales) as sales\n , sum(returns) as returns\n , sum(profit) as profit\n from \n (select 'store channel' as channel\n , 'store' || s_store_id as id\n , sales\n , returns\n , (profit - profit_loss) as profit\n from ssr\n union all\n select 'catalog channel' as channel\n , 'catalog_page' || cp_catalog_page_id as id\n , sales\n , returns\n , (profit - profit_loss) as profit\n from csr\n union all\n select 'web channel' as channel\n , 'web_site' || web_site_id as id\n , sales\n , returns\n , (profit - profit_loss) as profit\n from wsr\n ) x\n group by rollup (channel, id)\n order by channel\n ,id\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q50.sql", "content": "-- CometBench-DS query 50 derived from TPC-DS query 50 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect \n s_store_name\n ,s_company_id\n ,s_street_number\n ,s_street_name\n ,s_street_type\n ,s_suite_number\n ,s_city\n ,s_county\n ,s_state\n ,s_zip\n ,sum(case when (sr_returned_date_sk - ss_sold_date_sk <= 30 ) then 1 else 0 end) as `30 days` \n ,sum(case when (sr_returned_date_sk - ss_sold_date_sk > 30) and \n (sr_returned_date_sk - ss_sold_date_sk <= 60) then 1 else 0 end ) as `31-60 days` \n ,sum(case when (sr_returned_date_sk - ss_sold_date_sk > 60) and \n (sr_returned_date_sk - ss_sold_date_sk <= 90) then 1 else 0 end) as `61-90 days` \n ,sum(case when (sr_returned_date_sk - ss_sold_date_sk > 90) and\n (sr_returned_date_sk - ss_sold_date_sk <= 120) then 1 else 0 end) as `91-120 days` \n ,sum(case when (sr_returned_date_sk - ss_sold_date_sk > 120) then 1 else 0 end) as `>120 days` \nfrom\n store_sales\n ,store_returns\n ,store\n ,date_dim d1\n ,date_dim d2\nwhere\n d2.d_year = 2002\nand d2.d_moy = 8\nand ss_ticket_number = sr_ticket_number\nand ss_item_sk = sr_item_sk\nand ss_sold_date_sk = d1.d_date_sk\nand sr_returned_date_sk = d2.d_date_sk\nand ss_customer_sk = sr_customer_sk\nand ss_store_sk = s_store_sk\ngroup by\n s_store_name\n ,s_company_id\n ,s_street_number\n ,s_street_name\n ,s_street_type\n ,s_suite_number\n ,s_city\n ,s_county\n ,s_state\n ,s_zip\norder by s_store_name\n ,s_company_id\n ,s_street_number\n ,s_street_name\n ,s_street_type\n ,s_suite_number\n ,s_city\n ,s_county\n ,s_state\n ,s_zip\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q51.sql", "content": "-- CometBench-DS query 51 derived from TPC-DS query 51 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nWITH web_v1 as (\nselect\n ws_item_sk item_sk, d_date,\n sum(sum(ws_sales_price))\n over (partition by ws_item_sk order by d_date rows between unbounded preceding and current row) cume_sales\nfrom web_sales\n ,date_dim\nwhere ws_sold_date_sk=d_date_sk\n and d_month_seq between 1215 and 1215+11\n and ws_item_sk is not NULL\ngroup by ws_item_sk, d_date),\nstore_v1 as (\nselect\n ss_item_sk item_sk, d_date,\n sum(sum(ss_sales_price))\n over (partition by ss_item_sk order by d_date rows between unbounded preceding and current row) cume_sales\nfrom store_sales\n ,date_dim\nwhere ss_sold_date_sk=d_date_sk\n and d_month_seq between 1215 and 1215+11\n and ss_item_sk is not NULL\ngroup by ss_item_sk, d_date)\n select *\nfrom (select item_sk\n ,d_date\n ,web_sales\n ,store_sales\n ,max(web_sales)\n over (partition by item_sk order by d_date rows between unbounded preceding and current row) web_cumulative\n ,max(store_sales)\n over (partition by item_sk order by d_date rows between unbounded preceding and current row) store_cumulative\n from (select case when web.item_sk is not null then web.item_sk else store.item_sk end item_sk\n ,case when web.d_date is not null then web.d_date else store.d_date end d_date\n ,web.cume_sales web_sales\n ,store.cume_sales store_sales\n from web_v1 web full outer join store_v1 store on (web.item_sk = store.item_sk\n and web.d_date = store.d_date)\n )x )y\nwhere web_cumulative > store_cumulative\norder by item_sk\n ,d_date\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q52.sql", "content": "-- CometBench-DS query 52 derived from TPC-DS query 52 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect dt.d_year\n \t,item.i_brand_id brand_id\n \t,item.i_brand brand\n \t,sum(ss_ext_sales_price) ext_price\n from date_dim dt\n ,store_sales\n ,item\n where dt.d_date_sk = store_sales.ss_sold_date_sk\n and store_sales.ss_item_sk = item.i_item_sk\n and item.i_manager_id = 1\n and dt.d_moy=11\n and dt.d_year=2000\n group by dt.d_year\n \t,item.i_brand\n \t,item.i_brand_id\n order by dt.d_year\n \t,ext_price desc\n \t,brand_id\n LIMIT 100 ;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q53.sql", "content": "-- CometBench-DS query 53 derived from TPC-DS query 53 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect * from \n(select i_manufact_id,\nsum(ss_sales_price) sum_sales,\navg(sum(ss_sales_price)) over (partition by i_manufact_id) avg_quarterly_sales\nfrom item, store_sales, date_dim, store\nwhere ss_item_sk = i_item_sk and\nss_sold_date_sk = d_date_sk and\nss_store_sk = s_store_sk and\nd_month_seq in (1197,1197+1,1197+2,1197+3,1197+4,1197+5,1197+6,1197+7,1197+8,1197+9,1197+10,1197+11) and\n((i_category in ('Books','Children','Electronics') and\ni_class in ('personal','portable','reference','self-help') and\ni_brand in ('scholaramalgamalg #14','scholaramalgamalg #7',\n\t\t'exportiunivamalg #9','scholaramalgamalg #9'))\nor(i_category in ('Women','Music','Men') and\ni_class in ('accessories','classical','fragrances','pants') and\ni_brand in ('amalgimporto #1','edu packscholar #1','exportiimporto #1',\n\t\t'importoamalg #1')))\ngroup by i_manufact_id, d_qoy ) tmp1\nwhere case when avg_quarterly_sales > 0 \n\tthen abs (sum_sales - avg_quarterly_sales)/ avg_quarterly_sales \n\telse null end > 0.1\norder by avg_quarterly_sales,\n\t sum_sales,\n\t i_manufact_id\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q54.sql", "content": "-- CometBench-DS query 54 derived from TPC-DS query 54 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith my_customers as (\n select distinct c_customer_sk\n , c_current_addr_sk\n from \n ( select cs_sold_date_sk sold_date_sk,\n cs_bill_customer_sk customer_sk,\n cs_item_sk item_sk\n from catalog_sales\n union all\n select ws_sold_date_sk sold_date_sk,\n ws_bill_customer_sk customer_sk,\n ws_item_sk item_sk\n from web_sales\n ) cs_or_ws_sales,\n item,\n date_dim,\n customer\n where sold_date_sk = d_date_sk\n and item_sk = i_item_sk\n and i_category = 'Men'\n and i_class = 'shirts'\n and c_customer_sk = cs_or_ws_sales.customer_sk\n and d_moy = 4\n and d_year = 1998\n )\n , my_revenue as (\n select c_customer_sk,\n sum(ss_ext_sales_price) as revenue\n from my_customers,\n store_sales,\n customer_address,\n store,\n date_dim\n where c_current_addr_sk = ca_address_sk\n and ca_county = s_county\n and ca_state = s_state\n and ss_sold_date_sk = d_date_sk\n and c_customer_sk = ss_customer_sk\n and d_month_seq between (select distinct d_month_seq+1\n from date_dim where d_year = 1998 and d_moy = 4)\n and (select distinct d_month_seq+3\n from date_dim where d_year = 1998 and d_moy = 4)\n group by c_customer_sk\n )\n , segments as\n (select cast((revenue/50) as int) as segment\n from my_revenue\n )\n select segment, count(*) as num_customers, segment*50 as segment_base\n from segments\n group by segment\n order by segment, num_customers\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q55.sql", "content": "-- CometBench-DS query 55 derived from TPC-DS query 55 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect i_brand_id brand_id, i_brand brand,\n \tsum(ss_ext_sales_price) ext_price\n from date_dim, store_sales, item\n where d_date_sk = ss_sold_date_sk\n \tand ss_item_sk = i_item_sk\n \tand i_manager_id=20\n \tand d_moy=12\n \tand d_year=1998\n group by i_brand, i_brand_id\n order by ext_price desc, i_brand_id\n LIMIT 100 ;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q56.sql", "content": "-- CometBench-DS query 56 derived from TPC-DS query 56 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith ss as (\n select i_item_id,sum(ss_ext_sales_price) total_sales\n from\n \tstore_sales,\n \tdate_dim,\n customer_address,\n item\n where i_item_id in (select\n i_item_id\nfrom item\nwhere i_color in ('powder','goldenrod','bisque'))\n and ss_item_sk = i_item_sk\n and ss_sold_date_sk = d_date_sk\n and d_year = 1998\n and d_moy = 5\n and ss_addr_sk = ca_address_sk\n and ca_gmt_offset = -5 \n group by i_item_id),\n cs as (\n select i_item_id,sum(cs_ext_sales_price) total_sales\n from\n \tcatalog_sales,\n \tdate_dim,\n customer_address,\n item\n where\n i_item_id in (select\n i_item_id\nfrom item\nwhere i_color in ('powder','goldenrod','bisque'))\n and cs_item_sk = i_item_sk\n and cs_sold_date_sk = d_date_sk\n and d_year = 1998\n and d_moy = 5\n and cs_bill_addr_sk = ca_address_sk\n and ca_gmt_offset = -5 \n group by i_item_id),\n ws as (\n select i_item_id,sum(ws_ext_sales_price) total_sales\n from\n \tweb_sales,\n \tdate_dim,\n customer_address,\n item\n where\n i_item_id in (select\n i_item_id\nfrom item\nwhere i_color in ('powder','goldenrod','bisque'))\n and ws_item_sk = i_item_sk\n and ws_sold_date_sk = d_date_sk\n and d_year = 1998\n and d_moy = 5\n and ws_bill_addr_sk = ca_address_sk\n and ca_gmt_offset = -5\n group by i_item_id)\n select i_item_id ,sum(total_sales) total_sales\n from (select * from ss \n union all\n select * from cs \n union all\n select * from ws) tmp1\n group by i_item_id\n order by total_sales,\n i_item_id\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q57.sql", "content": "-- CometBench-DS query 57 derived from TPC-DS query 57 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith v1 as(\n select i_category, i_brand,\n cc_name,\n d_year, d_moy,\n sum(cs_sales_price) sum_sales,\n avg(sum(cs_sales_price)) over\n (partition by i_category, i_brand,\n cc_name, d_year)\n avg_monthly_sales,\n rank() over\n (partition by i_category, i_brand,\n cc_name\n order by d_year, d_moy) rn\n from item, catalog_sales, date_dim, call_center\n where cs_item_sk = i_item_sk and\n cs_sold_date_sk = d_date_sk and\n cc_call_center_sk= cs_call_center_sk and\n (\n d_year = 2000 or\n ( d_year = 2000-1 and d_moy =12) or\n ( d_year = 2000+1 and d_moy =1)\n )\n group by i_category, i_brand,\n cc_name , d_year, d_moy),\n v2 as(\n select v1.cc_name\n ,v1.d_year, v1.d_moy\n ,v1.avg_monthly_sales\n ,v1.sum_sales, v1_lag.sum_sales psum, v1_lead.sum_sales nsum\n from v1, v1 v1_lag, v1 v1_lead\n where v1.i_category = v1_lag.i_category and\n v1.i_category = v1_lead.i_category and\n v1.i_brand = v1_lag.i_brand and\n v1.i_brand = v1_lead.i_brand and\n v1. cc_name = v1_lag. cc_name and\n v1. cc_name = v1_lead. cc_name and\n v1.rn = v1_lag.rn + 1 and\n v1.rn = v1_lead.rn - 1)\n select *\n from v2\n where d_year = 2000 and\n avg_monthly_sales > 0 and\n case when avg_monthly_sales > 0 then abs(sum_sales - avg_monthly_sales) / avg_monthly_sales else null end > 0.1\n order by sum_sales - avg_monthly_sales, psum\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q58.sql", "content": "-- CometBench-DS query 58 derived from TPC-DS query 58 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith ss_items as\n (select i_item_id item_id\n ,sum(ss_ext_sales_price) ss_item_rev \n from store_sales\n ,item\n ,date_dim\n where ss_item_sk = i_item_sk\n and d_date in (select d_date\n from date_dim\n where d_week_seq = (select d_week_seq \n from date_dim\n where d_date = '2000-02-12'))\n and ss_sold_date_sk = d_date_sk\n group by i_item_id),\n cs_items as\n (select i_item_id item_id\n ,sum(cs_ext_sales_price) cs_item_rev\n from catalog_sales\n ,item\n ,date_dim\n where cs_item_sk = i_item_sk\n and d_date in (select d_date\n from date_dim\n where d_week_seq = (select d_week_seq \n from date_dim\n where d_date = '2000-02-12'))\n and cs_sold_date_sk = d_date_sk\n group by i_item_id),\n ws_items as\n (select i_item_id item_id\n ,sum(ws_ext_sales_price) ws_item_rev\n from web_sales\n ,item\n ,date_dim\n where ws_item_sk = i_item_sk\n and d_date in (select d_date\n from date_dim\n where d_week_seq =(select d_week_seq \n from date_dim\n where d_date = '2000-02-12'))\n and ws_sold_date_sk = d_date_sk\n group by i_item_id)\n select ss_items.item_id\n ,ss_item_rev\n ,ss_item_rev/((ss_item_rev+cs_item_rev+ws_item_rev)/3) * 100 ss_dev\n ,cs_item_rev\n ,cs_item_rev/((ss_item_rev+cs_item_rev+ws_item_rev)/3) * 100 cs_dev\n ,ws_item_rev\n ,ws_item_rev/((ss_item_rev+cs_item_rev+ws_item_rev)/3) * 100 ws_dev\n ,(ss_item_rev+cs_item_rev+ws_item_rev)/3 average\n from ss_items,cs_items,ws_items\n where ss_items.item_id=cs_items.item_id\n and ss_items.item_id=ws_items.item_id \n and ss_item_rev between 0.9 * cs_item_rev and 1.1 * cs_item_rev\n and ss_item_rev between 0.9 * ws_item_rev and 1.1 * ws_item_rev\n and cs_item_rev between 0.9 * ss_item_rev and 1.1 * ss_item_rev\n and cs_item_rev between 0.9 * ws_item_rev and 1.1 * ws_item_rev\n and ws_item_rev between 0.9 * ss_item_rev and 1.1 * ss_item_rev\n and ws_item_rev between 0.9 * cs_item_rev and 1.1 * cs_item_rev\n order by item_id\n ,ss_item_rev\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q59.sql", "content": "-- CometBench-DS query 59 derived from TPC-DS query 59 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith wss as \n (select d_week_seq,\n ss_store_sk,\n sum(case when (d_day_name='Sunday') then ss_sales_price else null end) sun_sales,\n sum(case when (d_day_name='Monday') then ss_sales_price else null end) mon_sales,\n sum(case when (d_day_name='Tuesday') then ss_sales_price else null end) tue_sales,\n sum(case when (d_day_name='Wednesday') then ss_sales_price else null end) wed_sales,\n sum(case when (d_day_name='Thursday') then ss_sales_price else null end) thu_sales,\n sum(case when (d_day_name='Friday') then ss_sales_price else null end) fri_sales,\n sum(case when (d_day_name='Saturday') then ss_sales_price else null end) sat_sales\n from store_sales,date_dim\n where d_date_sk = ss_sold_date_sk\n group by d_week_seq,ss_store_sk\n )\n select s_store_name1,s_store_id1,d_week_seq1\n ,sun_sales1/sun_sales2,mon_sales1/mon_sales2\n ,tue_sales1/tue_sales2,wed_sales1/wed_sales2,thu_sales1/thu_sales2\n ,fri_sales1/fri_sales2,sat_sales1/sat_sales2\n from\n (select s_store_name s_store_name1,wss.d_week_seq d_week_seq1\n ,s_store_id s_store_id1,sun_sales sun_sales1\n ,mon_sales mon_sales1,tue_sales tue_sales1\n ,wed_sales wed_sales1,thu_sales thu_sales1\n ,fri_sales fri_sales1,sat_sales sat_sales1\n from wss,store,date_dim d\n where d.d_week_seq = wss.d_week_seq and\n ss_store_sk = s_store_sk and \n d_month_seq between 1206 and 1206 + 11) y,\n (select s_store_name s_store_name2,wss.d_week_seq d_week_seq2\n ,s_store_id s_store_id2,sun_sales sun_sales2\n ,mon_sales mon_sales2,tue_sales tue_sales2\n ,wed_sales wed_sales2,thu_sales thu_sales2\n ,fri_sales fri_sales2,sat_sales sat_sales2\n from wss,store,date_dim d\n where d.d_week_seq = wss.d_week_seq and\n ss_store_sk = s_store_sk and \n d_month_seq between 1206+ 12 and 1206 + 23) x\n where s_store_id1=s_store_id2\n and d_week_seq1=d_week_seq2-52\n order by s_store_name1,s_store_id1,d_week_seq1\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q6.sql", "content": "-- CometBench-DS query 6 derived from TPC-DS query 6 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect a.ca_state state, count(*) cnt\n from customer_address a\n ,customer c\n ,store_sales s\n ,date_dim d\n ,item i\n where a.ca_address_sk = c.c_current_addr_sk\n \tand c.c_customer_sk = s.ss_customer_sk\n \tand s.ss_sold_date_sk = d.d_date_sk\n \tand s.ss_item_sk = i.i_item_sk\n \tand d.d_month_seq = \n \t (select distinct (d_month_seq)\n \t from date_dim\n where d_year = 1998\n \t and d_moy = 3 )\n \tand i.i_current_price > 1.2 * \n (select avg(j.i_current_price) \n \t from item j \n \t where j.i_category = i.i_category)\n group by a.ca_state\n having count(*) >= 10\n order by cnt, a.ca_state \n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q60.sql", "content": "-- CometBench-DS query 60 derived from TPC-DS query 60 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith ss as (\n select\n i_item_id,sum(ss_ext_sales_price) total_sales\n from\n \tstore_sales,\n \tdate_dim,\n customer_address,\n item\n where\n i_item_id in (select\n i_item_id\nfrom\n item\nwhere i_category in ('Shoes'))\n and ss_item_sk = i_item_sk\n and ss_sold_date_sk = d_date_sk\n and d_year = 2001\n and d_moy = 10\n and ss_addr_sk = ca_address_sk\n and ca_gmt_offset = -6 \n group by i_item_id),\n cs as (\n select\n i_item_id,sum(cs_ext_sales_price) total_sales\n from\n \tcatalog_sales,\n \tdate_dim,\n customer_address,\n item\n where\n i_item_id in (select\n i_item_id\nfrom\n item\nwhere i_category in ('Shoes'))\n and cs_item_sk = i_item_sk\n and cs_sold_date_sk = d_date_sk\n and d_year = 2001\n and d_moy = 10\n and cs_bill_addr_sk = ca_address_sk\n and ca_gmt_offset = -6 \n group by i_item_id),\n ws as (\n select\n i_item_id,sum(ws_ext_sales_price) total_sales\n from\n \tweb_sales,\n \tdate_dim,\n customer_address,\n item\n where\n i_item_id in (select\n i_item_id\nfrom\n item\nwhere i_category in ('Shoes'))\n and ws_item_sk = i_item_sk\n and ws_sold_date_sk = d_date_sk\n and d_year = 2001\n and d_moy = 10\n and ws_bill_addr_sk = ca_address_sk\n and ca_gmt_offset = -6\n group by i_item_id)\n select \n i_item_id\n,sum(total_sales) total_sales\n from (select * from ss \n union all\n select * from cs \n union all\n select * from ws) tmp1\n group by i_item_id\n order by i_item_id\n ,total_sales\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q61.sql", "content": "-- CometBench-DS query 61 derived from TPC-DS query 61 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect promotions,total,cast(promotions as decimal(15,4))/cast(total as decimal(15,4))*100\nfrom\n (select sum(ss_ext_sales_price) promotions\n from store_sales\n ,store\n ,promotion\n ,date_dim\n ,customer\n ,customer_address \n ,item\n where ss_sold_date_sk = d_date_sk\n and ss_store_sk = s_store_sk\n and ss_promo_sk = p_promo_sk\n and ss_customer_sk= c_customer_sk\n and ca_address_sk = c_current_addr_sk\n and ss_item_sk = i_item_sk \n and ca_gmt_offset = -6\n and i_category = 'Sports'\n and (p_channel_dmail = 'Y' or p_channel_email = 'Y' or p_channel_tv = 'Y')\n and s_gmt_offset = -6\n and d_year = 2002\n and d_moy = 11) promotional_sales,\n (select sum(ss_ext_sales_price) total\n from store_sales\n ,store\n ,date_dim\n ,customer\n ,customer_address\n ,item\n where ss_sold_date_sk = d_date_sk\n and ss_store_sk = s_store_sk\n and ss_customer_sk= c_customer_sk\n and ca_address_sk = c_current_addr_sk\n and ss_item_sk = i_item_sk\n and ca_gmt_offset = -6\n and i_category = 'Sports'\n and s_gmt_offset = -6\n and d_year = 2002\n and d_moy = 11) all_sales\norder by promotions, total\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q62.sql", "content": "-- CometBench-DS query 62 derived from TPC-DS query 62 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect \n substr(w_warehouse_name,1,20)\n ,sm_type\n ,web_name\n ,sum(case when (ws_ship_date_sk - ws_sold_date_sk <= 30 ) then 1 else 0 end) as `30 days` \n ,sum(case when (ws_ship_date_sk - ws_sold_date_sk > 30) and \n (ws_ship_date_sk - ws_sold_date_sk <= 60) then 1 else 0 end ) as `31-60 days` \n ,sum(case when (ws_ship_date_sk - ws_sold_date_sk > 60) and \n (ws_ship_date_sk - ws_sold_date_sk <= 90) then 1 else 0 end) as `61-90 days` \n ,sum(case when (ws_ship_date_sk - ws_sold_date_sk > 90) and\n (ws_ship_date_sk - ws_sold_date_sk <= 120) then 1 else 0 end) as `91-120 days` \n ,sum(case when (ws_ship_date_sk - ws_sold_date_sk > 120) then 1 else 0 end) as `>120 days` \nfrom\n web_sales\n ,warehouse\n ,ship_mode\n ,web_site\n ,date_dim\nwhere\n d_month_seq between 1217 and 1217 + 11\nand ws_ship_date_sk = d_date_sk\nand ws_warehouse_sk = w_warehouse_sk\nand ws_ship_mode_sk = sm_ship_mode_sk\nand ws_web_site_sk = web_site_sk\ngroup by\n substr(w_warehouse_name,1,20)\n ,sm_type\n ,web_name\norder by substr(w_warehouse_name,1,20)\n ,sm_type\n ,web_name\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q63.sql", "content": "-- CometBench-DS query 63 derived from TPC-DS query 63 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect * \nfrom (select i_manager_id\n ,sum(ss_sales_price) sum_sales\n ,avg(sum(ss_sales_price)) over (partition by i_manager_id) avg_monthly_sales\n from item\n ,store_sales\n ,date_dim\n ,store\n where ss_item_sk = i_item_sk\n and ss_sold_date_sk = d_date_sk\n and ss_store_sk = s_store_sk\n and d_month_seq in (1181,1181+1,1181+2,1181+3,1181+4,1181+5,1181+6,1181+7,1181+8,1181+9,1181+10,1181+11)\n and (( i_category in ('Books','Children','Electronics')\n and i_class in ('personal','portable','reference','self-help')\n and i_brand in ('scholaramalgamalg #14','scholaramalgamalg #7',\n\t\t 'exportiunivamalg #9','scholaramalgamalg #9'))\n or( i_category in ('Women','Music','Men')\n and i_class in ('accessories','classical','fragrances','pants')\n and i_brand in ('amalgimporto #1','edu packscholar #1','exportiimporto #1',\n\t\t 'importoamalg #1')))\ngroup by i_manager_id, d_moy) tmp1\nwhere case when avg_monthly_sales > 0 then abs (sum_sales - avg_monthly_sales) / avg_monthly_sales else null end > 0.1\norder by i_manager_id\n ,avg_monthly_sales\n ,sum_sales\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q64.sql", "content": "-- CometBench-DS query 64 derived from TPC-DS query 64 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith cs_ui as\n (select cs_item_sk\n ,sum(cs_ext_list_price) as sale,sum(cr_refunded_cash+cr_reversed_charge+cr_store_credit) as refund\n from catalog_sales\n ,catalog_returns\n where cs_item_sk = cr_item_sk\n and cs_order_number = cr_order_number\n group by cs_item_sk\n having sum(cs_ext_list_price)>2*sum(cr_refunded_cash+cr_reversed_charge+cr_store_credit)),\ncross_sales as\n (select i_product_name product_name\n ,i_item_sk item_sk\n ,s_store_name store_name\n ,s_zip store_zip\n ,ad1.ca_street_number b_street_number\n ,ad1.ca_street_name b_street_name\n ,ad1.ca_city b_city\n ,ad1.ca_zip b_zip\n ,ad2.ca_street_number c_street_number\n ,ad2.ca_street_name c_street_name\n ,ad2.ca_city c_city\n ,ad2.ca_zip c_zip\n ,d1.d_year as syear\n ,d2.d_year as fsyear\n ,d3.d_year s2year\n ,count(*) cnt\n ,sum(ss_wholesale_cost) s1\n ,sum(ss_list_price) s2\n ,sum(ss_coupon_amt) s3\n FROM store_sales\n ,store_returns\n ,cs_ui\n ,date_dim d1\n ,date_dim d2\n ,date_dim d3\n ,store\n ,customer\n ,customer_demographics cd1\n ,customer_demographics cd2\n ,promotion\n ,household_demographics hd1\n ,household_demographics hd2\n ,customer_address ad1\n ,customer_address ad2\n ,income_band ib1\n ,income_band ib2\n ,item\n WHERE ss_store_sk = s_store_sk AND\n ss_sold_date_sk = d1.d_date_sk AND\n ss_customer_sk = c_customer_sk AND\n ss_cdemo_sk= cd1.cd_demo_sk AND\n ss_hdemo_sk = hd1.hd_demo_sk AND\n ss_addr_sk = ad1.ca_address_sk and\n ss_item_sk = i_item_sk and\n ss_item_sk = sr_item_sk and\n ss_ticket_number = sr_ticket_number and\n ss_item_sk = cs_ui.cs_item_sk and\n c_current_cdemo_sk = cd2.cd_demo_sk AND\n c_current_hdemo_sk = hd2.hd_demo_sk AND\n c_current_addr_sk = ad2.ca_address_sk and\n c_first_sales_date_sk = d2.d_date_sk and\n c_first_shipto_date_sk = d3.d_date_sk and\n ss_promo_sk = p_promo_sk and\n hd1.hd_income_band_sk = ib1.ib_income_band_sk and\n hd2.hd_income_band_sk = ib2.ib_income_band_sk and\n cd1.cd_marital_status <> cd2.cd_marital_status and\n i_color in ('light','cyan','burnished','green','almond','smoke') and\n i_current_price between 22 and 22 + 10 and\n i_current_price between 22 + 1 and 22 + 15\ngroup by i_product_name\n ,i_item_sk\n ,s_store_name\n ,s_zip\n ,ad1.ca_street_number\n ,ad1.ca_street_name\n ,ad1.ca_city\n ,ad1.ca_zip\n ,ad2.ca_street_number\n ,ad2.ca_street_name\n ,ad2.ca_city\n ,ad2.ca_zip\n ,d1.d_year\n ,d2.d_year\n ,d3.d_year\n)\nselect cs1.product_name\n ,cs1.store_name\n ,cs1.store_zip\n ,cs1.b_street_number\n ,cs1.b_street_name\n ,cs1.b_city\n ,cs1.b_zip\n ,cs1.c_street_number\n ,cs1.c_street_name\n ,cs1.c_city\n ,cs1.c_zip\n ,cs1.syear\n ,cs1.cnt\n ,cs1.s1 as s11\n ,cs1.s2 as s21\n ,cs1.s3 as s31\n ,cs2.s1 as s12\n ,cs2.s2 as s22\n ,cs2.s3 as s32\n ,cs2.syear\n ,cs2.cnt\nfrom cross_sales cs1,cross_sales cs2\nwhere cs1.item_sk=cs2.item_sk and\n cs1.syear = 2001 and\n cs2.syear = 2001 + 1 and\n cs2.cnt <= cs1.cnt and\n cs1.store_name = cs2.store_name and\n cs1.store_zip = cs2.store_zip\norder by cs1.product_name\n ,cs1.store_name\n ,cs2.cnt\n ,cs1.s1\n ,cs2.s1;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q65.sql", "content": "-- CometBench-DS query 65 derived from TPC-DS query 65 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect \n\ts_store_name,\n\ti_item_desc,\n\tsc.revenue,\n\ti_current_price,\n\ti_wholesale_cost,\n\ti_brand\n from store, item,\n (select ss_store_sk, avg(revenue) as ave\n \tfrom\n \t (select ss_store_sk, ss_item_sk, \n \t\t sum(ss_sales_price) as revenue\n \t\tfrom store_sales, date_dim\n \t\twhere ss_sold_date_sk = d_date_sk and d_month_seq between 1186 and 1186+11\n \t\tgroup by ss_store_sk, ss_item_sk) sa\n \tgroup by ss_store_sk) sb,\n (select ss_store_sk, ss_item_sk, sum(ss_sales_price) as revenue\n \tfrom store_sales, date_dim\n \twhere ss_sold_date_sk = d_date_sk and d_month_seq between 1186 and 1186+11\n \tgroup by ss_store_sk, ss_item_sk) sc\n where sb.ss_store_sk = sc.ss_store_sk and \n sc.revenue <= 0.1 * sb.ave and\n s_store_sk = sc.ss_store_sk and\n i_item_sk = sc.ss_item_sk\n order by s_store_name, i_item_desc\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q66.sql", "content": "-- CometBench-DS query 66 derived from TPC-DS query 66 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect \n w_warehouse_name\n \t,w_warehouse_sq_ft\n \t,w_city\n \t,w_county\n \t,w_state\n \t,w_country\n ,ship_carriers\n ,year\n \t,sum(jan_sales) as jan_sales\n \t,sum(feb_sales) as feb_sales\n \t,sum(mar_sales) as mar_sales\n \t,sum(apr_sales) as apr_sales\n \t,sum(may_sales) as may_sales\n \t,sum(jun_sales) as jun_sales\n \t,sum(jul_sales) as jul_sales\n \t,sum(aug_sales) as aug_sales\n \t,sum(sep_sales) as sep_sales\n \t,sum(oct_sales) as oct_sales\n \t,sum(nov_sales) as nov_sales\n \t,sum(dec_sales) as dec_sales\n \t,sum(jan_sales/w_warehouse_sq_ft) as jan_sales_per_sq_foot\n \t,sum(feb_sales/w_warehouse_sq_ft) as feb_sales_per_sq_foot\n \t,sum(mar_sales/w_warehouse_sq_ft) as mar_sales_per_sq_foot\n \t,sum(apr_sales/w_warehouse_sq_ft) as apr_sales_per_sq_foot\n \t,sum(may_sales/w_warehouse_sq_ft) as may_sales_per_sq_foot\n \t,sum(jun_sales/w_warehouse_sq_ft) as jun_sales_per_sq_foot\n \t,sum(jul_sales/w_warehouse_sq_ft) as jul_sales_per_sq_foot\n \t,sum(aug_sales/w_warehouse_sq_ft) as aug_sales_per_sq_foot\n \t,sum(sep_sales/w_warehouse_sq_ft) as sep_sales_per_sq_foot\n \t,sum(oct_sales/w_warehouse_sq_ft) as oct_sales_per_sq_foot\n \t,sum(nov_sales/w_warehouse_sq_ft) as nov_sales_per_sq_foot\n \t,sum(dec_sales/w_warehouse_sq_ft) as dec_sales_per_sq_foot\n \t,sum(jan_net) as jan_net\n \t,sum(feb_net) as feb_net\n \t,sum(mar_net) as mar_net\n \t,sum(apr_net) as apr_net\n \t,sum(may_net) as may_net\n \t,sum(jun_net) as jun_net\n \t,sum(jul_net) as jul_net\n \t,sum(aug_net) as aug_net\n \t,sum(sep_net) as sep_net\n \t,sum(oct_net) as oct_net\n \t,sum(nov_net) as nov_net\n \t,sum(dec_net) as dec_net\n from (\n select \n \tw_warehouse_name\n \t,w_warehouse_sq_ft\n \t,w_city\n \t,w_county\n \t,w_state\n \t,w_country\n \t,'FEDEX' || ',' || 'GERMA' as ship_carriers\n ,d_year as year\n \t,sum(case when d_moy = 1 \n \t\tthen ws_ext_list_price* ws_quantity else 0 end) as jan_sales\n \t,sum(case when d_moy = 2 \n \t\tthen ws_ext_list_price* ws_quantity else 0 end) as feb_sales\n \t,sum(case when d_moy = 3 \n \t\tthen ws_ext_list_price* ws_quantity else 0 end) as mar_sales\n \t,sum(case when d_moy = 4 \n \t\tthen ws_ext_list_price* ws_quantity else 0 end) as apr_sales\n \t,sum(case when d_moy = 5 \n \t\tthen ws_ext_list_price* ws_quantity else 0 end) as may_sales\n \t,sum(case when d_moy = 6 \n \t\tthen ws_ext_list_price* ws_quantity else 0 end) as jun_sales\n \t,sum(case when d_moy = 7 \n \t\tthen ws_ext_list_price* ws_quantity else 0 end) as jul_sales\n \t,sum(case when d_moy = 8 \n \t\tthen ws_ext_list_price* ws_quantity else 0 end) as aug_sales\n \t,sum(case when d_moy = 9 \n \t\tthen ws_ext_list_price* ws_quantity else 0 end) as sep_sales\n \t,sum(case when d_moy = 10 \n \t\tthen ws_ext_list_price* ws_quantity else 0 end) as oct_sales\n \t,sum(case when d_moy = 11\n \t\tthen ws_ext_list_price* ws_quantity else 0 end) as nov_sales\n \t,sum(case when d_moy = 12\n \t\tthen ws_ext_list_price* ws_quantity else 0 end) as dec_sales\n \t,sum(case when d_moy = 1 \n \t\tthen ws_net_profit * ws_quantity else 0 end) as jan_net\n \t,sum(case when d_moy = 2\n \t\tthen ws_net_profit * ws_quantity else 0 end) as feb_net\n \t,sum(case when d_moy = 3 \n \t\tthen ws_net_profit * ws_quantity else 0 end) as mar_net\n \t,sum(case when d_moy = 4 \n \t\tthen ws_net_profit * ws_quantity else 0 end) as apr_net\n \t,sum(case when d_moy = 5 \n \t\tthen ws_net_profit * ws_quantity else 0 end) as may_net\n \t,sum(case when d_moy = 6 \n \t\tthen ws_net_profit * ws_quantity else 0 end) as jun_net\n \t,sum(case when d_moy = 7 \n \t\tthen ws_net_profit * ws_quantity else 0 end) as jul_net\n \t,sum(case when d_moy = 8 \n \t\tthen ws_net_profit * ws_quantity else 0 end) as aug_net\n \t,sum(case when d_moy = 9 \n \t\tthen ws_net_profit * ws_quantity else 0 end) as sep_net\n \t,sum(case when d_moy = 10 \n \t\tthen ws_net_profit * ws_quantity else 0 end) as oct_net\n \t,sum(case when d_moy = 11\n \t\tthen ws_net_profit * ws_quantity else 0 end) as nov_net\n \t,sum(case when d_moy = 12\n \t\tthen ws_net_profit * ws_quantity else 0 end) as dec_net\n from\n web_sales\n ,warehouse\n ,date_dim\n ,time_dim\n \t ,ship_mode\n where\n ws_warehouse_sk = w_warehouse_sk\n and ws_sold_date_sk = d_date_sk\n and ws_sold_time_sk = t_time_sk\n \tand ws_ship_mode_sk = sm_ship_mode_sk\n and d_year = 2001\n \tand t_time between 19072 and 19072+28800 \n \tand sm_carrier in ('FEDEX','GERMA')\n group by \n w_warehouse_name\n \t,w_warehouse_sq_ft\n \t,w_city\n \t,w_county\n \t,w_state\n \t,w_country\n ,d_year\n union all\n select \n \tw_warehouse_name\n \t,w_warehouse_sq_ft\n \t,w_city\n \t,w_county\n \t,w_state\n \t,w_country\n \t,'FEDEX' || ',' || 'GERMA' as ship_carriers\n ,d_year as year\n \t,sum(case when d_moy = 1 \n \t\tthen cs_sales_price* cs_quantity else 0 end) as jan_sales\n \t,sum(case when d_moy = 2 \n \t\tthen cs_sales_price* cs_quantity else 0 end) as feb_sales\n \t,sum(case when d_moy = 3 \n \t\tthen cs_sales_price* cs_quantity else 0 end) as mar_sales\n \t,sum(case when d_moy = 4 \n \t\tthen cs_sales_price* cs_quantity else 0 end) as apr_sales\n \t,sum(case when d_moy = 5 \n \t\tthen cs_sales_price* cs_quantity else 0 end) as may_sales\n \t,sum(case when d_moy = 6 \n \t\tthen cs_sales_price* cs_quantity else 0 end) as jun_sales\n \t,sum(case when d_moy = 7 \n \t\tthen cs_sales_price* cs_quantity else 0 end) as jul_sales\n \t,sum(case when d_moy = 8 \n \t\tthen cs_sales_price* cs_quantity else 0 end) as aug_sales\n \t,sum(case when d_moy = 9 \n \t\tthen cs_sales_price* cs_quantity else 0 end) as sep_sales\n \t,sum(case when d_moy = 10 \n \t\tthen cs_sales_price* cs_quantity else 0 end) as oct_sales\n \t,sum(case when d_moy = 11\n \t\tthen cs_sales_price* cs_quantity else 0 end) as nov_sales\n \t,sum(case when d_moy = 12\n \t\tthen cs_sales_price* cs_quantity else 0 end) as dec_sales\n \t,sum(case when d_moy = 1 \n \t\tthen cs_net_paid * cs_quantity else 0 end) as jan_net\n \t,sum(case when d_moy = 2 \n \t\tthen cs_net_paid * cs_quantity else 0 end) as feb_net\n \t,sum(case when d_moy = 3 \n \t\tthen cs_net_paid * cs_quantity else 0 end) as mar_net\n \t,sum(case when d_moy = 4 \n \t\tthen cs_net_paid * cs_quantity else 0 end) as apr_net\n \t,sum(case when d_moy = 5 \n \t\tthen cs_net_paid * cs_quantity else 0 end) as may_net\n \t,sum(case when d_moy = 6 \n \t\tthen cs_net_paid * cs_quantity else 0 end) as jun_net\n \t,sum(case when d_moy = 7 \n \t\tthen cs_net_paid * cs_quantity else 0 end) as jul_net\n \t,sum(case when d_moy = 8 \n \t\tthen cs_net_paid * cs_quantity else 0 end) as aug_net\n \t,sum(case when d_moy = 9 \n \t\tthen cs_net_paid * cs_quantity else 0 end) as sep_net\n \t,sum(case when d_moy = 10 \n \t\tthen cs_net_paid * cs_quantity else 0 end) as oct_net\n \t,sum(case when d_moy = 11\n \t\tthen cs_net_paid * cs_quantity else 0 end) as nov_net\n \t,sum(case when d_moy = 12\n \t\tthen cs_net_paid * cs_quantity else 0 end) as dec_net\n from\n catalog_sales\n ,warehouse\n ,date_dim\n ,time_dim\n \t ,ship_mode\n where\n cs_warehouse_sk = w_warehouse_sk\n and cs_sold_date_sk = d_date_sk\n and cs_sold_time_sk = t_time_sk\n \tand cs_ship_mode_sk = sm_ship_mode_sk\n and d_year = 2001\n \tand t_time between 19072 AND 19072+28800 \n \tand sm_carrier in ('FEDEX','GERMA')\n group by \n w_warehouse_name\n \t,w_warehouse_sq_ft\n \t,w_city\n \t,w_county\n \t,w_state\n \t,w_country\n ,d_year\n ) x\n group by \n w_warehouse_name\n \t,w_warehouse_sq_ft\n \t,w_city\n \t,w_county\n \t,w_state\n \t,w_country\n \t,ship_carriers\n ,year\n order by w_warehouse_name\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q67.sql", "content": "-- CometBench-DS query 67 derived from TPC-DS query 67 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect *\nfrom (select i_category\n ,i_class\n ,i_brand\n ,i_product_name\n ,d_year\n ,d_qoy\n ,d_moy\n ,s_store_id\n ,sumsales\n ,rank() over (partition by i_category order by sumsales desc) rk\n from (select i_category\n ,i_class\n ,i_brand\n ,i_product_name\n ,d_year\n ,d_qoy\n ,d_moy\n ,s_store_id\n ,sum(coalesce(ss_sales_price*ss_quantity,0)) sumsales\n from store_sales\n ,date_dim\n ,store\n ,item\n where ss_sold_date_sk=d_date_sk\n and ss_item_sk=i_item_sk\n and ss_store_sk = s_store_sk\n and d_month_seq between 1194 and 1194+11\n group by rollup(i_category, i_class, i_brand, i_product_name, d_year, d_qoy, d_moy,s_store_id))dw1) dw2\nwhere rk <= 100\norder by i_category\n ,i_class\n ,i_brand\n ,i_product_name\n ,d_year\n ,d_qoy\n ,d_moy\n ,s_store_id\n ,sumsales\n ,rk\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q68.sql", "content": "-- CometBench-DS query 68 derived from TPC-DS query 68 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect c_last_name\n ,c_first_name\n ,ca_city\n ,bought_city\n ,ss_ticket_number\n ,extended_price\n ,extended_tax\n ,list_price\n from (select ss_ticket_number\n ,ss_customer_sk\n ,ca_city bought_city\n ,sum(ss_ext_sales_price) extended_price \n ,sum(ss_ext_list_price) list_price\n ,sum(ss_ext_tax) extended_tax \n from store_sales\n ,date_dim\n ,store\n ,household_demographics\n ,customer_address \n where store_sales.ss_sold_date_sk = date_dim.d_date_sk\n and store_sales.ss_store_sk = store.s_store_sk \n and store_sales.ss_hdemo_sk = household_demographics.hd_demo_sk\n and store_sales.ss_addr_sk = customer_address.ca_address_sk\n and date_dim.d_dom between 1 and 2 \n and (household_demographics.hd_dep_count = 8 or\n household_demographics.hd_vehicle_count= 3)\n and date_dim.d_year in (2000,2000+1,2000+2)\n and store.s_city in ('Midway','Fairview')\n group by ss_ticket_number\n ,ss_customer_sk\n ,ss_addr_sk,ca_city) dn\n ,customer\n ,customer_address current_addr\n where ss_customer_sk = c_customer_sk\n and customer.c_current_addr_sk = current_addr.ca_address_sk\n and current_addr.ca_city <> bought_city\n order by c_last_name\n ,ss_ticket_number\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q69.sql", "content": "-- CometBench-DS query 69 derived from TPC-DS query 69 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect \n cd_gender,\n cd_marital_status,\n cd_education_status,\n count(*) cnt1,\n cd_purchase_estimate,\n count(*) cnt2,\n cd_credit_rating,\n count(*) cnt3\n from\n customer c,customer_address ca,customer_demographics\n where\n c.c_current_addr_sk = ca.ca_address_sk and\n ca_state in ('IN','VA','MS') and\n cd_demo_sk = c.c_current_cdemo_sk and \n exists (select *\n from store_sales,date_dim\n where c.c_customer_sk = ss_customer_sk and\n ss_sold_date_sk = d_date_sk and\n d_year = 2002 and\n d_moy between 2 and 2+2) and\n (not exists (select *\n from web_sales,date_dim\n where c.c_customer_sk = ws_bill_customer_sk and\n ws_sold_date_sk = d_date_sk and\n d_year = 2002 and\n d_moy between 2 and 2+2) and\n not exists (select * \n from catalog_sales,date_dim\n where c.c_customer_sk = cs_ship_customer_sk and\n cs_sold_date_sk = d_date_sk and\n d_year = 2002 and\n d_moy between 2 and 2+2))\n group by cd_gender,\n cd_marital_status,\n cd_education_status,\n cd_purchase_estimate,\n cd_credit_rating\n order by cd_gender,\n cd_marital_status,\n cd_education_status,\n cd_purchase_estimate,\n cd_credit_rating\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q7.sql", "content": "-- CometBench-DS query 7 derived from TPC-DS query 7 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect i_item_id, \n avg(ss_quantity) agg1,\n avg(ss_list_price) agg2,\n avg(ss_coupon_amt) agg3,\n avg(ss_sales_price) agg4 \n from store_sales, customer_demographics, date_dim, item, promotion\n where ss_sold_date_sk = d_date_sk and\n ss_item_sk = i_item_sk and\n ss_cdemo_sk = cd_demo_sk and\n ss_promo_sk = p_promo_sk and\n cd_gender = 'M' and \n cd_marital_status = 'M' and\n cd_education_status = '4 yr Degree' and\n (p_channel_email = 'N' or p_channel_event = 'N') and\n d_year = 2001 \n group by i_item_id\n order by i_item_id\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q70.sql", "content": "-- CometBench-DS query 70 derived from TPC-DS query 70 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect \n sum(ss_net_profit) as total_sum\n ,s_state\n ,s_county\n ,grouping(s_state)+grouping(s_county) as lochierarchy\n ,rank() over (\n \tpartition by grouping(s_state)+grouping(s_county),\n \tcase when grouping(s_county) = 0 then s_state end \n \torder by sum(ss_net_profit) desc) as rank_within_parent\n from\n store_sales\n ,date_dim d1\n ,store\n where\n d1.d_month_seq between 1180 and 1180+11\n and d1.d_date_sk = ss_sold_date_sk\n and s_store_sk = ss_store_sk\n and s_state in\n ( select s_state\n from (select s_state as s_state,\n \t\t\t rank() over ( partition by s_state order by sum(ss_net_profit) desc) as ranking\n from store_sales, store, date_dim\n where d_month_seq between 1180 and 1180+11\n \t\t\t and d_date_sk = ss_sold_date_sk\n \t\t\t and s_store_sk = ss_store_sk\n group by s_state\n ) tmp1 \n where ranking <= 5\n )\n group by rollup(s_state,s_county)\n order by\n lochierarchy desc\n ,case when lochierarchy = 0 then s_state end\n ,rank_within_parent\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q71.sql", "content": "-- CometBench-DS query 71 derived from TPC-DS query 71 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect i_brand_id brand_id, i_brand brand,t_hour,t_minute,\n \tsum(ext_price) ext_price\n from item, (select ws_ext_sales_price as ext_price, \n ws_sold_date_sk as sold_date_sk,\n ws_item_sk as sold_item_sk,\n ws_sold_time_sk as time_sk \n from web_sales,date_dim\n where d_date_sk = ws_sold_date_sk\n and d_moy=11\n and d_year=2001\n union all\n select cs_ext_sales_price as ext_price,\n cs_sold_date_sk as sold_date_sk,\n cs_item_sk as sold_item_sk,\n cs_sold_time_sk as time_sk\n from catalog_sales,date_dim\n where d_date_sk = cs_sold_date_sk\n and d_moy=11\n and d_year=2001\n union all\n select ss_ext_sales_price as ext_price,\n ss_sold_date_sk as sold_date_sk,\n ss_item_sk as sold_item_sk,\n ss_sold_time_sk as time_sk\n from store_sales,date_dim\n where d_date_sk = ss_sold_date_sk\n and d_moy=11\n and d_year=2001\n ) tmp,time_dim\n where\n sold_item_sk = i_item_sk\n and i_manager_id=1\n and time_sk = t_time_sk\n and (t_meal_time = 'breakfast' or t_meal_time = 'dinner')\n group by i_brand, i_brand_id,t_hour,t_minute\n order by ext_price desc, i_brand_id\n ;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q72.sql", "content": "-- CometBench-DS query 72 derived from TPC-DS query 72 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect i_item_desc\n ,w_warehouse_name\n ,d1.d_week_seq\n ,sum(case when p_promo_sk is null then 1 else 0 end) no_promo\n ,sum(case when p_promo_sk is not null then 1 else 0 end) promo\n ,count(*) total_cnt\nfrom catalog_sales\njoin inventory on (cs_item_sk = inv_item_sk)\njoin warehouse on (w_warehouse_sk=inv_warehouse_sk)\njoin item on (i_item_sk = cs_item_sk)\njoin customer_demographics on (cs_bill_cdemo_sk = cd_demo_sk)\njoin household_demographics on (cs_bill_hdemo_sk = hd_demo_sk)\njoin date_dim d1 on (cs_sold_date_sk = d1.d_date_sk)\njoin date_dim d2 on (inv_date_sk = d2.d_date_sk)\njoin date_dim d3 on (cs_ship_date_sk = d3.d_date_sk)\nleft outer join promotion on (cs_promo_sk=p_promo_sk)\nleft outer join catalog_returns on (cr_item_sk = cs_item_sk and cr_order_number = cs_order_number)\nwhere d1.d_week_seq = d2.d_week_seq\n and inv_quantity_on_hand < cs_quantity \n and d3.d_date > d1.d_date + 5\n and hd_buy_potential = '501-1000'\n and d1.d_year = 1999\n and cd_marital_status = 'S'\ngroup by i_item_desc,w_warehouse_name,d1.d_week_seq\norder by total_cnt desc, i_item_desc, w_warehouse_name, d_week_seq\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q73.sql", "content": "-- CometBench-DS query 73 derived from TPC-DS query 73 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect c_last_name\n ,c_first_name\n ,c_salutation\n ,c_preferred_cust_flag \n ,ss_ticket_number\n ,cnt from\n (select ss_ticket_number\n ,ss_customer_sk\n ,count(*) cnt\n from store_sales,date_dim,store,household_demographics\n where store_sales.ss_sold_date_sk = date_dim.d_date_sk\n and store_sales.ss_store_sk = store.s_store_sk \n and store_sales.ss_hdemo_sk = household_demographics.hd_demo_sk\n and date_dim.d_dom between 1 and 2 \n and (household_demographics.hd_buy_potential = '1001-5000' or\n household_demographics.hd_buy_potential = '5001-10000')\n and household_demographics.hd_vehicle_count > 0\n and case when household_demographics.hd_vehicle_count > 0 then \n household_demographics.hd_dep_count/ household_demographics.hd_vehicle_count else null end > 1\n and date_dim.d_year in (1999,1999+1,1999+2)\n and store.s_county in ('Williamson County','Williamson County','Williamson County','Williamson County')\n group by ss_ticket_number,ss_customer_sk) dj,customer\n where ss_customer_sk = c_customer_sk\n and cnt between 1 and 5\n order by cnt desc, c_last_name asc;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q74.sql", "content": "-- CometBench-DS query 74 derived from TPC-DS query 74 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith year_total as (\n select c_customer_id customer_id\n ,c_first_name customer_first_name\n ,c_last_name customer_last_name\n ,d_year as year\n ,stddev_samp(ss_net_paid) year_total\n ,'s' sale_type\n from customer\n ,store_sales\n ,date_dim\n where c_customer_sk = ss_customer_sk\n and ss_sold_date_sk = d_date_sk\n and d_year in (2001,2001+1)\n group by c_customer_id\n ,c_first_name\n ,c_last_name\n ,d_year\n union all\n select c_customer_id customer_id\n ,c_first_name customer_first_name\n ,c_last_name customer_last_name\n ,d_year as year\n ,stddev_samp(ws_net_paid) year_total\n ,'w' sale_type\n from customer\n ,web_sales\n ,date_dim\n where c_customer_sk = ws_bill_customer_sk\n and ws_sold_date_sk = d_date_sk\n and d_year in (2001,2001+1)\n group by c_customer_id\n ,c_first_name\n ,c_last_name\n ,d_year\n )\n select \n t_s_secyear.customer_id, t_s_secyear.customer_first_name, t_s_secyear.customer_last_name\n from year_total t_s_firstyear\n ,year_total t_s_secyear\n ,year_total t_w_firstyear\n ,year_total t_w_secyear\n where t_s_secyear.customer_id = t_s_firstyear.customer_id\n and t_s_firstyear.customer_id = t_w_secyear.customer_id\n and t_s_firstyear.customer_id = t_w_firstyear.customer_id\n and t_s_firstyear.sale_type = 's'\n and t_w_firstyear.sale_type = 'w'\n and t_s_secyear.sale_type = 's'\n and t_w_secyear.sale_type = 'w'\n and t_s_firstyear.year = 2001\n and t_s_secyear.year = 2001+1\n and t_w_firstyear.year = 2001\n and t_w_secyear.year = 2001+1\n and t_s_firstyear.year_total > 0\n and t_w_firstyear.year_total > 0\n and case when t_w_firstyear.year_total > 0 then t_w_secyear.year_total / t_w_firstyear.year_total else null end\n > case when t_s_firstyear.year_total > 0 then t_s_secyear.year_total / t_s_firstyear.year_total else null end\n order by 3,2,1\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q75.sql", "content": "-- CometBench-DS query 75 derived from TPC-DS query 75 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nWITH all_sales AS (\n SELECT d_year\n ,i_brand_id\n ,i_class_id\n ,i_category_id\n ,i_manufact_id\n ,SUM(sales_cnt) AS sales_cnt\n ,SUM(sales_amt) AS sales_amt\n FROM (SELECT d_year\n ,i_brand_id\n ,i_class_id\n ,i_category_id\n ,i_manufact_id\n ,cs_quantity - COALESCE(cr_return_quantity,0) AS sales_cnt\n ,cs_ext_sales_price - COALESCE(cr_return_amount,0.0) AS sales_amt\n FROM catalog_sales JOIN item ON i_item_sk=cs_item_sk\n JOIN date_dim ON d_date_sk=cs_sold_date_sk\n LEFT JOIN catalog_returns ON (cs_order_number=cr_order_number \n AND cs_item_sk=cr_item_sk)\n WHERE i_category='Shoes'\n UNION\n SELECT d_year\n ,i_brand_id\n ,i_class_id\n ,i_category_id\n ,i_manufact_id\n ,ss_quantity - COALESCE(sr_return_quantity,0) AS sales_cnt\n ,ss_ext_sales_price - COALESCE(sr_return_amt,0.0) AS sales_amt\n FROM store_sales JOIN item ON i_item_sk=ss_item_sk\n JOIN date_dim ON d_date_sk=ss_sold_date_sk\n LEFT JOIN store_returns ON (ss_ticket_number=sr_ticket_number \n AND ss_item_sk=sr_item_sk)\n WHERE i_category='Shoes'\n UNION\n SELECT d_year\n ,i_brand_id\n ,i_class_id\n ,i_category_id\n ,i_manufact_id\n ,ws_quantity - COALESCE(wr_return_quantity,0) AS sales_cnt\n ,ws_ext_sales_price - COALESCE(wr_return_amt,0.0) AS sales_amt\n FROM web_sales JOIN item ON i_item_sk=ws_item_sk\n JOIN date_dim ON d_date_sk=ws_sold_date_sk\n LEFT JOIN web_returns ON (ws_order_number=wr_order_number \n AND ws_item_sk=wr_item_sk)\n WHERE i_category='Shoes') sales_detail\n GROUP BY d_year, i_brand_id, i_class_id, i_category_id, i_manufact_id)\n SELECT prev_yr.d_year AS prev_year\n ,curr_yr.d_year AS year\n ,curr_yr.i_brand_id\n ,curr_yr.i_class_id\n ,curr_yr.i_category_id\n ,curr_yr.i_manufact_id\n ,prev_yr.sales_cnt AS prev_yr_cnt\n ,curr_yr.sales_cnt AS curr_yr_cnt\n ,curr_yr.sales_cnt-prev_yr.sales_cnt AS sales_cnt_diff\n ,curr_yr.sales_amt-prev_yr.sales_amt AS sales_amt_diff\n FROM all_sales curr_yr, all_sales prev_yr\n WHERE curr_yr.i_brand_id=prev_yr.i_brand_id\n AND curr_yr.i_class_id=prev_yr.i_class_id\n AND curr_yr.i_category_id=prev_yr.i_category_id\n AND curr_yr.i_manufact_id=prev_yr.i_manufact_id\n AND curr_yr.d_year=2000\n AND prev_yr.d_year=2000-1\n AND CAST(curr_yr.sales_cnt AS DECIMAL(17,2))/CAST(prev_yr.sales_cnt AS DECIMAL(17,2))<0.9\n ORDER BY sales_cnt_diff,sales_amt_diff\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q76.sql", "content": "-- CometBench-DS query 76 derived from TPC-DS query 76 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect channel, col_name, d_year, d_qoy, i_category, COUNT(*) sales_cnt, SUM(ext_sales_price) sales_amt FROM (\n SELECT 'store' as channel, 'ss_customer_sk' col_name, d_year, d_qoy, i_category, ss_ext_sales_price ext_sales_price\n FROM store_sales, item, date_dim\n WHERE ss_customer_sk IS NULL\n AND ss_sold_date_sk=d_date_sk\n AND ss_item_sk=i_item_sk\n UNION ALL\n SELECT 'web' as channel, 'ws_ship_hdemo_sk' col_name, d_year, d_qoy, i_category, ws_ext_sales_price ext_sales_price\n FROM web_sales, item, date_dim\n WHERE ws_ship_hdemo_sk IS NULL\n AND ws_sold_date_sk=d_date_sk\n AND ws_item_sk=i_item_sk\n UNION ALL\n SELECT 'catalog' as channel, 'cs_bill_customer_sk' col_name, d_year, d_qoy, i_category, cs_ext_sales_price ext_sales_price\n FROM catalog_sales, item, date_dim\n WHERE cs_bill_customer_sk IS NULL\n AND cs_sold_date_sk=d_date_sk\n AND cs_item_sk=i_item_sk) foo\nGROUP BY channel, col_name, d_year, d_qoy, i_category\nORDER BY channel, col_name, d_year, d_qoy, i_category\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q77.sql", "content": "-- CometBench-DS query 77 derived from TPC-DS query 77 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith ss as\n (select s_store_sk,\n sum(ss_ext_sales_price) as sales,\n sum(ss_net_profit) as profit\n from store_sales,\n date_dim,\n store\n where ss_sold_date_sk = d_date_sk\n and d_date between cast('2001-08-11' as date) \n and (cast('2001-08-11' as date) + INTERVAL '30 DAYS') \n and ss_store_sk = s_store_sk\n group by s_store_sk)\n ,\n sr as\n (select s_store_sk,\n sum(sr_return_amt) as returns,\n sum(sr_net_loss) as profit_loss\n from store_returns,\n date_dim,\n store\n where sr_returned_date_sk = d_date_sk\n and d_date between cast('2001-08-11' as date)\n and (cast('2001-08-11' as date) + INTERVAL '30 DAYS')\n and sr_store_sk = s_store_sk\n group by s_store_sk), \n cs as\n (select cs_call_center_sk,\n sum(cs_ext_sales_price) as sales,\n sum(cs_net_profit) as profit\n from catalog_sales,\n date_dim\n where cs_sold_date_sk = d_date_sk\n and d_date between cast('2001-08-11' as date)\n and (cast('2001-08-11' as date) + INTERVAL '30 DAYS')\n group by cs_call_center_sk \n ), \n cr as\n (select cr_call_center_sk,\n sum(cr_return_amount) as returns,\n sum(cr_net_loss) as profit_loss\n from catalog_returns,\n date_dim\n where cr_returned_date_sk = d_date_sk\n and d_date between cast('2001-08-11' as date)\n and (cast('2001-08-11' as date) + INTERVAL '30 DAYS')\n group by cr_call_center_sk\n ), \n ws as\n ( select wp_web_page_sk,\n sum(ws_ext_sales_price) as sales,\n sum(ws_net_profit) as profit\n from web_sales,\n date_dim,\n web_page\n where ws_sold_date_sk = d_date_sk\n and d_date between cast('2001-08-11' as date)\n and (cast('2001-08-11' as date) + INTERVAL '30 DAYS')\n and ws_web_page_sk = wp_web_page_sk\n group by wp_web_page_sk), \n wr as\n (select wp_web_page_sk,\n sum(wr_return_amt) as returns,\n sum(wr_net_loss) as profit_loss\n from web_returns,\n date_dim,\n web_page\n where wr_returned_date_sk = d_date_sk\n and d_date between cast('2001-08-11' as date)\n and (cast('2001-08-11' as date) + INTERVAL '30 DAYS')\n and wr_web_page_sk = wp_web_page_sk\n group by wp_web_page_sk)\n select channel\n , id\n , sum(sales) as sales\n , sum(returns) as returns\n , sum(profit) as profit\n from \n (select 'store channel' as channel\n , ss.s_store_sk as id\n , sales\n , coalesce(returns, 0) as returns\n , (profit - coalesce(profit_loss,0)) as profit\n from ss left join sr\n on ss.s_store_sk = sr.s_store_sk\n union all\n select 'catalog channel' as channel\n , cs_call_center_sk as id\n , sales\n , returns\n , (profit - profit_loss) as profit\n from cs\n , cr\n union all\n select 'web channel' as channel\n , ws.wp_web_page_sk as id\n , sales\n , coalesce(returns, 0) returns\n , (profit - coalesce(profit_loss,0)) as profit\n from ws left join wr\n on ws.wp_web_page_sk = wr.wp_web_page_sk\n ) x\n group by rollup (channel, id)\n order by channel\n ,id\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q78.sql", "content": "-- CometBench-DS query 78 derived from TPC-DS query 78 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith ws as\n (select d_year AS ws_sold_year, ws_item_sk,\n ws_bill_customer_sk ws_customer_sk,\n sum(ws_quantity) ws_qty,\n sum(ws_wholesale_cost) ws_wc,\n sum(ws_sales_price) ws_sp\n from web_sales\n left join web_returns on wr_order_number=ws_order_number and ws_item_sk=wr_item_sk\n join date_dim on ws_sold_date_sk = d_date_sk\n where wr_order_number is null\n group by d_year, ws_item_sk, ws_bill_customer_sk\n ),\ncs as\n (select d_year AS cs_sold_year, cs_item_sk,\n cs_bill_customer_sk cs_customer_sk,\n sum(cs_quantity) cs_qty,\n sum(cs_wholesale_cost) cs_wc,\n sum(cs_sales_price) cs_sp\n from catalog_sales\n left join catalog_returns on cr_order_number=cs_order_number and cs_item_sk=cr_item_sk\n join date_dim on cs_sold_date_sk = d_date_sk\n where cr_order_number is null\n group by d_year, cs_item_sk, cs_bill_customer_sk\n ),\nss as\n (select d_year AS ss_sold_year, ss_item_sk,\n ss_customer_sk,\n sum(ss_quantity) ss_qty,\n sum(ss_wholesale_cost) ss_wc,\n sum(ss_sales_price) ss_sp\n from store_sales\n left join store_returns on sr_ticket_number=ss_ticket_number and ss_item_sk=sr_item_sk\n join date_dim on ss_sold_date_sk = d_date_sk\n where sr_ticket_number is null\n group by d_year, ss_item_sk, ss_customer_sk\n )\n select \nss_customer_sk,\nround(ss_qty/(coalesce(ws_qty,0)+coalesce(cs_qty,0)),2) ratio,\nss_qty store_qty, ss_wc store_wholesale_cost, ss_sp store_sales_price,\ncoalesce(ws_qty,0)+coalesce(cs_qty,0) other_chan_qty,\ncoalesce(ws_wc,0)+coalesce(cs_wc,0) other_chan_wholesale_cost,\ncoalesce(ws_sp,0)+coalesce(cs_sp,0) other_chan_sales_price\nfrom ss\nleft join ws on (ws_sold_year=ss_sold_year and ws_item_sk=ss_item_sk and ws_customer_sk=ss_customer_sk)\nleft join cs on (cs_sold_year=ss_sold_year and cs_item_sk=ss_item_sk and cs_customer_sk=ss_customer_sk)\nwhere (coalesce(ws_qty,0)>0 or coalesce(cs_qty, 0)>0) and ss_sold_year=2001\norder by \n ss_customer_sk,\n ss_qty desc, ss_wc desc, ss_sp desc,\n other_chan_qty,\n other_chan_wholesale_cost,\n other_chan_sales_price,\n ratio\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q79.sql", "content": "-- CometBench-DS query 79 derived from TPC-DS query 79 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect \n c_last_name,c_first_name,substr(s_city,1,30),ss_ticket_number,amt,profit\n from\n (select ss_ticket_number\n ,ss_customer_sk\n ,store.s_city\n ,sum(ss_coupon_amt) amt\n ,sum(ss_net_profit) profit\n from store_sales,date_dim,store,household_demographics\n where store_sales.ss_sold_date_sk = date_dim.d_date_sk\n and store_sales.ss_store_sk = store.s_store_sk \n and store_sales.ss_hdemo_sk = household_demographics.hd_demo_sk\n and (household_demographics.hd_dep_count = 0 or household_demographics.hd_vehicle_count > 4)\n and date_dim.d_dow = 1\n and date_dim.d_year in (1999,1999+1,1999+2) \n and store.s_number_employees between 200 and 295\n group by ss_ticket_number,ss_customer_sk,ss_addr_sk,store.s_city) ms,customer\n where ss_customer_sk = c_customer_sk\n order by c_last_name,c_first_name,substr(s_city,1,30), profit\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q8.sql", "content": "-- CometBench-DS query 8 derived from TPC-DS query 8 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect s_store_name\n ,sum(ss_net_profit)\n from store_sales\n ,date_dim\n ,store,\n (select ca_zip\n from (\n SELECT substr(ca_zip,1,5) ca_zip\n FROM customer_address\n WHERE substr(ca_zip,1,5) IN (\n '19100','41548','51640','49699','88329','55986',\n '85119','19510','61020','95452','26235',\n '51102','16733','42819','27823','90192',\n '31905','28865','62197','23750','81398',\n '95288','45114','82060','12313','25218',\n '64386','46400','77230','69271','43672',\n '36521','34217','13017','27936','42766',\n '59233','26060','27477','39981','93402',\n '74270','13932','51731','71642','17710',\n '85156','21679','70840','67191','39214',\n '35273','27293','17128','15458','31615',\n '60706','67657','54092','32775','14683',\n '32206','62543','43053','11297','58216',\n '49410','14710','24501','79057','77038',\n '91286','32334','46298','18326','67213',\n '65382','40315','56115','80162','55956',\n '81583','73588','32513','62880','12201',\n '11592','17014','83832','61796','57872',\n '78829','69912','48524','22016','26905',\n '48511','92168','63051','25748','89786',\n '98827','86404','53029','37524','14039',\n '50078','34487','70142','18697','40129',\n '60642','42810','62667','57183','46414',\n '58463','71211','46364','34851','54884',\n '25382','25239','74126','21568','84204',\n '13607','82518','32982','36953','86001',\n '79278','21745','64444','35199','83181',\n '73255','86177','98043','90392','13882',\n '47084','17859','89526','42072','20233',\n '52745','75000','22044','77013','24182',\n '52554','56138','43440','86100','48791',\n '21883','17096','15965','31196','74903',\n '19810','35763','92020','55176','54433',\n '68063','71919','44384','16612','32109',\n '28207','14762','89933','10930','27616',\n '56809','14244','22733','33177','29784',\n '74968','37887','11299','34692','85843',\n '83663','95421','19323','17406','69264',\n '28341','50150','79121','73974','92917',\n '21229','32254','97408','46011','37169',\n '18146','27296','62927','68812','47734',\n '86572','12620','80252','50173','27261',\n '29534','23488','42184','23695','45868',\n '12910','23429','29052','63228','30731',\n '15747','25827','22332','62349','56661',\n '44652','51862','57007','22773','40361',\n '65238','19327','17282','44708','35484',\n '34064','11148','92729','22995','18833',\n '77528','48917','17256','93166','68576',\n '71096','56499','35096','80551','82424',\n '17700','32748','78969','46820','57725',\n '46179','54677','98097','62869','83959',\n '66728','19716','48326','27420','53458',\n '69056','84216','36688','63957','41469',\n '66843','18024','81950','21911','58387',\n '58103','19813','34581','55347','17171',\n '35914','75043','75088','80541','26802',\n '28849','22356','57721','77084','46385',\n '59255','29308','65885','70673','13306',\n '68788','87335','40987','31654','67560',\n '92309','78116','65961','45018','16548',\n '67092','21818','33716','49449','86150',\n '12156','27574','43201','50977','52839',\n '33234','86611','71494','17823','57172',\n '59869','34086','51052','11320','39717',\n '79604','24672','70555','38378','91135',\n '15567','21606','74994','77168','38607',\n '27384','68328','88944','40203','37893',\n '42726','83549','48739','55652','27543',\n '23109','98908','28831','45011','47525',\n '43870','79404','35780','42136','49317',\n '14574','99586','21107','14302','83882',\n '81272','92552','14916','87533','86518',\n '17862','30741','96288','57886','30304',\n '24201','79457','36728','49833','35182',\n '20108','39858','10804','47042','20439',\n '54708','59027','82499','75311','26548',\n '53406','92060','41152','60446','33129',\n '43979','16903','60319','35550','33887',\n '25463','40343','20726','44429')\n intersect\n select ca_zip\n from (SELECT substr(ca_zip,1,5) ca_zip,count(*) cnt\n FROM customer_address, customer\n WHERE ca_address_sk = c_current_addr_sk and\n c_preferred_cust_flag='Y'\n group by ca_zip\n having count(*) > 10)A1)A2) V1\n where ss_store_sk = s_store_sk\n and ss_sold_date_sk = d_date_sk\n and d_qoy = 1 and d_year = 2000\n and (substr(s_zip,1,2) = substr(V1.ca_zip,1,2))\n group by s_store_name\n order by s_store_name\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q80.sql", "content": "-- CometBench-DS query 80 derived from TPC-DS query 80 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith ssr as\n (select s_store_id as store_id,\n sum(ss_ext_sales_price) as sales,\n sum(coalesce(sr_return_amt, 0)) as returns,\n sum(ss_net_profit - coalesce(sr_net_loss, 0)) as profit\n from store_sales left outer join store_returns on\n (ss_item_sk = sr_item_sk and ss_ticket_number = sr_ticket_number),\n date_dim,\n store,\n item,\n promotion\n where ss_sold_date_sk = d_date_sk\n and d_date between cast('2002-08-04' as date) \n and (cast('2002-08-04' as date) + INTERVAL '30 DAYS')\n and ss_store_sk = s_store_sk\n and ss_item_sk = i_item_sk\n and i_current_price > 50\n and ss_promo_sk = p_promo_sk\n and p_channel_tv = 'N'\n group by s_store_id)\n ,\n csr as\n (select cp_catalog_page_id as catalog_page_id,\n sum(cs_ext_sales_price) as sales,\n sum(coalesce(cr_return_amount, 0)) as returns,\n sum(cs_net_profit - coalesce(cr_net_loss, 0)) as profit\n from catalog_sales left outer join catalog_returns on\n (cs_item_sk = cr_item_sk and cs_order_number = cr_order_number),\n date_dim,\n catalog_page,\n item,\n promotion\n where cs_sold_date_sk = d_date_sk\n and d_date between cast('2002-08-04' as date)\n and (cast('2002-08-04' as date) + INTERVAL '30 DAYS')\n and cs_catalog_page_sk = cp_catalog_page_sk\n and cs_item_sk = i_item_sk\n and i_current_price > 50\n and cs_promo_sk = p_promo_sk\n and p_channel_tv = 'N'\ngroup by cp_catalog_page_id)\n ,\n wsr as\n (select web_site_id,\n sum(ws_ext_sales_price) as sales,\n sum(coalesce(wr_return_amt, 0)) as returns,\n sum(ws_net_profit - coalesce(wr_net_loss, 0)) as profit\n from web_sales left outer join web_returns on\n (ws_item_sk = wr_item_sk and ws_order_number = wr_order_number),\n date_dim,\n web_site,\n item,\n promotion\n where ws_sold_date_sk = d_date_sk\n and d_date between cast('2002-08-04' as date)\n and (cast('2002-08-04' as date) + INTERVAL '30 DAYS')\n and ws_web_site_sk = web_site_sk\n and ws_item_sk = i_item_sk\n and i_current_price > 50\n and ws_promo_sk = p_promo_sk\n and p_channel_tv = 'N'\ngroup by web_site_id)\n select channel\n , id\n , sum(sales) as sales\n , sum(returns) as returns\n , sum(profit) as profit\n from \n (select 'store channel' as channel\n , 'store' || store_id as id\n , sales\n , returns\n , profit\n from ssr\n union all\n select 'catalog channel' as channel\n , 'catalog_page' || catalog_page_id as id\n , sales\n , returns\n , profit\n from csr\n union all\n select 'web channel' as channel\n , 'web_site' || web_site_id as id\n , sales\n , returns\n , profit\n from wsr\n ) x\n group by rollup (channel, id)\n order by channel\n ,id\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q81.sql", "content": "-- CometBench-DS query 81 derived from TPC-DS query 81 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith customer_total_return as\n (select cr_returning_customer_sk as ctr_customer_sk\n ,ca_state as ctr_state, \n \tsum(cr_return_amt_inc_tax) as ctr_total_return\n from catalog_returns\n ,date_dim\n ,customer_address\n where cr_returned_date_sk = d_date_sk \n and d_year =1998\n and cr_returning_addr_sk = ca_address_sk \n group by cr_returning_customer_sk\n ,ca_state )\n select c_customer_id,c_salutation,c_first_name,c_last_name,ca_street_number,ca_street_name\n ,ca_street_type,ca_suite_number,ca_city,ca_county,ca_state,ca_zip,ca_country,ca_gmt_offset\n ,ca_location_type,ctr_total_return\n from customer_total_return ctr1\n ,customer_address\n ,customer\n where ctr1.ctr_total_return > (select avg(ctr_total_return)*1.2\n \t\t\t from customer_total_return ctr2 \n \t where ctr1.ctr_state = ctr2.ctr_state)\n and ca_address_sk = c_current_addr_sk\n and ca_state = 'TX'\n and ctr1.ctr_customer_sk = c_customer_sk\n order by c_customer_id,c_salutation,c_first_name,c_last_name,ca_street_number,ca_street_name\n ,ca_street_type,ca_suite_number,ca_city,ca_county,ca_state,ca_zip,ca_country,ca_gmt_offset\n ,ca_location_type,ctr_total_return\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q82.sql", "content": "-- CometBench-DS query 82 derived from TPC-DS query 82 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect i_item_id\n ,i_item_desc\n ,i_current_price\n from item, inventory, date_dim, store_sales\n where i_current_price between 69 and 69+30\n and inv_item_sk = i_item_sk\n and d_date_sk=inv_date_sk\n and d_date between cast('1998-06-06' as date) and (cast('1998-06-06' as date) + INTERVAL '60 DAYS')\n and i_manufact_id in (105,513,180,137)\n and inv_quantity_on_hand between 100 and 500\n and ss_item_sk = i_item_sk\n group by i_item_id,i_item_desc,i_current_price\n order by i_item_id\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q83.sql", "content": "-- CometBench-DS query 83 derived from TPC-DS query 83 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith sr_items as\n (select i_item_id item_id,\n sum(sr_return_quantity) sr_item_qty\n from store_returns,\n item,\n date_dim\n where sr_item_sk = i_item_sk\n and d_date in \n\t(select d_date\n\tfrom date_dim\n\twhere d_week_seq in \n\t\t(select d_week_seq\n\t\tfrom date_dim\n\t where d_date in ('2000-04-29','2000-09-09','2000-11-02')))\n and sr_returned_date_sk = d_date_sk\n group by i_item_id),\n cr_items as\n (select i_item_id item_id,\n sum(cr_return_quantity) cr_item_qty\n from catalog_returns,\n item,\n date_dim\n where cr_item_sk = i_item_sk\n and d_date in \n\t(select d_date\n\tfrom date_dim\n\twhere d_week_seq in \n\t\t(select d_week_seq\n\t\tfrom date_dim\n\t where d_date in ('2000-04-29','2000-09-09','2000-11-02')))\n and cr_returned_date_sk = d_date_sk\n group by i_item_id),\n wr_items as\n (select i_item_id item_id,\n sum(wr_return_quantity) wr_item_qty\n from web_returns,\n item,\n date_dim\n where wr_item_sk = i_item_sk\n and d_date in \n\t(select d_date\n\tfrom date_dim\n\twhere d_week_seq in \n\t\t(select d_week_seq\n\t\tfrom date_dim\n\t\twhere d_date in ('2000-04-29','2000-09-09','2000-11-02')))\n and wr_returned_date_sk = d_date_sk\n group by i_item_id)\n select sr_items.item_id\n ,sr_item_qty\n ,sr_item_qty/(sr_item_qty+cr_item_qty+wr_item_qty)/3.0 * 100 sr_dev\n ,cr_item_qty\n ,cr_item_qty/(sr_item_qty+cr_item_qty+wr_item_qty)/3.0 * 100 cr_dev\n ,wr_item_qty\n ,wr_item_qty/(sr_item_qty+cr_item_qty+wr_item_qty)/3.0 * 100 wr_dev\n ,(sr_item_qty+cr_item_qty+wr_item_qty)/3.0 average\n from sr_items\n ,cr_items\n ,wr_items\n where sr_items.item_id=cr_items.item_id\n and sr_items.item_id=wr_items.item_id \n order by sr_items.item_id\n ,sr_item_qty\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q84.sql", "content": "-- CometBench-DS query 84 derived from TPC-DS query 84 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect c_customer_id as customer_id\n , coalesce(c_last_name,'') || ', ' || coalesce(c_first_name,'') as customername\n from customer\n ,customer_address\n ,customer_demographics\n ,household_demographics\n ,income_band\n ,store_returns\n where ca_city\t = 'White Oak'\n and c_current_addr_sk = ca_address_sk\n and ib_lower_bound >= 45626\n and ib_upper_bound <= 45626 + 50000\n and ib_income_band_sk = hd_income_band_sk\n and cd_demo_sk = c_current_cdemo_sk\n and hd_demo_sk = c_current_hdemo_sk\n and sr_cdemo_sk = cd_demo_sk\n order by c_customer_id\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q85.sql", "content": "-- CometBench-DS query 85 derived from TPC-DS query 85 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect substr(r_reason_desc,1,20)\n ,avg(ws_quantity)\n ,avg(wr_refunded_cash)\n ,avg(wr_fee)\n from web_sales, web_returns, web_page, customer_demographics cd1,\n customer_demographics cd2, customer_address, date_dim, reason \n where ws_web_page_sk = wp_web_page_sk\n and ws_item_sk = wr_item_sk\n and ws_order_number = wr_order_number\n and ws_sold_date_sk = d_date_sk and d_year = 2001\n and cd1.cd_demo_sk = wr_refunded_cdemo_sk \n and cd2.cd_demo_sk = wr_returning_cdemo_sk\n and ca_address_sk = wr_refunded_addr_sk\n and r_reason_sk = wr_reason_sk\n and\n (\n (\n cd1.cd_marital_status = 'D'\n and\n cd1.cd_marital_status = cd2.cd_marital_status\n and\n cd1.cd_education_status = 'Primary'\n and \n cd1.cd_education_status = cd2.cd_education_status\n and\n ws_sales_price between 100.00 and 150.00\n )\n or\n (\n cd1.cd_marital_status = 'U'\n and\n cd1.cd_marital_status = cd2.cd_marital_status\n and\n cd1.cd_education_status = 'Unknown' \n and\n cd1.cd_education_status = cd2.cd_education_status\n and\n ws_sales_price between 50.00 and 100.00\n )\n or\n (\n cd1.cd_marital_status = 'M'\n and\n cd1.cd_marital_status = cd2.cd_marital_status\n and\n cd1.cd_education_status = 'Advanced Degree'\n and\n cd1.cd_education_status = cd2.cd_education_status\n and\n ws_sales_price between 150.00 and 200.00\n )\n )\n and\n (\n (\n ca_country = 'United States'\n and\n ca_state in ('SC', 'IN', 'VA')\n and ws_net_profit between 100 and 200 \n )\n or\n (\n ca_country = 'United States'\n and\n ca_state in ('WA', 'KS', 'KY')\n and ws_net_profit between 150 and 300 \n )\n or\n (\n ca_country = 'United States'\n and\n ca_state in ('SD', 'WI', 'NE')\n and ws_net_profit between 50 and 250 \n )\n )\ngroup by r_reason_desc\norder by substr(r_reason_desc,1,20)\n ,avg(ws_quantity)\n ,avg(wr_refunded_cash)\n ,avg(wr_fee)\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q86.sql", "content": "-- CometBench-DS query 86 derived from TPC-DS query 86 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect \n sum(ws_net_paid) as total_sum\n ,i_category\n ,i_class\n ,grouping(i_category)+grouping(i_class) as lochierarchy\n ,rank() over (\n \tpartition by grouping(i_category)+grouping(i_class),\n \tcase when grouping(i_class) = 0 then i_category end \n \torder by sum(ws_net_paid) desc) as rank_within_parent\n from\n web_sales\n ,date_dim d1\n ,item\n where\n d1.d_month_seq between 1205 and 1205+11\n and d1.d_date_sk = ws_sold_date_sk\n and i_item_sk = ws_item_sk\n group by rollup(i_category,i_class)\n order by\n lochierarchy desc,\n case when lochierarchy = 0 then i_category end,\n rank_within_parent\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q87.sql", "content": "-- CometBench-DS query 87 derived from TPC-DS query 87 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect count(*) \nfrom ((select distinct c_last_name, c_first_name, d_date\n from store_sales, date_dim, customer\n where store_sales.ss_sold_date_sk = date_dim.d_date_sk\n and store_sales.ss_customer_sk = customer.c_customer_sk\n and d_month_seq between 1189 and 1189+11)\n except\n (select distinct c_last_name, c_first_name, d_date\n from catalog_sales, date_dim, customer\n where catalog_sales.cs_sold_date_sk = date_dim.d_date_sk\n and catalog_sales.cs_bill_customer_sk = customer.c_customer_sk\n and d_month_seq between 1189 and 1189+11)\n except\n (select distinct c_last_name, c_first_name, d_date\n from web_sales, date_dim, customer\n where web_sales.ws_sold_date_sk = date_dim.d_date_sk\n and web_sales.ws_bill_customer_sk = customer.c_customer_sk\n and d_month_seq between 1189 and 1189+11)\n) cool_cust\n;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q88.sql", "content": "-- CometBench-DS query 88 derived from TPC-DS query 88 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect *\nfrom\n (select count(*) h8_30_to_9\n from store_sales, household_demographics , time_dim, store\n where ss_sold_time_sk = time_dim.t_time_sk \n and ss_hdemo_sk = household_demographics.hd_demo_sk \n and ss_store_sk = s_store_sk\n and time_dim.t_hour = 8\n and time_dim.t_minute >= 30\n and ((household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or\n (household_demographics.hd_dep_count = 1 and household_demographics.hd_vehicle_count<=1+2) or\n (household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2)) \n and store.s_store_name = 'ese') s1,\n (select count(*) h9_to_9_30 \n from store_sales, household_demographics , time_dim, store\n where ss_sold_time_sk = time_dim.t_time_sk\n and ss_hdemo_sk = household_demographics.hd_demo_sk\n and ss_store_sk = s_store_sk \n and time_dim.t_hour = 9 \n and time_dim.t_minute < 30\n and ((household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or\n (household_demographics.hd_dep_count = 1 and household_demographics.hd_vehicle_count<=1+2) or\n (household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2))\n and store.s_store_name = 'ese') s2,\n (select count(*) h9_30_to_10 \n from store_sales, household_demographics , time_dim, store\n where ss_sold_time_sk = time_dim.t_time_sk\n and ss_hdemo_sk = household_demographics.hd_demo_sk\n and ss_store_sk = s_store_sk\n and time_dim.t_hour = 9\n and time_dim.t_minute >= 30\n and ((household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or\n (household_demographics.hd_dep_count = 1 and household_demographics.hd_vehicle_count<=1+2) or\n (household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2))\n and store.s_store_name = 'ese') s3,\n (select count(*) h10_to_10_30\n from store_sales, household_demographics , time_dim, store\n where ss_sold_time_sk = time_dim.t_time_sk\n and ss_hdemo_sk = household_demographics.hd_demo_sk\n and ss_store_sk = s_store_sk\n and time_dim.t_hour = 10 \n and time_dim.t_minute < 30\n and ((household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or\n (household_demographics.hd_dep_count = 1 and household_demographics.hd_vehicle_count<=1+2) or\n (household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2))\n and store.s_store_name = 'ese') s4,\n (select count(*) h10_30_to_11\n from store_sales, household_demographics , time_dim, store\n where ss_sold_time_sk = time_dim.t_time_sk\n and ss_hdemo_sk = household_demographics.hd_demo_sk\n and ss_store_sk = s_store_sk\n and time_dim.t_hour = 10 \n and time_dim.t_minute >= 30\n and ((household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or\n (household_demographics.hd_dep_count = 1 and household_demographics.hd_vehicle_count<=1+2) or\n (household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2))\n and store.s_store_name = 'ese') s5,\n (select count(*) h11_to_11_30\n from store_sales, household_demographics , time_dim, store\n where ss_sold_time_sk = time_dim.t_time_sk\n and ss_hdemo_sk = household_demographics.hd_demo_sk\n and ss_store_sk = s_store_sk \n and time_dim.t_hour = 11\n and time_dim.t_minute < 30\n and ((household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or\n (household_demographics.hd_dep_count = 1 and household_demographics.hd_vehicle_count<=1+2) or\n (household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2))\n and store.s_store_name = 'ese') s6,\n (select count(*) h11_30_to_12\n from store_sales, household_demographics , time_dim, store\n where ss_sold_time_sk = time_dim.t_time_sk\n and ss_hdemo_sk = household_demographics.hd_demo_sk\n and ss_store_sk = s_store_sk\n and time_dim.t_hour = 11\n and time_dim.t_minute >= 30\n and ((household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or\n (household_demographics.hd_dep_count = 1 and household_demographics.hd_vehicle_count<=1+2) or\n (household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2))\n and store.s_store_name = 'ese') s7,\n (select count(*) h12_to_12_30\n from store_sales, household_demographics , time_dim, store\n where ss_sold_time_sk = time_dim.t_time_sk\n and ss_hdemo_sk = household_demographics.hd_demo_sk\n and ss_store_sk = s_store_sk\n and time_dim.t_hour = 12\n and time_dim.t_minute < 30\n and ((household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or\n (household_demographics.hd_dep_count = 1 and household_demographics.hd_vehicle_count<=1+2) or\n (household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2))\n and store.s_store_name = 'ese') s8\n;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q89.sql", "content": "-- CometBench-DS query 89 derived from TPC-DS query 89 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect *\nfrom(\nselect i_category, i_class, i_brand,\n s_store_name, s_company_name,\n d_moy,\n sum(ss_sales_price) sum_sales,\n avg(sum(ss_sales_price)) over\n (partition by i_category, i_brand, s_store_name, s_company_name)\n avg_monthly_sales\nfrom item, store_sales, date_dim, store\nwhere ss_item_sk = i_item_sk and\n ss_sold_date_sk = d_date_sk and\n ss_store_sk = s_store_sk and\n d_year in (2001) and\n ((i_category in ('Children','Jewelry','Home') and\n i_class in ('infants','birdal','flatware')\n )\n or (i_category in ('Electronics','Music','Books') and\n i_class in ('audio','classical','science') \n ))\ngroup by i_category, i_class, i_brand,\n s_store_name, s_company_name, d_moy) tmp1\nwhere case when (avg_monthly_sales <> 0) then (abs(sum_sales - avg_monthly_sales) / avg_monthly_sales) else null end > 0.1\norder by sum_sales - avg_monthly_sales, s_store_name\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q9.sql", "content": "-- CometBench-DS query 9 derived from TPC-DS query 9 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect case when (select count(*) \n from store_sales \n where ss_quantity between 1 and 20) > 31002\n then (select avg(ss_ext_discount_amt) \n from store_sales \n where ss_quantity between 1 and 20) \n else (select avg(ss_net_profit)\n from store_sales\n where ss_quantity between 1 and 20) end bucket1 ,\n case when (select count(*)\n from store_sales\n where ss_quantity between 21 and 40) > 588\n then (select avg(ss_ext_discount_amt)\n from store_sales\n where ss_quantity between 21 and 40) \n else (select avg(ss_net_profit)\n from store_sales\n where ss_quantity between 21 and 40) end bucket2,\n case when (select count(*)\n from store_sales\n where ss_quantity between 41 and 60) > 2456\n then (select avg(ss_ext_discount_amt)\n from store_sales\n where ss_quantity between 41 and 60)\n else (select avg(ss_net_profit)\n from store_sales\n where ss_quantity between 41 and 60) end bucket3,\n case when (select count(*)\n from store_sales\n where ss_quantity between 61 and 80) > 21645\n then (select avg(ss_ext_discount_amt)\n from store_sales\n where ss_quantity between 61 and 80)\n else (select avg(ss_net_profit)\n from store_sales\n where ss_quantity between 61 and 80) end bucket4,\n case when (select count(*)\n from store_sales\n where ss_quantity between 81 and 100) > 20553\n then (select avg(ss_ext_discount_amt)\n from store_sales\n where ss_quantity between 81 and 100)\n else (select avg(ss_net_profit)\n from store_sales\n where ss_quantity between 81 and 100) end bucket5\nfrom reason\nwhere r_reason_sk = 1\n;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q90.sql", "content": "-- CometBench-DS query 90 derived from TPC-DS query 90 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect cast(amc as decimal(15,4))/cast(pmc as decimal(15,4)) am_pm_ratio\n from ( select count(*) amc\n from web_sales, household_demographics , time_dim, web_page\n where ws_sold_time_sk = time_dim.t_time_sk\n and ws_ship_hdemo_sk = household_demographics.hd_demo_sk\n and ws_web_page_sk = web_page.wp_web_page_sk\n and time_dim.t_hour between 9 and 9+1\n and household_demographics.hd_dep_count = 2\n and web_page.wp_char_count between 5000 and 5200) at,\n ( select count(*) pmc\n from web_sales, household_demographics , time_dim, web_page\n where ws_sold_time_sk = time_dim.t_time_sk\n and ws_ship_hdemo_sk = household_demographics.hd_demo_sk\n and ws_web_page_sk = web_page.wp_web_page_sk\n and time_dim.t_hour between 15 and 15+1\n and household_demographics.hd_dep_count = 2\n and web_page.wp_char_count between 5000 and 5200) pt\n order by am_pm_ratio\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q91.sql", "content": "-- CometBench-DS query 91 derived from TPC-DS query 91 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect \n cc_call_center_id Call_Center,\n cc_name Call_Center_Name,\n cc_manager Manager,\n sum(cr_net_loss) Returns_Loss\nfrom\n call_center,\n catalog_returns,\n date_dim,\n customer,\n customer_address,\n customer_demographics,\n household_demographics\nwhere\n cr_call_center_sk = cc_call_center_sk\nand cr_returned_date_sk = d_date_sk\nand cr_returning_customer_sk= c_customer_sk\nand cd_demo_sk = c_current_cdemo_sk\nand hd_demo_sk = c_current_hdemo_sk\nand ca_address_sk = c_current_addr_sk\nand d_year = 2002 \nand d_moy = 11\nand ( (cd_marital_status = 'M' and cd_education_status = 'Unknown')\n or(cd_marital_status = 'W' and cd_education_status = 'Advanced Degree'))\nand hd_buy_potential like 'Unknown%'\nand ca_gmt_offset = -6\ngroup by cc_call_center_id,cc_name,cc_manager,cd_marital_status,cd_education_status\norder by sum(cr_net_loss) desc;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q92.sql", "content": "-- CometBench-DS query 92 derived from TPC-DS query 92 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect \n sum(ws_ext_discount_amt) as `Excess Discount Amount` \nfrom \n web_sales \n ,item \n ,date_dim\nwhere\ni_manufact_id = 914\nand i_item_sk = ws_item_sk \nand d_date between '2001-01-25' and \n (cast('2001-01-25' as date) + INTERVAL '90 DAYS')\nand d_date_sk = ws_sold_date_sk \nand ws_ext_discount_amt \n > ( \n SELECT \n 1.3 * avg(ws_ext_discount_amt) \n FROM \n web_sales \n ,date_dim\n WHERE \n ws_item_sk = i_item_sk \n and d_date between '2001-01-25' and\n (cast('2001-01-25' as date) + INTERVAL '90 DAYS')\n and d_date_sk = ws_sold_date_sk \n ) \norder by sum(ws_ext_discount_amt)\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q93.sql", "content": "-- CometBench-DS query 93 derived from TPC-DS query 93 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect ss_customer_sk\n ,sum(act_sales) sumsales\n from (select ss_item_sk\n ,ss_ticket_number\n ,ss_customer_sk\n ,case when sr_return_quantity is not null then (ss_quantity-sr_return_quantity)*ss_sales_price\n else (ss_quantity*ss_sales_price) end act_sales\n from store_sales left outer join store_returns on (sr_item_sk = ss_item_sk\n and sr_ticket_number = ss_ticket_number)\n ,reason\n where sr_reason_sk = r_reason_sk\n and r_reason_desc = 'Did not get it on time') t\n group by ss_customer_sk\n order by sumsales, ss_customer_sk\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q94.sql", "content": "-- CometBench-DS query 94 derived from TPC-DS query 94 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect \n count(distinct ws_order_number) as `order count`\n ,sum(ws_ext_ship_cost) as `total shipping cost`\n ,sum(ws_net_profit) as `total net profit`\nfrom\n web_sales ws1\n ,date_dim\n ,customer_address\n ,web_site\nwhere\n d_date between '1999-4-01' and \n (cast('1999-4-01' as date) + INTERVAL '60 DAYS')\nand ws1.ws_ship_date_sk = d_date_sk\nand ws1.ws_ship_addr_sk = ca_address_sk\nand ca_state = 'WI'\nand ws1.ws_web_site_sk = web_site_sk\nand web_company_name = 'pri'\nand exists (select *\n from web_sales ws2\n where ws1.ws_order_number = ws2.ws_order_number\n and ws1.ws_warehouse_sk <> ws2.ws_warehouse_sk)\nand not exists(select *\n from web_returns wr1\n where ws1.ws_order_number = wr1.wr_order_number)\norder by count(distinct ws_order_number)\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q95.sql", "content": "-- CometBench-DS query 95 derived from TPC-DS query 95 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith ws_wh as\n(select ws1.ws_order_number,ws1.ws_warehouse_sk wh1,ws2.ws_warehouse_sk wh2\n from web_sales ws1,web_sales ws2\n where ws1.ws_order_number = ws2.ws_order_number\n and ws1.ws_warehouse_sk <> ws2.ws_warehouse_sk)\n select \n count(distinct ws_order_number) as `order count`\n ,sum(ws_ext_ship_cost) as `total shipping cost`\n ,sum(ws_net_profit) as `total net profit`\nfrom\n web_sales ws1\n ,date_dim\n ,customer_address\n ,web_site\nwhere\n d_date between '2002-5-01' and \n (cast('2002-5-01' as date) + INTERVAL '60 DAYS')\nand ws1.ws_ship_date_sk = d_date_sk\nand ws1.ws_ship_addr_sk = ca_address_sk\nand ca_state = 'MA'\nand ws1.ws_web_site_sk = web_site_sk\nand web_company_name = 'pri'\nand ws1.ws_order_number in (select ws_order_number\n from ws_wh)\nand ws1.ws_order_number in (select wr_order_number\n from web_returns,ws_wh\n where wr_order_number = ws_wh.ws_order_number)\norder by count(distinct ws_order_number)\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q96.sql", "content": "-- CometBench-DS query 96 derived from TPC-DS query 96 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect count(*) \nfrom store_sales\n ,household_demographics \n ,time_dim, store\nwhere ss_sold_time_sk = time_dim.t_time_sk \n and ss_hdemo_sk = household_demographics.hd_demo_sk \n and ss_store_sk = s_store_sk\n and time_dim.t_hour = 8\n and time_dim.t_minute >= 30\n and household_demographics.hd_dep_count = 5\n and store.s_store_name = 'ese'\norder by count(*)\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q97.sql", "content": "-- CometBench-DS query 97 derived from TPC-DS query 97 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nwith ssci as (\nselect ss_customer_sk customer_sk\n ,ss_item_sk item_sk\nfrom store_sales,date_dim\nwhere ss_sold_date_sk = d_date_sk\n and d_month_seq between 1211 and 1211 + 11\ngroup by ss_customer_sk\n ,ss_item_sk),\ncsci as(\n select cs_bill_customer_sk customer_sk\n ,cs_item_sk item_sk\nfrom catalog_sales,date_dim\nwhere cs_sold_date_sk = d_date_sk\n and d_month_seq between 1211 and 1211 + 11\ngroup by cs_bill_customer_sk\n ,cs_item_sk)\n select sum(case when ssci.customer_sk is not null and csci.customer_sk is null then 1 else 0 end) store_only\n ,sum(case when ssci.customer_sk is null and csci.customer_sk is not null then 1 else 0 end) catalog_only\n ,sum(case when ssci.customer_sk is not null and csci.customer_sk is not null then 1 else 0 end) store_and_catalog\nfrom ssci full outer join csci on (ssci.customer_sk=csci.customer_sk\n and ssci.item_sk = csci.item_sk)\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q98.sql", "content": "-- CometBench-DS query 98 derived from TPC-DS query 98 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect i_item_id\n ,i_item_desc \n ,i_category \n ,i_class \n ,i_current_price\n ,sum(ss_ext_sales_price) as itemrevenue \n ,sum(ss_ext_sales_price)*100/sum(sum(ss_ext_sales_price)) over\n (partition by i_class) as revenueratio\nfrom\t\n\tstore_sales\n \t,item \n \t,date_dim\nwhere \n\tss_item_sk = i_item_sk \n \tand i_category in ('Shoes', 'Music', 'Men')\n \tand ss_sold_date_sk = d_date_sk\n\tand d_date between cast('2000-01-05' as date) \n\t\t\t\tand (cast('2000-01-05' as date) + INTERVAL '30 DAYS')\ngroup by \n\ti_item_id\n ,i_item_desc \n ,i_category\n ,i_class\n ,i_current_price\norder by \n\ti_category\n ,i_class\n ,i_item_id\n ,i_item_desc\n ,revenueratio;\n\n" }, { "path": "benchmarks/tpc/queries/tpcds/q99.sql", "content": "-- CometBench-DS query 99 derived from TPC-DS query 99 under the terms of the TPC Fair Use Policy.\n-- TPC-DS queries are Copyright 2021 Transaction Processing Performance Council.\n-- This query was generated at scale factor 1.\nselect \n substr(w_warehouse_name,1,20)\n ,sm_type\n ,cc_name\n ,sum(case when (cs_ship_date_sk - cs_sold_date_sk <= 30 ) then 1 else 0 end) as `30 days` \n ,sum(case when (cs_ship_date_sk - cs_sold_date_sk > 30) and \n (cs_ship_date_sk - cs_sold_date_sk <= 60) then 1 else 0 end ) as `31-60 days` \n ,sum(case when (cs_ship_date_sk - cs_sold_date_sk > 60) and \n (cs_ship_date_sk - cs_sold_date_sk <= 90) then 1 else 0 end) as `61-90 days` \n ,sum(case when (cs_ship_date_sk - cs_sold_date_sk > 90) and\n (cs_ship_date_sk - cs_sold_date_sk <= 120) then 1 else 0 end) as `91-120 days` \n ,sum(case when (cs_ship_date_sk - cs_sold_date_sk > 120) then 1 else 0 end) as `>120 days` \nfrom\n catalog_sales\n ,warehouse\n ,ship_mode\n ,call_center\n ,date_dim\nwhere\n d_month_seq between 1188 and 1188 + 11\nand cs_ship_date_sk = d_date_sk\nand cs_warehouse_sk = w_warehouse_sk\nand cs_ship_mode_sk = sm_ship_mode_sk\nand cs_call_center_sk = cc_call_center_sk\ngroup by\n substr(w_warehouse_name,1,20)\n ,sm_type\n ,cc_name\norder by substr(w_warehouse_name,1,20)\n ,sm_type\n ,cc_name\n LIMIT 100;\n\n" }, { "path": "benchmarks/tpc/queries/tpch/q1.sql", "content": "-- CometBench-H query 1 derived from TPC-H query 1 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\nselect\n\tl_returnflag,\n\tl_linestatus,\n\tsum(l_quantity) as sum_qty,\n\tsum(l_extendedprice) as sum_base_price,\n\tsum(l_extendedprice * (1 - l_discount)) as sum_disc_price,\n\tsum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) as sum_charge,\n\tavg(l_quantity) as avg_qty,\n\tavg(l_extendedprice) as avg_price,\n\tavg(l_discount) as avg_disc,\n\tcount(*) as count_order\nfrom\n\tlineitem\nwhere\n\tl_shipdate <= date '1998-12-01' - interval '68 days'\ngroup by\n\tl_returnflag,\n\tl_linestatus\norder by\n\tl_returnflag,\n\tl_linestatus;\n" }, { "path": "benchmarks/tpc/queries/tpch/q10.sql", "content": "-- CometBench-H query 10 derived from TPC-H query 10 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\nselect\n\tc_custkey,\n\tc_name,\n\tsum(l_extendedprice * (1 - l_discount)) as revenue,\n\tc_acctbal,\n\tn_name,\n\tc_address,\n\tc_phone,\n\tc_comment\nfrom\n\tcustomer,\n\torders,\n\tlineitem,\n\tnation\nwhere\n\tc_custkey = o_custkey\n\tand l_orderkey = o_orderkey\n\tand o_orderdate >= date '1993-07-01'\n\tand o_orderdate < date '1993-07-01' + interval '3' month\n\tand l_returnflag = 'R'\n\tand c_nationkey = n_nationkey\ngroup by\n\tc_custkey,\n\tc_name,\n\tc_acctbal,\n\tc_phone,\n\tn_name,\n\tc_address,\n\tc_comment\norder by\n\trevenue desc limit 20;\n" }, { "path": "benchmarks/tpc/queries/tpch/q11.sql", "content": "-- CometBench-H query 11 derived from TPC-H query 11 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\nselect\n\tps_partkey,\n\tsum(ps_supplycost * ps_availqty) as value\nfrom\n\tpartsupp,\n\tsupplier,\n\tnation\nwhere\n\tps_suppkey = s_suppkey\n\tand s_nationkey = n_nationkey\n\tand n_name = 'ALGERIA'\ngroup by\n\tps_partkey having\n\t\tsum(ps_supplycost * ps_availqty) > (\n\t\t\tselect\n\t\t\t\tsum(ps_supplycost * ps_availqty) * 0.0001000000\n\t\t\tfrom\n\t\t\t\tpartsupp,\n\t\t\t\tsupplier,\n\t\t\t\tnation\n\t\t\twhere\n\t\t\t\tps_suppkey = s_suppkey\n\t\t\t\tand s_nationkey = n_nationkey\n\t\t\t\tand n_name = 'ALGERIA'\n\t\t)\norder by\n\tvalue desc;\n" }, { "path": "benchmarks/tpc/queries/tpch/q12.sql", "content": "-- CometBench-H query 12 derived from TPC-H query 12 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\nselect\n\tl_shipmode,\n\tsum(case\n\t\twhen o_orderpriority = '1-URGENT'\n\t\t\tor o_orderpriority = '2-HIGH'\n\t\t\tthen 1\n\t\telse 0\n\tend) as high_line_count,\n\tsum(case\n\t\twhen o_orderpriority <> '1-URGENT'\n\t\t\tand o_orderpriority <> '2-HIGH'\n\t\t\tthen 1\n\t\telse 0\n\tend) as low_line_count\nfrom\n\torders,\n\tlineitem\nwhere\n\to_orderkey = l_orderkey\n\tand l_shipmode in ('FOB', 'SHIP')\n\tand l_commitdate < l_receiptdate\n\tand l_shipdate < l_commitdate\n\tand l_receiptdate >= date '1995-01-01'\n\tand l_receiptdate < date '1995-01-01' + interval '1' year\ngroup by\n\tl_shipmode\norder by\n\tl_shipmode;\n" }, { "path": "benchmarks/tpc/queries/tpch/q13.sql", "content": "-- CometBench-H query 13 derived from TPC-H query 13 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\nselect\n\tc_count,\n\tcount(*) as custdist\nfrom\n\t(\n\t\tselect\n\t\t\tc_custkey,\n\t\t\tcount(o_orderkey)\n\t\tfrom\n\t\t\tcustomer left outer join orders on\n\t\t\t\tc_custkey = o_custkey\n\t\t\t\tand o_comment not like '%express%requests%'\n\t\tgroup by\n\t\t\tc_custkey\n\t) as c_orders (c_custkey, c_count)\ngroup by\n\tc_count\norder by\n\tcustdist desc,\n\tc_count desc;\n" }, { "path": "benchmarks/tpc/queries/tpch/q14.sql", "content": "-- CometBench-H query 14 derived from TPC-H query 14 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\nselect\n\t100.00 * sum(case\n\t\twhen p_type like 'PROMO%'\n\t\t\tthen l_extendedprice * (1 - l_discount)\n\t\telse 0\n\tend) / sum(l_extendedprice * (1 - l_discount)) as promo_revenue\nfrom\n\tlineitem,\n\tpart\nwhere\n\tl_partkey = p_partkey\n\tand l_shipdate >= date '1995-02-01'\n\tand l_shipdate < date '1995-02-01' + interval '1' month;\n" }, { "path": "benchmarks/tpc/queries/tpch/q15.sql", "content": "-- CometBench-H query 15 derived from TPC-H query 15 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\ncreate view revenue0 (supplier_no, total_revenue) as\n\tselect\n\t\tl_suppkey,\n\t\tsum(l_extendedprice * (1 - l_discount))\n\tfrom\n\t\tlineitem\n\twhere\n\t\tl_shipdate >= date '1996-08-01'\n\t\tand l_shipdate < date '1996-08-01' + interval '3' month\n\tgroup by\n\t\tl_suppkey;\nselect\n\ts_suppkey,\n\ts_name,\n\ts_address,\n\ts_phone,\n\ttotal_revenue\nfrom\n\tsupplier,\n\trevenue0\nwhere\n\ts_suppkey = supplier_no\n\tand total_revenue = (\n\t\tselect\n\t\t\tmax(total_revenue)\n\t\tfrom\n\t\t\trevenue0\n\t)\norder by\n\ts_suppkey;\ndrop view revenue0;\n" }, { "path": "benchmarks/tpc/queries/tpch/q16.sql", "content": "-- CometBench-H query 16 derived from TPC-H query 16 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\nselect\n\tp_brand,\n\tp_type,\n\tp_size,\n\tcount(distinct ps_suppkey) as supplier_cnt\nfrom\n\tpartsupp,\n\tpart\nwhere\n\tp_partkey = ps_partkey\n\tand p_brand <> 'Brand#14'\n\tand p_type not like 'SMALL PLATED%'\n\tand p_size in (14, 6, 5, 31, 49, 15, 41, 47)\n\tand ps_suppkey not in (\n\t\tselect\n\t\t\ts_suppkey\n\t\tfrom\n\t\t\tsupplier\n\t\twhere\n\t\t\ts_comment like '%Customer%Complaints%'\n\t)\ngroup by\n\tp_brand,\n\tp_type,\n\tp_size\norder by\n\tsupplier_cnt desc,\n\tp_brand,\n\tp_type,\n\tp_size;\n" }, { "path": "benchmarks/tpc/queries/tpch/q17.sql", "content": "-- CometBench-H query 17 derived from TPC-H query 17 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\nselect\n\tsum(l_extendedprice) / 7.0 as avg_yearly\nfrom\n\tlineitem,\n\tpart\nwhere\n\tp_partkey = l_partkey\n\tand p_brand = 'Brand#42'\n\tand p_container = 'LG BAG'\n\tand l_quantity < (\n\t\tselect\n\t\t\t0.2 * avg(l_quantity)\n\t\tfrom\n\t\t\tlineitem\n\t\twhere\n\t\t\tl_partkey = p_partkey\n\t);\n" }, { "path": "benchmarks/tpc/queries/tpch/q18.sql", "content": "-- CometBench-H query 18 derived from TPC-H query 18 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\nselect\n\tc_name,\n\tc_custkey,\n\to_orderkey,\n\to_orderdate,\n\to_totalprice,\n\tsum(l_quantity)\nfrom\n\tcustomer,\n\torders,\n\tlineitem\nwhere\n\to_orderkey in (\n\t\tselect\n\t\t\tl_orderkey\n\t\tfrom\n\t\t\tlineitem\n\t\tgroup by\n\t\t\tl_orderkey having\n\t\t\t\tsum(l_quantity) > 313\n\t)\n\tand c_custkey = o_custkey\n\tand o_orderkey = l_orderkey\ngroup by\n\tc_name,\n\tc_custkey,\n\to_orderkey,\n\to_orderdate,\n\to_totalprice\norder by\n\to_totalprice desc,\n\to_orderdate limit 100;\n" }, { "path": "benchmarks/tpc/queries/tpch/q19.sql", "content": "-- CometBench-H query 19 derived from TPC-H query 19 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\nselect\n\tsum(l_extendedprice* (1 - l_discount)) as revenue\nfrom\n\tlineitem,\n\tpart\nwhere\n\t(\n\t\tp_partkey = l_partkey\n\t\tand p_brand = 'Brand#21'\n\t\tand p_container in ('SM CASE', 'SM BOX', 'SM PACK', 'SM PKG')\n\t\tand l_quantity >= 8 and l_quantity <= 8 + 10\n\t\tand p_size between 1 and 5\n\t\tand l_shipmode in ('AIR', 'AIR REG')\n\t\tand l_shipinstruct = 'DELIVER IN PERSON'\n\t)\n\tor\n\t(\n\t\tp_partkey = l_partkey\n\t\tand p_brand = 'Brand#13'\n\t\tand p_container in ('MED BAG', 'MED BOX', 'MED PKG', 'MED PACK')\n\t\tand l_quantity >= 20 and l_quantity <= 20 + 10\n\t\tand p_size between 1 and 10\n\t\tand l_shipmode in ('AIR', 'AIR REG')\n\t\tand l_shipinstruct = 'DELIVER IN PERSON'\n\t)\n\tor\n\t(\n\t\tp_partkey = l_partkey\n\t\tand p_brand = 'Brand#52'\n\t\tand p_container in ('LG CASE', 'LG BOX', 'LG PACK', 'LG PKG')\n\t\tand l_quantity >= 30 and l_quantity <= 30 + 10\n\t\tand p_size between 1 and 15\n\t\tand l_shipmode in ('AIR', 'AIR REG')\n\t\tand l_shipinstruct = 'DELIVER IN PERSON'\n\t);\n" }, { "path": "benchmarks/tpc/queries/tpch/q2.sql", "content": "-- CometBench-H query 2 derived from TPC-H query 2 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\nselect\n\ts_acctbal,\n\ts_name,\n\tn_name,\n\tp_partkey,\n\tp_mfgr,\n\ts_address,\n\ts_phone,\n\ts_comment\nfrom\n\tpart,\n\tsupplier,\n\tpartsupp,\n\tnation,\n\tregion\nwhere\n\tp_partkey = ps_partkey\n\tand s_suppkey = ps_suppkey\n\tand p_size = 48\n\tand p_type like '%TIN'\n\tand s_nationkey = n_nationkey\n\tand n_regionkey = r_regionkey\n\tand r_name = 'ASIA'\n\tand ps_supplycost = (\n\t\tselect\n\t\t\tmin(ps_supplycost)\n\t\tfrom\n\t\t\tpartsupp,\n\t\t\tsupplier,\n\t\t\tnation,\n\t\t\tregion\n\t\twhere\n\t\t\tp_partkey = ps_partkey\n\t\t\tand s_suppkey = ps_suppkey\n\t\t\tand s_nationkey = n_nationkey\n\t\t\tand n_regionkey = r_regionkey\n\t\t\tand r_name = 'ASIA'\n\t)\norder by\n\ts_acctbal desc,\n\tn_name,\n\ts_name,\n\tp_partkey limit 100;\n" }, { "path": "benchmarks/tpc/queries/tpch/q20.sql", "content": "-- CometBench-H query 20 derived from TPC-H query 20 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\nselect\n\ts_name,\n\ts_address\nfrom\n\tsupplier,\n\tnation\nwhere\n\ts_suppkey in (\n\t\tselect\n\t\t\tps_suppkey\n\t\tfrom\n\t\t\tpartsupp\n\t\twhere\n\t\t\tps_partkey in (\n\t\t\t\tselect\n\t\t\t\t\tp_partkey\n\t\t\t\tfrom\n\t\t\t\t\tpart\n\t\t\t\twhere\n\t\t\t\t\tp_name like 'blanched%'\n\t\t\t)\n\t\t\tand ps_availqty > (\n\t\t\t\tselect\n\t\t\t\t\t0.5 * sum(l_quantity)\n\t\t\t\tfrom\n\t\t\t\t\tlineitem\n\t\t\t\twhere\n\t\t\t\t\tl_partkey = ps_partkey\n\t\t\t\t\tand l_suppkey = ps_suppkey\n\t\t\t\t\tand l_shipdate >= date '1993-01-01'\n\t\t\t\t\tand l_shipdate < date '1993-01-01' + interval '1' year\n\t\t\t)\n\t)\n\tand s_nationkey = n_nationkey\n\tand n_name = 'KENYA'\norder by\n\ts_name;\n" }, { "path": "benchmarks/tpc/queries/tpch/q21.sql", "content": "-- CometBench-H query 21 derived from TPC-H query 21 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\nselect\n\ts_name,\n\tcount(*) as numwait\nfrom\n\tsupplier,\n\tlineitem l1,\n\torders,\n\tnation\nwhere\n\ts_suppkey = l1.l_suppkey\n\tand o_orderkey = l1.l_orderkey\n\tand o_orderstatus = 'F'\n\tand l1.l_receiptdate > l1.l_commitdate\n\tand exists (\n\t\tselect\n\t\t\t*\n\t\tfrom\n\t\t\tlineitem l2\n\t\twhere\n\t\t\tl2.l_orderkey = l1.l_orderkey\n\t\t\tand l2.l_suppkey <> l1.l_suppkey\n\t)\n\tand not exists (\n\t\tselect\n\t\t\t*\n\t\tfrom\n\t\t\tlineitem l3\n\t\twhere\n\t\t\tl3.l_orderkey = l1.l_orderkey\n\t\t\tand l3.l_suppkey <> l1.l_suppkey\n\t\t\tand l3.l_receiptdate > l3.l_commitdate\n\t)\n\tand s_nationkey = n_nationkey\n\tand n_name = 'ARGENTINA'\ngroup by\n\ts_name\norder by\n\tnumwait desc,\n\ts_name limit 100;\n" }, { "path": "benchmarks/tpc/queries/tpch/q22.sql", "content": "-- CometBench-H query 22 derived from TPC-H query 22 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\nselect\n\tcntrycode,\n\tcount(*) as numcust,\n\tsum(c_acctbal) as totacctbal\nfrom\n\t(\n\t\tselect\n\t\t\tsubstring(c_phone from 1 for 2) as cntrycode,\n\t\t\tc_acctbal\n\t\tfrom\n\t\t\tcustomer\n\t\twhere\n\t\t\tsubstring(c_phone from 1 for 2) in\n\t\t\t\t('24', '34', '16', '30', '33', '14', '13')\n\t\t\tand c_acctbal > (\n\t\t\t\tselect\n\t\t\t\t\tavg(c_acctbal)\n\t\t\t\tfrom\n\t\t\t\t\tcustomer\n\t\t\t\twhere\n\t\t\t\t\tc_acctbal > 0.00\n\t\t\t\t\tand substring(c_phone from 1 for 2) in\n\t\t\t\t\t\t('24', '34', '16', '30', '33', '14', '13')\n\t\t\t)\n\t\t\tand not exists (\n\t\t\t\tselect\n\t\t\t\t\t*\n\t\t\t\tfrom\n\t\t\t\t\torders\n\t\t\t\twhere\n\t\t\t\t\to_custkey = c_custkey\n\t\t\t)\n\t) as custsale\ngroup by\n\tcntrycode\norder by\n\tcntrycode;\n" }, { "path": "benchmarks/tpc/queries/tpch/q3.sql", "content": "-- CometBench-H query 3 derived from TPC-H query 3 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\nselect\n\tl_orderkey,\n\tsum(l_extendedprice * (1 - l_discount)) as revenue,\n\to_orderdate,\n\to_shippriority\nfrom\n\tcustomer,\n\torders,\n\tlineitem\nwhere\n\tc_mktsegment = 'BUILDING'\n\tand c_custkey = o_custkey\n\tand l_orderkey = o_orderkey\n\tand o_orderdate < date '1995-03-15'\n\tand l_shipdate > date '1995-03-15'\ngroup by\n\tl_orderkey,\n\to_orderdate,\n\to_shippriority\norder by\n\trevenue desc,\n\to_orderdate limit 10;\n" }, { "path": "benchmarks/tpc/queries/tpch/q4.sql", "content": "-- CometBench-H query 4 derived from TPC-H query 4 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\nselect\n\to_orderpriority,\n\tcount(*) as order_count\nfrom\n\torders\nwhere\n\to_orderdate >= date '1995-04-01'\n\tand o_orderdate < date '1995-04-01' + interval '3' month\n\tand exists (\n\t\tselect\n\t\t\t*\n\t\tfrom\n\t\t\tlineitem\n\t\twhere\n\t\t\tl_orderkey = o_orderkey\n\t\t\tand l_commitdate < l_receiptdate\n\t)\ngroup by\n\to_orderpriority\norder by\n\to_orderpriority;\n" }, { "path": "benchmarks/tpc/queries/tpch/q5.sql", "content": "-- CometBench-H query 5 derived from TPC-H query 5 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\nselect\n\tn_name,\n\tsum(l_extendedprice * (1 - l_discount)) as revenue\nfrom\n\tcustomer,\n\torders,\n\tlineitem,\n\tsupplier,\n\tnation,\n\tregion\nwhere\n\tc_custkey = o_custkey\n\tand l_orderkey = o_orderkey\n\tand l_suppkey = s_suppkey\n\tand c_nationkey = s_nationkey\n\tand s_nationkey = n_nationkey\n\tand n_regionkey = r_regionkey\n\tand r_name = 'AFRICA'\n\tand o_orderdate >= date '1994-01-01'\n\tand o_orderdate < date '1994-01-01' + interval '1' year\ngroup by\n\tn_name\norder by\n\trevenue desc;\n" }, { "path": "benchmarks/tpc/queries/tpch/q6.sql", "content": "-- CometBench-H query 6 derived from TPC-H query 6 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\nselect\n\tsum(l_extendedprice * l_discount) as revenue\nfrom\n\tlineitem\nwhere\n\tl_shipdate >= date '1994-01-01'\n\tand l_shipdate < date '1994-01-01' + interval '1' year\n\tand l_discount between 0.04 - 0.01 and 0.04 + 0.01\n\tand l_quantity < 24;\n" }, { "path": "benchmarks/tpc/queries/tpch/q7.sql", "content": "-- CometBench-H query 7 derived from TPC-H query 7 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\nselect\n\tsupp_nation,\n\tcust_nation,\n\tl_year,\n\tsum(volume) as revenue\nfrom\n\t(\n\t\tselect\n\t\t\tn1.n_name as supp_nation,\n\t\t\tn2.n_name as cust_nation,\n\t\t\textract(year from l_shipdate) as l_year,\n\t\t\tl_extendedprice * (1 - l_discount) as volume\n\t\tfrom\n\t\t\tsupplier,\n\t\t\tlineitem,\n\t\t\torders,\n\t\t\tcustomer,\n\t\t\tnation n1,\n\t\t\tnation n2\n\t\twhere\n\t\t\ts_suppkey = l_suppkey\n\t\t\tand o_orderkey = l_orderkey\n\t\t\tand c_custkey = o_custkey\n\t\t\tand s_nationkey = n1.n_nationkey\n\t\t\tand c_nationkey = n2.n_nationkey\n\t\t\tand (\n\t\t\t\t(n1.n_name = 'GERMANY' and n2.n_name = 'IRAQ')\n\t\t\t\tor (n1.n_name = 'IRAQ' and n2.n_name = 'GERMANY')\n\t\t\t)\n\t\t\tand l_shipdate between date '1995-01-01' and date '1996-12-31'\n\t) as shipping\ngroup by\n\tsupp_nation,\n\tcust_nation,\n\tl_year\norder by\n\tsupp_nation,\n\tcust_nation,\n\tl_year;\n" }, { "path": "benchmarks/tpc/queries/tpch/q8.sql", "content": "-- CometBench-H query 8 derived from TPC-H query 8 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\nselect\n\to_year,\n\tsum(case\n\t\twhen nation = 'IRAQ' then volume\n\t\telse 0\n\tend) / sum(volume) as mkt_share\nfrom\n\t(\n\t\tselect\n\t\t\textract(year from o_orderdate) as o_year,\n\t\t\tl_extendedprice * (1 - l_discount) as volume,\n\t\t\tn2.n_name as nation\n\t\tfrom\n\t\t\tpart,\n\t\t\tsupplier,\n\t\t\tlineitem,\n\t\t\torders,\n\t\t\tcustomer,\n\t\t\tnation n1,\n\t\t\tnation n2,\n\t\t\tregion\n\t\twhere\n\t\t\tp_partkey = l_partkey\n\t\t\tand s_suppkey = l_suppkey\n\t\t\tand l_orderkey = o_orderkey\n\t\t\tand o_custkey = c_custkey\n\t\t\tand c_nationkey = n1.n_nationkey\n\t\t\tand n1.n_regionkey = r_regionkey\n\t\t\tand r_name = 'MIDDLE EAST'\n\t\t\tand s_nationkey = n2.n_nationkey\n\t\t\tand o_orderdate between date '1995-01-01' and date '1996-12-31'\n\t\t\tand p_type = 'LARGE PLATED STEEL'\n\t) as all_nations\ngroup by\n\to_year\norder by\n\to_year;\n" }, { "path": "benchmarks/tpc/queries/tpch/q9.sql", "content": "-- CometBench-H query 9 derived from TPC-H query 9 under the terms of the TPC Fair Use Policy.\n-- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council.\nselect\n\tnation,\n\to_year,\n\tsum(amount) as sum_profit\nfrom\n\t(\n\t\tselect\n\t\t\tn_name as nation,\n\t\t\textract(year from o_orderdate) as o_year,\n\t\t\tl_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity as amount\n\t\tfrom\n\t\t\tpart,\n\t\t\tsupplier,\n\t\t\tlineitem,\n\t\t\tpartsupp,\n\t\t\torders,\n\t\t\tnation\n\t\twhere\n\t\t\ts_suppkey = l_suppkey\n\t\t\tand ps_suppkey = l_suppkey\n\t\t\tand ps_partkey = l_partkey\n\t\t\tand p_partkey = l_partkey\n\t\t\tand o_orderkey = l_orderkey\n\t\t\tand s_nationkey = n_nationkey\n\t\t\tand p_name like '%moccasin%'\n\t) as profit\ngroup by\n\tnation,\n\to_year\norder by\n\tnation,\n\to_year desc;\n" }, { "path": "benchmarks/tpc/run.py", "content": "#!/usr/bin/env python3\n#\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n\"\"\"Consolidated TPC benchmark runner for Spark-based engines.\n\nUsage:\n python3 run.py --engine comet --benchmark tpch\n python3 run.py --engine comet --benchmark tpcds --iterations 3\n python3 run.py --engine comet-iceberg --benchmark tpch\n python3 run.py --engine comet --benchmark tpch --dry-run\n python3 run.py --engine spark --benchmark tpch --no-restart\n\"\"\"\n\nimport argparse\nimport os\nimport re\nimport subprocess\nimport sys\n\n# ---------------------------------------------------------------------------\n# TOML loading – prefer stdlib tomllib (3.11+), else minimal fallback\n# ---------------------------------------------------------------------------\n\ntry:\n import tomllib # Python 3.11+\n\n def load_toml(path):\n with open(path, \"rb\") as f:\n return tomllib.load(f)\n\nexcept ModuleNotFoundError:\n\n def _parse_toml(text):\n \"\"\"Minimal TOML parser supporting tables, quoted-key strings, plain\n strings, arrays of strings, booleans, and comments. Sufficient for\n the engine config files used by this runner.\"\"\"\n root = {}\n current = root\n for line in text.splitlines():\n line = line.strip()\n if not line or line.startswith(\"#\"):\n continue\n # Table header: [env.defaults] or [spark_conf]\n m = re.match(r\"^\\[([^\\]]+)\\]$\", line)\n if m:\n keys = m.group(1).split(\".\")\n current = root\n for k in keys:\n current = current.setdefault(k, {})\n continue\n # Key = value\n m = re.match(r'^(\"(?:[^\"\\\\]|\\\\.)*\"|[A-Za-z0-9_.]+)\\s*=\\s*(.+)$', line)\n if not m:\n continue\n raw_key, raw_val = m.group(1), m.group(2).strip()\n key = raw_key.strip('\"')\n val = _parse_value(raw_val)\n current[key] = val\n return root\n\n def _parse_value(raw):\n if raw == \"true\":\n return True\n if raw == \"false\":\n return False\n if raw.startswith('\"') and raw.endswith('\"'):\n return raw[1:-1]\n if raw.startswith(\"[\"):\n # Simple array of strings\n items = []\n for m in re.finditer(r'\"((?:[^\"\\\\]|\\\\.)*)\"', raw):\n items.append(m.group(1))\n return items\n if raw.startswith(\"{\"):\n # Inline table: { KEY = \"VAL\", ... }\n tbl = {}\n for m in re.finditer(r'([A-Za-z0-9_]+)\\s*=\\s*\"((?:[^\"\\\\]|\\\\.)*)\"', raw):\n tbl[m.group(1)] = m.group(2)\n return tbl\n return raw\n\n def load_toml(path):\n with open(path, \"r\") as f:\n return _parse_toml(f.read())\n\n\n# ---------------------------------------------------------------------------\n# Common Spark configuration (shared across all engines)\n# ---------------------------------------------------------------------------\n\nCOMMON_SPARK_CONF = {\n \"spark.driver.memory\": \"8G\",\n \"spark.executor.memory\": \"16g\",\n \"spark.memory.offHeap.enabled\": \"true\",\n \"spark.memory.offHeap.size\": \"16g\",\n \"spark.eventLog.enabled\": \"true\",\n \"spark.eventLog.dir\": os.environ.get(\"SPARK_EVENT_LOG_DIR\", \"/tmp/spark-events\"),\n \"spark.hadoop.fs.s3a.impl\": \"org.apache.hadoop.fs.s3a.S3AFileSystem\",\n \"spark.hadoop.fs.s3a.aws.credentials.provider\": \"com.amazonaws.auth.DefaultAWSCredentialsProviderChain\",\n}\n\n# ---------------------------------------------------------------------------\n# Benchmark profiles\n# ---------------------------------------------------------------------------\n\nBENCHMARK_PROFILES = {\n \"tpch\": {\n \"executor_instances\": \"2\",\n \"executor_cores\": \"8\",\n \"max_cores\": \"16\",\n \"data_env\": \"TPCH_DATA\",\n \"format\": \"parquet\",\n },\n \"tpcds\": {\n \"executor_instances\": \"2\",\n \"executor_cores\": \"8\",\n \"max_cores\": \"16\",\n \"data_env\": \"TPCDS_DATA\",\n \"format\": None, # omit --format for TPC-DS\n },\n}\n\n# ---------------------------------------------------------------------------\n# Helpers\n# ---------------------------------------------------------------------------\n\n\ndef resolve_env(value):\n \"\"\"Expand $VAR and ${VAR} references using os.environ.\"\"\"\n if not isinstance(value, str):\n return value\n return re.sub(\n r\"\\$\\{([^}]+)\\}|\\$([A-Za-z_][A-Za-z0-9_]*)\",\n lambda m: os.environ.get(m.group(1) or m.group(2), \"\"),\n value,\n )\n\n\ndef resolve_env_in_list(lst):\n return [resolve_env(v) for v in lst]\n\n\ndef load_engine_config(engine_name):\n \"\"\"Load and return the TOML config for the given engine.\"\"\"\n script_dir = os.path.dirname(os.path.abspath(__file__))\n config_path = os.path.join(script_dir, \"engines\", f\"{engine_name}.toml\")\n if not os.path.exists(config_path):\n available = sorted(\n f.removesuffix(\".toml\")\n for f in os.listdir(os.path.join(script_dir, \"engines\"))\n if f.endswith(\".toml\")\n )\n print(f\"Error: Unknown engine '{engine_name}'\", file=sys.stderr)\n print(f\"Available engines: {', '.join(available)}\", file=sys.stderr)\n sys.exit(1)\n return load_toml(config_path)\n\n\ndef apply_env_defaults(config):\n \"\"\"Set environment variable defaults from [env.defaults].\"\"\"\n defaults = config.get(\"env\", {}).get(\"defaults\", {})\n for key, val in defaults.items():\n if key not in os.environ:\n os.environ[key] = val\n\n\ndef apply_env_exports(config):\n \"\"\"Export environment variables from [env.exports].\"\"\"\n exports = config.get(\"env\", {}).get(\"exports\", {})\n for key, val in exports.items():\n os.environ[key] = val\n\n\ndef check_required_env(config):\n \"\"\"Validate that required environment variables are set.\"\"\"\n required = config.get(\"env\", {}).get(\"required\", [])\n missing = [v for v in required if not os.environ.get(v)]\n if missing:\n print(\n f\"Error: Required environment variable(s) not set: {', '.join(missing)}\",\n file=sys.stderr,\n )\n sys.exit(1)\n\n\ndef check_common_env():\n \"\"\"Validate SPARK_HOME and SPARK_MASTER are set.\"\"\"\n for var in (\"SPARK_HOME\", \"SPARK_MASTER\"):\n if not os.environ.get(var):\n print(f\"Error: {var} is not set\", file=sys.stderr)\n sys.exit(1)\n\n\ndef check_benchmark_env(config, benchmark):\n \"\"\"Validate benchmark-specific environment variables.\"\"\"\n profile = BENCHMARK_PROFILES[benchmark]\n use_iceberg = config.get(\"tpcbench_args\", {}).get(\"use_iceberg\", False)\n\n required = []\n if not use_iceberg:\n required.append(profile[\"data_env\"])\n\n missing = [v for v in required if not os.environ.get(v)]\n if missing:\n print(\n f\"Error: Required environment variable(s) not set for \"\n f\"{benchmark}: {', '.join(missing)}\",\n file=sys.stderr,\n )\n sys.exit(1)\n\n # Default ICEBERG_DATABASE to the benchmark name if not already set\n if use_iceberg and not os.environ.get(\"ICEBERG_DATABASE\"):\n os.environ[\"ICEBERG_DATABASE\"] = benchmark\n\n\ndef build_spark_submit_cmd(config, benchmark, args):\n \"\"\"Build the spark-submit command list.\"\"\"\n spark_home = os.environ[\"SPARK_HOME\"]\n spark_master = os.environ[\"SPARK_MASTER\"]\n profile = BENCHMARK_PROFILES[benchmark]\n\n cmd = [os.path.join(spark_home, \"bin\", \"spark-submit\")]\n cmd += [\"--master\", spark_master]\n\n # --jars\n jars = config.get(\"spark_submit\", {}).get(\"jars\", [])\n if jars:\n cmd += [\"--jars\", \",\".join(resolve_env_in_list(jars))]\n\n # --driver-class-path\n driver_cp = config.get(\"spark_submit\", {}).get(\"driver_class_path\", [])\n if driver_cp:\n cmd += [\"--driver-class-path\", \":\".join(resolve_env_in_list(driver_cp))]\n\n # Merge spark confs: common + benchmark profile + engine overrides\n conf = dict(COMMON_SPARK_CONF)\n conf[\"spark.executor.instances\"] = profile[\"executor_instances\"]\n conf[\"spark.executor.cores\"] = profile[\"executor_cores\"]\n conf[\"spark.cores.max\"] = profile[\"max_cores\"]\n\n engine_conf = config.get(\"spark_conf\", {})\n for key, val in engine_conf.items():\n if isinstance(val, bool):\n val = \"true\" if val else \"false\"\n conf[resolve_env(key)] = resolve_env(str(val))\n\n # JFR profiling: append to extraJavaOptions (preserving any existing values)\n if args.jfr:\n jfr_dir = args.jfr_dir\n driver_jfr = (\n f\"-XX:StartFlightRecording=disk=true,dumponexit=true,\"\n f\"filename={jfr_dir}/driver.jfr,settings=profile\"\n )\n executor_jfr = (\n f\"-XX:StartFlightRecording=disk=true,dumponexit=true,\"\n f\"filename={jfr_dir}/executor.jfr,settings=profile\"\n )\n for spark_key, jfr_opts in [\n (\"spark.driver.extraJavaOptions\", driver_jfr),\n (\"spark.executor.extraJavaOptions\", executor_jfr),\n ]:\n existing = conf.get(spark_key, \"\")\n conf[spark_key] = f\"{existing} {jfr_opts}\".strip()\n\n # async-profiler: attach as a Java agent via -agentpath\n if args.async_profiler:\n ap_home = os.environ.get(\"ASYNC_PROFILER_HOME\", \"\")\n if not ap_home:\n print(\n \"Error: ASYNC_PROFILER_HOME is not set. \"\n \"Set it to the async-profiler installation directory.\",\n file=sys.stderr,\n )\n sys.exit(1)\n lib_ext = \"dylib\" if sys.platform == \"darwin\" else \"so\"\n ap_lib = os.path.join(ap_home, \"lib\", f\"libasyncProfiler.{lib_ext}\")\n ap_dir = args.async_profiler_dir\n ap_event = args.async_profiler_event\n ap_fmt = args.async_profiler_format\n ext = {\"flamegraph\": \"html\", \"jfr\": \"jfr\", \"collapsed\": \"txt\", \"text\": \"txt\"}[ap_fmt]\n\n driver_ap = (\n f\"-agentpath:{ap_lib}=start,event={ap_event},\"\n f\"{ap_fmt},file={ap_dir}/driver.{ext}\"\n )\n executor_ap = (\n f\"-agentpath:{ap_lib}=start,event={ap_event},\"\n f\"{ap_fmt},file={ap_dir}/executor.{ext}\"\n )\n for spark_key, ap_opts in [\n (\"spark.driver.extraJavaOptions\", driver_ap),\n (\"spark.executor.extraJavaOptions\", executor_ap),\n ]:\n existing = conf.get(spark_key, \"\")\n conf[spark_key] = f\"{existing} {ap_opts}\".strip()\n\n for key, val in sorted(conf.items()):\n cmd += [\"--conf\", f\"{key}={val}\"]\n\n # tpcbench.py path\n cmd.append(\"tpcbench.py\")\n\n # tpcbench args\n engine_name = config.get(\"engine\", {}).get(\"name\", args.engine)\n cmd += [\"--name\", engine_name]\n cmd += [\"--benchmark\", benchmark]\n\n use_iceberg = config.get(\"tpcbench_args\", {}).get(\"use_iceberg\", False)\n if use_iceberg:\n cmd += [\"--catalog\", resolve_env(\"${ICEBERG_CATALOG}\")]\n cmd += [\"--database\", resolve_env(\"${ICEBERG_DATABASE}\")]\n else:\n data_var = profile[\"data_env\"]\n data_val = os.environ.get(data_var, \"\")\n cmd += [\"--data\", data_val]\n\n cmd += [\"--output\", args.output]\n cmd += [\"--iterations\", str(args.iterations)]\n\n if args.query is not None:\n cmd += [\"--query\", str(args.query)]\n\n if profile[\"format\"] and not use_iceberg:\n cmd += [\"--format\", profile[\"format\"]]\n\n return cmd\n\n\ndef restart_spark():\n \"\"\"Stop and start Spark master and worker.\"\"\"\n spark_home = os.environ[\"SPARK_HOME\"]\n sbin = os.path.join(spark_home, \"sbin\")\n spark_master = os.environ[\"SPARK_MASTER\"]\n\n # Stop (ignore errors)\n subprocess.run(\n [os.path.join(sbin, \"stop-master.sh\")],\n stdout=subprocess.DEVNULL,\n stderr=subprocess.DEVNULL,\n )\n subprocess.run(\n [os.path.join(sbin, \"stop-worker.sh\")],\n stdout=subprocess.DEVNULL,\n stderr=subprocess.DEVNULL,\n )\n\n # Start (check errors)\n r = subprocess.run([os.path.join(sbin, \"start-master.sh\")])\n if r.returncode != 0:\n print(\"Error: Failed to start Spark master\", file=sys.stderr)\n sys.exit(1)\n\n r = subprocess.run([os.path.join(sbin, \"start-worker.sh\"), spark_master])\n if r.returncode != 0:\n print(\"Error: Failed to start Spark worker\", file=sys.stderr)\n sys.exit(1)\n\n\ndef main():\n parser = argparse.ArgumentParser(\n description=\"Consolidated TPC benchmark runner for Spark-based engines.\"\n )\n parser.add_argument(\n \"--engine\",\n required=True,\n help=\"Engine name (matches a TOML file in engines/)\",\n )\n parser.add_argument(\n \"--benchmark\",\n required=True,\n choices=[\"tpch\", \"tpcds\"],\n help=\"Benchmark to run\",\n )\n parser.add_argument(\n \"--iterations\", type=int, default=1, help=\"Number of iterations (default: 1)\"\n )\n parser.add_argument(\n \"--output\", default=\".\", help=\"Output directory (default: .)\"\n )\n parser.add_argument(\n \"--query\", type=int, default=None, help=\"Run a single query number\"\n )\n parser.add_argument(\n \"--no-restart\",\n action=\"store_true\",\n help=\"Skip Spark master/worker restart\",\n )\n parser.add_argument(\n \"--dry-run\",\n action=\"store_true\",\n help=\"Print the spark-submit command without executing\",\n )\n parser.add_argument(\n \"--jfr\",\n action=\"store_true\",\n help=\"Enable Java Flight Recorder profiling for driver and executors\",\n )\n parser.add_argument(\n \"--jfr-dir\",\n default=\"/results/jfr\",\n help=\"Directory for JFR output files (default: /results/jfr)\",\n )\n parser.add_argument(\n \"--async-profiler\",\n action=\"store_true\",\n help=\"Enable async-profiler for driver and executors (profiles Java + native code)\",\n )\n parser.add_argument(\n \"--async-profiler-dir\",\n default=\"/results/async-profiler\",\n help=\"Directory for async-profiler output files (default: /results/async-profiler)\",\n )\n parser.add_argument(\n \"--async-profiler-event\",\n default=\"cpu\",\n help=\"async-profiler event type: cpu, wall, alloc, lock, etc. (default: cpu)\",\n )\n parser.add_argument(\n \"--async-profiler-format\",\n default=\"flamegraph\",\n choices=[\"flamegraph\", \"jfr\", \"collapsed\", \"text\"],\n help=\"async-profiler output format (default: flamegraph)\",\n )\n args = parser.parse_args()\n\n config = load_engine_config(args.engine)\n\n # Apply env defaults and exports before validation\n apply_env_defaults(config)\n apply_env_exports(config)\n\n check_common_env()\n check_required_env(config)\n check_benchmark_env(config, args.benchmark)\n\n # Restart Spark unless --no-restart or --dry-run\n if not args.no_restart and not args.dry_run:\n restart_spark()\n\n # Create profiling output directories (skip for dry-run)\n if not args.dry_run:\n if args.jfr:\n os.makedirs(args.jfr_dir, exist_ok=True)\n if args.async_profiler:\n os.makedirs(args.async_profiler_dir, exist_ok=True)\n\n cmd = build_spark_submit_cmd(config, args.benchmark, args)\n\n if args.dry_run:\n # Group paired arguments (e.g. --conf key=value) on one line\n parts = []\n i = 0\n while i < len(cmd):\n token = cmd[i]\n if token.startswith(\"--\") and i + 1 < len(cmd) and not cmd[i + 1].startswith(\"--\"):\n parts.append(f\"{token} {cmd[i + 1]}\")\n i += 2\n else:\n parts.append(token)\n i += 1\n print(\" \\\\\\n \".join(parts))\n else:\n r = subprocess.run(cmd)\n sys.exit(r.returncode)\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "benchmarks/tpc/tpcbench.py", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n\"\"\"\nTPC-H / TPC-DS benchmark runner.\n\nSupports two data sources:\n - Files: use --data with --format (parquet, csv, json) and optional --options\n - Iceberg tables: use --catalog and --database to specify the catalog location\n\"\"\"\n\nimport argparse\nfrom datetime import datetime\nimport hashlib\nimport json\nimport os\nfrom pyspark.sql import SparkSession\nimport time\nfrom typing import Dict\n\n\ndef dedup_columns(df):\n \"\"\"Rename duplicate column aliases: a, a, b, b -> a, a_1, b, b_1\"\"\"\n counts = {}\n new_cols = []\n for c in df.columns:\n if c not in counts:\n counts[c] = 0\n new_cols.append(c)\n else:\n counts[c] += 1\n new_cols.append(f\"{c}_{counts[c]}\")\n return df.toDF(*new_cols)\n\n\ndef result_hash(rows):\n \"\"\"Compute a deterministic MD5 hash from collected rows.\"\"\"\n sorted_rows = sorted(rows, key=lambda r: str(r))\n h = hashlib.md5()\n for row in sorted_rows:\n h.update(str(row).encode(\"utf-8\"))\n return h.hexdigest()\n\n\ndef main(\n benchmark: str,\n data_path: str,\n catalog: str,\n database: str,\n iterations: int,\n output: str,\n name: str,\n format: str,\n query_num: int = None,\n write_path: str = None,\n options: Dict[str, str] = None,\n):\n if options is None:\n options = {}\n\n query_path = os.path.join(\n os.path.dirname(os.path.abspath(__file__)), \"queries\", benchmark\n )\n\n spark = SparkSession.builder \\\n .appName(f\"{name} benchmark derived from {benchmark}\") \\\n .getOrCreate()\n\n # Define tables for each benchmark\n if benchmark == \"tpch\":\n num_queries = 22\n table_names = [\n \"customer\", \"lineitem\", \"nation\", \"orders\",\n \"part\", \"partsupp\", \"region\", \"supplier\"\n ]\n elif benchmark == \"tpcds\":\n num_queries = 99\n table_names = [\n \"call_center\", \"catalog_page\", \"catalog_returns\", \"catalog_sales\",\n \"customer\", \"customer_address\", \"customer_demographics\", \"date_dim\",\n \"time_dim\", \"household_demographics\", \"income_band\", \"inventory\",\n \"item\", \"promotion\", \"reason\", \"ship_mode\", \"store\", \"store_returns\",\n \"store_sales\", \"warehouse\", \"web_page\", \"web_returns\", \"web_sales\",\n \"web_site\"\n ]\n else:\n raise ValueError(f\"Invalid benchmark: {benchmark}\")\n\n # Register tables from either files or Iceberg catalog\n using_iceberg = catalog is not None\n for table in table_names:\n if using_iceberg:\n source = f\"{catalog}.{database}.{table}\"\n print(f\"Registering table {table} from {source}\")\n df = spark.table(source)\n else:\n # Support both \"customer/\" and \"customer.parquet/\" layouts\n source = f\"{data_path}/{table}.{format}\"\n if not os.path.exists(source):\n source = f\"{data_path}/{table}\"\n print(f\"Registering table {table} from {source}\")\n df = spark.read.format(format).options(**options).load(source)\n df.createOrReplaceTempView(table)\n\n conf_dict = {k: v for k, v in spark.sparkContext.getConf().getAll()}\n\n results = {\n 'engine': 'datafusion-comet',\n 'benchmark': benchmark,\n 'spark_conf': conf_dict,\n }\n if using_iceberg:\n results['catalog'] = catalog\n results['database'] = database\n else:\n results['data_path'] = data_path\n\n for iteration in range(iterations):\n print(f\"\\n{'='*60}\")\n print(f\"Starting iteration {iteration + 1} of {iterations}\")\n print(f\"{'='*60}\")\n iter_start_time = time.time()\n\n # Determine which queries to run\n if query_num is not None:\n if query_num < 1 or query_num > num_queries:\n raise ValueError(\n f\"Query number {query_num} out of range. \"\n f\"Valid: 1-{num_queries} for {benchmark}\"\n )\n queries_to_run = [query_num]\n else:\n queries_to_run = range(1, num_queries + 1)\n\n for query in queries_to_run:\n spark.sparkContext.setJobDescription(f\"{benchmark} q{query}\")\n\n path = f\"{query_path}/q{query}.sql\"\n print(f\"\\nRunning query {query} from {path}\")\n\n with open(path, \"r\") as f:\n text = f.read()\n queries = text.split(\";\")\n\n start_time = time.time()\n for sql in queries:\n sql = sql.strip().replace(\"create view\", \"create temp view\")\n if len(sql) > 0:\n print(f\"Executing: {sql[:100]}...\")\n df = spark.sql(sql)\n df.explain(\"formatted\")\n\n if write_path is not None:\n if len(df.columns) > 0:\n output_path = f\"{write_path}/q{query}\"\n deduped = dedup_columns(df)\n deduped.orderBy(*deduped.columns).coalesce(1).write.mode(\"overwrite\").parquet(output_path)\n print(f\"Results written to {output_path}\")\n else:\n rows = df.collect()\n row_count = len(rows)\n row_hash = result_hash(rows)\n print(f\"Query {query} returned {row_count} rows, hash={row_hash}\")\n\n end_time = time.time()\n elapsed = end_time - start_time\n print(f\"Query {query} took {elapsed:.2f} seconds\")\n\n query_result = results.setdefault(query, {\"durations\": []})\n query_result[\"durations\"].append(round(elapsed, 3))\n if \"row_count\" not in query_result and not write_path:\n query_result[\"row_count\"] = row_count\n query_result[\"result_hash\"] = row_hash\n\n iter_end_time = time.time()\n print(f\"\\nIteration {iteration + 1} took {iter_end_time - iter_start_time:.2f} seconds\")\n\n # Write results\n result_str = json.dumps(results, indent=4)\n current_time_millis = int(datetime.now().timestamp() * 1000)\n results_path = f\"{output}/{name}-{benchmark}-{current_time_millis}.json\"\n print(f\"\\nWriting results to {results_path}\")\n with open(results_path, \"w\") as f:\n f.write(result_str)\n\n spark.stop()\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(\n description=\"TPC-H/TPC-DS benchmark runner for files or Iceberg tables\"\n )\n parser.add_argument(\n \"--benchmark\", required=True,\n help=\"Benchmark to run (tpch or tpcds)\"\n )\n\n # Data source - mutually exclusive: either file path or Iceberg catalog\n source_group = parser.add_mutually_exclusive_group(required=True)\n source_group.add_argument(\n \"--data\",\n help=\"Path to data files\"\n )\n source_group.add_argument(\n \"--catalog\",\n help=\"Iceberg catalog name\"\n )\n\n # Options for file-based reading\n parser.add_argument(\n \"--format\", default=\"parquet\",\n help=\"Input file format: parquet, csv, json (only used with --data)\"\n )\n parser.add_argument(\n \"--options\", type=json.loads, default={},\n help='Spark reader options as JSON string, e.g., \\'{\"header\": \"true\"}\\' (only used with --data)'\n )\n\n # Options for Iceberg\n parser.add_argument(\n \"--database\", default=\"tpch\",\n help=\"Database containing TPC tables (only used with --catalog)\"\n )\n\n parser.add_argument(\n \"--iterations\", type=int, default=1,\n help=\"Number of iterations\"\n )\n parser.add_argument(\n \"--output\", required=True,\n help=\"Path to write results JSON\"\n )\n parser.add_argument(\n \"--name\", required=True,\n help=\"Prefix for result file\"\n )\n parser.add_argument(\n \"--query\", type=int,\n help=\"Specific query number (1-based). If omitted, run all.\"\n )\n parser.add_argument(\n \"--write\",\n help=\"Path to save query results as Parquet\"\n )\n args = parser.parse_args()\n\n main(\n args.benchmark,\n args.data,\n args.catalog,\n args.database,\n args.iterations,\n args.output,\n args.name,\n args.format,\n args.query,\n args.write,\n args.options,\n )\n" }, { "path": "common/pom.xml", "content": "\n\n\n\n\n\n 4.0.0\n \n org.apache.datafusion\n comet-parent-spark${spark.version.short}_${scala.binary.version}\n 0.15.0-SNAPSHOT\n ../pom.xml\n \n\n comet-common-spark${spark.version.short}_${scala.binary.version}\n comet-common\n\n \n \n false\n \n\n \n \n org.apache.spark\n spark-sql_${scala.binary.version}\n \n \n org.apache.parquet\n parquet-column\n \n \n org.apache.parquet\n parquet-hadoop\n \n \n org.apache.parquet\n parquet-format-structures\n \n \n org.apache.arrow\n arrow-vector\n \n \n org.apache.arrow\n arrow-memory-unsafe\n \n \n org.apache.arrow\n arrow-c-data\n \n \n org.scala-lang.modules\n scala-collection-compat_${scala.binary.version}\n \n \n junit\n junit\n test\n \n \n org.assertj\n assertj-core\n test\n \n \n\n \n \n \n io.github.git-commit-id\n git-commit-id-maven-plugin\n ${git-commit-id-maven-plugin.version}\n \n \n get-the-git-infos\n \n revision\n \n initialize\n \n \n \n true\n ${project.build.outputDirectory}/comet-git-info.properties\n full\n \n ^git.branch$\n ^git.build.*$\n ^git.commit.id.(abbrev|full)$\n ^git.remote.*$\n \n \n \n \n org.apache.maven.plugins\n maven-shade-plugin\n \n \n package\n \n shade\n \n \n true\n true\n false\n true\n \n \n \n org.apache.arrow:*\n \n \n \n \n *:*\n \n **/*.thrift\n git.properties\n log4j.properties\n log4j2.properties\n arrow-git.properties\n \n \n \n org.apache.arrow:arrow-vector\n \n \n codegen/**\n \n \n \n \n \n org.apache.arrow\n ${comet.shade.packageName}.arrow\n \n \n org/apache/arrow/c/jni/JniWrapper\n org/apache/arrow/c/jni/PrivateData\n org/apache/arrow/c/jni/CDataJniException\n \n org/apache/arrow/c/ArrayStreamExporter$ExportedArrayStreamPrivateData\n \n \n \n \n \n \n \n \n \n net.alchim31.maven\n scala-maven-plugin\n \n \n org.codehaus.mojo\n build-helper-maven-plugin\n \n \n add-shim-source\n generate-sources\n \n add-source\n \n \n \n src/main/${shims.majorVerSrc}\n src/main/${shims.minorVerSrc}\n \n \n \n \n \n \n \n \n ${project.basedir}/src/main/resources\n \n \n ${project.basedir}/../native/target/x86_64-apple-darwin/release\n \n libcomet.dylib\n \n org/apache/comet/darwin/x86_64\n \n \n ${project.basedir}/../native/target/aarch64-apple-darwin/release\n \n libcomet.dylib\n \n org/apache/comet/darwin/aarch64\n \n \n ${jni.dir}\n \n libcomet.dylib\n libcomet.so\n comet.dll\n \n org/apache/comet/${platform}/${arch}\n \n \n \n\n\n" }, { "path": "common/src/main/java/org/apache/arrow/c/AbstractCometSchemaImporter.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.arrow.c;\n\nimport org.apache.arrow.memory.BufferAllocator;\nimport org.apache.arrow.vector.FieldVector;\nimport org.apache.arrow.vector.types.pojo.Field;\n\nimport org.apache.comet.IcebergApi;\n\n/** This is a simple wrapper around SchemaImporter to make it accessible from Java Arrow. */\npublic abstract class AbstractCometSchemaImporter {\n private final BufferAllocator allocator;\n private final SchemaImporter importer;\n private final CDataDictionaryProvider provider = new CDataDictionaryProvider();\n\n public AbstractCometSchemaImporter(BufferAllocator allocator) {\n this.allocator = allocator;\n this.importer = new SchemaImporter(allocator);\n }\n\n public BufferAllocator getAllocator() {\n return allocator;\n }\n\n public CDataDictionaryProvider getProvider() {\n return provider;\n }\n\n public Field importField(ArrowSchema schema) {\n try {\n return importer.importField(schema, provider);\n } finally {\n schema.release();\n schema.close();\n }\n }\n\n /**\n * Imports data from ArrowArray/ArrowSchema into a FieldVector. This is basically the same as Java\n * Arrow `Data.importVector`. `Data.importVector` initiates `SchemaImporter` internally which is\n * used to fill dictionary ids for dictionary encoded vectors. Every call to `importVector` will\n * begin with dictionary ids starting from 0. So, separate calls to `importVector` will overwrite\n * dictionary ids. To avoid this, we need to use the same `SchemaImporter` instance for all calls\n * to `importVector`.\n */\n public FieldVector importVector(ArrowArray array, ArrowSchema schema) {\n Field field = importField(schema);\n FieldVector vector = field.createVector(allocator);\n Data.importIntoVector(allocator, array, vector, provider);\n\n return vector;\n }\n\n @IcebergApi\n public void close() {\n provider.close();\n }\n}\n" }, { "path": "common/src/main/java/org/apache/arrow/c/ArrowImporter.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.arrow.c;\n\nimport org.apache.arrow.memory.BufferAllocator;\nimport org.apache.arrow.vector.FieldVector;\nimport org.apache.arrow.vector.types.pojo.Field;\n\n/**\n * This class is used to import Arrow schema and array from native execution/shuffle. We cannot use\n * Arrow's Java API to import schema and array directly because Arrow's Java API `Data.importField`\n * initiates a new `SchemaImporter` for each field. Each `SchemaImporter` maintains an internal\n * dictionary id counter. So the dictionary ids for multiple dictionary columns will conflict with\n * each other and cause data corruption.\n */\npublic class ArrowImporter {\n private final SchemaImporter importer;\n private final BufferAllocator allocator;\n\n public ArrowImporter(BufferAllocator allocator) {\n this.allocator = allocator;\n this.importer = new SchemaImporter(allocator);\n }\n\n Field importField(ArrowSchema schema, CDataDictionaryProvider provider) {\n try {\n return importer.importField(schema, provider);\n } finally {\n schema.release();\n schema.close();\n }\n }\n\n public FieldVector importVector(\n ArrowArray array, ArrowSchema schema, CDataDictionaryProvider provider) {\n Field field = importField(schema, provider);\n FieldVector vector = field.createVector(allocator);\n ArrayImporter importer = new ArrayImporter(allocator, vector, provider);\n importer.importArray(array);\n return vector;\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/CometNativeException.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet;\n\n/** Parent class for all exceptions thrown from Comet native side. */\npublic class CometNativeException extends CometRuntimeException {\n public CometNativeException(String message) {\n super(message);\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/CometOutOfMemoryError.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet;\n\n/** OOM error specific for Comet memory management */\npublic class CometOutOfMemoryError extends OutOfMemoryError {\n public CometOutOfMemoryError(String msg) {\n super(msg);\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/CometRuntimeException.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet;\n\n/** The parent class for all Comet runtime exceptions */\npublic class CometRuntimeException extends RuntimeException {\n public CometRuntimeException(String message) {\n super(message);\n }\n\n public CometRuntimeException(String message, Throwable cause) {\n super(message, cause);\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/CometSchemaImporter.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet;\n\nimport org.apache.arrow.c.*;\nimport org.apache.arrow.memory.BufferAllocator;\n\n/** This is a simple wrapper around SchemaImporter to make it accessible from Java Arrow. */\n@IcebergApi\npublic class CometSchemaImporter extends AbstractCometSchemaImporter {\n @IcebergApi\n public CometSchemaImporter(BufferAllocator allocator) {\n super(allocator);\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/IcebergApi.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet;\n\nimport java.lang.annotation.Documented;\nimport java.lang.annotation.ElementType;\nimport java.lang.annotation.Retention;\nimport java.lang.annotation.RetentionPolicy;\nimport java.lang.annotation.Target;\n\n/**\n * Indicates that the annotated element is part of the public API used by Apache Iceberg.\n *\n *

This annotation marks classes, methods, constructors, and fields that form the contract\n * between Comet and Iceberg. Changes to these APIs may break Iceberg's Comet integration, so\n * contributors should exercise caution and consider backward compatibility when modifying annotated\n * elements.\n *\n *

The Iceberg integration uses Comet's native Parquet reader for accelerated vectorized reads.\n * See the contributor guide documentation for details on how Iceberg uses these APIs.\n *\n * @see Apache Iceberg\n */\n@Documented\n@Retention(RetentionPolicy.RUNTIME)\n@Target({ElementType.TYPE, ElementType.METHOD, ElementType.CONSTRUCTOR, ElementType.FIELD})\npublic @interface IcebergApi {}\n" }, { "path": "common/src/main/java/org/apache/comet/NativeBase.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet;\n\nimport java.io.BufferedReader;\nimport java.io.File;\nimport java.io.FilenameFilter;\nimport java.io.IOException;\nimport java.io.InputStream;\nimport java.io.InputStreamReader;\nimport java.nio.file.Files;\nimport java.nio.file.StandardCopyOption;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport org.apache.spark.sql.comet.util.Utils;\n\nimport static org.apache.comet.Constants.LOG_CONF_NAME;\nimport static org.apache.comet.Constants.LOG_CONF_PATH;\nimport static org.apache.comet.Constants.LOG_LEVEL_ENV;\n\n/** Base class for JNI bindings. MUST be inherited by all classes that introduce JNI APIs. */\npublic abstract class NativeBase {\n static final String ARROW_UNSAFE_MEMORY_ACCESS = \"arrow.enable_unsafe_memory_access\";\n static final String ARROW_NULL_CHECK_FOR_GET = \"arrow.enable_null_check_for_get\";\n\n private static final Logger LOG = LoggerFactory.getLogger(NativeBase.class);\n private static final String NATIVE_LIB_NAME = \"comet\";\n\n private static final String libraryToLoad = System.mapLibraryName(NATIVE_LIB_NAME);\n private static boolean loaded = false;\n private static volatile Throwable loadErr = null;\n private static final String searchPattern = \"libcomet-\";\n\n static {\n try {\n load();\n } catch (Throwable th) {\n LOG.warn(\"Failed to load comet library\", th);\n // logging may not be initialized yet, so also write to stderr\n System.err.println(\"Failed to load comet library: \" + th.getMessage());\n loadErr = th;\n }\n }\n\n public static synchronized boolean isLoaded() throws Throwable {\n if (loadErr != null) {\n throw loadErr;\n }\n return loaded;\n }\n\n // Only for testing\n static synchronized void setLoaded(boolean b) {\n loaded = b;\n }\n\n static synchronized void load() {\n if (loaded) {\n return;\n }\n\n cleanupOldTempLibs();\n\n // Check if the arch used by JDK is the same as arch on the host machine, in particular,\n // whether x86_64 JDK is used in arm64 Mac\n if (!checkArch()) {\n LOG.warn(\n \"Comet is disabled. JDK compiled for x86_64 is used in a Mac based on Apple Silicon. \"\n + \"In order to use Comet, Please install a JDK version for ARM64 architecture\");\n return;\n }\n\n // Try to load Comet library from the java.library.path.\n try {\n System.loadLibrary(NATIVE_LIB_NAME);\n loaded = true;\n } catch (UnsatisfiedLinkError ex) {\n // Doesn't exist, so proceed to loading bundled library.\n bundleLoadLibrary();\n }\n\n initWithLogConf();\n // Only set the Arrow properties when debugging mode is off\n if (!(boolean) CometConf.COMET_DEBUG_ENABLED().get()) {\n setArrowProperties();\n }\n }\n\n /**\n * Use the bundled native libraries. Functionally equivalent to System.loadLibrary.\n */\n private static void bundleLoadLibrary() {\n String resourceName = resourceName();\n InputStream is = NativeBase.class.getResourceAsStream(resourceName);\n if (is == null) {\n throw new UnsupportedOperationException(\n \"Unsupported OS/arch, cannot find \"\n + resourceName\n + \". Please try building from source.\");\n }\n\n File tempLib = null;\n File tempLibLock = null;\n try {\n // Create the .lck file first to avoid a race condition\n // with other concurrently running Java processes using Comet.\n tempLibLock = File.createTempFile(searchPattern, \".\" + os().libExtension + \".lck\");\n tempLib = new File(tempLibLock.getAbsolutePath().replaceFirst(\".lck$\", \"\"));\n // copy to tempLib\n Files.copy(is, tempLib.toPath(), StandardCopyOption.REPLACE_EXISTING);\n System.load(tempLib.getAbsolutePath());\n loaded = true;\n } catch (IOException e) {\n throw new IllegalStateException(\"Cannot unpack libcomet: \" + e);\n } finally {\n if (!loaded) {\n if (tempLib != null && tempLib.exists()) {\n if (!tempLib.delete()) {\n LOG.error(\n \"Cannot unpack libcomet / cannot delete a temporary native library \" + tempLib);\n }\n }\n if (tempLibLock != null && tempLibLock.exists()) {\n if (!tempLibLock.delete()) {\n LOG.error(\n \"Cannot unpack libcomet / cannot delete a temporary lock file \" + tempLibLock);\n }\n }\n } else {\n tempLib.deleteOnExit();\n tempLibLock.deleteOnExit();\n }\n }\n }\n\n private static void initWithLogConf() {\n String logConfPath = System.getProperty(LOG_CONF_PATH(), Utils.getConfPath(LOG_CONF_NAME()));\n String logLevel = System.getenv(LOG_LEVEL_ENV());\n\n // If both the system property and the environmental variable failed to find a log\n // configuration, then fall back to using the deployed default\n if (logConfPath == null) {\n LOG.info(\n \"Couldn't locate log file from either COMET_CONF_DIR or comet.log.file.path. \"\n + \"Using default log configuration with {} log level which emits to stderr\",\n logLevel == null ? \"INFO\" : logLevel);\n logConfPath = \"\";\n } else {\n // Ignore log level if a log configuration file is specified\n if (logLevel != null) {\n LOG.warn(\"Ignoring log level {} because a log configuration file is specified\", logLevel);\n }\n\n LOG.info(\"Using {} for native library logging\", logConfPath);\n }\n init(logConfPath, logLevel);\n }\n\n private static void cleanupOldTempLibs() {\n String tempFolder = new File(System.getProperty(\"java.io.tmpdir\")).getAbsolutePath();\n File dir = new File(tempFolder);\n\n File[] tempLibFiles =\n dir.listFiles(\n new FilenameFilter() {\n public boolean accept(File dir, String name) {\n return name.startsWith(searchPattern) && !name.endsWith(\".lck\");\n }\n });\n\n if (tempLibFiles != null) {\n for (File tempLibFile : tempLibFiles) {\n File lckFile = new File(tempLibFile.getAbsolutePath() + \".lck\");\n if (!lckFile.exists()) {\n try {\n tempLibFile.delete();\n } catch (SecurityException e) {\n LOG.error(\"Failed to delete old temp lib\", e);\n }\n }\n }\n }\n }\n\n // Set Arrow related properties upon initializing native, such as enabling unsafe memory access\n // as well as disabling null check for get, for performance reasons.\n private static void setArrowProperties() {\n setPropertyIfNull(ARROW_UNSAFE_MEMORY_ACCESS, \"true\");\n setPropertyIfNull(ARROW_NULL_CHECK_FOR_GET, \"false\");\n }\n\n private static void setPropertyIfNull(String key, String value) {\n if (System.getProperty(key) == null) {\n LOG.info(\"Setting system property {} to {}\", key, value);\n System.setProperty(key, value);\n } else {\n LOG.info(\n \"Skip setting system property {} to {}, because it is already set to {}\",\n key,\n value,\n System.getProperty(key));\n }\n }\n\n private enum OS {\n // Even on Windows, the default compiler from cpptasks (gcc) uses .so as a shared lib extension\n WINDOWS(\"win32\", \"so\"),\n LINUX(\"linux\", \"so\"),\n MAC(\"darwin\", \"dylib\"),\n SOLARIS(\"solaris\", \"so\");\n public final String name, libExtension;\n\n OS(String name, String libExtension) {\n this.name = name;\n this.libExtension = libExtension;\n }\n }\n\n private static String arch() {\n return System.getProperty(\"os.arch\");\n }\n\n private static OS os() {\n String osName = System.getProperty(\"os.name\");\n if (osName.contains(\"Linux\")) {\n return OS.LINUX;\n } else if (osName.contains(\"Mac\")) {\n return OS.MAC;\n } else if (osName.contains(\"Windows\")) {\n return OS.WINDOWS;\n } else if (osName.contains(\"Solaris\") || osName.contains(\"SunOS\")) {\n return OS.SOLARIS;\n } else {\n throw new UnsupportedOperationException(\"Unsupported operating system: \" + osName);\n }\n }\n\n // For some reason users will get JVM crash when running Comet that is compiled for `aarch64`\n // using a JVM that is compiled against `amd64`. Here we check if that is the case and fallback\n // to Spark accordingly.\n private static boolean checkArch() {\n if (os() == OS.MAC) {\n try {\n String javaArch = arch();\n Process process = Runtime.getRuntime().exec(\"uname -a\");\n if (process.waitFor() == 0) {\n BufferedReader in = new BufferedReader(new InputStreamReader(process.getInputStream()));\n String line;\n while ((line = in.readLine()) != null) {\n if (javaArch.equals(\"x86_64\") && line.contains(\"ARM64\")) {\n return false;\n }\n }\n }\n } catch (IOException | InterruptedException e) {\n LOG.warn(\"Error parsing host architecture\", e);\n }\n }\n\n return true;\n }\n\n private static String resourceName() {\n OS os = os();\n String packagePrefix = NativeBase.class.getPackage().getName().replace('.', '/');\n\n return \"/\" + packagePrefix + \"/\" + os.name + \"/\" + arch() + \"/\" + libraryToLoad;\n }\n\n /**\n * Initialize the native library through JNI.\n *\n * @param logConfPath location to the native log configuration file\n */\n static native void init(String logConfPath, String logLevel);\n\n /**\n * Check if a specific feature is enabled in the native library.\n *\n * @param featureName The name of the feature to check (e.g., \"hdfs\", \"jemalloc\", \"hdfs-opendal\")\n * @return true if the feature is enabled, false otherwise\n */\n public static native boolean isFeatureEnabled(String featureName);\n}\n" }, { "path": "common/src/main/java/org/apache/comet/ParquetRuntimeException.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet;\n\n/** The parent class for the subset of Comet runtime exceptions related to Parquet. */\npublic class ParquetRuntimeException extends CometRuntimeException {\n public ParquetRuntimeException(String message) {\n super(message);\n }\n\n public ParquetRuntimeException(String message, Throwable cause) {\n super(message, cause);\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/exceptions/CometQueryExecutionException.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.exceptions;\n\nimport org.apache.comet.CometNativeException;\n\n/**\n * Exception thrown from Comet native execution containing JSON-encoded error information. The\n * message contains a JSON object with the following structure:\n *\n * 

\n * {\n *   \"errorType\": \"DivideByZero\",\n *   \"errorClass\": \"DIVIDE_BY_ZERO\",\n *   \"params\": { ... },\n *   \"context\": { \"sqlText\": \"...\", \"startOffset\": 0, \"stopOffset\": 10 },\n *   \"hint\": \"Use `try_divide` to tolerate divisor being 0\"\n * }\n *
\n *\n * CometExecIterator parses this JSON and converts it to the appropriate Spark exception by calling\n * the corresponding QueryExecutionErrors.* method.\n */\npublic final class CometQueryExecutionException extends CometNativeException {\n\n /**\n * Creates a new CometQueryExecutionException with a JSON-encoded error message.\n *\n * @param jsonMessage JSON string containing error information\n */\n public CometQueryExecutionException(String jsonMessage) {\n super(jsonMessage);\n }\n\n /**\n * Returns true if the message appears to be JSON-formatted. This is used to distinguish between\n * JSON-encoded errors and legacy error messages.\n *\n * @return true if message starts with '{' and ends with '}'\n */\n public boolean isJsonMessage() {\n String msg = getMessage();\n if (msg == null) return false;\n String trimmed = msg.trim();\n return trimmed.startsWith(\"{\") && trimmed.endsWith(\"}\");\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/AbstractColumnReader.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport org.apache.parquet.column.ColumnDescriptor;\nimport org.apache.parquet.schema.Type;\nimport org.apache.spark.sql.types.DataType;\nimport org.apache.spark.sql.types.TimestampNTZType$;\n\nimport org.apache.comet.CometConf;\nimport org.apache.comet.IcebergApi;\nimport org.apache.comet.vector.CometVector;\n\n/** Base class for Comet Parquet column reader implementations. */\n@IcebergApi\npublic abstract class AbstractColumnReader implements AutoCloseable {\n protected static final Logger LOG = LoggerFactory.getLogger(AbstractColumnReader.class);\n\n /** The Spark data type. */\n protected final DataType type;\n\n /** The Spark data type. */\n protected final Type fieldType;\n\n /** Parquet column descriptor. */\n protected final ColumnDescriptor descriptor;\n\n /**\n * Whether to always return 128 bit decimals, regardless of its precision. If false, this will\n * return 32, 64 or 128 bit decimals depending on the precision.\n */\n protected final boolean useDecimal128;\n\n /**\n * Whether to return dates/timestamps that were written with legacy hybrid (Julian + Gregorian)\n * calendar as it is. If this is true, Comet will return them as it is, instead of rebasing them\n * to the new Proleptic Gregorian calendar. If this is false, Comet will throw exceptions when\n * seeing these dates/timestamps.\n */\n protected final boolean useLegacyDateTimestamp;\n\n /** The size of one batch, gets updated by 'readBatch' */\n protected int batchSize;\n\n /** A pointer to the native implementation of ColumnReader. */\n @IcebergApi protected long nativeHandle;\n\n AbstractColumnReader(\n DataType type,\n Type fieldType,\n ColumnDescriptor descriptor,\n boolean useDecimal128,\n boolean useLegacyDateTimestamp) {\n this.type = type;\n this.fieldType = fieldType;\n this.descriptor = descriptor;\n this.useDecimal128 = useDecimal128;\n this.useLegacyDateTimestamp = useLegacyDateTimestamp;\n }\n\n AbstractColumnReader(\n DataType type,\n ColumnDescriptor descriptor,\n boolean useDecimal128,\n boolean useLegacyDateTimestamp) {\n this(type, null, descriptor, useDecimal128, useLegacyDateTimestamp);\n TypeUtil.checkParquetType(descriptor, type);\n }\n\n ColumnDescriptor getDescriptor() {\n return descriptor;\n }\n\n String getPath() {\n return String.join(\".\", this.descriptor.getPath());\n }\n\n /**\n * Set the batch size of this reader to be 'batchSize'. Also initializes the native column reader.\n */\n @IcebergApi\n public void setBatchSize(int batchSize) {\n assert nativeHandle == 0\n : \"Native column reader shouldn't be initialized before \" + \"'setBatchSize' is called\";\n this.batchSize = batchSize;\n initNative();\n }\n\n /**\n * Reads a batch of 'total' new rows.\n *\n * @param total the total number of rows to read\n */\n public abstract void readBatch(int total);\n\n /** Returns the {@link CometVector} read by this reader. */\n public abstract CometVector currentBatch();\n\n @IcebergApi\n @Override\n public void close() {\n if (nativeHandle != 0) {\n LOG.debug(\"Closing the column reader\");\n Native.closeColumnReader(nativeHandle);\n nativeHandle = 0;\n }\n }\n\n protected void initNative() {\n LOG.debug(\"initializing the native column reader\");\n DataType readType = (boolean) CometConf.COMET_SCHEMA_EVOLUTION_ENABLED().get() ? type : null;\n boolean useLegacyDateTimestampOrNTZ =\n useLegacyDateTimestamp || type == TimestampNTZType$.MODULE$;\n nativeHandle =\n Utils.initColumnReader(\n descriptor, readType, batchSize, useDecimal128, useLegacyDateTimestampOrNTZ);\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/ArrowConstantColumnReader.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.math.BigDecimal;\n\nimport org.apache.arrow.memory.BufferAllocator;\nimport org.apache.arrow.memory.RootAllocator;\nimport org.apache.arrow.vector.*;\nimport org.apache.spark.sql.catalyst.InternalRow;\nimport org.apache.spark.sql.catalyst.util.ResolveDefaultColumns;\nimport org.apache.spark.sql.types.*;\nimport org.apache.spark.unsafe.types.UTF8String;\n\nimport org.apache.comet.vector.CometPlainVector;\nimport org.apache.comet.vector.CometVector;\n\n/**\n * A column reader that returns constant vectors using Arrow Java vectors directly (no native\n * mutable buffers). Used for partition columns and missing columns in the native_iceberg_compat\n * scan path.\n *\n *

The vector is filled with the constant value repeated for every row in the batch. This is\n * necessary because the underlying Arrow vector's buffers must be large enough to match the\n * reported value count — otherwise variable-width types (strings, binary) would have undersized\n * offset buffers, causing out-of-bounds reads on the native side.\n */\npublic class ArrowConstantColumnReader extends AbstractColumnReader {\n private final BufferAllocator allocator = new RootAllocator();\n\n private boolean isNull;\n private Object value;\n private FieldVector fieldVector;\n private CometPlainVector vector;\n private int currentSize;\n\n /** Constructor for missing columns (default values from schema). */\n ArrowConstantColumnReader(StructField field, int batchSize, boolean useDecimal128) {\n super(field.dataType(), TypeUtil.convertToParquet(field), useDecimal128, false);\n this.batchSize = batchSize;\n this.value =\n ResolveDefaultColumns.getExistenceDefaultValues(new StructType(new StructField[] {field}))[\n 0];\n initVector(value, batchSize);\n }\n\n /** Constructor for partition columns with values from a row. */\n ArrowConstantColumnReader(\n StructField field, int batchSize, InternalRow values, int index, boolean useDecimal128) {\n super(field.dataType(), TypeUtil.convertToParquet(field), useDecimal128, false);\n this.batchSize = batchSize;\n Object v = values.get(index, field.dataType());\n this.value = v;\n initVector(v, batchSize);\n }\n\n @Override\n public void setBatchSize(int batchSize) {\n close();\n this.batchSize = batchSize;\n initVector(value, batchSize);\n }\n\n @Override\n public void readBatch(int total) {\n if (total != currentSize) {\n close();\n initVector(value, total);\n }\n }\n\n @Override\n public CometVector currentBatch() {\n return vector;\n }\n\n @Override\n public void close() {\n if (vector != null) {\n vector.close();\n vector = null;\n }\n if (fieldVector != null) {\n fieldVector.close();\n fieldVector = null;\n }\n }\n\n private void initVector(Object value, int count) {\n currentSize = count;\n if (value == null) {\n isNull = true;\n fieldVector = createNullVector(count);\n } else {\n isNull = false;\n fieldVector = createFilledVector(value, count);\n }\n vector = new CometPlainVector(fieldVector, useDecimal128, false, true);\n }\n\n /** Creates a vector of the correct type with {@code count} null values. */\n private FieldVector createNullVector(int count) {\n String name = \"constant\";\n FieldVector v;\n if (type == DataTypes.BooleanType) {\n v = new BitVector(name, allocator);\n } else if (type == DataTypes.ByteType) {\n v = new TinyIntVector(name, allocator);\n } else if (type == DataTypes.ShortType) {\n v = new SmallIntVector(name, allocator);\n } else if (type == DataTypes.IntegerType || type == DataTypes.DateType) {\n v = new IntVector(name, allocator);\n } else if (type == DataTypes.LongType\n || type == DataTypes.TimestampType\n || type == TimestampNTZType$.MODULE$) {\n v = new BigIntVector(name, allocator);\n } else if (type == DataTypes.FloatType) {\n v = new Float4Vector(name, allocator);\n } else if (type == DataTypes.DoubleType) {\n v = new Float8Vector(name, allocator);\n } else if (type == DataTypes.BinaryType) {\n v = new VarBinaryVector(name, allocator);\n } else if (type == DataTypes.StringType) {\n v = new VarCharVector(name, allocator);\n } else if (type instanceof DecimalType) {\n DecimalType dt = (DecimalType) type;\n if (!useDecimal128 && dt.precision() <= Decimal.MAX_INT_DIGITS()) {\n v = new IntVector(name, allocator);\n } else if (!useDecimal128 && dt.precision() <= Decimal.MAX_LONG_DIGITS()) {\n v = new BigIntVector(name, allocator);\n } else {\n v = new DecimalVector(name, allocator, dt.precision(), dt.scale());\n }\n } else {\n throw new UnsupportedOperationException(\"Unsupported Spark type: \" + type);\n }\n v.setValueCount(count);\n return v;\n }\n\n /** Creates a vector filled with {@code count} copies of the given value. */\n private FieldVector createFilledVector(Object value, int count) {\n String name = \"constant\";\n if (type == DataTypes.BooleanType) {\n BitVector v = new BitVector(name, allocator);\n v.allocateNew(count);\n int bit = (boolean) value ? 1 : 0;\n for (int i = 0; i < count; i++) v.setSafe(i, bit);\n v.setValueCount(count);\n return v;\n } else if (type == DataTypes.ByteType) {\n TinyIntVector v = new TinyIntVector(name, allocator);\n v.allocateNew(count);\n byte val = (byte) value;\n for (int i = 0; i < count; i++) v.setSafe(i, val);\n v.setValueCount(count);\n return v;\n } else if (type == DataTypes.ShortType) {\n SmallIntVector v = new SmallIntVector(name, allocator);\n v.allocateNew(count);\n short val = (short) value;\n for (int i = 0; i < count; i++) v.setSafe(i, val);\n v.setValueCount(count);\n return v;\n } else if (type == DataTypes.IntegerType || type == DataTypes.DateType) {\n IntVector v = new IntVector(name, allocator);\n v.allocateNew(count);\n int val = (int) value;\n for (int i = 0; i < count; i++) v.setSafe(i, val);\n v.setValueCount(count);\n return v;\n } else if (type == DataTypes.LongType\n || type == DataTypes.TimestampType\n || type == TimestampNTZType$.MODULE$) {\n BigIntVector v = new BigIntVector(name, allocator);\n v.allocateNew(count);\n long val = (long) value;\n for (int i = 0; i < count; i++) v.setSafe(i, val);\n v.setValueCount(count);\n return v;\n } else if (type == DataTypes.FloatType) {\n Float4Vector v = new Float4Vector(name, allocator);\n v.allocateNew(count);\n float val = (float) value;\n for (int i = 0; i < count; i++) v.setSafe(i, val);\n v.setValueCount(count);\n return v;\n } else if (type == DataTypes.DoubleType) {\n Float8Vector v = new Float8Vector(name, allocator);\n v.allocateNew(count);\n double val = (double) value;\n for (int i = 0; i < count; i++) v.setSafe(i, val);\n v.setValueCount(count);\n return v;\n } else if (type == DataTypes.BinaryType) {\n VarBinaryVector v = new VarBinaryVector(name, allocator);\n v.allocateNew(count);\n byte[] bytes = (byte[]) value;\n for (int i = 0; i < count; i++) v.setSafe(i, bytes, 0, bytes.length);\n v.setValueCount(count);\n return v;\n } else if (type == DataTypes.StringType) {\n VarCharVector v = new VarCharVector(name, allocator);\n v.allocateNew(count);\n byte[] bytes = ((UTF8String) value).getBytes();\n for (int i = 0; i < count; i++) v.setSafe(i, bytes, 0, bytes.length);\n v.setValueCount(count);\n return v;\n } else if (type instanceof DecimalType) {\n DecimalType dt = (DecimalType) type;\n Decimal d = (Decimal) value;\n if (!useDecimal128 && dt.precision() <= Decimal.MAX_INT_DIGITS()) {\n IntVector v = new IntVector(name, allocator);\n v.allocateNew(count);\n int val = (int) d.toUnscaledLong();\n for (int i = 0; i < count; i++) v.setSafe(i, val);\n v.setValueCount(count);\n return v;\n } else if (!useDecimal128 && dt.precision() <= Decimal.MAX_LONG_DIGITS()) {\n BigIntVector v = new BigIntVector(name, allocator);\n v.allocateNew(count);\n long val = d.toUnscaledLong();\n for (int i = 0; i < count; i++) v.setSafe(i, val);\n v.setValueCount(count);\n return v;\n } else {\n DecimalVector v = new DecimalVector(name, allocator, dt.precision(), dt.scale());\n v.allocateNew(count);\n BigDecimal bd = d.toJavaBigDecimal();\n for (int i = 0; i < count; i++) v.setSafe(i, bd);\n v.setValueCount(count);\n return v;\n }\n } else {\n throw new UnsupportedOperationException(\"Unsupported Spark type: \" + type);\n }\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/ArrowRowIndexColumnReader.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport org.apache.arrow.memory.BufferAllocator;\nimport org.apache.arrow.memory.RootAllocator;\nimport org.apache.arrow.vector.BigIntVector;\nimport org.apache.spark.sql.types.*;\n\nimport org.apache.comet.vector.CometPlainVector;\nimport org.apache.comet.vector.CometVector;\n\n/**\n * A column reader that computes row indices in Java and creates Arrow BigIntVectors directly (no\n * native mutable buffers). Used for the row index metadata column in the native_iceberg_compat scan\n * path.\n *\n *

The {@code indices} array contains alternating pairs of (start_index, count) representing\n * ranges of sequential row indices within each row group.\n */\npublic class ArrowRowIndexColumnReader extends AbstractColumnReader {\n private final BufferAllocator allocator = new RootAllocator();\n\n /** Alternating (start_index, count) pairs from row groups. */\n private final long[] indices;\n\n /** Number of row indices consumed so far across batches. */\n private long offset;\n\n private BigIntVector fieldVector;\n private CometPlainVector vector;\n\n public ArrowRowIndexColumnReader(StructField field, int batchSize, long[] indices) {\n super(field.dataType(), TypeUtil.convertToParquet(field), false, false);\n this.indices = indices;\n this.batchSize = batchSize;\n }\n\n @Override\n public void setBatchSize(int batchSize) {\n close();\n this.batchSize = batchSize;\n }\n\n @Override\n public void readBatch(int total) {\n close();\n\n fieldVector = new BigIntVector(\"row_index\", allocator);\n fieldVector.allocateNew(total);\n\n // Port of Rust set_indices: iterate (start, count) pairs, skip offset rows, fill up to total.\n long skipped = 0;\n int filled = 0;\n for (int i = 0; i < indices.length && filled < total; i += 2) {\n long index = indices[i];\n long count = indices[i + 1];\n long skip = Math.min(count, offset - skipped);\n skipped += skip;\n if (count == skip) {\n continue;\n }\n long remaining = Math.min(count - skip, total - filled);\n for (long j = 0; j < remaining; j++) {\n fieldVector.setSafe(filled, index + skip + j);\n filled++;\n }\n }\n offset += filled;\n\n fieldVector.setValueCount(filled);\n vector = new CometPlainVector(fieldVector, false, false, false);\n vector.setNumValues(filled);\n }\n\n @Override\n public CometVector currentBatch() {\n return vector;\n }\n\n @Override\n public void close() {\n if (vector != null) {\n vector.close();\n vector = null;\n }\n if (fieldVector != null) {\n fieldVector.close();\n fieldVector = null;\n }\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/BloomFilterReader.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.io.IOException;\nimport java.util.HashMap;\nimport java.util.Map;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport org.apache.parquet.column.values.bloomfilter.BlockSplitBloomFilter;\nimport org.apache.parquet.column.values.bloomfilter.BloomFilter;\nimport org.apache.parquet.crypto.AesCipher;\nimport org.apache.parquet.crypto.InternalColumnDecryptionSetup;\nimport org.apache.parquet.crypto.InternalFileDecryptor;\nimport org.apache.parquet.crypto.ModuleCipherFactory;\nimport org.apache.parquet.crypto.ParquetCryptoRuntimeException;\nimport org.apache.parquet.filter2.predicate.FilterPredicate;\nimport org.apache.parquet.filter2.predicate.Operators;\nimport org.apache.parquet.filter2.predicate.UserDefinedPredicate;\nimport org.apache.parquet.format.BlockCipher;\nimport org.apache.parquet.format.BloomFilterHeader;\nimport org.apache.parquet.format.Util;\nimport org.apache.parquet.hadoop.metadata.BlockMetaData;\nimport org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;\nimport org.apache.parquet.hadoop.metadata.ColumnPath;\nimport org.apache.parquet.io.SeekableInputStream;\n\npublic class BloomFilterReader implements FilterPredicate.Visitor {\n private static final Logger LOG = LoggerFactory.getLogger(BloomFilterReader.class);\n private static final boolean BLOCK_MIGHT_MATCH = false;\n private static final boolean BLOCK_CANNOT_MATCH = true;\n\n private final Map columns;\n private final Map cache = new HashMap<>();\n private final InternalFileDecryptor fileDecryptor;\n private final SeekableInputStream inputStream;\n\n BloomFilterReader(\n BlockMetaData block, InternalFileDecryptor fileDecryptor, SeekableInputStream inputStream) {\n this.columns = new HashMap<>();\n for (ColumnChunkMetaData column : block.getColumns()) {\n columns.put(column.getPath(), column);\n }\n this.fileDecryptor = fileDecryptor;\n this.inputStream = inputStream;\n }\n\n @Override\n public > Boolean visit(Operators.Eq eq) {\n T value = eq.getValue();\n\n if (value == null) {\n // the bloom filter bitset contains only non-null values so isn't helpful. this\n // could check the column stats, but the StatisticsFilter is responsible\n return BLOCK_MIGHT_MATCH;\n }\n\n Operators.Column filterColumn = eq.getColumn();\n ColumnChunkMetaData meta = columns.get(filterColumn.getColumnPath());\n if (meta == null) {\n // the column isn't in this file so all values are null, but the value\n // must be non-null because of the above check.\n return BLOCK_CANNOT_MATCH;\n }\n\n try {\n BloomFilter bloomFilter = readBloomFilter(meta);\n if (bloomFilter != null && !bloomFilter.findHash(bloomFilter.hash(value))) {\n return BLOCK_CANNOT_MATCH;\n }\n } catch (RuntimeException e) {\n LOG.warn(e.getMessage());\n return BLOCK_MIGHT_MATCH;\n }\n\n return BLOCK_MIGHT_MATCH;\n }\n\n @Override\n public > Boolean visit(Operators.NotEq notEq) {\n return BLOCK_MIGHT_MATCH;\n }\n\n @Override\n public > Boolean visit(Operators.Lt lt) {\n return BLOCK_MIGHT_MATCH;\n }\n\n @Override\n public > Boolean visit(Operators.LtEq ltEq) {\n return BLOCK_MIGHT_MATCH;\n }\n\n @Override\n public > Boolean visit(Operators.Gt gt) {\n return BLOCK_MIGHT_MATCH;\n }\n\n @Override\n public > Boolean visit(Operators.GtEq gtEq) {\n return BLOCK_MIGHT_MATCH;\n }\n\n @Override\n public Boolean visit(Operators.And and) {\n return and.getLeft().accept(this) || and.getRight().accept(this);\n }\n\n @Override\n public Boolean visit(Operators.Or or) {\n return or.getLeft().accept(this) && or.getRight().accept(this);\n }\n\n @Override\n public Boolean visit(Operators.Not not) {\n throw new IllegalArgumentException(\n \"This predicate \"\n + not\n + \" contains a not! Did you forget\"\n + \" to run this predicate through LogicalInverseRewriter?\");\n }\n\n @Override\n public , U extends UserDefinedPredicate> Boolean visit(\n Operators.UserDefined udp) {\n return visit(udp, false);\n }\n\n @Override\n public , U extends UserDefinedPredicate> Boolean visit(\n Operators.LogicalNotUserDefined udp) {\n return visit(udp.getUserDefined(), true);\n }\n\n private , U extends UserDefinedPredicate> Boolean visit(\n Operators.UserDefined ud, boolean inverted) {\n return BLOCK_MIGHT_MATCH;\n }\n\n BloomFilter readBloomFilter(ColumnChunkMetaData meta) {\n if (cache.containsKey(meta.getPath())) {\n return cache.get(meta.getPath());\n }\n try {\n if (!cache.containsKey(meta.getPath())) {\n BloomFilter bloomFilter = readBloomFilterInternal(meta);\n if (bloomFilter == null) {\n return null;\n }\n\n cache.put(meta.getPath(), bloomFilter);\n }\n return cache.get(meta.getPath());\n } catch (IOException e) {\n LOG.error(\"Failed to read Bloom filter data\", e);\n }\n\n return null;\n }\n\n private BloomFilter readBloomFilterInternal(ColumnChunkMetaData meta) throws IOException {\n long bloomFilterOffset = meta.getBloomFilterOffset();\n if (bloomFilterOffset < 0) {\n return null;\n }\n\n // Prepare to decrypt Bloom filter (for encrypted columns)\n BlockCipher.Decryptor bloomFilterDecryptor = null;\n byte[] bloomFilterHeaderAAD = null;\n byte[] bloomFilterBitsetAAD = null;\n if (null != fileDecryptor && !fileDecryptor.plaintextFile()) {\n InternalColumnDecryptionSetup columnDecryptionSetup =\n fileDecryptor.getColumnSetup(meta.getPath());\n if (columnDecryptionSetup.isEncrypted()) {\n bloomFilterDecryptor = columnDecryptionSetup.getMetaDataDecryptor();\n bloomFilterHeaderAAD =\n AesCipher.createModuleAAD(\n fileDecryptor.getFileAAD(),\n ModuleCipherFactory.ModuleType.BloomFilterHeader,\n meta.getRowGroupOrdinal(),\n columnDecryptionSetup.getOrdinal(),\n -1);\n bloomFilterBitsetAAD =\n AesCipher.createModuleAAD(\n fileDecryptor.getFileAAD(),\n ModuleCipherFactory.ModuleType.BloomFilterBitset,\n meta.getRowGroupOrdinal(),\n columnDecryptionSetup.getOrdinal(),\n -1);\n }\n }\n\n // Read Bloom filter data header.\n inputStream.seek(bloomFilterOffset);\n BloomFilterHeader bloomFilterHeader;\n try {\n bloomFilterHeader =\n Util.readBloomFilterHeader(inputStream, bloomFilterDecryptor, bloomFilterHeaderAAD);\n } catch (IOException e) {\n LOG.warn(\"read no bloom filter\");\n return null;\n }\n\n int numBytes = bloomFilterHeader.getNumBytes();\n if (numBytes <= 0 || numBytes > BlockSplitBloomFilter.UPPER_BOUND_BYTES) {\n LOG.warn(\"the read bloom filter size is wrong, size is {}\", bloomFilterHeader.getNumBytes());\n return null;\n }\n\n if (!bloomFilterHeader.getHash().isSetXXHASH()\n || !bloomFilterHeader.getAlgorithm().isSetBLOCK()\n || !bloomFilterHeader.getCompression().isSetUNCOMPRESSED()) {\n LOG.warn(\n \"the read bloom filter is not supported yet, algorithm = {}, hash = {}, \"\n + \"compression = {}\",\n bloomFilterHeader.getAlgorithm(),\n bloomFilterHeader.getHash(),\n bloomFilterHeader.getCompression());\n return null;\n }\n\n byte[] bitset;\n if (null == bloomFilterDecryptor) {\n bitset = new byte[numBytes];\n inputStream.readFully(bitset);\n } else {\n bitset = bloomFilterDecryptor.decrypt(inputStream, bloomFilterBitsetAAD);\n if (bitset.length != numBytes) {\n throw new ParquetCryptoRuntimeException(\"Wrong length of decrypted bloom filter bitset\");\n }\n }\n return new BlockSplitBloomFilter(bitset);\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/ColumnIndexReader.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.io.IOException;\nimport java.util.Collections;\nimport java.util.HashMap;\nimport java.util.Map;\nimport java.util.Set;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport org.apache.parquet.crypto.AesCipher;\nimport org.apache.parquet.crypto.InternalColumnDecryptionSetup;\nimport org.apache.parquet.crypto.InternalFileDecryptor;\nimport org.apache.parquet.crypto.ModuleCipherFactory;\nimport org.apache.parquet.format.BlockCipher;\nimport org.apache.parquet.format.Util;\nimport org.apache.parquet.format.converter.ParquetMetadataConverter;\nimport org.apache.parquet.hadoop.metadata.BlockMetaData;\nimport org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;\nimport org.apache.parquet.hadoop.metadata.ColumnPath;\nimport org.apache.parquet.internal.column.columnindex.ColumnIndex;\nimport org.apache.parquet.internal.column.columnindex.OffsetIndex;\nimport org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore;\nimport org.apache.parquet.internal.hadoop.metadata.IndexReference;\nimport org.apache.parquet.io.SeekableInputStream;\n\nclass ColumnIndexReader implements ColumnIndexStore {\n private static final Logger LOG = LoggerFactory.getLogger(ColumnIndexReader.class);\n\n // Used for columns are not in this parquet file\n private static final IndexStore MISSING_INDEX_STORE =\n new IndexStore() {\n @Override\n public ColumnIndex getColumnIndex() {\n return null;\n }\n\n @Override\n public OffsetIndex getOffsetIndex() {\n return null;\n }\n };\n\n private static final ColumnIndexReader EMPTY =\n new ColumnIndexReader(new BlockMetaData(), Collections.emptySet(), null, null) {\n @Override\n public ColumnIndex getColumnIndex(ColumnPath column) {\n return null;\n }\n\n @Override\n public OffsetIndex getOffsetIndex(ColumnPath column) {\n throw new MissingOffsetIndexException(column);\n }\n };\n\n private final InternalFileDecryptor fileDecryptor;\n private final SeekableInputStream inputStream;\n private final Map store;\n\n /**\n * Creates a column index store which lazily reads column/offset indexes for the columns in paths.\n * Paths are the set of columns used for the projection.\n */\n static ColumnIndexReader create(\n BlockMetaData block,\n Set paths,\n InternalFileDecryptor fileDecryptor,\n SeekableInputStream inputStream) {\n try {\n return new ColumnIndexReader(block, paths, fileDecryptor, inputStream);\n } catch (MissingOffsetIndexException e) {\n return EMPTY;\n }\n }\n\n private ColumnIndexReader(\n BlockMetaData block,\n Set paths,\n InternalFileDecryptor fileDecryptor,\n SeekableInputStream inputStream) {\n this.fileDecryptor = fileDecryptor;\n this.inputStream = inputStream;\n Map store = new HashMap<>();\n for (ColumnChunkMetaData column : block.getColumns()) {\n ColumnPath path = column.getPath();\n if (paths.contains(path)) {\n store.put(path, new IndexStoreImpl(column));\n }\n }\n this.store = store;\n }\n\n @Override\n public ColumnIndex getColumnIndex(ColumnPath column) {\n return store.getOrDefault(column, MISSING_INDEX_STORE).getColumnIndex();\n }\n\n @Override\n public OffsetIndex getOffsetIndex(ColumnPath column) {\n return store.getOrDefault(column, MISSING_INDEX_STORE).getOffsetIndex();\n }\n\n private interface IndexStore {\n ColumnIndex getColumnIndex();\n\n OffsetIndex getOffsetIndex();\n }\n\n private class IndexStoreImpl implements IndexStore {\n private final ColumnChunkMetaData meta;\n private ColumnIndex columnIndex;\n private boolean columnIndexRead;\n private final OffsetIndex offsetIndex;\n\n IndexStoreImpl(ColumnChunkMetaData meta) {\n this.meta = meta;\n OffsetIndex oi;\n try {\n oi = readOffsetIndex(meta);\n } catch (IOException e) {\n // If the I/O issue still stands it will fail the reading later;\n // otherwise we fail the filtering only with a missing offset index.\n LOG.warn(\"Unable to read offset index for column {}\", meta.getPath(), e);\n oi = null;\n }\n if (oi == null) {\n throw new MissingOffsetIndexException(meta.getPath());\n }\n offsetIndex = oi;\n }\n\n @Override\n public ColumnIndex getColumnIndex() {\n if (!columnIndexRead) {\n try {\n columnIndex = readColumnIndex(meta);\n } catch (IOException e) {\n // If the I/O issue still stands it will fail the reading later;\n // otherwise we fail the filtering only with a missing column index.\n LOG.warn(\"Unable to read column index for column {}\", meta.getPath(), e);\n }\n columnIndexRead = true;\n }\n return columnIndex;\n }\n\n @Override\n public OffsetIndex getOffsetIndex() {\n return offsetIndex;\n }\n }\n\n // Visible for testing\n ColumnIndex readColumnIndex(ColumnChunkMetaData column) throws IOException {\n IndexReference ref = column.getColumnIndexReference();\n if (ref == null) {\n return null;\n }\n inputStream.seek(ref.getOffset());\n\n BlockCipher.Decryptor columnIndexDecryptor = null;\n byte[] columnIndexAAD = null;\n if (null != fileDecryptor && !fileDecryptor.plaintextFile()) {\n InternalColumnDecryptionSetup columnDecryptionSetup =\n fileDecryptor.getColumnSetup(column.getPath());\n if (columnDecryptionSetup.isEncrypted()) {\n columnIndexDecryptor = columnDecryptionSetup.getMetaDataDecryptor();\n columnIndexAAD =\n AesCipher.createModuleAAD(\n fileDecryptor.getFileAAD(),\n ModuleCipherFactory.ModuleType.ColumnIndex,\n column.getRowGroupOrdinal(),\n columnDecryptionSetup.getOrdinal(),\n -1);\n }\n }\n return ParquetMetadataConverter.fromParquetColumnIndex(\n column.getPrimitiveType(),\n Util.readColumnIndex(inputStream, columnIndexDecryptor, columnIndexAAD));\n }\n\n // Visible for testing\n OffsetIndex readOffsetIndex(ColumnChunkMetaData column) throws IOException {\n IndexReference ref = column.getOffsetIndexReference();\n if (ref == null) {\n return null;\n }\n inputStream.seek(ref.getOffset());\n\n BlockCipher.Decryptor offsetIndexDecryptor = null;\n byte[] offsetIndexAAD = null;\n if (null != fileDecryptor && !fileDecryptor.plaintextFile()) {\n InternalColumnDecryptionSetup columnDecryptionSetup =\n fileDecryptor.getColumnSetup(column.getPath());\n if (columnDecryptionSetup.isEncrypted()) {\n offsetIndexDecryptor = columnDecryptionSetup.getMetaDataDecryptor();\n offsetIndexAAD =\n AesCipher.createModuleAAD(\n fileDecryptor.getFileAAD(),\n ModuleCipherFactory.ModuleType.OffsetIndex,\n column.getRowGroupOrdinal(),\n columnDecryptionSetup.getOrdinal(),\n -1);\n }\n }\n return ParquetMetadataConverter.fromParquetOffsetIndex(\n Util.readOffsetIndex(inputStream, offsetIndexDecryptor, offsetIndexAAD));\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/ColumnPageReader.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.io.IOException;\nimport java.util.ArrayDeque;\nimport java.util.List;\nimport java.util.Queue;\n\nimport org.apache.parquet.bytes.BytesInput;\nimport org.apache.parquet.column.page.DataPage;\nimport org.apache.parquet.column.page.DataPageV1;\nimport org.apache.parquet.column.page.DataPageV2;\nimport org.apache.parquet.column.page.DictionaryPage;\nimport org.apache.parquet.column.page.PageReader;\nimport org.apache.parquet.compression.CompressionCodecFactory;\nimport org.apache.parquet.crypto.AesCipher;\nimport org.apache.parquet.crypto.ModuleCipherFactory;\nimport org.apache.parquet.format.BlockCipher;\nimport org.apache.parquet.internal.column.columnindex.OffsetIndex;\nimport org.apache.parquet.io.ParquetDecodingException;\n\npublic class ColumnPageReader implements PageReader {\n private final CompressionCodecFactory.BytesInputDecompressor decompressor;\n private final long valueCount;\n private final Queue compressedPages;\n private final DictionaryPage compressedDictionaryPage;\n\n private final OffsetIndex offsetIndex;\n private final long rowCount;\n private int pageIndex = 0;\n\n private final BlockCipher.Decryptor blockDecryptor;\n private final byte[] dataPageAAD;\n private final byte[] dictionaryPageAAD;\n\n ColumnPageReader(\n CompressionCodecFactory.BytesInputDecompressor decompressor,\n List compressedPages,\n DictionaryPage compressedDictionaryPage,\n OffsetIndex offsetIndex,\n long rowCount,\n BlockCipher.Decryptor blockDecryptor,\n byte[] fileAAD,\n int rowGroupOrdinal,\n int columnOrdinal) {\n this.decompressor = decompressor;\n this.compressedPages = new ArrayDeque<>(compressedPages);\n this.compressedDictionaryPage = compressedDictionaryPage;\n long count = 0;\n for (DataPage p : compressedPages) {\n count += p.getValueCount();\n }\n this.valueCount = count;\n this.offsetIndex = offsetIndex;\n this.rowCount = rowCount;\n this.blockDecryptor = blockDecryptor;\n\n if (blockDecryptor != null) {\n dataPageAAD =\n AesCipher.createModuleAAD(\n fileAAD, ModuleCipherFactory.ModuleType.DataPage, rowGroupOrdinal, columnOrdinal, 0);\n dictionaryPageAAD =\n AesCipher.createModuleAAD(\n fileAAD,\n ModuleCipherFactory.ModuleType.DictionaryPage,\n rowGroupOrdinal,\n columnOrdinal,\n -1);\n } else {\n dataPageAAD = null;\n dictionaryPageAAD = null;\n }\n }\n\n @Override\n public long getTotalValueCount() {\n return valueCount;\n }\n\n /** Returns the total value count of the current page. */\n public int getPageValueCount() {\n return compressedPages.element().getValueCount();\n }\n\n /** Skips the current page so it won't be returned by {@link #readPage()} */\n public void skipPage() {\n compressedPages.poll();\n pageIndex++;\n }\n\n @Override\n public DataPage readPage() {\n final DataPage compressedPage = compressedPages.poll();\n if (compressedPage == null) {\n return null;\n }\n final int currentPageIndex = pageIndex++;\n\n if (null != blockDecryptor) {\n AesCipher.quickUpdatePageAAD(dataPageAAD, getPageOrdinal(currentPageIndex));\n }\n\n return compressedPage.accept(\n new DataPage.Visitor() {\n @Override\n public DataPage visit(DataPageV1 dataPageV1) {\n try {\n BytesInput bytes = dataPageV1.getBytes();\n if (null != blockDecryptor) {\n bytes = BytesInput.from(blockDecryptor.decrypt(bytes.toByteArray(), dataPageAAD));\n }\n BytesInput decompressed =\n decompressor.decompress(bytes, dataPageV1.getUncompressedSize());\n\n final DataPageV1 decompressedPage;\n if (offsetIndex == null) {\n decompressedPage =\n new DataPageV1(\n decompressed,\n dataPageV1.getValueCount(),\n dataPageV1.getUncompressedSize(),\n dataPageV1.getStatistics(),\n dataPageV1.getRlEncoding(),\n dataPageV1.getDlEncoding(),\n dataPageV1.getValueEncoding());\n } else {\n long firstRowIndex = offsetIndex.getFirstRowIndex(currentPageIndex);\n decompressedPage =\n new DataPageV1(\n decompressed,\n dataPageV1.getValueCount(),\n dataPageV1.getUncompressedSize(),\n firstRowIndex,\n Math.toIntExact(\n offsetIndex.getLastRowIndex(currentPageIndex, rowCount)\n - firstRowIndex\n + 1),\n dataPageV1.getStatistics(),\n dataPageV1.getRlEncoding(),\n dataPageV1.getDlEncoding(),\n dataPageV1.getValueEncoding());\n }\n if (dataPageV1.getCrc().isPresent()) {\n decompressedPage.setCrc(dataPageV1.getCrc().getAsInt());\n }\n return decompressedPage;\n } catch (IOException e) {\n throw new ParquetDecodingException(\"could not decompress page\", e);\n }\n }\n\n @Override\n public DataPage visit(DataPageV2 dataPageV2) {\n if (!dataPageV2.isCompressed() && offsetIndex == null && null == blockDecryptor) {\n return dataPageV2;\n }\n BytesInput pageBytes = dataPageV2.getData();\n\n if (null != blockDecryptor) {\n try {\n pageBytes =\n BytesInput.from(blockDecryptor.decrypt(pageBytes.toByteArray(), dataPageAAD));\n } catch (IOException e) {\n throw new ParquetDecodingException(\n \"could not convert page ByteInput to byte array\", e);\n }\n }\n if (dataPageV2.isCompressed()) {\n int uncompressedSize =\n Math.toIntExact(\n dataPageV2.getUncompressedSize()\n - dataPageV2.getDefinitionLevels().size()\n - dataPageV2.getRepetitionLevels().size());\n try {\n pageBytes = decompressor.decompress(pageBytes, uncompressedSize);\n } catch (IOException e) {\n throw new ParquetDecodingException(\"could not decompress page\", e);\n }\n }\n\n if (offsetIndex == null) {\n return DataPageV2.uncompressed(\n dataPageV2.getRowCount(),\n dataPageV2.getNullCount(),\n dataPageV2.getValueCount(),\n dataPageV2.getRepetitionLevels(),\n dataPageV2.getDefinitionLevels(),\n dataPageV2.getDataEncoding(),\n pageBytes,\n dataPageV2.getStatistics());\n } else {\n return DataPageV2.uncompressed(\n dataPageV2.getRowCount(),\n dataPageV2.getNullCount(),\n dataPageV2.getValueCount(),\n offsetIndex.getFirstRowIndex(currentPageIndex),\n dataPageV2.getRepetitionLevels(),\n dataPageV2.getDefinitionLevels(),\n dataPageV2.getDataEncoding(),\n pageBytes,\n dataPageV2.getStatistics());\n }\n }\n });\n }\n\n @Override\n public DictionaryPage readDictionaryPage() {\n if (compressedDictionaryPage == null) {\n return null;\n }\n try {\n BytesInput bytes = compressedDictionaryPage.getBytes();\n if (null != blockDecryptor) {\n bytes = BytesInput.from(blockDecryptor.decrypt(bytes.toByteArray(), dictionaryPageAAD));\n }\n DictionaryPage decompressedPage =\n new DictionaryPage(\n decompressor.decompress(bytes, compressedDictionaryPage.getUncompressedSize()),\n compressedDictionaryPage.getDictionarySize(),\n compressedDictionaryPage.getEncoding());\n if (compressedDictionaryPage.getCrc().isPresent()) {\n decompressedPage.setCrc(compressedDictionaryPage.getCrc().getAsInt());\n }\n return decompressedPage;\n } catch (IOException e) {\n throw new ParquetDecodingException(\"Could not decompress dictionary page\", e);\n }\n }\n\n private int getPageOrdinal(int currentPageIndex) {\n return offsetIndex == null ? currentPageIndex : offsetIndex.getPageOrdinal(currentPageIndex);\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/ColumnReader.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.io.IOException;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport org.apache.arrow.c.ArrowArray;\nimport org.apache.arrow.c.ArrowSchema;\nimport org.apache.arrow.memory.BufferAllocator;\nimport org.apache.arrow.memory.RootAllocator;\nimport org.apache.arrow.vector.FieldVector;\nimport org.apache.arrow.vector.dictionary.Dictionary;\nimport org.apache.arrow.vector.types.pojo.DictionaryEncoding;\nimport org.apache.parquet.column.ColumnDescriptor;\nimport org.apache.parquet.column.Encoding;\nimport org.apache.parquet.column.page.DataPage;\nimport org.apache.parquet.column.page.DataPageV1;\nimport org.apache.parquet.column.page.DataPageV2;\nimport org.apache.parquet.column.page.DictionaryPage;\nimport org.apache.parquet.column.page.PageReader;\nimport org.apache.parquet.schema.LogicalTypeAnnotation;\nimport org.apache.spark.sql.types.DataType;\n\nimport org.apache.comet.CometSchemaImporter;\nimport org.apache.comet.IcebergApi;\nimport org.apache.comet.vector.CometDecodedVector;\nimport org.apache.comet.vector.CometDictionary;\nimport org.apache.comet.vector.CometDictionaryVector;\nimport org.apache.comet.vector.CometPlainVector;\nimport org.apache.comet.vector.CometVector;\n\n@IcebergApi\npublic class ColumnReader extends AbstractColumnReader {\n protected static final Logger LOG = LoggerFactory.getLogger(ColumnReader.class);\n protected final BufferAllocator ALLOCATOR = new RootAllocator();\n\n /**\n * The current Comet vector holding all the values read by this column reader. Owned by this\n * reader and MUST be closed after use.\n */\n private CometDecodedVector currentVector;\n\n /** Dictionary values for this column. Only set if the column is using dictionary encoding. */\n protected CometDictionary dictionary;\n\n /** Reader for dictionary & data pages in the current column chunk. */\n protected PageReader pageReader;\n\n /** Whether the first data page has been loaded. */\n private boolean firstPageLoaded = false;\n\n /**\n * The number of nulls in the current batch, used when we are skipping importing of Arrow vectors,\n * in which case we'll simply update the null count of the existing vectors.\n */\n int currentNumNulls;\n\n /**\n * The number of values in the current batch, used when we are skipping importing of Arrow\n * vectors, in which case we'll simply update the null count of the existing vectors.\n */\n int currentNumValues;\n\n /**\n * Whether the last loaded vector contains any null value. This is used to determine if we can\n * skip vector reloading. If the flag is false, Arrow C API will skip to import the validity\n * buffer, and therefore we cannot skip vector reloading.\n */\n boolean hadNull;\n\n private final CometSchemaImporter importer;\n\n private ArrowArray array = null;\n private ArrowSchema schema = null;\n\n ColumnReader(\n DataType type,\n ColumnDescriptor descriptor,\n CometSchemaImporter importer,\n int batchSize,\n boolean useDecimal128,\n boolean useLegacyDateTimestamp) {\n super(type, descriptor, useDecimal128, useLegacyDateTimestamp);\n assert batchSize > 0 : \"Batch size must be positive, found \" + batchSize;\n this.batchSize = batchSize;\n this.importer = importer;\n initNative();\n }\n\n /**\n * Set the page reader for a new column chunk to read. Expects to call `readBatch` after this.\n *\n * @param pageReader the page reader for the new column chunk\n * @see Comet Issue #2079\n */\n @IcebergApi\n public void setPageReader(PageReader pageReader) throws IOException {\n this.pageReader = pageReader;\n\n DictionaryPage dictionaryPage = pageReader.readDictionaryPage();\n if (dictionaryPage != null) {\n LOG.debug(\"dictionary page encoding = {}\", dictionaryPage.getEncoding());\n Native.setDictionaryPage(\n nativeHandle,\n dictionaryPage.getDictionarySize(),\n dictionaryPage.getBytes().toByteArray(),\n dictionaryPage.getEncoding().ordinal());\n }\n }\n\n /** This method is called from Apache Iceberg. */\n @IcebergApi\n public void setRowGroupReader(RowGroupReader rowGroupReader, ParquetColumnSpec columnSpec)\n throws IOException {\n ColumnDescriptor descriptor = Utils.buildColumnDescriptor(columnSpec);\n setPageReader(rowGroupReader.getPageReader(descriptor));\n }\n\n @Override\n public void readBatch(int total) {\n LOG.debug(\"Start to batch of size = \" + total);\n\n if (!firstPageLoaded) {\n readPage();\n firstPageLoaded = true;\n }\n\n // Now first reset the current columnar batch so that it can be used to fill in a new batch\n // of values. Then, keep reading more data pages (via 'readBatch') until the current batch is\n // full, or we have read 'total' number of values.\n Native.resetBatch(nativeHandle);\n\n int left = total, nullsRead = 0;\n while (left > 0) {\n int[] array = Native.readBatch(nativeHandle, left);\n int valuesRead = array[0];\n nullsRead += array[1];\n if (valuesRead < left) {\n readPage();\n }\n left -= valuesRead;\n }\n\n this.currentNumValues = total;\n this.currentNumNulls = nullsRead;\n }\n\n /** Returns the {@link CometVector} read by this reader. */\n @Override\n public CometVector currentBatch() {\n return loadVector();\n }\n\n @Override\n public void close() {\n if (currentVector != null) {\n currentVector.close();\n currentVector = null;\n }\n super.close();\n }\n\n /** Returns a decoded {@link CometDecodedVector Comet vector}. */\n public CometDecodedVector loadVector() {\n LOG.debug(\"Reloading vector\");\n\n // Close the previous vector first to release struct memory allocated to import Arrow array &\n // schema from native side, through the C data interface\n if (currentVector != null) {\n currentVector.close();\n }\n\n LogicalTypeAnnotation logicalTypeAnnotation =\n descriptor.getPrimitiveType().getLogicalTypeAnnotation();\n boolean isUuid =\n logicalTypeAnnotation instanceof LogicalTypeAnnotation.UUIDLogicalTypeAnnotation;\n\n array = ArrowArray.allocateNew(ALLOCATOR);\n schema = ArrowSchema.allocateNew(ALLOCATOR);\n\n long arrayAddr = array.memoryAddress();\n long schemaAddr = schema.memoryAddress();\n\n Native.currentBatch(nativeHandle, arrayAddr, schemaAddr);\n\n FieldVector vector = importer.importVector(array, schema);\n\n DictionaryEncoding dictionaryEncoding = vector.getField().getDictionary();\n\n CometPlainVector cometVector = new CometPlainVector(vector, useDecimal128);\n\n // Update whether the current vector contains any null values. This is used in the following\n // batch(s) to determine whether we can skip loading the native vector.\n hadNull = cometVector.hasNull();\n\n if (dictionaryEncoding == null) {\n if (dictionary != null) {\n // This means the column was using dictionary encoding but now has fall-back to plain\n // encoding, on the native side. Setting 'dictionary' to null here, so we can use it as\n // a condition to check if we can re-use vector later.\n dictionary = null;\n }\n // Either the column is not dictionary encoded, or it was using dictionary encoding but\n // a new data page has switched back to use plain encoding. For both cases we should\n // return plain vector.\n currentVector = cometVector;\n return currentVector;\n }\n\n // We should already re-initiate `CometDictionary` here because `Data.importVector` API will\n // release the previous dictionary vector and create a new one.\n Dictionary arrowDictionary = importer.getProvider().lookup(dictionaryEncoding.getId());\n CometPlainVector dictionaryVector =\n new CometPlainVector(arrowDictionary.getVector(), useDecimal128, isUuid);\n if (dictionary != null) {\n dictionary.setDictionaryVector(dictionaryVector);\n } else {\n dictionary = new CometDictionary(dictionaryVector);\n }\n\n currentVector =\n new CometDictionaryVector(\n cometVector, dictionary, importer.getProvider(), useDecimal128, false, isUuid);\n\n currentVector =\n new CometDictionaryVector(cometVector, dictionary, importer.getProvider(), useDecimal128);\n return currentVector;\n }\n\n protected void readPage() {\n DataPage page = pageReader.readPage();\n if (page == null) {\n throw new RuntimeException(\"overreading: returned DataPage is null\");\n }\n ;\n int pageValueCount = page.getValueCount();\n page.accept(\n new DataPage.Visitor() {\n @Override\n public Void visit(DataPageV1 dataPageV1) {\n LOG.debug(\"data page encoding = {}\", dataPageV1.getValueEncoding());\n if (dataPageV1.getDlEncoding() != Encoding.RLE\n && descriptor.getMaxDefinitionLevel() != 0) {\n throw new UnsupportedOperationException(\n \"Unsupported encoding: \" + dataPageV1.getDlEncoding());\n }\n if (!isValidValueEncoding(dataPageV1.getValueEncoding())) {\n throw new UnsupportedOperationException(\n \"Unsupported value encoding: \" + dataPageV1.getValueEncoding());\n }\n try {\n byte[] array = dataPageV1.getBytes().toByteArray();\n Native.setPageV1(\n nativeHandle, pageValueCount, array, dataPageV1.getValueEncoding().ordinal());\n } catch (IOException e) {\n throw new RuntimeException(e);\n }\n return null;\n }\n\n @Override\n public Void visit(DataPageV2 dataPageV2) {\n if (!isValidValueEncoding(dataPageV2.getDataEncoding())) {\n throw new UnsupportedOperationException(\n \"Unsupported encoding: \" + dataPageV2.getDataEncoding());\n }\n try {\n Native.setPageV2(\n nativeHandle,\n pageValueCount,\n dataPageV2.getDefinitionLevels().toByteArray(),\n dataPageV2.getRepetitionLevels().toByteArray(),\n dataPageV2.getData().toByteArray(),\n dataPageV2.getDataEncoding().ordinal());\n } catch (IOException e) {\n throw new RuntimeException(e);\n }\n return null;\n }\n });\n }\n\n @SuppressWarnings(\"deprecation\")\n private boolean isValidValueEncoding(Encoding encoding) {\n switch (encoding) {\n case PLAIN:\n case RLE_DICTIONARY:\n case PLAIN_DICTIONARY:\n return true;\n default:\n return false;\n }\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/CometFileKeyUnwrapper.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.util.concurrent.ConcurrentHashMap;\n\nimport org.apache.hadoop.conf.Configuration;\nimport org.apache.hadoop.fs.Path;\nimport org.apache.parquet.crypto.DecryptionKeyRetriever;\nimport org.apache.parquet.crypto.DecryptionPropertiesFactory;\nimport org.apache.parquet.crypto.FileDecryptionProperties;\nimport org.apache.parquet.crypto.ParquetCryptoRuntimeException;\n\n// spotless:off\n/*\n * Architecture Overview:\n *\n * JVM Side | Native Side\n * ┌─────────────────────────────────────┐ | ┌─────────────────────────────────────┐\n * │ CometFileKeyUnwrapper │ | │ Parquet File Reading │\n * │ │ | │ │\n * │ ┌─────────────────────────────┐ │ | │ ┌─────────────────────────────┐ │\n * │ │ hadoopConf │ │ | │ │ file1.parquet │ │\n * │ │ (Configuration) │ │ | │ │ file2.parquet │ │\n * │ └─────────────────────────────┘ │ | │ │ file3.parquet │ │\n * │ │ │ | │ └─────────────────────────────┘ │\n * │ ▼ │ | │ │ │\n * │ ┌─────────────────────────────┐ │ | │ │ │\n * │ │ factoryCache │ │ | │ ▼ │\n * │ │ (many-to-one mapping) │ │ | │ ┌─────────────────────────────┐ │\n * │ │ │ │ | │ │ Parse file metadata & │ │\n * │ │ file1 ──┐ │ │ | │ │ extract keyMetadata │ │\n * │ │ file2 ──┼─► DecryptionProps │ │ | │ └─────────────────────────────┘ │\n * │ │ file3 ──┘ Factory │ │ | │ │ │\n * │ └─────────────────────────────┘ │ | │ │ │\n * │ │ │ | │ ▼ │\n * │ ▼ │ | │ ╔═════════════════════════════╗ │\n * │ ┌─────────────────────────────┐ │ | │ ║ JNI CALL: ║ │\n * │ │ retrieverCache │ │ | │ ║ getKey(filePath, ║ │\n * │ │ filePath -> KeyRetriever │◄───┼───┼───┼──║ keyMetadata) ║ │\n * │ └─────────────────────────────┘ │ | │ ╚═════════════════════════════╝ │\n * │ │ │ | │ │\n * │ ▼ │ | │ │\n * │ ┌─────────────────────────────┐ │ | │ │\n * │ │ DecryptionKeyRetriever │ │ | │ │\n * │ │ .getKey(keyMetadata) │ │ | │ │\n * │ └─────────────────────────────┘ │ | │ │\n * │ │ │ | │ │\n * │ ▼ │ | │ │\n * │ ┌─────────────────────────────┐ │ | │ ┌─────────────────────────────┐ │\n * │ │ return key bytes │────┼───┼───┼─►│ Use key for decryption │ │\n * │ └─────────────────────────────┘ │ | │ │ of parquet data │ │\n * └─────────────────────────────────────┘ | │ └─────────────────────────────┘ │\n * | └─────────────────────────────────────┘\n * |\n * JNI Boundary\n *\n * Setup Phase (storeDecryptionKeyRetriever):\n * 1. hadoopConf → DecryptionPropertiesFactory (cached in factoryCache)\n * 2. Factory + filePath → DecryptionKeyRetriever (cached in retrieverCache)\n *\n * Runtime Phase (getKey):\n * 3. Native code calls getKey(filePath, keyMetadata) ──► JVM\n * 4. Retrieve cached DecryptionKeyRetriever for filePath\n * 5. KeyRetriever.getKey(keyMetadata) → decrypted key bytes\n * 6. Return key bytes ──► Native code for parquet decryption\n */\n// spotless:on\n\n/**\n * Helper class to access DecryptionKeyRetriever.getKey from native code via JNI. This class handles\n * the complexity of creating and caching properly configured DecryptionKeyRetriever instances using\n * DecryptionPropertiesFactory. The life of this object is meant to map to a single Comet plan, so\n * associated with CometExecIterator.\n */\npublic class CometFileKeyUnwrapper {\n\n // Each file path gets a unique DecryptionKeyRetriever\n private final ConcurrentHashMap retrieverCache =\n new ConcurrentHashMap<>();\n\n // Cache the factory since we should be using the same hadoopConf for every file in this scan.\n private DecryptionPropertiesFactory factory = null;\n // Cache the hadoopConf just to assert the assumption above.\n private Configuration conf = null;\n\n /**\n * Normalizes S3 URI schemes to a canonical form. S3 can be accessed via multiple schemes (s3://,\n * s3a://, s3n://) that refer to the same logical filesystem. This method ensures consistent cache\n * lookups regardless of which scheme is used.\n *\n * @param filePath The file path that may contain an S3 URI\n * @return The file path with normalized S3 scheme (s3a://)\n */\n private String normalizeS3Scheme(final String filePath) {\n // Normalize s3:// and s3n:// to s3a:// for consistent cache lookups\n // This handles the case where ObjectStoreUrl uses s3:// but Spark uses s3a://\n String s3Prefix = \"s3://\";\n String s3nPrefix = \"s3n://\";\n if (filePath.startsWith(s3Prefix)) {\n return \"s3a://\" + filePath.substring(s3Prefix.length());\n } else if (filePath.startsWith(s3nPrefix)) {\n return \"s3a://\" + filePath.substring(s3nPrefix.length());\n }\n return filePath;\n }\n\n /**\n * Creates and stores a DecryptionKeyRetriever instance for the given file path.\n *\n * @param filePath The path to the Parquet file\n * @param hadoopConf The Hadoop Configuration to use for this file path\n */\n public void storeDecryptionKeyRetriever(final String filePath, final Configuration hadoopConf) {\n final String normalizedPath = normalizeS3Scheme(filePath);\n // Use DecryptionPropertiesFactory.loadFactory to get the factory and then call\n // getFileDecryptionProperties\n if (factory == null) {\n factory = DecryptionPropertiesFactory.loadFactory(hadoopConf);\n conf = hadoopConf;\n } else {\n // Check the assumption that all files have the same hadoopConf and thus same Factory\n assert (conf == hadoopConf);\n }\n Path path = new Path(filePath);\n FileDecryptionProperties decryptionProperties =\n factory.getFileDecryptionProperties(hadoopConf, path);\n\n DecryptionKeyRetriever keyRetriever = decryptionProperties.getKeyRetriever();\n retrieverCache.put(normalizedPath, keyRetriever);\n }\n\n /**\n * Gets the decryption key for the given key metadata using the cached DecryptionKeyRetriever for\n * the specified file path.\n *\n * @param filePath The path to the Parquet file\n * @param keyMetadata The key metadata bytes from the Parquet file\n * @return The decrypted key bytes\n * @throws ParquetCryptoRuntimeException if key unwrapping fails\n */\n public byte[] getKey(final String filePath, final byte[] keyMetadata)\n throws ParquetCryptoRuntimeException {\n final String normalizedPath = normalizeS3Scheme(filePath);\n DecryptionKeyRetriever keyRetriever = retrieverCache.get(normalizedPath);\n if (keyRetriever == null) {\n throw new ParquetCryptoRuntimeException(\n \"Failed to find DecryptionKeyRetriever for path: \" + filePath);\n }\n return keyRetriever.getKey(keyMetadata);\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/CometInputFile.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.io.IOException;\nimport java.util.regex.Matcher;\nimport java.util.regex.Pattern;\n\nimport org.apache.hadoop.conf.Configuration;\nimport org.apache.hadoop.fs.FSDataInputStream;\nimport org.apache.hadoop.fs.FileStatus;\nimport org.apache.hadoop.fs.FileSystem;\nimport org.apache.hadoop.fs.FutureDataInputStreamBuilder;\nimport org.apache.hadoop.fs.Path;\nimport org.apache.hadoop.util.VersionInfo;\nimport org.apache.parquet.hadoop.util.HadoopStreams;\nimport org.apache.parquet.io.InputFile;\nimport org.apache.parquet.io.SeekableInputStream;\n\n/**\n * A Parquet {@link InputFile} implementation that's similar to {@link\n * org.apache.parquet.hadoop.util.HadoopInputFile}, but with optimizations introduced in Hadoop 3.x,\n * for S3 specifically.\n */\npublic class CometInputFile implements InputFile {\n private static final String MAJOR_MINOR_REGEX = \"^(\\\\d+)\\\\.(\\\\d+)(\\\\..*)?$\";\n private static final Pattern VERSION_MATCHER = Pattern.compile(MAJOR_MINOR_REGEX);\n\n private final FileSystem fs;\n private final FileStatus stat;\n private final Configuration conf;\n\n public static CometInputFile fromPath(Path path, Configuration conf) throws IOException {\n FileSystem fs = path.getFileSystem(conf);\n return new CometInputFile(fs, fs.getFileStatus(path), conf);\n }\n\n private CometInputFile(FileSystem fs, FileStatus stat, Configuration conf) {\n this.fs = fs;\n this.stat = stat;\n this.conf = conf;\n }\n\n @Override\n public long getLength() {\n return stat.getLen();\n }\n\n public Configuration getConf() {\n return this.conf;\n }\n\n public FileSystem getFileSystem() {\n return this.fs;\n }\n\n public Path getPath() {\n return stat.getPath();\n }\n\n @Override\n public SeekableInputStream newStream() throws IOException {\n FSDataInputStream stream;\n try {\n if (isAtLeastHadoop33()) {\n // If Hadoop version is >= 3.3.x, we'll use the 'openFile' API which can save a\n // HEAD request from cloud storages like S3\n FutureDataInputStreamBuilder inputStreamBuilder =\n fs.openFile(stat.getPath()).withFileStatus(stat);\n\n if (stat.getPath().toString().startsWith(\"s3a\")) {\n // Switch to random S3 input policy so that we don't do sequential read on the entire\n // S3 object. By default, the policy is normal which does sequential read until a back\n // seek happens, which in our case will never happen.\n inputStreamBuilder =\n inputStreamBuilder.opt(\"fs.s3a.experimental.input.fadvise\", \"random\");\n }\n stream = inputStreamBuilder.build().get();\n } else {\n stream = fs.open(stat.getPath());\n }\n } catch (Exception e) {\n throw new IOException(\"Error when opening file \" + stat.getPath(), e);\n }\n return HadoopStreams.wrap(stream);\n }\n\n public SeekableInputStream newStream(long offset, long length) throws IOException {\n try {\n FSDataInputStream stream;\n if (isAtLeastHadoop33()) {\n FutureDataInputStreamBuilder inputStreamBuilder =\n fs.openFile(stat.getPath()).withFileStatus(stat);\n\n if (stat.getPath().toString().startsWith(\"s3a\")) {\n // Switch to random S3 input policy so that we don't do sequential read on the entire\n // S3 object. By default, the policy is normal which does sequential read until a back\n // seek happens, which in our case will never happen.\n //\n // Also set read ahead length equal to the column chunk length so we don't have to open\n // multiple S3 http connections.\n inputStreamBuilder =\n inputStreamBuilder\n .opt(\"fs.s3a.experimental.input.fadvise\", \"random\")\n .opt(\"fs.s3a.readahead.range\", Long.toString(length));\n }\n\n stream = inputStreamBuilder.build().get();\n } else {\n stream = fs.open(stat.getPath());\n }\n return HadoopStreams.wrap(stream);\n } catch (Exception e) {\n throw new IOException(\n \"Error when opening file \" + stat.getPath() + \", offset=\" + offset + \", length=\" + length,\n e);\n }\n }\n\n @Override\n public String toString() {\n return stat.getPath().toString();\n }\n\n private static boolean isAtLeastHadoop33() {\n String version = VersionInfo.getVersion();\n return CometInputFile.isAtLeastHadoop33(version);\n }\n\n static boolean isAtLeastHadoop33(String version) {\n Matcher matcher = VERSION_MATCHER.matcher(version);\n if (matcher.matches()) {\n if (matcher.group(1).equals(\"3\")) {\n int minorVersion = Integer.parseInt(matcher.group(2));\n return minorVersion >= 3;\n }\n }\n return false;\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/DictionaryPageReader.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.io.IOException;\nimport java.util.HashMap;\nimport java.util.Map;\nimport java.util.Optional;\nimport java.util.concurrent.ConcurrentHashMap;\n\nimport org.apache.parquet.ParquetReadOptions;\nimport org.apache.parquet.bytes.BytesInput;\nimport org.apache.parquet.column.ColumnDescriptor;\nimport org.apache.parquet.column.page.DictionaryPage;\nimport org.apache.parquet.column.page.DictionaryPageReadStore;\nimport org.apache.parquet.compression.CompressionCodecFactory;\nimport org.apache.parquet.crypto.AesCipher;\nimport org.apache.parquet.crypto.InternalColumnDecryptionSetup;\nimport org.apache.parquet.crypto.InternalFileDecryptor;\nimport org.apache.parquet.crypto.ModuleCipherFactory;\nimport org.apache.parquet.format.BlockCipher;\nimport org.apache.parquet.format.DictionaryPageHeader;\nimport org.apache.parquet.format.PageHeader;\nimport org.apache.parquet.format.Util;\nimport org.apache.parquet.hadoop.metadata.BlockMetaData;\nimport org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;\nimport org.apache.parquet.io.ParquetDecodingException;\nimport org.apache.parquet.io.SeekableInputStream;\n\npublic class DictionaryPageReader implements DictionaryPageReadStore {\n private final Map> cache;\n private final InternalFileDecryptor fileDecryptor;\n private final SeekableInputStream inputStream;\n private final ParquetReadOptions options;\n private final Map columns;\n\n DictionaryPageReader(\n BlockMetaData block,\n InternalFileDecryptor fileDecryptor,\n SeekableInputStream inputStream,\n ParquetReadOptions options) {\n this.columns = new HashMap<>();\n this.cache = new ConcurrentHashMap<>();\n this.fileDecryptor = fileDecryptor;\n this.inputStream = inputStream;\n this.options = options;\n\n for (ColumnChunkMetaData column : block.getColumns()) {\n columns.put(column.getPath().toDotString(), column);\n }\n }\n\n @Override\n public DictionaryPage readDictionaryPage(ColumnDescriptor descriptor) {\n String dotPath = String.join(\".\", descriptor.getPath());\n ColumnChunkMetaData column = columns.get(dotPath);\n\n if (column == null) {\n throw new ParquetDecodingException(\"Failed to load dictionary, unknown column: \" + dotPath);\n }\n\n return cache\n .computeIfAbsent(\n dotPath,\n key -> {\n try {\n final DictionaryPage dict =\n column.hasDictionaryPage() ? readDictionary(column) : null;\n\n // Copy the dictionary to ensure it can be reused if it is returned\n // more than once. This can happen when a DictionaryFilter has two or\n // more predicates for the same column. Cache misses as well.\n return (dict != null) ? Optional.of(reusableCopy(dict)) : Optional.empty();\n } catch (IOException e) {\n throw new ParquetDecodingException(\"Failed to read dictionary\", e);\n }\n })\n .orElse(null);\n }\n\n DictionaryPage readDictionary(ColumnChunkMetaData meta) throws IOException {\n if (!meta.hasDictionaryPage()) {\n return null;\n }\n\n if (inputStream.getPos() != meta.getStartingPos()) {\n inputStream.seek(meta.getStartingPos());\n }\n\n boolean encryptedColumn = false;\n InternalColumnDecryptionSetup columnDecryptionSetup = null;\n byte[] dictionaryPageAAD = null;\n BlockCipher.Decryptor pageDecryptor = null;\n if (null != fileDecryptor && !fileDecryptor.plaintextFile()) {\n columnDecryptionSetup = fileDecryptor.getColumnSetup(meta.getPath());\n if (columnDecryptionSetup.isEncrypted()) {\n encryptedColumn = true;\n }\n }\n\n PageHeader pageHeader;\n if (!encryptedColumn) {\n pageHeader = Util.readPageHeader(inputStream);\n } else {\n byte[] dictionaryPageHeaderAAD =\n AesCipher.createModuleAAD(\n fileDecryptor.getFileAAD(),\n ModuleCipherFactory.ModuleType.DictionaryPageHeader,\n meta.getRowGroupOrdinal(),\n columnDecryptionSetup.getOrdinal(),\n -1);\n pageHeader =\n Util.readPageHeader(\n inputStream, columnDecryptionSetup.getMetaDataDecryptor(), dictionaryPageHeaderAAD);\n dictionaryPageAAD =\n AesCipher.createModuleAAD(\n fileDecryptor.getFileAAD(),\n ModuleCipherFactory.ModuleType.DictionaryPage,\n meta.getRowGroupOrdinal(),\n columnDecryptionSetup.getOrdinal(),\n -1);\n pageDecryptor = columnDecryptionSetup.getDataDecryptor();\n }\n\n if (!pageHeader.isSetDictionary_page_header()) {\n return null;\n }\n\n DictionaryPage compressedPage =\n readCompressedDictionary(pageHeader, inputStream, pageDecryptor, dictionaryPageAAD);\n CompressionCodecFactory.BytesInputDecompressor decompressor =\n options.getCodecFactory().getDecompressor(meta.getCodec());\n\n return new DictionaryPage(\n decompressor.decompress(compressedPage.getBytes(), compressedPage.getUncompressedSize()),\n compressedPage.getDictionarySize(),\n compressedPage.getEncoding());\n }\n\n private DictionaryPage readCompressedDictionary(\n PageHeader pageHeader,\n SeekableInputStream fin,\n BlockCipher.Decryptor pageDecryptor,\n byte[] dictionaryPageAAD)\n throws IOException {\n DictionaryPageHeader dictHeader = pageHeader.getDictionary_page_header();\n\n int uncompressedPageSize = pageHeader.getUncompressed_page_size();\n int compressedPageSize = pageHeader.getCompressed_page_size();\n\n byte[] dictPageBytes = new byte[compressedPageSize];\n fin.readFully(dictPageBytes);\n\n BytesInput bin = BytesInput.from(dictPageBytes);\n\n if (null != pageDecryptor) {\n bin = BytesInput.from(pageDecryptor.decrypt(bin.toByteArray(), dictionaryPageAAD));\n }\n\n return new DictionaryPage(\n bin,\n uncompressedPageSize,\n dictHeader.getNum_values(),\n org.apache.parquet.column.Encoding.valueOf(dictHeader.getEncoding().name()));\n }\n\n private static DictionaryPage reusableCopy(DictionaryPage dict) throws IOException {\n return new DictionaryPage(\n BytesInput.from(dict.getBytes().toByteArray()),\n dict.getDictionarySize(),\n dict.getEncoding());\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/FileReader.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.io.Closeable;\nimport java.io.IOException;\nimport java.io.InputStream;\nimport java.lang.reflect.Method;\nimport java.net.URI;\nimport java.nio.ByteBuffer;\nimport java.util.ArrayList;\nimport java.util.Arrays;\nimport java.util.Collection;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.concurrent.ExecutionException;\nimport java.util.concurrent.ExecutorService;\nimport java.util.concurrent.Future;\nimport java.util.stream.Collectors;\nimport java.util.stream.Stream;\nimport java.util.zip.CRC32;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport org.apache.hadoop.conf.Configuration;\nimport org.apache.parquet.HadoopReadOptions;\nimport org.apache.parquet.ParquetReadOptions;\nimport org.apache.parquet.Preconditions;\nimport org.apache.parquet.bytes.ByteBufferInputStream;\nimport org.apache.parquet.bytes.BytesInput;\nimport org.apache.parquet.bytes.BytesUtils;\nimport org.apache.parquet.column.ColumnDescriptor;\nimport org.apache.parquet.column.page.DataPage;\nimport org.apache.parquet.column.page.DataPageV1;\nimport org.apache.parquet.column.page.DataPageV2;\nimport org.apache.parquet.column.page.DictionaryPage;\nimport org.apache.parquet.column.page.PageReadStore;\nimport org.apache.parquet.compression.CompressionCodecFactory;\nimport org.apache.parquet.crypto.AesCipher;\nimport org.apache.parquet.crypto.EncryptionPropertiesFactory;\nimport org.apache.parquet.crypto.FileDecryptionProperties;\nimport org.apache.parquet.crypto.InternalColumnDecryptionSetup;\nimport org.apache.parquet.crypto.InternalFileDecryptor;\nimport org.apache.parquet.crypto.ModuleCipherFactory;\nimport org.apache.parquet.crypto.ParquetCryptoRuntimeException;\nimport org.apache.parquet.filter2.compat.FilterCompat;\nimport org.apache.parquet.format.BlockCipher;\nimport org.apache.parquet.format.DataPageHeader;\nimport org.apache.parquet.format.DataPageHeaderV2;\nimport org.apache.parquet.format.DictionaryPageHeader;\nimport org.apache.parquet.format.FileCryptoMetaData;\nimport org.apache.parquet.format.PageHeader;\nimport org.apache.parquet.format.Util;\nimport org.apache.parquet.format.converter.ParquetMetadataConverter;\nimport org.apache.parquet.hadoop.ParquetInputFormat;\nimport org.apache.parquet.hadoop.metadata.BlockMetaData;\nimport org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;\nimport org.apache.parquet.hadoop.metadata.ColumnPath;\nimport org.apache.parquet.hadoop.metadata.FileMetaData;\nimport org.apache.parquet.hadoop.metadata.ParquetMetadata;\nimport org.apache.parquet.hadoop.util.counters.BenchmarkCounter;\nimport org.apache.parquet.internal.column.columnindex.OffsetIndex;\nimport org.apache.parquet.internal.filter2.columnindex.ColumnIndexFilter;\nimport org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore;\nimport org.apache.parquet.internal.filter2.columnindex.RowRanges;\nimport org.apache.parquet.io.InputFile;\nimport org.apache.parquet.io.ParquetDecodingException;\nimport org.apache.parquet.io.SeekableInputStream;\nimport org.apache.parquet.schema.MessageType;\nimport org.apache.parquet.schema.PrimitiveType;\nimport org.apache.spark.sql.execution.metric.SQLMetric;\n\nimport org.apache.comet.IcebergApi;\n\nimport static org.apache.parquet.hadoop.ParquetFileWriter.EFMAGIC;\nimport static org.apache.parquet.hadoop.ParquetFileWriter.MAGIC;\n\nimport static org.apache.comet.parquet.RowGroupFilter.FilterLevel.BLOOMFILTER;\nimport static org.apache.comet.parquet.RowGroupFilter.FilterLevel.DICTIONARY;\nimport static org.apache.comet.parquet.RowGroupFilter.FilterLevel.STATISTICS;\n\n/**\n * A Parquet file reader. Mostly followed {@code ParquetFileReader} in {@code parquet-mr}, but with\n * customizations & optimizations for Comet.\n */\n@IcebergApi\npublic class FileReader implements Closeable {\n private static final Logger LOG = LoggerFactory.getLogger(FileReader.class);\n\n private final ParquetMetadataConverter converter;\n private final SeekableInputStream f;\n private final InputFile file;\n private final Map metrics;\n private final Map paths = new HashMap<>();\n private final FileMetaData fileMetaData; // may be null\n private final List blocks;\n private final List blockIndexStores;\n private final List blockRowRanges;\n private final CRC32 crc;\n private final ParquetMetadata footer;\n\n /**\n * Read configurations come from two options: - options: these are options defined & specified\n * from 'parquet-mr' library - cometOptions: these are Comet-specific options, for the features\n * introduced in Comet's Parquet implementation\n */\n private final ParquetReadOptions options;\n\n private final ReadOptions cometOptions;\n\n private int currentBlock = 0;\n private RowGroupReader currentRowGroup = null;\n private InternalFileDecryptor fileDecryptor;\n\n FileReader(InputFile file, ParquetReadOptions options, ReadOptions cometOptions)\n throws IOException {\n this(file, null, options, cometOptions, null);\n }\n\n /** This constructor is called from Apache Iceberg. */\n @IcebergApi\n public FileReader(\n WrappedInputFile file,\n ReadOptions cometOptions,\n Map properties,\n Long start,\n Long length,\n byte[] fileEncryptionKey,\n byte[] fileAADPrefix)\n throws IOException {\n ParquetReadOptions options =\n buildParquetReadOptions(\n new Configuration(), properties, start, length, fileEncryptionKey, fileAADPrefix);\n this.converter = new ParquetMetadataConverter(options);\n this.file = file;\n this.f = file.newStream();\n this.options = options;\n this.cometOptions = cometOptions;\n this.metrics = null;\n try {\n this.footer = readFooter(file, options, f, converter);\n } catch (Exception e) {\n // In case that reading footer throws an exception in the constructor, the new stream\n // should be closed. Otherwise, there's no way to close this outside.\n f.close();\n throw e;\n }\n this.fileMetaData = footer.getFileMetaData();\n this.fileDecryptor = fileMetaData.getFileDecryptor(); // must be called before filterRowGroups!\n if (null != fileDecryptor && fileDecryptor.plaintextFile()) {\n this.fileDecryptor = null; // Plaintext file. No need in decryptor\n }\n\n this.blocks = footer.getBlocks(); // filter row group in iceberg\n this.blockIndexStores = listWithNulls(this.blocks.size());\n this.blockRowRanges = listWithNulls(this.blocks.size());\n for (ColumnDescriptor col : footer.getFileMetaData().getSchema().getColumns()) {\n paths.put(ColumnPath.get(col.getPath()), col);\n }\n this.crc = options.usePageChecksumVerification() ? new CRC32() : null;\n }\n\n FileReader(\n InputFile file,\n ParquetReadOptions options,\n ReadOptions cometOptions,\n Map metrics)\n throws IOException {\n this(file, null, options, cometOptions, metrics);\n }\n\n FileReader(\n InputFile file,\n ParquetMetadata footer,\n ParquetReadOptions options,\n ReadOptions cometOptions,\n Map metrics)\n throws IOException {\n this.converter = new ParquetMetadataConverter(options);\n this.file = file;\n this.f = file.newStream();\n this.options = options;\n this.cometOptions = cometOptions;\n this.metrics = metrics;\n if (footer == null) {\n try {\n footer = readFooter(file, options, f, converter);\n } catch (Exception e) {\n // In case that reading footer throws an exception in the constructor, the new stream\n // should be closed. Otherwise, there's no way to close this outside.\n f.close();\n throw e;\n }\n }\n this.footer = footer;\n this.fileMetaData = footer.getFileMetaData();\n this.fileDecryptor = fileMetaData.getFileDecryptor(); // must be called before filterRowGroups!\n if (null != fileDecryptor && fileDecryptor.plaintextFile()) {\n this.fileDecryptor = null; // Plaintext file. No need in decryptor\n }\n\n this.blocks = filterRowGroups(footer.getBlocks());\n this.blockIndexStores = listWithNulls(this.blocks.size());\n this.blockRowRanges = listWithNulls(this.blocks.size());\n for (ColumnDescriptor col : footer.getFileMetaData().getSchema().getColumns()) {\n paths.put(ColumnPath.get(col.getPath()), col);\n }\n this.crc = options.usePageChecksumVerification() ? new CRC32() : null;\n }\n\n /** Returns the footer of the Parquet file being read. */\n ParquetMetadata getFooter() {\n return this.footer;\n }\n\n /** Returns the metadata of the Parquet file being read. */\n FileMetaData getFileMetaData() {\n return this.fileMetaData;\n }\n\n /** Returns the input stream of the Parquet file being read. */\n public SeekableInputStream getInputStream() {\n return this.f;\n }\n\n /** Returns the Parquet options for reading the file. */\n public ParquetReadOptions getOptions() {\n return this.options;\n }\n\n /** Returns all the row groups of this reader (after applying row group filtering). */\n public List getRowGroups() {\n return blocks;\n }\n\n /** Sets the projected columns to be read later via {@link #readNextRowGroup()} */\n public void setRequestedSchema(List projection) {\n paths.clear();\n for (ColumnDescriptor col : projection) {\n paths.put(ColumnPath.get(col.getPath()), col);\n }\n }\n\n /** This method is called from Apache Iceberg. */\n @IcebergApi\n public void setRequestedSchemaFromSpecs(List specList) {\n paths.clear();\n for (ParquetColumnSpec colSpec : specList) {\n ColumnDescriptor descriptor = Utils.buildColumnDescriptor(colSpec);\n paths.put(ColumnPath.get(colSpec.getPath()), descriptor);\n }\n }\n\n private static ParquetReadOptions buildParquetReadOptions(\n Configuration conf,\n Map properties,\n Long start,\n Long length,\n byte[] fileEncryptionKey,\n byte[] fileAADPrefix) {\n\n // Iceberg remove these read properties when building the ParquetReadOptions.\n // We want build the exact same ParquetReadOptions as Iceberg's.\n Collection readPropertiesToRemove =\n Set.of(\n ParquetInputFormat.UNBOUND_RECORD_FILTER,\n ParquetInputFormat.FILTER_PREDICATE,\n ParquetInputFormat.READ_SUPPORT_CLASS,\n EncryptionPropertiesFactory.CRYPTO_FACTORY_CLASS_PROPERTY_NAME);\n\n for (String property : readPropertiesToRemove) {\n conf.unset(property);\n }\n\n ParquetReadOptions.Builder optionsBuilder = HadoopReadOptions.builder(conf);\n for (Map.Entry entry : properties.entrySet()) {\n optionsBuilder.set(entry.getKey(), entry.getValue());\n }\n\n if (start != null && length != null) {\n optionsBuilder.withRange(start, start + length);\n }\n\n if (fileEncryptionKey != null) {\n FileDecryptionProperties fileDecryptionProperties =\n FileDecryptionProperties.builder()\n .withFooterKey(fileEncryptionKey)\n .withAADPrefix(fileAADPrefix)\n .build();\n optionsBuilder.withDecryption(fileDecryptionProperties);\n }\n\n return optionsBuilder.build();\n }\n\n /**\n * Gets the total number of records across all row groups (after applying row group filtering).\n */\n public long getRecordCount() {\n long total = 0;\n for (BlockMetaData block : blocks) {\n total += block.getRowCount();\n }\n return total;\n }\n\n /**\n * Gets the total number of records across all row groups (after applying both row group filtering\n * and page-level column index filtering).\n */\n public long getFilteredRecordCount() {\n if (!options.useColumnIndexFilter()\n || !FilterCompat.isFilteringRequired(options.getRecordFilter())) {\n return getRecordCount();\n }\n long total = 0;\n for (int i = 0, n = blocks.size(); i < n; ++i) {\n total += getRowRanges(i).rowCount();\n }\n return total;\n }\n\n /** Skips the next row group. Returns false if there's no row group to skip. Otherwise, true. */\n @IcebergApi\n public boolean skipNextRowGroup() {\n return advanceToNextBlock();\n }\n\n /**\n * Returns the next row group to read (after applying row group filtering), or null if there's no\n * more row group.\n */\n @IcebergApi\n public RowGroupReader readNextRowGroup() throws IOException {\n if (currentBlock == blocks.size()) {\n return null;\n }\n BlockMetaData block = blocks.get(currentBlock);\n if (block.getRowCount() == 0) {\n throw new RuntimeException(\"Illegal row group of 0 rows\");\n }\n this.currentRowGroup = new RowGroupReader(block.getRowCount(), block.getRowIndexOffset());\n // prepare the list of consecutive parts to read them in one scan\n List allParts = new ArrayList<>();\n ConsecutivePartList currentParts = null;\n for (ColumnChunkMetaData mc : block.getColumns()) {\n ColumnPath pathKey = mc.getPath();\n ColumnDescriptor columnDescriptor = paths.get(pathKey);\n if (columnDescriptor != null) {\n BenchmarkCounter.incrementTotalBytes(mc.getTotalSize());\n long startingPos = mc.getStartingPos();\n boolean mergeRanges = cometOptions.isIOMergeRangesEnabled();\n int mergeRangeDelta = cometOptions.getIOMergeRangesDelta();\n\n // start a new list if -\n // it is the first part or\n // the part is consecutive or\n // the part is not consecutive but within the merge range\n if (currentParts == null\n || (!mergeRanges && currentParts.endPos() != startingPos)\n || (mergeRanges && startingPos - currentParts.endPos() > mergeRangeDelta)) {\n currentParts = new ConsecutivePartList(startingPos);\n allParts.add(currentParts);\n }\n // if we are in a consecutive part list and there is a gap in between the parts,\n // we treat the gap as a skippable chunk\n long delta = startingPos - currentParts.endPos();\n if (mergeRanges && delta > 0 && delta <= mergeRangeDelta) {\n // add a chunk that will be skipped because it has no column descriptor\n currentParts.addChunk(new ChunkDescriptor(null, null, startingPos, delta));\n }\n currentParts.addChunk(\n new ChunkDescriptor(columnDescriptor, mc, startingPos, mc.getTotalSize()));\n }\n }\n // actually read all the chunks\n return readChunks(block, allParts, new ChunkListBuilder());\n }\n\n /**\n * Returns the next row group to read (after applying both row group filtering and page level\n * column index filtering), or null if there's no more row group.\n */\n public PageReadStore readNextFilteredRowGroup() throws IOException {\n if (currentBlock == blocks.size()) {\n return null;\n }\n if (!options.useColumnIndexFilter()\n || !FilterCompat.isFilteringRequired(options.getRecordFilter())) {\n return readNextRowGroup();\n }\n BlockMetaData block = blocks.get(currentBlock);\n if (block.getRowCount() == 0) {\n throw new RuntimeException(\"Illegal row group of 0 rows\");\n }\n ColumnIndexStore ciStore = getColumnIndexReader(currentBlock);\n RowRanges rowRanges = getRowRanges(currentBlock);\n long rowCount = rowRanges.rowCount();\n if (rowCount == 0) {\n // There are no matching rows -> skipping this row-group\n advanceToNextBlock();\n return readNextFilteredRowGroup();\n }\n if (rowCount == block.getRowCount()) {\n // All rows are matching -> fall back to the non-filtering path\n return readNextRowGroup();\n }\n\n this.currentRowGroup = new RowGroupReader(rowRanges);\n // prepare the list of consecutive parts to read them in one scan\n ChunkListBuilder builder = new ChunkListBuilder();\n List allParts = new ArrayList<>();\n ConsecutivePartList currentParts = null;\n for (ColumnChunkMetaData mc : block.getColumns()) {\n ColumnPath pathKey = mc.getPath();\n ColumnDescriptor columnDescriptor = paths.get(pathKey);\n if (columnDescriptor != null) {\n OffsetIndex offsetIndex = ciStore.getOffsetIndex(mc.getPath());\n IndexFilter indexFilter = new IndexFilter(rowRanges, offsetIndex, block.getRowCount());\n OffsetIndex filteredOffsetIndex = indexFilter.filterOffsetIndex();\n for (IndexFilter.OffsetRange range :\n indexFilter.calculateOffsetRanges(filteredOffsetIndex, mc)) {\n BenchmarkCounter.incrementTotalBytes(range.length);\n long startingPos = range.offset;\n // first part or not consecutive => new list\n if (currentParts == null || currentParts.endPos() != startingPos) {\n currentParts = new ConsecutivePartList(startingPos);\n allParts.add(currentParts);\n }\n ChunkDescriptor chunkDescriptor =\n new ChunkDescriptor(columnDescriptor, mc, startingPos, range.length);\n currentParts.addChunk(chunkDescriptor);\n builder.setOffsetIndex(chunkDescriptor, filteredOffsetIndex);\n }\n }\n }\n // actually read all the chunks\n return readChunks(block, allParts, builder);\n }\n\n // Visible for testing\n ColumnIndexReader getColumnIndexReader(int blockIndex) {\n ColumnIndexReader ciStore = blockIndexStores.get(blockIndex);\n if (ciStore == null) {\n ciStore = ColumnIndexReader.create(blocks.get(blockIndex), paths.keySet(), fileDecryptor, f);\n blockIndexStores.set(blockIndex, ciStore);\n }\n return ciStore;\n }\n\n private RowGroupReader readChunks(\n BlockMetaData block, List allParts, ChunkListBuilder builder)\n throws IOException {\n if (shouldReadParallel()) {\n readAllPartsParallel(allParts, builder);\n } else {\n for (ConsecutivePartList consecutiveChunks : allParts) {\n consecutiveChunks.readAll(f, builder);\n }\n }\n for (Chunk chunk : builder.build()) {\n readChunkPages(chunk, block);\n }\n\n advanceToNextBlock();\n\n return currentRowGroup;\n }\n\n private boolean shouldReadParallel() {\n if (file instanceof CometInputFile) {\n URI uri = ((CometInputFile) file).getPath().toUri();\n return shouldReadParallel(cometOptions, uri.getScheme());\n }\n\n return false;\n }\n\n static boolean shouldReadParallel(ReadOptions options, String scheme) {\n return options.isParallelIOEnabled() && shouldReadParallelForScheme(scheme);\n }\n\n private static boolean shouldReadParallelForScheme(String scheme) {\n if (scheme == null) {\n return false;\n }\n\n switch (scheme) {\n case \"s3a\":\n // Only enable parallel read for S3, so far.\n return true;\n default:\n return false;\n }\n }\n\n static class ReadRange {\n\n long offset = 0;\n long length = 0;\n List buffers = new ArrayList<>();\n\n @Override\n public String toString() {\n return \"ReadRange{\"\n + \"offset=\"\n + offset\n + \", length=\"\n + length\n + \", numBuffers=\"\n + buffers.size()\n + '}';\n }\n }\n\n List getReadRanges(List allParts, int nBuffers) {\n int nThreads = cometOptions.parallelIOThreadPoolSize();\n long buffersPerThread = nBuffers / nThreads + 1;\n boolean adjustSkew = cometOptions.adjustReadRangesSkew();\n List allRanges = new ArrayList<>();\n for (ConsecutivePartList consecutiveChunk : allParts) {\n ReadRange readRange = null;\n long offset = consecutiveChunk.offset;\n for (int i = 0; i < consecutiveChunk.buffers.size(); i++) {\n if ((adjustSkew && (i % buffersPerThread == 0)) || i == 0) {\n readRange = new ReadRange();\n allRanges.add(readRange);\n readRange.offset = offset;\n }\n ByteBuffer b = consecutiveChunk.buffers.get(i);\n readRange.length += b.capacity();\n readRange.buffers.add(b);\n offset += b.capacity();\n }\n }\n if (LOG.isDebugEnabled()) {\n StringBuilder sb = new StringBuilder();\n for (int i = 0; i < allRanges.size(); i++) {\n sb.append(allRanges.get(i).toString());\n if (i < allRanges.size() - 1) {\n sb.append(\",\");\n }\n }\n LOG.debug(\"Read Ranges: {}\", sb);\n }\n return allRanges;\n }\n\n private void readAllRangesParallel(List allRanges) {\n int nThreads = cometOptions.parallelIOThreadPoolSize();\n ExecutorService threadPool = CometFileReaderThreadPool.getOrCreateThreadPool(nThreads);\n List> futures = new ArrayList<>();\n\n for (ReadRange readRange : allRanges) {\n futures.add(\n threadPool.submit(\n () -> {\n SeekableInputStream inputStream = null;\n try {\n if (file instanceof CometInputFile) {\n // limit the max read ahead to length of the range\n inputStream =\n (((CometInputFile) file).newStream(readRange.offset, readRange.length));\n LOG.debug(\n \"Opened new input file: {}, at offset: {}\",\n ((CometInputFile) file).getPath().getName(),\n readRange.offset);\n } else {\n inputStream = file.newStream();\n }\n long curPos = readRange.offset;\n for (ByteBuffer buffer : readRange.buffers) {\n inputStream.seek(curPos);\n LOG.debug(\n \"Thread: {} Offset: {} Size: {}\",\n Thread.currentThread().getId(),\n curPos,\n buffer.capacity());\n inputStream.readFully(buffer);\n buffer.flip();\n curPos += buffer.capacity();\n } // for\n } finally {\n if (inputStream != null) {\n inputStream.close();\n }\n }\n\n return null;\n }));\n }\n for (Future future : futures) {\n try {\n future.get();\n } catch (InterruptedException | ExecutionException e) {\n throw new RuntimeException(e);\n }\n }\n }\n\n /**\n * Read all the consecutive part list objects in parallel.\n *\n * @param allParts all consecutive parts\n * @param builder chunk list builder\n */\n public void readAllPartsParallel(List allParts, ChunkListBuilder builder)\n throws IOException {\n int nBuffers = 0;\n for (ConsecutivePartList consecutiveChunks : allParts) {\n consecutiveChunks.allocateReadBuffers();\n nBuffers += consecutiveChunks.buffers.size();\n }\n List allRanges = getReadRanges(allParts, nBuffers);\n\n long startNs = System.nanoTime();\n readAllRangesParallel(allRanges);\n\n for (ConsecutivePartList consecutiveChunks : allParts) {\n consecutiveChunks.setReadMetrics(startNs);\n ByteBufferInputStream stream;\n stream = ByteBufferInputStream.wrap(consecutiveChunks.buffers);\n // report in a counter the data we just scanned\n BenchmarkCounter.incrementBytesRead(consecutiveChunks.length);\n for (int i = 0; i < consecutiveChunks.chunks.size(); i++) {\n ChunkDescriptor descriptor = consecutiveChunks.chunks.get(i);\n if (descriptor.col != null) {\n builder.add(descriptor, stream.sliceBuffers(descriptor.size));\n } else {\n stream.skipFully(descriptor.size);\n }\n }\n }\n }\n\n private void readChunkPages(Chunk chunk, BlockMetaData block) throws IOException {\n if (fileDecryptor == null || fileDecryptor.plaintextFile()) {\n currentRowGroup.addColumn(chunk.descriptor.col, chunk.readAllPages());\n return;\n }\n // Encrypted file\n ColumnPath columnPath = ColumnPath.get(chunk.descriptor.col.getPath());\n InternalColumnDecryptionSetup columnDecryptionSetup = fileDecryptor.getColumnSetup(columnPath);\n if (!columnDecryptionSetup.isEncrypted()) { // plaintext column\n currentRowGroup.addColumn(chunk.descriptor.col, chunk.readAllPages());\n } else { // encrypted column\n currentRowGroup.addColumn(\n chunk.descriptor.col,\n chunk.readAllPages(\n columnDecryptionSetup.getMetaDataDecryptor(),\n columnDecryptionSetup.getDataDecryptor(),\n fileDecryptor.getFileAAD(),\n block.getOrdinal(),\n columnDecryptionSetup.getOrdinal()));\n }\n }\n\n private boolean advanceToNextBlock() {\n if (currentBlock == blocks.size()) {\n return false;\n }\n // update the current block and instantiate a dictionary reader for it\n ++currentBlock;\n return true;\n }\n\n public long[] getRowIndices() {\n return getRowIndices(blocks);\n }\n\n public static long[] getRowIndices(List blocks) {\n long[] rowIndices = new long[blocks.size() * 2];\n for (int i = 0, n = blocks.size(); i < n; i++) {\n BlockMetaData block = blocks.get(i);\n rowIndices[i * 2] = getRowIndexOffset(block);\n rowIndices[i * 2 + 1] = block.getRowCount();\n }\n return rowIndices;\n }\n\n // Uses reflection to get row index offset from a Parquet block metadata.\n //\n // The reason reflection is used here is that some Spark versions still depend on a\n // Parquet version where the method `getRowIndexOffset` is not public.\n public static long getRowIndexOffset(BlockMetaData metaData) {\n try {\n Method method = BlockMetaData.class.getMethod(\"getRowIndexOffset\");\n method.setAccessible(true);\n return (long) method.invoke(metaData);\n } catch (Exception e) {\n throw new RuntimeException(\"Error when calling getRowIndexOffset\", e);\n }\n }\n\n private RowRanges getRowRanges(int blockIndex) {\n Preconditions.checkState(\n FilterCompat.isFilteringRequired(options.getRecordFilter()),\n \"Should not be invoked if filter is null or NOOP\");\n RowRanges rowRanges = blockRowRanges.get(blockIndex);\n if (rowRanges == null) {\n rowRanges =\n ColumnIndexFilter.calculateRowRanges(\n options.getRecordFilter(),\n getColumnIndexReader(blockIndex),\n paths.keySet(),\n blocks.get(blockIndex).getRowCount());\n blockRowRanges.set(blockIndex, rowRanges);\n }\n return rowRanges;\n }\n\n private static ParquetMetadata readFooter(\n InputFile file,\n ParquetReadOptions options,\n SeekableInputStream f,\n ParquetMetadataConverter converter)\n throws IOException {\n long fileLen = file.getLength();\n String filePath = file.toString();\n LOG.debug(\"File length {}\", fileLen);\n\n int FOOTER_LENGTH_SIZE = 4;\n\n // MAGIC + data + footer + footerIndex + MAGIC\n if (fileLen < MAGIC.length + FOOTER_LENGTH_SIZE + MAGIC.length) {\n throw new RuntimeException(\n filePath + \" is not a Parquet file (length is too low: \" + fileLen + \")\");\n }\n\n // Read footer length and magic string - with a single seek\n byte[] magic = new byte[MAGIC.length];\n long fileMetadataLengthIndex = fileLen - magic.length - FOOTER_LENGTH_SIZE;\n LOG.debug(\"reading footer index at {}\", fileMetadataLengthIndex);\n f.seek(fileMetadataLengthIndex);\n int fileMetadataLength = BytesUtils.readIntLittleEndian(f);\n f.readFully(magic);\n\n boolean encryptedFooterMode;\n if (Arrays.equals(MAGIC, magic)) {\n encryptedFooterMode = false;\n } else if (Arrays.equals(EFMAGIC, magic)) {\n encryptedFooterMode = true;\n } else {\n throw new RuntimeException(\n filePath\n + \" is not a Parquet file. Expected magic number \"\n + \"at tail, but found \"\n + Arrays.toString(magic));\n }\n\n long fileMetadataIndex = fileMetadataLengthIndex - fileMetadataLength;\n LOG.debug(\"read footer length: {}, footer index: {}\", fileMetadataLength, fileMetadataIndex);\n if (fileMetadataIndex < magic.length || fileMetadataIndex >= fileMetadataLengthIndex) {\n throw new RuntimeException(\n \"corrupted file: the footer index is not within the file: \" + fileMetadataIndex);\n }\n f.seek(fileMetadataIndex);\n\n FileDecryptionProperties fileDecryptionProperties = options.getDecryptionProperties();\n InternalFileDecryptor fileDecryptor = null;\n if (null != fileDecryptionProperties) {\n fileDecryptor = new InternalFileDecryptor(fileDecryptionProperties);\n }\n\n // Read all the footer bytes in one time to avoid multiple read operations,\n // since it can be pretty time consuming for a single read operation in HDFS.\n byte[] footerBytes = new byte[fileMetadataLength];\n f.readFully(footerBytes);\n ByteBuffer footerBytesBuffer = ByteBuffer.wrap(footerBytes);\n LOG.debug(\"Finished to read all footer bytes.\");\n InputStream footerBytesStream = ByteBufferInputStream.wrap(footerBytesBuffer);\n\n // Regular file, or encrypted file with plaintext footer\n if (!encryptedFooterMode) {\n return converter.readParquetMetadata(\n footerBytesStream, options.getMetadataFilter(), fileDecryptor, false, fileMetadataLength);\n }\n\n // Encrypted file with encrypted footer\n if (fileDecryptor == null) {\n throw new ParquetCryptoRuntimeException(\n \"Trying to read file with encrypted footer. \" + \"No keys available\");\n }\n FileCryptoMetaData fileCryptoMetaData = Util.readFileCryptoMetaData(footerBytesStream);\n fileDecryptor.setFileCryptoMetaData(\n fileCryptoMetaData.getEncryption_algorithm(), true, fileCryptoMetaData.getKey_metadata());\n // footer length is required only for signed plaintext footers\n return converter.readParquetMetadata(\n footerBytesStream, options.getMetadataFilter(), fileDecryptor, true, 0);\n }\n\n private List filterRowGroups(List blocks) {\n return filterRowGroups(options, blocks, this);\n }\n\n public static List filterRowGroups(\n ParquetReadOptions options, List blocks, FileReader fileReader) {\n FilterCompat.Filter recordFilter = options.getRecordFilter();\n if (FilterCompat.isFilteringRequired(recordFilter)) {\n // set up data filters based on configured levels\n List levels = new ArrayList<>();\n\n if (options.useStatsFilter()) {\n levels.add(STATISTICS);\n }\n\n if (options.useDictionaryFilter()) {\n levels.add(DICTIONARY);\n }\n\n if (options.useBloomFilter()) {\n levels.add(BLOOMFILTER);\n }\n return RowGroupFilter.filterRowGroups(levels, recordFilter, blocks, fileReader);\n }\n\n return blocks;\n }\n\n public static List filterRowGroups(\n ParquetReadOptions options, List blocks, MessageType schema) {\n FilterCompat.Filter recordFilter = options.getRecordFilter();\n if (FilterCompat.isFilteringRequired(recordFilter)) {\n // set up data filters based on configured levels\n List levels = new ArrayList<>();\n\n if (options.useStatsFilter()) {\n levels.add(STATISTICS);\n }\n\n if (options.useDictionaryFilter()) {\n levels.add(DICTIONARY);\n }\n\n if (options.useBloomFilter()) {\n levels.add(BLOOMFILTER);\n }\n return RowGroupFilter.filterRowGroups(levels, recordFilter, blocks, schema);\n }\n\n return blocks;\n }\n\n private static List listWithNulls(int size) {\n return Stream.generate(() -> (T) null).limit(size).collect(Collectors.toList());\n }\n\n public void closeStream() throws IOException {\n if (f != null) {\n f.close();\n }\n }\n\n @IcebergApi\n @Override\n public void close() throws IOException {\n try {\n if (f != null) {\n f.close();\n }\n } finally {\n options.getCodecFactory().release();\n }\n }\n\n /**\n * Builder to concatenate the buffers of the discontinuous parts for the same column. These parts\n * are generated as a result of the column-index based filtering when some pages might be skipped\n * at reading.\n */\n private class ChunkListBuilder {\n private class ChunkData {\n final List buffers = new ArrayList<>();\n OffsetIndex offsetIndex;\n }\n\n private final Map map = new HashMap<>();\n\n void add(ChunkDescriptor descriptor, List buffers) {\n ChunkListBuilder.ChunkData data = map.get(descriptor);\n if (data == null) {\n data = new ChunkData();\n map.put(descriptor, data);\n }\n data.buffers.addAll(buffers);\n }\n\n void setOffsetIndex(ChunkDescriptor descriptor, OffsetIndex offsetIndex) {\n ChunkData data = map.get(descriptor);\n if (data == null) {\n data = new ChunkData();\n map.put(descriptor, data);\n }\n data.offsetIndex = offsetIndex;\n }\n\n List build() {\n List chunks = new ArrayList<>();\n for (Map.Entry entry : map.entrySet()) {\n ChunkDescriptor descriptor = entry.getKey();\n ChunkData data = entry.getValue();\n chunks.add(new Chunk(descriptor, data.buffers, data.offsetIndex));\n }\n return chunks;\n }\n }\n\n /** The data for a column chunk */\n private class Chunk {\n private final ChunkDescriptor descriptor;\n private final ByteBufferInputStream stream;\n final OffsetIndex offsetIndex;\n\n /**\n * @param descriptor descriptor for the chunk\n * @param buffers ByteBuffers that contain the chunk\n * @param offsetIndex the offset index for this column; might be null\n */\n Chunk(ChunkDescriptor descriptor, List buffers, OffsetIndex offsetIndex) {\n this.descriptor = descriptor;\n this.stream = ByteBufferInputStream.wrap(buffers);\n this.offsetIndex = offsetIndex;\n }\n\n protected PageHeader readPageHeader(BlockCipher.Decryptor blockDecryptor, byte[] pageHeaderAAD)\n throws IOException {\n return Util.readPageHeader(stream, blockDecryptor, pageHeaderAAD);\n }\n\n /**\n * Calculate checksum of input bytes, throw decoding exception if it does not match the provided\n * reference crc\n */\n private void verifyCrc(int referenceCrc, byte[] bytes, String exceptionMsg) {\n crc.reset();\n crc.update(bytes);\n if (crc.getValue() != ((long) referenceCrc & 0xffffffffL)) {\n throw new ParquetDecodingException(exceptionMsg);\n }\n }\n\n private ColumnPageReader readAllPages() throws IOException {\n return readAllPages(null, null, null, -1, -1);\n }\n\n private ColumnPageReader readAllPages(\n BlockCipher.Decryptor headerBlockDecryptor,\n BlockCipher.Decryptor pageBlockDecryptor,\n byte[] aadPrefix,\n int rowGroupOrdinal,\n int columnOrdinal)\n throws IOException {\n List pagesInChunk = new ArrayList<>();\n DictionaryPage dictionaryPage = null;\n PrimitiveType type =\n fileMetaData.getSchema().getType(descriptor.col.getPath()).asPrimitiveType();\n\n long valuesCountReadSoFar = 0;\n int dataPageCountReadSoFar = 0;\n byte[] dataPageHeaderAAD = null;\n if (null != headerBlockDecryptor) {\n dataPageHeaderAAD =\n AesCipher.createModuleAAD(\n aadPrefix,\n ModuleCipherFactory.ModuleType.DataPageHeader,\n rowGroupOrdinal,\n columnOrdinal,\n getPageOrdinal(dataPageCountReadSoFar));\n }\n while (hasMorePages(valuesCountReadSoFar, dataPageCountReadSoFar)) {\n byte[] pageHeaderAAD = dataPageHeaderAAD;\n if (null != headerBlockDecryptor) {\n // Important: this verifies file integrity (makes sure dictionary page had not been\n // removed)\n if (null == dictionaryPage && descriptor.metadata.hasDictionaryPage()) {\n pageHeaderAAD =\n AesCipher.createModuleAAD(\n aadPrefix,\n ModuleCipherFactory.ModuleType.DictionaryPageHeader,\n rowGroupOrdinal,\n columnOrdinal,\n -1);\n } else {\n int pageOrdinal = getPageOrdinal(dataPageCountReadSoFar);\n AesCipher.quickUpdatePageAAD(dataPageHeaderAAD, pageOrdinal);\n }\n }\n\n PageHeader pageHeader = readPageHeader(headerBlockDecryptor, pageHeaderAAD);\n int uncompressedPageSize = pageHeader.getUncompressed_page_size();\n int compressedPageSize = pageHeader.getCompressed_page_size();\n final BytesInput pageBytes;\n switch (pageHeader.type) {\n case DICTIONARY_PAGE:\n // there is only one dictionary page per column chunk\n if (dictionaryPage != null) {\n throw new ParquetDecodingException(\n \"more than one dictionary page in column \" + descriptor.col);\n }\n pageBytes = this.readAsBytesInput(compressedPageSize);\n if (options.usePageChecksumVerification() && pageHeader.isSetCrc()) {\n verifyCrc(\n pageHeader.getCrc(),\n pageBytes.toByteArray(),\n \"could not verify dictionary page integrity, CRC checksum verification failed\");\n }\n DictionaryPageHeader dicHeader = pageHeader.getDictionary_page_header();\n dictionaryPage =\n new DictionaryPage(\n pageBytes,\n uncompressedPageSize,\n dicHeader.getNum_values(),\n converter.getEncoding(dicHeader.getEncoding()));\n // Copy crc to new page, used for testing\n if (pageHeader.isSetCrc()) {\n dictionaryPage.setCrc(pageHeader.getCrc());\n }\n break;\n\n case DATA_PAGE:\n DataPageHeader dataHeaderV1 = pageHeader.getData_page_header();\n pageBytes = this.readAsBytesInput(compressedPageSize);\n if (options.usePageChecksumVerification() && pageHeader.isSetCrc()) {\n verifyCrc(\n pageHeader.getCrc(),\n pageBytes.toByteArray(),\n \"could not verify page integrity, CRC checksum verification failed\");\n }\n DataPageV1 dataPageV1 =\n new DataPageV1(\n pageBytes,\n dataHeaderV1.getNum_values(),\n uncompressedPageSize,\n converter.fromParquetStatistics(\n getFileMetaData().getCreatedBy(), dataHeaderV1.getStatistics(), type),\n converter.getEncoding(dataHeaderV1.getRepetition_level_encoding()),\n converter.getEncoding(dataHeaderV1.getDefinition_level_encoding()),\n converter.getEncoding(dataHeaderV1.getEncoding()));\n // Copy crc to new page, used for testing\n if (pageHeader.isSetCrc()) {\n dataPageV1.setCrc(pageHeader.getCrc());\n }\n pagesInChunk.add(dataPageV1);\n valuesCountReadSoFar += dataHeaderV1.getNum_values();\n ++dataPageCountReadSoFar;\n break;\n\n case DATA_PAGE_V2:\n DataPageHeaderV2 dataHeaderV2 = pageHeader.getData_page_header_v2();\n int dataSize =\n compressedPageSize\n - dataHeaderV2.getRepetition_levels_byte_length()\n - dataHeaderV2.getDefinition_levels_byte_length();\n pagesInChunk.add(\n new DataPageV2(\n dataHeaderV2.getNum_rows(),\n dataHeaderV2.getNum_nulls(),\n dataHeaderV2.getNum_values(),\n this.readAsBytesInput(dataHeaderV2.getRepetition_levels_byte_length()),\n this.readAsBytesInput(dataHeaderV2.getDefinition_levels_byte_length()),\n converter.getEncoding(dataHeaderV2.getEncoding()),\n this.readAsBytesInput(dataSize),\n uncompressedPageSize,\n converter.fromParquetStatistics(\n getFileMetaData().getCreatedBy(), dataHeaderV2.getStatistics(), type),\n dataHeaderV2.isIs_compressed()));\n valuesCountReadSoFar += dataHeaderV2.getNum_values();\n ++dataPageCountReadSoFar;\n break;\n\n default:\n LOG.debug(\n \"skipping page of type {} of size {}\", pageHeader.getType(), compressedPageSize);\n stream.skipFully(compressedPageSize);\n break;\n }\n }\n if (offsetIndex == null && valuesCountReadSoFar != descriptor.metadata.getValueCount()) {\n // Would be nice to have a CorruptParquetFileException or something as a subclass?\n throw new IOException(\n \"Expected \"\n + descriptor.metadata.getValueCount()\n + \" values in column chunk at \"\n + file\n + \" offset \"\n + descriptor.metadata.getFirstDataPageOffset()\n + \" but got \"\n + valuesCountReadSoFar\n + \" values instead over \"\n + pagesInChunk.size()\n + \" pages ending at file offset \"\n + (descriptor.fileOffset + stream.position()));\n }\n CompressionCodecFactory.BytesInputDecompressor decompressor =\n options.getCodecFactory().getDecompressor(descriptor.metadata.getCodec());\n return new ColumnPageReader(\n decompressor,\n pagesInChunk,\n dictionaryPage,\n offsetIndex,\n blocks.get(currentBlock).getRowCount(),\n pageBlockDecryptor,\n aadPrefix,\n rowGroupOrdinal,\n columnOrdinal);\n }\n\n private boolean hasMorePages(long valuesCountReadSoFar, int dataPageCountReadSoFar) {\n return offsetIndex == null\n ? valuesCountReadSoFar < descriptor.metadata.getValueCount()\n : dataPageCountReadSoFar < offsetIndex.getPageCount();\n }\n\n private int getPageOrdinal(int dataPageCountReadSoFar) {\n if (null == offsetIndex) {\n return dataPageCountReadSoFar;\n }\n\n return offsetIndex.getPageOrdinal(dataPageCountReadSoFar);\n }\n\n /**\n * @param size the size of the page\n * @return the page\n * @throws IOException if there is an error while reading from the file stream\n */\n public BytesInput readAsBytesInput(int size) throws IOException {\n return BytesInput.from(stream.sliceBuffers(size));\n }\n }\n\n /**\n * Describes a list of consecutive parts to be read at once. A consecutive part may contain whole\n * column chunks or only parts of them (some pages).\n */\n private class ConsecutivePartList {\n private final long offset;\n private final List chunks = new ArrayList<>();\n private long length;\n private final SQLMetric fileReadTimeMetric;\n private final SQLMetric fileReadSizeMetric;\n private final SQLMetric readThroughput;\n List buffers;\n\n /**\n * Constructor\n *\n * @param offset where the first chunk starts\n */\n ConsecutivePartList(long offset) {\n if (metrics != null) {\n this.fileReadTimeMetric = metrics.get(\"ParquetInputFileReadTime\");\n this.fileReadSizeMetric = metrics.get(\"ParquetInputFileReadSize\");\n this.readThroughput = metrics.get(\"ParquetInputFileReadThroughput\");\n } else {\n this.fileReadTimeMetric = null;\n this.fileReadSizeMetric = null;\n this.readThroughput = null;\n }\n this.offset = offset;\n }\n\n /**\n * Adds a chunk to the list. It must be consecutive to the previous chunk.\n *\n * @param descriptor a chunk descriptor\n */\n public void addChunk(ChunkDescriptor descriptor) {\n chunks.add(descriptor);\n length += descriptor.size;\n }\n\n private void allocateReadBuffers() {\n int fullAllocations = Math.toIntExact(length / options.getMaxAllocationSize());\n int lastAllocationSize = Math.toIntExact(length % options.getMaxAllocationSize());\n\n int numAllocations = fullAllocations + (lastAllocationSize > 0 ? 1 : 0);\n this.buffers = new ArrayList<>(numAllocations);\n\n for (int i = 0; i < fullAllocations; i += 1) {\n this.buffers.add(options.getAllocator().allocate(options.getMaxAllocationSize()));\n }\n\n if (lastAllocationSize > 0) {\n this.buffers.add(options.getAllocator().allocate(lastAllocationSize));\n }\n }\n\n /**\n * @param f file to read the chunks from\n * @param builder used to build chunk list to read the pages for the different columns\n * @throws IOException if there is an error while reading from the stream\n */\n public void readAll(SeekableInputStream f, ChunkListBuilder builder) throws IOException {\n f.seek(offset);\n\n allocateReadBuffers();\n long startNs = System.nanoTime();\n\n for (ByteBuffer buffer : buffers) {\n f.readFully(buffer);\n buffer.flip();\n }\n setReadMetrics(startNs);\n\n // report in a counter the data we just scanned\n BenchmarkCounter.incrementBytesRead(length);\n ByteBufferInputStream stream = ByteBufferInputStream.wrap(buffers);\n for (int i = 0; i < chunks.size(); i++) {\n ChunkDescriptor descriptor = chunks.get(i);\n if (descriptor.col != null) {\n builder.add(descriptor, stream.sliceBuffers(descriptor.size));\n } else {\n stream.skipFully(descriptor.size);\n }\n }\n }\n\n private void setReadMetrics(long startNs) {\n long totalFileReadTimeNs = System.nanoTime() - startNs;\n double sizeInMb = ((double) length) / (1024 * 1024);\n double timeInSec = ((double) totalFileReadTimeNs) / 1000_0000_0000L;\n double throughput = sizeInMb / timeInSec;\n LOG.debug(\n \"Comet: File Read stats: Length: {} MB, Time: {} secs, throughput: {} MB/sec \",\n sizeInMb,\n timeInSec,\n throughput);\n if (fileReadTimeMetric != null) {\n fileReadTimeMetric.add(totalFileReadTimeNs);\n }\n if (fileReadSizeMetric != null) {\n fileReadSizeMetric.add(length);\n }\n if (readThroughput != null) {\n readThroughput.set(throughput);\n }\n }\n\n /**\n * End position of the last byte of these chunks\n *\n * @return the position following the last byte of these chunks\n */\n public long endPos() {\n return offset + length;\n }\n }\n\n /** Information needed to read a column chunk or a part of it. */\n private static class ChunkDescriptor {\n\n private final ColumnDescriptor col;\n private final ColumnChunkMetaData metadata;\n private final long fileOffset;\n private final long size;\n\n /**\n * @param col column this chunk is part of\n * @param metadata metadata for the column\n * @param fileOffset offset in the file where this chunk starts\n * @param size size of the chunk\n */\n ChunkDescriptor(\n ColumnDescriptor col, ColumnChunkMetaData metadata, long fileOffset, long size) {\n this.col = col;\n this.metadata = metadata;\n this.fileOffset = fileOffset;\n this.size = size;\n }\n\n @Override\n public int hashCode() {\n return col.hashCode();\n }\n\n @Override\n public boolean equals(Object obj) {\n if (this == obj) {\n return true;\n } else if (obj instanceof ChunkDescriptor) {\n return col.equals(((ChunkDescriptor) obj).col);\n } else {\n return false;\n }\n }\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/FooterReader.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.io.IOException;\nimport java.net.URI;\nimport java.net.URISyntaxException;\n\nimport org.apache.hadoop.conf.Configuration;\nimport org.apache.hadoop.fs.Path;\nimport org.apache.parquet.HadoopReadOptions;\nimport org.apache.parquet.ParquetReadOptions;\nimport org.apache.parquet.hadoop.metadata.ParquetMetadata;\nimport org.apache.spark.sql.execution.datasources.PartitionedFile;\n\n/**\n * Copied from Spark's `ParquetFooterReader` in order to avoid shading issue around Parquet.\n *\n *

`FooterReader` is a util class which encapsulates the helper methods of reading parquet file\n * footer.\n */\npublic class FooterReader {\n public static ParquetMetadata readFooter(Configuration configuration, PartitionedFile file)\n throws IOException, URISyntaxException {\n long start = file.start();\n long length = file.length();\n Path filePath = new Path(new URI(file.filePath().toString()));\n CometInputFile inputFile = CometInputFile.fromPath(filePath, configuration);\n ParquetReadOptions readOptions =\n HadoopReadOptions.builder(inputFile.getConf(), inputFile.getPath())\n .withRange(start, start + length)\n .build();\n ReadOptions cometReadOptions = ReadOptions.builder(configuration).build();\n // Use try-with-resources to ensure fd is closed.\n try (FileReader fileReader = new FileReader(inputFile, readOptions, cometReadOptions)) {\n return fileReader.getFooter();\n }\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/IcebergCometNativeBatchReader.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.util.Map;\n\nimport org.apache.hadoop.conf.Configuration;\nimport org.apache.spark.sql.catalyst.InternalRow;\nimport org.apache.spark.sql.execution.metric.SQLMetric;\nimport org.apache.spark.sql.types.StructType;\n\n/**\n * A specialized NativeBatchReader for Iceberg that accepts ParquetMetadata as a Thrift encoded byte\n * array . This allows Iceberg to pass metadata in serialized form with a two-step initialization\n * pattern.\n */\npublic class IcebergCometNativeBatchReader extends NativeBatchReader {\n\n public IcebergCometNativeBatchReader(StructType requiredSchema) {\n super();\n this.sparkSchema = requiredSchema;\n }\n\n /** Initialize the reader using FileInfo instead of PartitionedFile. */\n public void init(\n Configuration conf,\n FileInfo fileInfo,\n byte[] parquetMetadataBytes,\n byte[] nativeFilter,\n int capacity,\n StructType dataSchema,\n boolean isCaseSensitive,\n boolean useFieldId,\n boolean ignoreMissingIds,\n boolean useLegacyDateTimestamp,\n StructType partitionSchema,\n InternalRow partitionValues,\n AbstractColumnReader[] preInitializedReaders,\n Map metrics)\n throws Throwable {\n\n // Set parent fields\n this.conf = conf;\n this.fileInfo = fileInfo;\n this.footer = new ParquetMetadataSerializer().deserialize(parquetMetadataBytes);\n this.nativeFilter = nativeFilter;\n this.capacity = capacity;\n this.dataSchema = dataSchema;\n this.isCaseSensitive = isCaseSensitive;\n this.useFieldId = useFieldId;\n this.ignoreMissingIds = ignoreMissingIds;\n this.useLegacyDateTimestamp = useLegacyDateTimestamp;\n this.partitionSchema = partitionSchema;\n this.partitionValues = partitionValues;\n this.preInitializedReaders = preInitializedReaders;\n this.metrics.clear();\n if (metrics != null) {\n this.metrics.putAll(metrics);\n }\n\n // Call parent init method\n super.init();\n }\n\n public StructType getSparkSchema() {\n return this.sparkSchema;\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/IndexFilter.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.util.ArrayList;\nimport java.util.List;\n\nimport org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;\nimport org.apache.parquet.internal.column.columnindex.OffsetIndex;\nimport org.apache.parquet.internal.filter2.columnindex.RowRanges;\n\npublic class IndexFilter {\n private final RowRanges rowRanges;\n private final OffsetIndex offsetIndex;\n private final long totalRowCount;\n\n public IndexFilter(RowRanges rowRanges, OffsetIndex offsetIndex, long totalRowCount) {\n this.rowRanges = rowRanges;\n this.offsetIndex = offsetIndex;\n this.totalRowCount = totalRowCount;\n }\n\n OffsetIndex filterOffsetIndex() {\n List indexMap = new ArrayList<>();\n for (int i = 0, n = offsetIndex.getPageCount(); i < n; ++i) {\n long from = offsetIndex.getFirstRowIndex(i);\n if (rowRanges.isOverlapping(from, offsetIndex.getLastRowIndex(i, totalRowCount))) {\n indexMap.add(i);\n }\n }\n\n int[] indexArray = new int[indexMap.size()];\n for (int i = 0; i < indexArray.length; i++) {\n indexArray[i] = indexMap.get(i);\n }\n return new FilteredOffsetIndex(offsetIndex, indexArray);\n }\n\n List calculateOffsetRanges(OffsetIndex filteredOffsetIndex, ColumnChunkMetaData cm) {\n List ranges = new ArrayList<>();\n long firstPageOffset = offsetIndex.getOffset(0);\n int n = filteredOffsetIndex.getPageCount();\n\n if (n > 0) {\n OffsetRange currentRange = null;\n\n // Add a range for the dictionary page if required\n long rowGroupOffset = cm.getStartingPos();\n if (rowGroupOffset < firstPageOffset) {\n currentRange = new OffsetRange(rowGroupOffset, (int) (firstPageOffset - rowGroupOffset));\n ranges.add(currentRange);\n }\n\n for (int i = 0; i < n; ++i) {\n long offset = filteredOffsetIndex.getOffset(i);\n int length = filteredOffsetIndex.getCompressedPageSize(i);\n if (currentRange == null || !currentRange.extend(offset, length)) {\n currentRange = new OffsetRange(offset, length);\n ranges.add(currentRange);\n }\n }\n }\n return ranges;\n }\n\n private static class FilteredOffsetIndex implements OffsetIndex {\n private final OffsetIndex offsetIndex;\n private final int[] indexMap;\n\n private FilteredOffsetIndex(OffsetIndex offsetIndex, int[] indexMap) {\n this.offsetIndex = offsetIndex;\n this.indexMap = indexMap;\n }\n\n @Override\n public int getPageOrdinal(int pageIndex) {\n return indexMap[pageIndex];\n }\n\n @Override\n public int getPageCount() {\n return indexMap.length;\n }\n\n @Override\n public long getOffset(int pageIndex) {\n return offsetIndex.getOffset(indexMap[pageIndex]);\n }\n\n @Override\n public int getCompressedPageSize(int pageIndex) {\n return offsetIndex.getCompressedPageSize(indexMap[pageIndex]);\n }\n\n @Override\n public long getFirstRowIndex(int pageIndex) {\n return offsetIndex.getFirstRowIndex(indexMap[pageIndex]);\n }\n\n @Override\n public long getLastRowIndex(int pageIndex, long totalRowCount) {\n int nextIndex = indexMap[pageIndex] + 1;\n return (nextIndex >= offsetIndex.getPageCount()\n ? totalRowCount\n : offsetIndex.getFirstRowIndex(nextIndex))\n - 1;\n }\n }\n\n static class OffsetRange {\n final long offset;\n long length;\n\n private OffsetRange(long offset, int length) {\n this.offset = offset;\n this.length = length;\n }\n\n private boolean extend(long offset, int length) {\n if (this.offset + this.length == offset) {\n this.length += length;\n return true;\n } else {\n return false;\n }\n }\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/LazyColumnReader.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.io.IOException;\n\nimport org.apache.parquet.column.ColumnDescriptor;\nimport org.apache.parquet.column.page.PageReader;\nimport org.apache.spark.sql.types.DataType;\n\nimport org.apache.comet.CometSchemaImporter;\nimport org.apache.comet.vector.CometLazyVector;\nimport org.apache.comet.vector.CometVector;\n\npublic class LazyColumnReader extends ColumnReader {\n\n // Remember the largest skipped index for sanity checking.\n private int lastSkippedRowId = Integer.MAX_VALUE;\n\n // Track whether the underlying page is drained.\n private boolean isPageDrained = true;\n\n // Leftover number of rows that did not skip in the previous batch.\n private int numRowsToSkipFromPrevBatch;\n\n // The lazy vector being updated.\n private final CometLazyVector vector;\n\n LazyColumnReader(\n DataType sparkReadType,\n ColumnDescriptor descriptor,\n CometSchemaImporter importer,\n int batchSize,\n boolean useDecimal128,\n boolean useLegacyDateTimestamp) {\n super(sparkReadType, descriptor, importer, batchSize, useDecimal128, useLegacyDateTimestamp);\n this.batchSize = 0; // the batch size is set later in `readBatch`\n this.vector = new CometLazyVector(sparkReadType, this, useDecimal128);\n }\n\n @Override\n public void setPageReader(PageReader pageReader) throws IOException {\n super.setPageReader(pageReader);\n lastSkippedRowId = Integer.MAX_VALUE;\n isPageDrained = true;\n numRowsToSkipFromPrevBatch = 0;\n currentNumValues = batchSize;\n }\n\n /**\n * Lazily read a batch of 'total' rows for this column. The includes: 1) Skip any unused rows from\n * the previous batch 2) Reset the native columnar batch 3) Reset tracking variables\n *\n * @param total the number of rows in the batch. MUST be <= the number of rows available in this\n * column chunk.\n */\n @Override\n public void readBatch(int total) {\n // Before starting a new batch, take care of the remaining rows to skip from the previous batch.\n tryPageSkip(batchSize);\n numRowsToSkipFromPrevBatch += batchSize - currentNumValues;\n\n // Now first reset the current columnar batch so that it can be used to fill in a new batch\n // of values. Then, keep reading more data pages (via 'readBatch') until the current batch is\n // full, or we have read 'total' number of values.\n Native.resetBatch(nativeHandle);\n\n batchSize = total;\n currentNumValues = 0;\n lastSkippedRowId = -1;\n }\n\n @Override\n public CometVector currentBatch() {\n return vector;\n }\n\n /** Read all rows up to the `batchSize`. Expects no rows are skipped so far. */\n public void readAllBatch() {\n // All rows should be read without any skips so far\n assert (lastSkippedRowId == -1);\n\n readBatch(batchSize - 1, 0);\n }\n\n /**\n * Read at least up to `rowId`. It may read beyond `rowId` if enough rows available in the page.\n * It may skip reading rows before `rowId`. In case `rowId` is already read, return immediately.\n *\n * @param rowId the row index in the batch to read.\n * @return true if `rowId` is newly materialized, or false if `rowId` is already materialized.\n */\n public boolean materializeUpToIfNecessary(int rowId) {\n // Not allowed reading rowId if it may have skipped previously.\n assert (rowId > lastSkippedRowId);\n\n // If `rowId` is already materialized, return immediately.\n if (rowId < currentNumValues) return false;\n\n int numRowsWholePageSkipped = tryPageSkip(rowId);\n readBatch(rowId, numRowsWholePageSkipped);\n return true;\n }\n\n /**\n * Read up to `rowId` (inclusive). If the whole pages are skipped previously in `tryPageSkip()`,\n * pad the number of whole page skipped rows with nulls to the underlying vector before reading.\n *\n * @param rowId the row index in the batch to read.\n * @param numNullRowsToPad the number of nulls to pad before reading.\n */\n private void readBatch(int rowId, int numNullRowsToPad) {\n if (numRowsToSkipFromPrevBatch > 0) {\n // Reaches here only when starting a new batch and the page is previously drained\n readPage();\n isPageDrained = false;\n Native.skipBatch(nativeHandle, numRowsToSkipFromPrevBatch, true);\n numRowsToSkipFromPrevBatch = 0;\n }\n while (rowId >= currentNumValues) {\n int numRowsToRead = batchSize - currentNumValues;\n if (isPageDrained) {\n readPage();\n }\n int[] array = Native.readBatch(nativeHandle, numRowsToRead, numNullRowsToPad);\n int read = array[0];\n isPageDrained = read < numRowsToRead;\n currentNumValues += read;\n currentNumNulls += array[1];\n // No need to update numNullRowsToPad. numNullRowsToPad > 0 means there were whole page skips.\n // That guarantees that the Native.readBatch can read up to rowId in the current page.\n }\n }\n\n /**\n * Try to skip until `rowId` (exclusive). If possible, it skips whole underlying pages without\n * decompressing. In that case, it returns early at the page end, so that the next iteration can\n * lazily decide to `readPage()` or `tryPageSkip()` again.\n *\n * @param rowId the row index in the batch that it tries to skip up until (exclusive).\n * @return the number of rows that the whole page skips were applied.\n */\n private int tryPageSkip(int rowId) {\n int total = rowId - currentNumValues;\n int wholePageSkipped = 0;\n if (total > 0) {\n // First try to skip from the non-drained underlying page.\n int skipped = isPageDrained ? 0 : Native.skipBatch(nativeHandle, total);\n total -= skipped;\n isPageDrained = total > 0;\n if (isPageDrained) {\n ColumnPageReader columnPageReader = (ColumnPageReader) pageReader;\n // It is always `columnPageReader.getPageValueCount() > numRowsToSkipFromPriorBatch`\n int pageValueCount = columnPageReader.getPageValueCount() - numRowsToSkipFromPrevBatch;\n while (pageValueCount <= total) {\n // skip the entire page if the next page is small enough\n columnPageReader.skipPage();\n numRowsToSkipFromPrevBatch = 0;\n total -= pageValueCount;\n wholePageSkipped += pageValueCount;\n pageValueCount = columnPageReader.getPageValueCount();\n }\n }\n\n currentNumValues += skipped + wholePageSkipped;\n currentNumNulls += skipped;\n lastSkippedRowId = currentNumValues - 1;\n }\n return wholePageSkipped;\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/Native.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.util.Map;\n\nimport org.apache.comet.IcebergApi;\nimport org.apache.comet.NativeBase;\n\npublic final class Native extends NativeBase {\n public static int[] readBatch(long handle, int batchSize) {\n return readBatch(handle, batchSize, 0);\n }\n\n public static int skipBatch(long handle, int batchSize) {\n return skipBatch(handle, batchSize, false);\n }\n\n /** Native APIs * */\n\n /**\n * Creates a reader for a primitive Parquet column.\n *\n * @param physicalTypeId id for Parquet physical type\n * @param logicalTypeId id for Parquet logical type\n * @param expectedPhysicalTypeId id for Parquet physical type, converted from Spark read type.\n * This is used for type promotion.\n * @param path the path from the root schema to the column, derived from the method\n * 'ColumnDescriptor#getPath()'.\n * @param maxDl the maximum definition level of the primitive column\n * @param maxRl the maximum repetition level of the primitive column\n * @param bitWidth (only set when logical type is INT) the bit width for the integer type (INT8,\n * INT16, INT32, etc)\n * @param isSigned (only set when logical type is INT) whether it is signed or unsigned int.\n * @param typeLength number of bytes required to store a value of the type, only set when the\n * physical type is FIXED_LEN_BYTE_ARRAY, otherwise it's 0.\n * @param precision (only set when logical type is DECIMAL) precision of the decimal type\n * @param expectedPrecision (only set when logical type is DECIMAL) precision of the decimal type\n * from Spark read schema. This is used for type promotion.\n * @param scale (only set when logical type is DECIMAL) scale of the decimal type\n * @param tu (only set when logical type is TIMESTAMP) unit for the timestamp\n * @param isAdjustedUtc (only set when logical type is TIMESTAMP) whether the timestamp is\n * adjusted to UTC or not\n * @param batchSize the batch size for the columnar read\n * @param useDecimal128 whether to always return 128 bit decimal regardless of precision\n * @param useLegacyDateTimestampOrNTZ whether to read legacy dates/timestamps as it is\n * @return a pointer to a native Parquet column reader created\n */\n public static native long initColumnReader(\n int physicalTypeId,\n int logicalTypeId,\n int expectedPhysicalTypeId,\n String[] path,\n int maxDl,\n int maxRl,\n int bitWidth,\n int expectedBitWidth,\n boolean isSigned,\n int typeLength,\n int precision,\n int expectedPrecision,\n int scale,\n int expectedScale,\n int tu,\n boolean isAdjustedUtc,\n int batchSize,\n boolean useDecimal128,\n boolean useLegacyDateTimestampOrNTZ);\n\n /**\n * Pass a Parquet dictionary page to the native column reader. Note this should only be called\n * once per Parquet column chunk. Otherwise it'll panic.\n *\n * @param handle the handle to the native Parquet column reader\n * @param dictionaryValueCount the number of values in this dictionary\n * @param dictionaryData the actual dictionary page data, including repetition/definition levels\n * as well as values\n * @param encoding the encoding used by the dictionary\n */\n public static native void setDictionaryPage(\n long handle, int dictionaryValueCount, byte[] dictionaryData, int encoding);\n\n /**\n * Passes a Parquet data page V1 to the native column reader.\n *\n * @param handle the handle to the native Parquet column reader\n * @param pageValueCount the number of values in this data page\n * @param pageData the actual page data, which should only contain PLAIN-encoded values.\n * @param valueEncoding the encoding used by the values\n */\n public static native void setPageV1(\n long handle, int pageValueCount, byte[] pageData, int valueEncoding);\n\n /**\n * Passes a Parquet data page V2 to the native column reader.\n *\n * @param handle the handle to the native Parquet column reader\n * @param pageValueCount the number of values in this data page\n * @param defLevelData the data for definition levels\n * @param repLevelData the data for repetition levels\n * @param valueData the data for values\n * @param valueEncoding the encoding used by the values\n */\n public static native void setPageV2(\n long handle,\n int pageValueCount,\n byte[] defLevelData,\n byte[] repLevelData,\n byte[] valueData,\n int valueEncoding);\n\n /**\n * Reset the current columnar batch. This will clear all the content of the batch as well as any\n * internal state such as the current offset.\n *\n * @param handle the handle to the native Parquet column reader\n */\n @IcebergApi\n public static native void resetBatch(long handle);\n\n /**\n * Reads at most 'batchSize' number of rows from the native Parquet column reader. Returns a tuple\n * where the first element is the actual number of rows read (including both nulls and non-nulls),\n * and the second element is the number of nulls read.\n *\n *

If the returned value is < 'batchSize' then it means the current page has been completely\n * drained. In this case, the caller should call {@link Native#setPageV1} or {@link\n * Native#setPageV2} before the next 'readBatch' call.\n *\n *

Note that the current page could also be drained if the returned value = 'batchSize', i.e.,\n * the remaining number of rows in the page is exactly equal to 'batchSize'. In this case, the\n * next 'readBatch' call will return 0 and the caller should call {@link Native#setPageV1} or\n * {@link Native#setPageV2} next.\n *\n *

If `nullPadSize` > 0, it pads nulls into the underlying vector before the values will be\n * read into.\n *\n * @param handle the handle to the native Parquet column reader\n * @param batchSize the number of rows to be read\n * @param nullPadSize the number of nulls to pad before reading.\n * @return a tuple: (the actual number of rows read, the number of nulls read)\n */\n public static native int[] readBatch(long handle, int batchSize, int nullPadSize);\n\n /**\n * Skips at most 'batchSize' number of rows from the native Parquet column reader, and returns the\n * actual number of rows skipped.\n *\n *

If the returned value is < 'batchSize' then it means the current page has been completely\n * drained. In this case, the caller should call {@link Native#setPageV1} or {@link\n * Native#setPageV2} before the next 'skipBatch' call.\n *\n *

Note that the current page could also be drained if the returned value = 'batchSize', i.e.,\n * the remaining number of rows in the page is exactly equal to 'batchSize'. In this case, the\n * next 'skipBatch' call will return 0 and the caller should call {@link Native#setPageV1} or\n * {@link Native#setPageV2} next.\n *\n * @param handle the handle to the native Parquet column reader\n * @param batchSize the number of rows to skip in the current page\n * @param discard if true, discard read rows without padding nulls into the underlying vector\n * @return the actual number of rows skipped\n */\n public static native int skipBatch(long handle, int batchSize, boolean discard);\n\n /**\n * Returns the current batch constructed via 'readBatch'\n *\n * @param handle the handle to the native Parquet column reader\n * @param arrayAddr the memory address to the ArrowArray struct\n * @param schemaAddr the memory address to the ArrowSchema struct\n */\n public static native void currentBatch(long handle, long arrayAddr, long schemaAddr);\n\n /**\n * Closes the native Parquet column reader and releases all resources associated with it.\n *\n * @param handle the handle to the native Parquet column reader\n */\n public static native void closeColumnReader(long handle);\n\n ///////////// Arrow Native Parquet Reader APIs\n // TODO: Add partitionValues(?), improve requiredColumns to use a projection mask that corresponds\n // to arrow.\n // Add batch size, datetimeRebaseModeSpec, metrics(how?)...\n\n /**\n * Verify that object store options are valid. An exception will be thrown if the provided options\n * are not valid.\n */\n public static native void validateObjectStoreConfig(\n String filePath, Map objectStoreOptions);\n\n /**\n * Initialize a record batch reader for a PartitionedFile\n *\n * @param filePath\n * @param starts\n * @param lengths\n * @return a handle to the record batch reader, used in subsequent calls.\n */\n public static native long initRecordBatchReader(\n String filePath,\n long fileSize,\n long[] starts,\n long[] lengths,\n byte[] filter,\n byte[] requiredSchema,\n byte[] dataSchema,\n String sessionTimezone,\n int batchSize,\n boolean caseSensitive,\n Map objectStoreOptions,\n CometFileKeyUnwrapper keyUnwrapper,\n Object metricsNode);\n\n // arrow native version of read batch\n\n /**\n * Read the next batch of data into memory on native side\n *\n * @param handle\n * @return the number of rows read\n */\n public static native int readNextRecordBatch(long handle);\n\n // arrow native equivalent of currentBatch. 'columnNum' is number of the column in the record\n // batch\n\n /**\n * Load the column corresponding to columnNum in the currently loaded record batch into JVM\n *\n * @param handle\n * @param columnNum\n * @param arrayAddr\n * @param schemaAddr\n */\n public static native void currentColumnBatch(\n long handle, int columnNum, long arrayAddr, long schemaAddr);\n\n // arrow native version to close record batch reader\n\n /**\n * Close the record batch reader. Free the resources\n *\n * @param handle\n */\n public static native void closeRecordBatchReader(long handle);\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/NativeBatchReader.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.io.ByteArrayOutputStream;\nimport java.io.Closeable;\nimport java.io.IOException;\nimport java.lang.reflect.InvocationTargetException;\nimport java.lang.reflect.Method;\nimport java.net.URI;\nimport java.net.URISyntaxException;\nimport java.nio.channels.Channels;\nimport java.util.*;\nimport java.util.stream.Collectors;\n\nimport scala.Option;\nimport scala.collection.Seq;\nimport scala.collection.mutable.Buffer;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport org.apache.arrow.memory.BufferAllocator;\nimport org.apache.arrow.memory.RootAllocator;\nimport org.apache.arrow.vector.ipc.WriteChannel;\nimport org.apache.arrow.vector.ipc.message.MessageSerializer;\nimport org.apache.arrow.vector.types.pojo.Schema;\nimport org.apache.hadoop.conf.Configuration;\nimport org.apache.hadoop.fs.Path;\nimport org.apache.hadoop.mapreduce.InputSplit;\nimport org.apache.hadoop.mapreduce.RecordReader;\nimport org.apache.hadoop.mapreduce.TaskAttemptContext;\nimport org.apache.parquet.HadoopReadOptions;\nimport org.apache.parquet.ParquetReadOptions;\nimport org.apache.parquet.Preconditions;\nimport org.apache.parquet.column.ColumnDescriptor;\nimport org.apache.parquet.hadoop.metadata.BlockMetaData;\nimport org.apache.parquet.hadoop.metadata.ParquetMetadata;\nimport org.apache.parquet.schema.GroupType;\nimport org.apache.parquet.schema.MessageType;\nimport org.apache.parquet.schema.Type;\nimport org.apache.spark.TaskContext;\nimport org.apache.spark.TaskContext$;\nimport org.apache.spark.executor.TaskMetrics;\nimport org.apache.spark.sql.catalyst.InternalRow;\nimport org.apache.spark.sql.comet.parquet.CometParquetReadSupport;\nimport org.apache.spark.sql.comet.util.Utils$;\nimport org.apache.spark.sql.errors.QueryExecutionErrors;\nimport org.apache.spark.sql.execution.datasources.PartitionedFile;\nimport org.apache.spark.sql.execution.datasources.parquet.ParquetColumn;\nimport org.apache.spark.sql.execution.datasources.parquet.ParquetToSparkSchemaConverter;\nimport org.apache.spark.sql.execution.datasources.parquet.ParquetUtils;\nimport org.apache.spark.sql.execution.metric.SQLMetric;\nimport org.apache.spark.sql.internal.SQLConf;\nimport org.apache.spark.sql.types.*;\nimport org.apache.spark.sql.vectorized.ColumnarBatch;\nimport org.apache.spark.util.AccumulatorV2;\n\nimport org.apache.comet.CometConf;\nimport org.apache.comet.CometSchemaImporter;\nimport org.apache.comet.objectstore.NativeConfig;\nimport org.apache.comet.shims.ShimBatchReader;\nimport org.apache.comet.shims.ShimFileFormat;\nimport org.apache.comet.vector.CometVector;\nimport org.apache.comet.vector.NativeUtil;\n\nimport static scala.jdk.javaapi.CollectionConverters.asJava;\n\n/**\n * A vectorized Parquet reader that reads a Parquet file in a batched fashion.\n *\n *

Example of how to use this:\n *\n * 

\n *   NativeBatchReader reader = new NativeBatchReader(parquetFile, batchSize);\n *   try {\n *     reader.init();\n *     while (reader.readBatch()) {\n *       ColumnarBatch batch = reader.currentBatch();\n *       // consume the batch\n *     }\n *   } finally { // resources associated with the reader should be released\n *     reader.close();\n *   }\n *
\n */\npublic class NativeBatchReader extends RecordReader implements Closeable {\n\n /**\n * A class that contains the necessary file information for reading a Parquet file. This class\n * provides an abstraction over PartitionedFile properties.\n */\n public static class FileInfo {\n private final long start;\n private final long length;\n private final String filePath;\n private final long fileSize;\n\n public FileInfo(long start, long length, String filePath, long fileSize)\n throws URISyntaxException {\n this.start = start;\n this.length = length;\n URI uri = new Path(filePath).toUri();\n if (uri.getScheme() == null) {\n uri = new Path(\"file://\" + filePath).toUri();\n }\n this.filePath = uri.toString();\n this.fileSize = fileSize;\n }\n\n public static FileInfo fromPartitionedFile(PartitionedFile file) throws URISyntaxException {\n return new FileInfo(file.start(), file.length(), file.filePath().toString(), file.fileSize());\n }\n\n public long start() {\n return start;\n }\n\n public long length() {\n return length;\n }\n\n public String filePath() {\n return filePath;\n }\n\n public long fileSize() {\n return fileSize;\n }\n\n public URI pathUri() throws URISyntaxException {\n return new URI(filePath);\n }\n }\n\n private static final Logger LOG = LoggerFactory.getLogger(NativeBatchReader.class);\n protected static final BufferAllocator ALLOCATOR = new RootAllocator();\n private NativeUtil nativeUtil = new NativeUtil();\n\n protected Configuration conf;\n protected int capacity;\n protected boolean isCaseSensitive;\n protected boolean useFieldId;\n protected boolean ignoreMissingIds;\n protected StructType partitionSchema;\n protected InternalRow partitionValues;\n protected PartitionedFile file;\n protected FileInfo fileInfo;\n protected final Map metrics;\n // Unfortunately CometMetricNode is from the \"spark\" package and cannot be used directly here\n // TODO: Move it to common package?\n protected Object metricsNode = null;\n\n protected StructType sparkSchema;\n protected StructType dataSchema;\n MessageType fileSchema;\n protected MessageType requestedSchema;\n protected CometVector[] vectors;\n protected AbstractColumnReader[] columnReaders;\n protected CometSchemaImporter importer;\n protected ColumnarBatch currentBatch;\n // private FileReader fileReader;\n protected boolean[] missingColumns;\n protected boolean isInitialized;\n protected ParquetMetadata footer;\n protected byte[] nativeFilter;\n protected AbstractColumnReader[] preInitializedReaders;\n\n private ParquetColumn parquetColumn;\n\n /**\n * Map from field name to spark schema index for efficient lookups during batch loading. Built\n * once during initialization and reused across all batch loads.\n */\n private Map sparkFieldIndexMap;\n\n /**\n * Whether the native scan should always return decimal represented by 128 bits, regardless of its\n * precision. Normally, this should be true if native execution is enabled, since Arrow compute\n * kernels doesn't support 32 and 64 bit decimals yet.\n */\n // TODO: (ARROW NATIVE)\n private boolean useDecimal128;\n\n /**\n * Whether to return dates/timestamps that were written with legacy hybrid (Julian + Gregorian)\n * calendar as it is. If this is true, Comet will return them as it is, instead of rebasing them\n * to the new Proleptic Gregorian calendar. If this is false, Comet will throw exceptions when\n * seeing these dates/timestamps.\n */\n // TODO: (ARROW NATIVE)\n protected boolean useLegacyDateTimestamp;\n\n /** The TaskContext object for executing this task. */\n private final TaskContext taskContext;\n\n private long totalRowCount = 0;\n private long handle;\n\n // Protected no-arg constructor for subclasses\n protected NativeBatchReader() {\n this.taskContext = TaskContext$.MODULE$.get();\n this.metrics = new HashMap<>();\n }\n\n // Only for testing\n public NativeBatchReader(String file, int capacity) {\n this(file, capacity, null, null);\n }\n\n // Only for testing\n public NativeBatchReader(\n String file, int capacity, StructType partitionSchema, InternalRow partitionValues) {\n this(new Configuration(), file, capacity, partitionSchema, partitionValues);\n }\n\n // Only for testing\n public NativeBatchReader(\n Configuration conf,\n String file,\n int capacity,\n StructType partitionSchema,\n InternalRow partitionValues) {\n\n this.conf = conf;\n this.capacity = capacity;\n this.isCaseSensitive = false;\n this.useFieldId = false;\n this.ignoreMissingIds = false;\n this.partitionSchema = partitionSchema;\n this.partitionValues = partitionValues;\n\n this.file = ShimBatchReader.newPartitionedFile(partitionValues, file);\n this.metrics = new HashMap<>();\n\n this.taskContext = TaskContext$.MODULE$.get();\n }\n\n private NativeBatchReader(AbstractColumnReader[] columnReaders) {\n // Todo: set useDecimal128 and useLazyMaterialization\n int numColumns = columnReaders.length;\n this.columnReaders = new AbstractColumnReader[numColumns];\n vectors = new CometVector[numColumns];\n currentBatch = new ColumnarBatch(vectors);\n // This constructor is used by Iceberg only. The columnReaders are\n // initialized in Iceberg, so no need to call the init()\n isInitialized = true;\n this.taskContext = TaskContext$.MODULE$.get();\n this.metrics = new HashMap<>();\n }\n\n NativeBatchReader(\n Configuration conf,\n PartitionedFile inputSplit,\n ParquetMetadata footer,\n byte[] nativeFilter,\n int capacity,\n StructType sparkSchema,\n StructType dataSchema,\n boolean isCaseSensitive,\n boolean useFieldId,\n boolean ignoreMissingIds,\n boolean useLegacyDateTimestamp,\n StructType partitionSchema,\n InternalRow partitionValues,\n Map metrics,\n Object metricsNode) {\n this.conf = conf;\n this.capacity = capacity;\n this.sparkSchema = sparkSchema;\n this.dataSchema = dataSchema;\n this.isCaseSensitive = isCaseSensitive;\n this.useFieldId = useFieldId;\n this.ignoreMissingIds = ignoreMissingIds;\n this.useLegacyDateTimestamp = useLegacyDateTimestamp;\n this.partitionSchema = partitionSchema;\n this.partitionValues = partitionValues;\n this.file = inputSplit;\n this.footer = footer;\n this.nativeFilter = nativeFilter;\n this.metrics = metrics;\n this.metricsNode = metricsNode;\n this.taskContext = TaskContext$.MODULE$.get();\n }\n\n /** Alternate constructor that accepts FileInfo instead of PartitionedFile. */\n NativeBatchReader(\n Configuration conf,\n FileInfo fileInfo,\n ParquetMetadata footer,\n byte[] nativeFilter,\n int capacity,\n StructType sparkSchema,\n StructType dataSchema,\n boolean isCaseSensitive,\n boolean useFieldId,\n boolean ignoreMissingIds,\n boolean useLegacyDateTimestamp,\n StructType partitionSchema,\n InternalRow partitionValues,\n Map metrics,\n Object metricsNode) {\n this.conf = conf;\n this.capacity = capacity;\n this.sparkSchema = sparkSchema;\n this.dataSchema = dataSchema;\n this.isCaseSensitive = isCaseSensitive;\n this.useFieldId = useFieldId;\n this.ignoreMissingIds = ignoreMissingIds;\n this.useLegacyDateTimestamp = useLegacyDateTimestamp;\n this.partitionSchema = partitionSchema;\n this.partitionValues = partitionValues;\n this.fileInfo = fileInfo;\n this.footer = footer;\n this.nativeFilter = nativeFilter;\n this.metrics = metrics;\n this.metricsNode = metricsNode;\n this.taskContext = TaskContext$.MODULE$.get();\n }\n\n /**\n * Initialize this reader. The reason we don't do it in the constructor is that we want to close\n * any resource hold by this reader when error happens during the initialization.\n */\n public void init() throws Throwable {\n\n useDecimal128 =\n conf.getBoolean(\n CometConf.COMET_USE_DECIMAL_128().key(),\n (Boolean) CometConf.COMET_USE_DECIMAL_128().defaultValue().get());\n\n // Use fileInfo if available, otherwise fall back to file\n long start = fileInfo != null ? fileInfo.start() : file.start();\n long length = fileInfo != null ? fileInfo.length() : file.length();\n String filePath = fileInfo != null ? fileInfo.filePath() : file.filePath().toString();\n long fileSize = fileInfo != null ? fileInfo.fileSize() : file.fileSize();\n URI pathUri = fileInfo != null ? fileInfo.pathUri() : file.pathUri();\n\n ParquetReadOptions.Builder builder = HadoopReadOptions.builder(conf, new Path(filePath));\n\n if (start >= 0 && length >= 0) {\n builder = builder.withRange(start, start + length);\n }\n ParquetReadOptions readOptions = builder.build();\n\n Map objectStoreOptions =\n asJava(NativeConfig.extractObjectStoreOptions(conf, pathUri));\n\n // TODO: enable off-heap buffer when they are ready\n ReadOptions cometReadOptions = ReadOptions.builder(conf).build();\n\n Path path = new Path(new URI(filePath));\n try (FileReader fileReader =\n new FileReader(\n CometInputFile.fromPath(path, conf), footer, readOptions, cometReadOptions, metrics)) {\n\n requestedSchema = footer.getFileMetaData().getSchema();\n fileSchema = requestedSchema;\n\n if (sparkSchema == null) {\n ParquetToSparkSchemaConverter converter = new ParquetToSparkSchemaConverter(conf);\n sparkSchema = converter.convert(requestedSchema);\n } else {\n requestedSchema =\n CometParquetReadSupport.clipParquetSchema(\n requestedSchema, sparkSchema, isCaseSensitive, useFieldId, ignoreMissingIds);\n if (requestedSchema.getFieldCount() != sparkSchema.size()) {\n throw new IllegalArgumentException(\n String.format(\n \"Spark schema has %d columns while \" + \"Parquet schema has %d columns\",\n sparkSchema.size(), requestedSchema.getFieldCount()));\n }\n }\n\n boolean caseSensitive =\n conf.getBoolean(\n SQLConf.CASE_SENSITIVE().key(),\n (boolean) SQLConf.CASE_SENSITIVE().defaultValue().get());\n // rename spark fields based on field_id so name of spark schema field matches the parquet\n // field name\n if (useFieldId && ParquetUtils.hasFieldIds(sparkSchema)) {\n sparkSchema =\n getSparkSchemaByFieldId(sparkSchema, requestedSchema.asGroupType(), caseSensitive);\n }\n\n this.parquetColumn = getParquetColumn(requestedSchema, this.sparkSchema);\n\n // Create Column readers\n List fields = requestedSchema.getFields();\n List fileFields = fileSchema.getFields();\n ParquetColumn[] parquetFields =\n asJava(parquetColumn.children()).toArray(new ParquetColumn[0]);\n int numColumns = fields.size();\n if (partitionSchema != null) numColumns += partitionSchema.size();\n columnReaders = new AbstractColumnReader[numColumns];\n\n // Initialize missing columns and use null vectors for them\n missingColumns = new boolean[numColumns];\n // We do not need the column index of the row index; but this method has the\n // side effect of throwing an exception if a column with the same name is\n // found which we do want (spark unit tests explicitly test for that).\n ShimFileFormat.findRowIndexColumnIndexInSchema(sparkSchema);\n StructField[] nonPartitionFields = sparkSchema.fields();\n boolean hasRowIndexColumn = false;\n // Ranges of rows to read (needed iff row indexes are being read)\n List blocks =\n FileReader.filterRowGroups(readOptions, footer.getBlocks(), fileReader);\n totalRowCount = fileReader.getFilteredRecordCount();\n if (totalRowCount == 0) {\n // all the data is filtered out.\n isInitialized = true;\n return;\n }\n long[] starts = new long[blocks.size()];\n long[] lengths = new long[blocks.size()];\n int blockIndex = 0;\n for (BlockMetaData block : blocks) {\n long blockStart = block.getStartingPos();\n long blockLength = block.getCompressedSize();\n starts[blockIndex] = blockStart;\n lengths[blockIndex] = blockLength;\n blockIndex++;\n }\n for (int i = 0; i < fields.size(); i++) {\n Type field = fields.get(i);\n Optional optFileField =\n fileFields.stream().filter(f -> f.getName().equals(field.getName())).findFirst();\n if (nonPartitionFields[i].name().equals(ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME())) {\n // Values of ROW_INDEX_TEMPORARY_COLUMN_NAME column are always populated with\n // generated row indexes, rather than read from the file.\n // TODO(SPARK-40059): Allow users to include columns named\n // FileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME in their schemas.\n long[] rowIndices = FileReader.getRowIndices(blocks);\n columnReaders[i] =\n new ArrowRowIndexColumnReader(nonPartitionFields[i], capacity, rowIndices);\n hasRowIndexColumn = true;\n missingColumns[i] = true;\n } else if (optFileField.isPresent()) {\n // The column we are reading may be a complex type in which case we check if each field in\n // the requested type is in the file type (and the same data type)\n // This makes the same check as Spark's VectorizedParquetReader\n checkColumn(parquetFields[i]);\n missingColumns[i] = false;\n } else {\n if (preInitializedReaders != null\n && i < preInitializedReaders.length\n && preInitializedReaders[i] != null) {\n columnReaders[i] = preInitializedReaders[i];\n missingColumns[i] = true;\n } else {\n if (field.getRepetition() == Type.Repetition.REQUIRED) {\n throw new IOException(\n \"Required column '\"\n + field.getName()\n + \"' is missing\"\n + \" in data file \"\n + filePath);\n }\n if (field.isPrimitive()) {\n ArrowConstantColumnReader reader =\n new ArrowConstantColumnReader(nonPartitionFields[i], capacity, useDecimal128);\n columnReaders[i] = reader;\n missingColumns[i] = true;\n } else {\n // the column requested is not in the file, but the native reader can handle that\n // and will return nulls for all rows requested\n missingColumns[i] = false;\n }\n }\n }\n }\n\n // Initialize constant readers for partition columns\n if (partitionSchema != null) {\n StructField[] partitionFields = partitionSchema.fields();\n for (int i = fields.size(); i < columnReaders.length; i++) {\n int fieldIndex = i - fields.size();\n StructField field = partitionFields[fieldIndex];\n ArrowConstantColumnReader reader =\n new ArrowConstantColumnReader(\n field, capacity, partitionValues, fieldIndex, useDecimal128);\n columnReaders[i] = reader;\n }\n }\n\n vectors = new CometVector[numColumns];\n currentBatch = new ColumnarBatch(vectors);\n\n // For test purpose only\n // If the last external accumulator is `NumRowGroupsAccumulator`, the row group number to read\n // will be updated to the accumulator. So we can check if the row groups are filtered or not\n // in test case.\n // Note that this tries to get thread local TaskContext object, if this is called at other\n // thread, it won't update the accumulator.\n if (taskContext != null) {\n Option> accu = getTaskAccumulator(taskContext.taskMetrics());\n if (accu.isDefined() && accu.get().getClass().getSimpleName().equals(\"NumRowGroupsAcc\")) {\n @SuppressWarnings(\"unchecked\")\n AccumulatorV2 intAccum = (AccumulatorV2) accu.get();\n intAccum.add(blocks.size());\n }\n }\n\n boolean encryptionEnabled = CometParquetUtils.encryptionEnabled(conf);\n\n // Create keyUnwrapper if encryption is enabled\n CometFileKeyUnwrapper keyUnwrapper = null;\n if (encryptionEnabled) {\n keyUnwrapper = new CometFileKeyUnwrapper();\n keyUnwrapper.storeDecryptionKeyRetriever(filePath, conf);\n }\n\n // Filter out columns with preinitialized readers from sparkSchema before making the\n // call to native\n if (preInitializedReaders != null) {\n StructType filteredSchema = new StructType();\n StructField[] sparkFields = sparkSchema.fields();\n // Build name map for efficient lookups\n Map fileFieldNameMap =\n caseSensitive\n ? buildCaseSensitiveNameMap(fileFields)\n : buildCaseInsensitiveNameMap(fileFields);\n for (int i = 0; i < sparkFields.length; i++) {\n // Keep the column if:\n // 1. It doesn't have a preinitialized reader, OR\n // 2. It has a preinitialized reader but exists in fileSchema\n boolean hasPreInitializedReader =\n i < preInitializedReaders.length && preInitializedReaders[i] != null;\n String fieldName =\n caseSensitive\n ? sparkFields[i].name()\n : sparkFields[i].name().toLowerCase(Locale.ROOT);\n boolean existsInFileSchema = fileFieldNameMap.containsKey(fieldName);\n if (!hasPreInitializedReader || existsInFileSchema) {\n filteredSchema = filteredSchema.add(sparkFields[i]);\n }\n }\n sparkSchema = filteredSchema;\n }\n\n // Native code uses \"UTC\" always as the timeZoneId when converting from spark to arrow schema.\n String timeZoneId = \"UTC\";\n Schema arrowSchema = Utils$.MODULE$.toArrowSchema(sparkSchema, timeZoneId);\n byte[] serializedRequestedArrowSchema = serializeArrowSchema(arrowSchema);\n Schema dataArrowSchema = Utils$.MODULE$.toArrowSchema(dataSchema, timeZoneId);\n byte[] serializedDataArrowSchema = serializeArrowSchema(dataArrowSchema);\n\n int batchSize =\n conf.getInt(\n CometConf.COMET_BATCH_SIZE().key(),\n (Integer) CometConf.COMET_BATCH_SIZE().defaultValue().get());\n this.handle =\n Native.initRecordBatchReader(\n filePath,\n fileSize,\n starts,\n lengths,\n hasRowIndexColumn ? null : nativeFilter,\n serializedRequestedArrowSchema,\n serializedDataArrowSchema,\n timeZoneId,\n batchSize,\n caseSensitive,\n objectStoreOptions,\n keyUnwrapper,\n metricsNode);\n\n // Build spark field index map for efficient lookups during batch loading\n StructField[] sparkFields = sparkSchema.fields();\n sparkFieldIndexMap = new HashMap<>();\n for (int j = 0; j < sparkFields.length; j++) {\n String fieldName =\n caseSensitive ? sparkFields[j].name() : sparkFields[j].name().toLowerCase(Locale.ROOT);\n sparkFieldIndexMap.put(fieldName, j);\n }\n }\n isInitialized = true;\n }\n\n private ParquetColumn getParquetColumn(MessageType schema, StructType sparkSchema) {\n // We use a different config from the config that is passed in.\n // This follows the setting used in Spark's SpecificParquetRecordReaderBase\n Configuration config = new Configuration(conf);\n config.setBoolean(SQLConf.PARQUET_BINARY_AS_STRING().key(), false);\n config.setBoolean(SQLConf.PARQUET_INT96_AS_TIMESTAMP().key(), false);\n config.setBoolean(SQLConf.CASE_SENSITIVE().key(), false);\n config.setBoolean(SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED().key(), false);\n config.setBoolean(SQLConf.LEGACY_PARQUET_NANOS_AS_LONG().key(), false);\n ParquetToSparkSchemaConverter converter = new ParquetToSparkSchemaConverter(config);\n return converter.convertParquetColumn(schema, Option.apply(sparkSchema));\n }\n\n private Map> getIdToParquetFieldMap(GroupType type) {\n return type.getFields().stream()\n .filter(f -> f.getId() != null)\n .collect(Collectors.groupingBy(f -> f.getId().intValue()));\n }\n\n private Map> getCaseSensitiveParquetFieldMap(GroupType schema) {\n return schema.getFields().stream().collect(Collectors.toMap(Type::getName, Arrays::asList));\n }\n\n private Map> getCaseInsensitiveParquetFieldMap(GroupType schema) {\n return schema.getFields().stream()\n .collect(Collectors.groupingBy(f -> f.getName().toLowerCase(Locale.ROOT)));\n }\n\n private Map buildCaseSensitiveNameMap(List types) {\n return types.stream().collect(Collectors.toMap(Type::getName, t -> t));\n }\n\n private Map buildCaseInsensitiveNameMap(List types) {\n return types.stream()\n .collect(Collectors.toMap(t -> t.getName().toLowerCase(Locale.ROOT), t -> t));\n }\n\n private Type getMatchingParquetFieldById(\n StructField f,\n Map> idToParquetFieldMap,\n Map> nameToParquetFieldMap,\n boolean isCaseSensitive) {\n List matched = null;\n int fieldId = 0;\n if (ParquetUtils.hasFieldId(f)) {\n fieldId = ParquetUtils.getFieldId(f);\n matched = idToParquetFieldMap.get(fieldId);\n } else {\n String fieldName = isCaseSensitive ? f.name() : f.name().toLowerCase(Locale.ROOT);\n matched = nameToParquetFieldMap.get(fieldName);\n }\n\n if (matched == null || matched.isEmpty()) {\n return null;\n }\n if (matched.size() > 1) {\n // Need to fail if there is ambiguity, i.e. more than one field is matched\n String parquetTypesString =\n matched.stream().map(Type::getName).collect(Collectors.joining(\"[\", \", \", \"]\"));\n throw QueryExecutionErrors.foundDuplicateFieldInFieldIdLookupModeError(\n fieldId, parquetTypesString);\n } else {\n return matched.get(0);\n }\n }\n\n // Derived from CometParquetReadSupport.matchFieldId\n private String getMatchingNameById(\n StructField f,\n Map> idToParquetFieldMap,\n Map> nameToParquetFieldMap,\n boolean isCaseSensitive) {\n Type matched =\n getMatchingParquetFieldById(f, idToParquetFieldMap, nameToParquetFieldMap, isCaseSensitive);\n\n // When there is no ID match, we use a fake name to avoid a name match by accident\n // We need this name to be unique as well, otherwise there will be type conflicts\n if (matched == null) {\n return CometParquetReadSupport.generateFakeColumnName();\n } else {\n return matched.getName();\n }\n }\n\n // clip ParquetGroup Type\n private StructType getSparkSchemaByFieldId(\n StructType schema, GroupType parquetSchema, boolean caseSensitive) {\n StructType newSchema = new StructType();\n Map> idToParquetFieldMap = getIdToParquetFieldMap(parquetSchema);\n Map> nameToParquetFieldMap =\n caseSensitive\n ? getCaseSensitiveParquetFieldMap(parquetSchema)\n : getCaseInsensitiveParquetFieldMap(parquetSchema);\n for (StructField f : schema.fields()) {\n DataType newDataType;\n String fieldName = isCaseSensitive ? f.name() : f.name().toLowerCase(Locale.ROOT);\n List parquetFieldList = nameToParquetFieldMap.get(fieldName);\n if (parquetFieldList == null) {\n newDataType = f.dataType();\n } else {\n Type fieldType = parquetFieldList.get(0);\n if (f.dataType() instanceof StructType) {\n newDataType =\n getSparkSchemaByFieldId(\n (StructType) f.dataType(), fieldType.asGroupType(), caseSensitive);\n } else {\n newDataType = getSparkTypeByFieldId(f.dataType(), fieldType, caseSensitive);\n }\n }\n String matchedName =\n getMatchingNameById(f, idToParquetFieldMap, nameToParquetFieldMap, isCaseSensitive);\n StructField newField = f.copy(matchedName, newDataType, f.nullable(), f.metadata());\n newSchema = newSchema.add(newField);\n }\n return newSchema;\n }\n\n private static boolean isPrimitiveCatalystType(DataType dataType) {\n return !(dataType instanceof ArrayType)\n && !(dataType instanceof MapType)\n && !(dataType instanceof StructType);\n }\n\n private DataType getSparkTypeByFieldId(\n DataType dataType, Type parquetType, boolean caseSensitive) {\n DataType newDataType;\n if (dataType instanceof StructType) {\n newDataType =\n getSparkSchemaByFieldId((StructType) dataType, parquetType.asGroupType(), caseSensitive);\n } else if (dataType instanceof ArrayType\n && !isPrimitiveCatalystType(((ArrayType) dataType).elementType())) {\n\n newDataType =\n getSparkArrayTypeByFieldId(\n (ArrayType) dataType, parquetType.asGroupType(), caseSensitive);\n } else if (dataType instanceof MapType) {\n MapType mapType = (MapType) dataType;\n DataType keyType = mapType.keyType();\n DataType valueType = mapType.valueType();\n DataType newKeyType;\n DataType newValueType;\n Type parquetMapType = parquetType.asGroupType().getFields().get(0);\n Type parquetKeyType = parquetMapType.asGroupType().getType(\"key\");\n Type parquetValueType = parquetMapType.asGroupType().getType(\"value\");\n if (keyType instanceof StructType) {\n newKeyType =\n getSparkSchemaByFieldId(\n (StructType) keyType, parquetKeyType.asGroupType(), caseSensitive);\n } else {\n newKeyType = keyType;\n }\n if (valueType instanceof StructType) {\n newValueType =\n getSparkSchemaByFieldId(\n (StructType) valueType, parquetValueType.asGroupType(), caseSensitive);\n } else {\n newValueType = valueType;\n }\n newDataType = new MapType(newKeyType, newValueType, mapType.valueContainsNull());\n } else {\n newDataType = dataType;\n }\n return newDataType;\n }\n\n private DataType getSparkArrayTypeByFieldId(\n ArrayType arrayType, GroupType parquetList, boolean caseSensitive) {\n DataType newDataType;\n DataType elementType = arrayType.elementType();\n DataType newElementType;\n Type parquetElementType;\n if (parquetList.getLogicalTypeAnnotation() == null\n && parquetList.isRepetition(Type.Repetition.REPEATED)) {\n parquetElementType = parquetList;\n } else {\n // we expect only non-primitive types here (see clipParquetListTypes for related logic)\n GroupType repeatedGroup = parquetList.asGroupType().getType(0).asGroupType();\n if (repeatedGroup.getFieldCount() > 1\n || Objects.equals(repeatedGroup.getName(), \"array\")\n || Objects.equals(repeatedGroup.getName(), parquetList.getName() + \"_tuple\")) {\n parquetElementType = repeatedGroup;\n } else {\n parquetElementType = repeatedGroup.getType(0);\n }\n }\n if (elementType instanceof StructType) {\n newElementType =\n getSparkSchemaByFieldId(\n (StructType) elementType, parquetElementType.asGroupType(), caseSensitive);\n } else {\n newElementType = getSparkTypeByFieldId(elementType, parquetElementType, caseSensitive);\n }\n newDataType = new ArrayType(newElementType, arrayType.containsNull());\n return newDataType;\n }\n\n private void checkParquetType(ParquetColumn column) throws IOException {\n String[] path = asJava(column.path()).toArray(new String[0]);\n if (containsPath(fileSchema, path)) {\n if (column.isPrimitive()) {\n ColumnDescriptor desc = column.descriptor().get();\n ColumnDescriptor fd = fileSchema.getColumnDescription(desc.getPath());\n TypeUtil.checkParquetType(fd, column.sparkType());\n } else {\n for (ParquetColumn childColumn : asJava(column.children())) {\n checkColumn(childColumn);\n }\n }\n } else { // A missing column which is either primitive or complex\n if (column.required()) {\n // check if we have a preinitialized column reader for this column.\n int columnIndex = getColumnIndexFromParquetColumn(column);\n if (columnIndex == -1\n || preInitializedReaders == null\n || columnIndex >= preInitializedReaders.length\n || preInitializedReaders[columnIndex] == null) {\n // Column is missing in data but the required data is non-nullable. This file is invalid.\n throw new IOException(\n \"Required column is missing in data file. Col: \" + Arrays.toString(path));\n }\n }\n }\n }\n\n /**\n * Get the column index in the requested schema for a given ParquetColumn. Returns -1 if not\n * found.\n */\n private int getColumnIndexFromParquetColumn(ParquetColumn column) {\n String[] targetPath = asJava(column.path()).toArray(new String[0]);\n if (targetPath.length == 0) {\n return -1;\n }\n\n // For top-level columns, match by name\n String columnName = targetPath[0];\n ParquetColumn[] parquetFields = asJava(parquetColumn.children()).toArray(new ParquetColumn[0]);\n for (int i = 0; i < parquetFields.length; i++) {\n String[] fieldPath = asJava(parquetFields[i].path()).toArray(new String[0]);\n if (fieldPath.length > 0 && fieldPath[0].equals(columnName)) {\n return i;\n }\n }\n return -1;\n }\n\n /**\n * Checks whether the given 'path' exists in 'parquetType'. The difference between this and {@link\n * MessageType#containsPath(String[])} is that the latter only support paths to leaf From Spark:\n * VectorizedParquetRecordReader Check whether a column from requested schema is missing from the\n * file schema, or whether it conforms to the type of the file schema.\n */\n private void checkColumn(ParquetColumn column) throws IOException {\n String[] path = asJava(column.path()).toArray(new String[0]);\n if (containsPath(fileSchema, path)) {\n if (column.isPrimitive()) {\n ColumnDescriptor desc = column.descriptor().get();\n ColumnDescriptor fd = fileSchema.getColumnDescription(desc.getPath());\n if (!fd.equals(desc)) {\n throw new UnsupportedOperationException(\"Schema evolution not supported.\");\n }\n } else {\n for (ParquetColumn childColumn : asJava(column.children())) {\n checkColumn(childColumn);\n }\n }\n } else { // A missing column which is either primitive or complex\n if (column.required()) {\n // Column is missing in data but the required data is non-nullable. This file is invalid.\n throw new IOException(\n \"Required column is missing in data file. Col: \" + Arrays.toString(path));\n }\n }\n }\n\n /**\n * Checks whether the given 'path' exists in 'parquetType'. The difference between this and {@link\n * MessageType#containsPath(String[])} is that the latter only support paths to leaf nodes, while\n * this support paths both to leaf and non-leaf nodes.\n */\n private boolean containsPath(Type parquetType, String[] path) {\n return containsPath(parquetType, path, 0);\n }\n\n private boolean containsPath(Type parquetType, String[] path, int depth) {\n if (path.length == depth) return true;\n if (parquetType instanceof GroupType) {\n String fieldName = path[depth];\n GroupType parquetGroupType = (GroupType) parquetType;\n if (parquetGroupType.containsField(fieldName)) {\n return containsPath(parquetGroupType.getType(fieldName), path, depth + 1);\n }\n }\n return false;\n }\n\n public void setSparkSchema(StructType schema) {\n this.sparkSchema = schema;\n }\n\n public AbstractColumnReader[] getColumnReaders() {\n return columnReaders;\n }\n\n @Override\n public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptContext)\n throws IOException, InterruptedException {\n // Do nothing. The initialization work is done in 'init' already.\n }\n\n @Override\n public boolean nextKeyValue() throws IOException {\n return nextBatch();\n }\n\n @Override\n public Void getCurrentKey() {\n return null;\n }\n\n @Override\n public ColumnarBatch getCurrentValue() {\n return currentBatch();\n }\n\n @Override\n public float getProgress() {\n return 0;\n }\n\n /**\n * Returns the current columnar batch being read.\n *\n *

Note that this must be called AFTER {@link NativeBatchReader#nextBatch()}.\n */\n public ColumnarBatch currentBatch() {\n return currentBatch;\n }\n\n /**\n * Loads the next batch of rows. This is called by Spark _and_ Iceberg\n *\n * @return true if there are no more rows to read, false otherwise.\n */\n public boolean nextBatch() throws IOException {\n Preconditions.checkState(isInitialized, \"init() should be called first!\");\n\n // if (rowsRead >= totalRowCount) return false;\n\n if (totalRowCount == 0) return false;\n\n int batchSize;\n\n try {\n batchSize = loadNextBatch();\n } catch (RuntimeException e) {\n // Spark will check certain exception e.g. `SchemaColumnConvertNotSupportedException`.\n throw e;\n } catch (Throwable e) {\n throw new IOException(e);\n }\n\n if (batchSize == 0) return false;\n\n long totalDecodeTime = 0, totalLoadTime = 0;\n for (int i = 0; i < columnReaders.length; i++) {\n AbstractColumnReader reader = columnReaders[i];\n long startNs = System.nanoTime();\n // TODO: read from native reader\n reader.readBatch(batchSize);\n // totalDecodeTime += System.nanoTime() - startNs;\n // startNs = System.nanoTime();\n vectors[i] = reader.currentBatch();\n totalLoadTime += System.nanoTime() - startNs;\n }\n\n // TODO: (ARROW NATIVE) Add Metrics\n // SQLMetric decodeMetric = metrics.get(\"ParquetNativeDecodeTime\");\n // if (decodeMetric != null) {\n // decodeMetric.add(totalDecodeTime);\n // }\n SQLMetric loadMetric = metrics.get(\"ParquetNativeLoadTime\");\n if (loadMetric != null) {\n loadMetric.add(totalLoadTime);\n }\n\n currentBatch.setNumRows(batchSize);\n return true;\n }\n\n @Override\n public void close() throws IOException {\n if (columnReaders != null) {\n for (AbstractColumnReader reader : columnReaders) {\n if (reader != null) {\n reader.close();\n }\n }\n }\n if (importer != null) {\n importer.close();\n importer = null;\n }\n nativeUtil.close();\n if (this.handle > 0) {\n Native.closeRecordBatchReader(this.handle);\n this.handle = 0;\n }\n }\n\n @SuppressWarnings(\"deprecation\")\n private int loadNextBatch() throws Throwable {\n\n for (ParquetColumn childColumn : asJava(parquetColumn.children())) {\n checkParquetType(childColumn);\n }\n\n int batchSize = Native.readNextRecordBatch(this.handle);\n if (batchSize == 0) {\n return batchSize;\n }\n if (importer != null) importer.close();\n importer = new CometSchemaImporter(ALLOCATOR);\n\n List fields = requestedSchema.getFields();\n StructField[] sparkFields = sparkSchema.fields();\n\n boolean caseSensitive =\n conf.getBoolean(\n SQLConf.CASE_SENSITIVE().key(),\n (boolean) SQLConf.CASE_SENSITIVE().defaultValue().get());\n\n for (int i = 0; i < fields.size(); i++) {\n if (!missingColumns[i]) {\n if (columnReaders[i] != null) columnReaders[i].close();\n // TODO: (ARROW NATIVE) handle tz, datetime & int96 rebase\n Type field = fields.get(i);\n\n // Find the corresponding spark field by matching field names using the prebuilt map\n String fieldName =\n caseSensitive ? field.getName() : field.getName().toLowerCase(Locale.ROOT);\n Integer sparkSchemaIndex = sparkFieldIndexMap.get(fieldName);\n\n if (sparkSchemaIndex == null) {\n throw new IOException(\n \"Could not find matching Spark field for Parquet field: \" + field.getName());\n }\n\n DataType dataType = sparkFields[sparkSchemaIndex].dataType();\n NativeColumnReader reader =\n new NativeColumnReader(\n this.handle,\n sparkSchemaIndex,\n dataType,\n field,\n null,\n importer,\n nativeUtil,\n capacity,\n useDecimal128,\n useLegacyDateTimestamp);\n columnReaders[i] = reader;\n }\n }\n return batchSize;\n }\n\n // Signature of externalAccums changed from returning a Buffer to returning a Seq. If comet is\n // expecting a Buffer but the Spark version returns a Seq or vice versa, we get a\n // method not found exception.\n @SuppressWarnings(\"unchecked\")\n private Option> getTaskAccumulator(TaskMetrics taskMetrics) {\n Method externalAccumsMethod;\n try {\n externalAccumsMethod = TaskMetrics.class.getDeclaredMethod(\"externalAccums\");\n externalAccumsMethod.setAccessible(true);\n String returnType = externalAccumsMethod.getReturnType().getName();\n if (returnType.equals(\"scala.collection.mutable.Buffer\")) {\n return ((Buffer>) externalAccumsMethod.invoke(taskMetrics))\n .lastOption();\n } else if (returnType.equals(\"scala.collection.Seq\")) {\n return ((Seq>) externalAccumsMethod.invoke(taskMetrics)).lastOption();\n } else {\n return Option.apply(null); // None\n }\n } catch (NoSuchMethodException | InvocationTargetException | IllegalAccessException e) {\n return Option.apply(null); // None\n }\n }\n\n private byte[] serializeArrowSchema(Schema schema) throws IOException {\n ByteArrayOutputStream out = new ByteArrayOutputStream();\n WriteChannel writeChannel = new WriteChannel(Channels.newChannel(out));\n MessageSerializer.serialize(writeChannel, schema);\n return out.toByteArray();\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/NativeColumnReader.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport org.apache.arrow.c.ArrowArray;\nimport org.apache.arrow.c.ArrowSchema;\nimport org.apache.arrow.memory.BufferAllocator;\nimport org.apache.arrow.memory.RootAllocator;\nimport org.apache.parquet.column.ColumnDescriptor;\nimport org.apache.parquet.schema.Type;\nimport org.apache.spark.sql.types.DataType;\n\nimport org.apache.comet.CometSchemaImporter;\nimport org.apache.comet.vector.*;\n\nimport static scala.jdk.javaapi.CollectionConverters.*;\n\n// TODO: extend ColumnReader instead of AbstractColumnReader to reduce code duplication\npublic class NativeColumnReader extends AbstractColumnReader {\n protected static final Logger LOG = LoggerFactory.getLogger(NativeColumnReader.class);\n protected final BufferAllocator ALLOCATOR = new RootAllocator();\n\n /**\n * The current Comet vector holding all the values read by this column reader. Owned by this\n * reader and MUST be closed after use.\n */\n private CometDecodedVector currentVector;\n\n /** Dictionary values for this column. Only set if the column is using dictionary encoding. */\n protected CometDictionary dictionary;\n\n /**\n * The number of values in the current batch, used when we are skipping importing of Arrow\n * vectors, in which case we'll simply update the null count of the existing vectors.\n */\n int currentNumValues;\n\n /**\n * Whether the last loaded vector contains any null value. This is used to determine if we can\n * skip vector reloading. If the flag is false, Arrow C API will skip to import the validity\n * buffer, and therefore we cannot skip vector reloading.\n */\n boolean hadNull;\n\n private final CometSchemaImporter importer;\n private final NativeUtil nativeUtil;\n\n private ArrowArray array = null;\n private ArrowSchema schema = null;\n\n private long nativeBatchHandle = 0xDEADBEEFL;\n private final int columnNum;\n\n NativeColumnReader(\n long nativeBatchHandle,\n int columnNum,\n DataType type,\n Type fieldType,\n ColumnDescriptor descriptor,\n CometSchemaImporter importer,\n NativeUtil nativeUtil,\n int batchSize,\n boolean useDecimal128,\n boolean useLegacyDateTimestamp) {\n super(type, fieldType, descriptor, useDecimal128, useLegacyDateTimestamp);\n assert batchSize > 0 : \"Batch size must be positive, found \" + batchSize;\n this.batchSize = batchSize;\n this.nativeUtil = nativeUtil;\n this.importer = importer;\n this.nativeBatchHandle = nativeBatchHandle;\n this.columnNum = columnNum;\n initNative();\n }\n\n @Override\n // Override in order to avoid creation of JVM side column readers\n protected void initNative() {\n LOG.debug(\n \"Native column reader {} is initialized\", String.join(\".\", this.type.catalogString()));\n nativeHandle = 0;\n }\n\n @Override\n public void readBatch(int total) {\n LOG.debug(\"Reading column batch of size = {}\", total);\n\n this.currentNumValues = total;\n }\n\n /** Returns the {@link CometVector} read by this reader. */\n @Override\n public CometVector currentBatch() {\n return loadVector();\n }\n\n @Override\n public void close() {\n if (currentVector != null) {\n currentVector.close();\n currentVector = null;\n }\n super.close();\n }\n\n /** Returns a decoded {@link CometDecodedVector Comet vector}. */\n public CometDecodedVector loadVector() {\n\n LOG.debug(\"Loading vector for next batch\");\n\n // Close the previous vector first to release struct memory allocated to import Arrow array &\n // schema from native side, through the C data interface\n if (currentVector != null) {\n currentVector.close();\n }\n\n // TODO: ARROW NATIVE : Handle Uuid?\n\n array = ArrowArray.allocateNew(ALLOCATOR);\n schema = ArrowSchema.allocateNew(ALLOCATOR);\n\n long arrayAddr = array.memoryAddress();\n long schemaAddr = schema.memoryAddress();\n\n Native.currentColumnBatch(nativeBatchHandle, columnNum, arrayAddr, schemaAddr);\n\n ArrowArray[] arrays = {array};\n ArrowSchema[] schemas = {schema};\n\n CometDecodedVector cometVector =\n (CometDecodedVector) asJava(nativeUtil.importVector(arrays, schemas)).get(0);\n\n // Update whether the current vector contains any null values. This is used in the following\n // batch(s) to determine whether we can skip loading the native vector.\n hadNull = cometVector.hasNull();\n\n currentVector = cometVector;\n return currentVector;\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/ParquetColumnSpec.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.util.Map;\n\nimport org.apache.comet.IcebergApi;\n\n/**\n * Parquet ColumnSpec encapsulates the information withing a Parquet ColumnDescriptor. Utility\n * methods can convert from and to a ColumnDescriptor The only purpose of this class is to allow\n * passing of Column descriptors between Comet and Iceberg. This is required because Iceberg shades\n * Parquet, changing the package of Parquet classes and making then incompatible with Comet.\n */\n@IcebergApi\npublic class ParquetColumnSpec {\n\n private final int fieldId;\n private final String[] path;\n private final String physicalType;\n private final int typeLength;\n private final boolean isRepeated;\n private final int maxDefinitionLevel;\n private final int maxRepetitionLevel;\n\n // Logical type info\n private String logicalTypeName;\n private Map logicalTypeParams;\n\n @IcebergApi\n public ParquetColumnSpec(\n int fieldId,\n String[] path,\n String physicalType,\n int typeLength,\n boolean isRepeated,\n int maxDefinitionLevel,\n int maxRepetitionLevel,\n String logicalTypeName,\n Map logicalTypeParams) {\n this.fieldId = fieldId;\n this.path = path;\n this.physicalType = physicalType;\n this.typeLength = typeLength;\n this.isRepeated = isRepeated;\n this.maxDefinitionLevel = maxDefinitionLevel;\n this.maxRepetitionLevel = maxRepetitionLevel;\n this.logicalTypeName = logicalTypeName;\n this.logicalTypeParams = logicalTypeParams;\n }\n\n @IcebergApi\n public int getFieldId() {\n return fieldId;\n }\n\n @IcebergApi\n public String[] getPath() {\n return path;\n }\n\n @IcebergApi\n public String getPhysicalType() {\n return physicalType;\n }\n\n @IcebergApi\n public int getTypeLength() {\n return typeLength;\n }\n\n public boolean isRepeated() {\n return isRepeated;\n }\n\n @IcebergApi\n public int getMaxRepetitionLevel() {\n return maxRepetitionLevel;\n }\n\n @IcebergApi\n public int getMaxDefinitionLevel() {\n return maxDefinitionLevel;\n }\n\n @IcebergApi\n public String getLogicalTypeName() {\n return logicalTypeName;\n }\n\n @IcebergApi\n public Map getLogicalTypeParams() {\n return logicalTypeParams;\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/ParquetMetadataSerializer.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.io.ByteArrayInputStream;\nimport java.io.ByteArrayOutputStream;\nimport java.io.IOException;\n\nimport org.apache.parquet.format.FileMetaData;\nimport org.apache.parquet.format.Util;\nimport org.apache.parquet.format.converter.ParquetMetadataConverter;\nimport org.apache.parquet.hadoop.metadata.ParquetMetadata;\n\n/**\n * Utility class for serializing and deserializing ParquetMetadata instances to/from byte arrays.\n * This uses the Parquet format's FileMetaData structure and the underlying Thrift compact protocol\n * for serialization.\n */\npublic class ParquetMetadataSerializer {\n\n private final ParquetMetadataConverter converter;\n\n public ParquetMetadataSerializer() {\n this.converter = new ParquetMetadataConverter();\n }\n\n public ParquetMetadataSerializer(ParquetMetadataConverter converter) {\n this.converter = converter;\n }\n\n /**\n * Serializes a ParquetMetadata instance to a byte array.\n *\n * @param metadata the ParquetMetadata to serialize\n * @return the serialized byte array\n * @throws IOException if an error occurs during serialization\n */\n public byte[] serialize(ParquetMetadata metadata) throws IOException {\n FileMetaData fileMetaData = converter.toParquetMetadata(1, metadata);\n ByteArrayOutputStream outputStream = new ByteArrayOutputStream();\n Util.writeFileMetaData(fileMetaData, outputStream);\n return outputStream.toByteArray();\n }\n\n /**\n * Deserializes a byte array back into a ParquetMetadata instance.\n *\n * @param bytes the serialized byte array\n * @return the deserialized ParquetMetadata\n * @throws IOException if an error occurs during deserialization\n */\n public ParquetMetadata deserialize(byte[] bytes) throws IOException {\n ByteArrayInputStream inputStream = new ByteArrayInputStream(bytes);\n FileMetaData fileMetaData = Util.readFileMetaData(inputStream);\n return converter.fromParquetMetadata(fileMetaData);\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/ReadOptions.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport org.apache.hadoop.conf.Configuration;\nimport org.apache.spark.SparkEnv;\nimport org.apache.spark.launcher.SparkLauncher;\n\nimport org.apache.comet.CometConf;\nimport org.apache.comet.IcebergApi;\n\n/**\n * Comet specific Parquet related read options.\n *\n *

TODO: merge this with {@link org.apache.parquet.HadoopReadOptions} once PARQUET-2203 is done.\n */\n@IcebergApi\npublic class ReadOptions {\n private static final Logger LOG = LoggerFactory.getLogger(ReadOptions.class);\n\n // Max number of concurrent tasks we expect. Used to autoconfigure S3 client connections\n public static final int S3A_MAX_EXPECTED_PARALLELISM = 32;\n // defined in hadoop-aws - org.apache.hadoop.fs.s3a.Constants.MAXIMUM_CONNECTIONS\n public static final String S3A_MAXIMUM_CONNECTIONS = \"fs.s3a.connection.maximum\";\n // default max connections in S3A - org.apache.hadoop.fs.s3a.Constants.DEFAULT_MAXIMUM_CONNECTIONS\n public static final int S3A_DEFAULT_MAX_HTTP_CONNECTIONS = 96;\n\n public static final String S3A_READAHEAD_RANGE = \"fs.s3a.readahead.range\";\n // Default read ahead range in Hadoop is 64K; we increase it to 1 MB\n public static final long COMET_DEFAULT_READAHEAD_RANGE = 1 * 1024 * 1024; // 1 MB\n\n private final boolean parallelIOEnabled;\n private final int parallelIOThreadPoolSize;\n private final boolean ioMergeRanges;\n private final int ioMergeRangesDelta;\n private final boolean adjustReadRangeSkew;\n\n ReadOptions(\n boolean parallelIOEnabled,\n int parallelIOThreadPoolSize,\n boolean ioMergeRanges,\n int ioMergeRangesDelta,\n boolean adjustReadRangeSkew) {\n this.parallelIOEnabled = parallelIOEnabled;\n this.parallelIOThreadPoolSize = parallelIOThreadPoolSize;\n this.ioMergeRanges = ioMergeRanges;\n this.ioMergeRangesDelta = ioMergeRangesDelta;\n this.adjustReadRangeSkew = adjustReadRangeSkew;\n }\n\n public boolean isParallelIOEnabled() {\n return this.parallelIOEnabled;\n }\n\n public int parallelIOThreadPoolSize() {\n return this.parallelIOThreadPoolSize;\n }\n\n public boolean isIOMergeRangesEnabled() {\n return ioMergeRanges;\n }\n\n public int getIOMergeRangesDelta() {\n return ioMergeRangesDelta;\n }\n\n public boolean adjustReadRangesSkew() {\n return adjustReadRangeSkew;\n }\n\n @IcebergApi\n public static Builder builder(Configuration conf) {\n return new Builder(conf);\n }\n\n @IcebergApi\n public static class Builder {\n private final Configuration conf;\n\n private boolean parallelIOEnabled;\n private int parallelIOThreadPoolSize;\n private boolean ioMergeRanges;\n private int ioMergeRangesDelta;\n private boolean adjustReadRangeSkew;\n\n /**\n * Whether to enable Parquet parallel IO when reading row groups. If true, Parquet reader will\n * use multiple threads to read multiple chunks of data from the current row group in parallel.\n */\n public Builder enableParallelIO(boolean b) {\n this.parallelIOEnabled = b;\n return this;\n }\n\n /**\n * Specify the number of threads to be used in parallel IO.\n *\n *

Note: this will only be effective if parallel IO is enabled (e.g., via {@link\n * #enableParallelIO(boolean)}).\n */\n public Builder withParallelIOThreadPoolSize(int numThreads) {\n this.parallelIOThreadPoolSize = numThreads;\n return this;\n }\n\n public Builder enableIOMergeRanges(boolean enableIOMergeRanges) {\n this.ioMergeRanges = enableIOMergeRanges;\n return this;\n }\n\n public Builder withIOMergeRangesDelta(int ioMergeRangesDelta) {\n this.ioMergeRangesDelta = ioMergeRangesDelta;\n return this;\n }\n\n public Builder adjustReadRangeSkew(boolean adjustReadRangeSkew) {\n this.adjustReadRangeSkew = adjustReadRangeSkew;\n return this;\n }\n\n @IcebergApi\n public ReadOptions build() {\n return new ReadOptions(\n parallelIOEnabled,\n parallelIOThreadPoolSize,\n ioMergeRanges,\n ioMergeRangesDelta,\n adjustReadRangeSkew);\n }\n\n @IcebergApi\n public Builder(Configuration conf) {\n this.conf = conf;\n this.parallelIOEnabled =\n conf.getBoolean(\n CometConf.COMET_PARQUET_PARALLEL_IO_ENABLED().key(),\n (Boolean) CometConf.COMET_PARQUET_PARALLEL_IO_ENABLED().defaultValue().get());\n this.parallelIOThreadPoolSize =\n conf.getInt(\n CometConf.COMET_PARQUET_PARALLEL_IO_THREADS().key(),\n (Integer) CometConf.COMET_PARQUET_PARALLEL_IO_THREADS().defaultValue().get());\n this.ioMergeRanges =\n conf.getBoolean(\n CometConf.COMET_IO_MERGE_RANGES().key(),\n (boolean) CometConf.COMET_IO_MERGE_RANGES().defaultValue().get());\n this.ioMergeRangesDelta =\n conf.getInt(\n CometConf.COMET_IO_MERGE_RANGES_DELTA().key(),\n (Integer) CometConf.COMET_IO_MERGE_RANGES_DELTA().defaultValue().get());\n this.adjustReadRangeSkew =\n conf.getBoolean(\n CometConf.COMET_IO_ADJUST_READRANGE_SKEW().key(),\n (Boolean) CometConf.COMET_IO_ADJUST_READRANGE_SKEW().defaultValue().get());\n // override some S3 defaults\n setS3Config();\n }\n\n // For paths to S3, if the s3 connection pool max is less than twice the product of\n // parallel reader threads * number of cores, then increase the connection pool max\n private void setS3Config() {\n int s3ConnectionsMax = S3A_DEFAULT_MAX_HTTP_CONNECTIONS;\n SparkEnv env = SparkEnv.get();\n // Use a default number of cores in case we are using the FileReader outside the context\n // of Spark.\n int numExecutorCores = S3A_MAX_EXPECTED_PARALLELISM;\n if (env != null) {\n numExecutorCores = env.conf().getInt(SparkLauncher.EXECUTOR_CORES, numExecutorCores);\n }\n int parallelReaderThreads = this.parallelIOEnabled ? this.parallelIOThreadPoolSize : 1;\n s3ConnectionsMax = Math.max(numExecutorCores * parallelReaderThreads * 2, s3ConnectionsMax);\n\n setS3ConfIfGreater(conf, S3A_MAXIMUM_CONNECTIONS, s3ConnectionsMax);\n setS3ConfIfGreater(conf, S3A_READAHEAD_RANGE, COMET_DEFAULT_READAHEAD_RANGE);\n }\n\n // Update the conf iff the new value is greater than the existing val\n private void setS3ConfIfGreater(Configuration conf, String key, int newVal) {\n int maxVal = newVal;\n String curr = conf.get(key);\n if (curr != null && !curr.isEmpty()) {\n maxVal = Math.max(Integer.parseInt(curr), newVal);\n }\n LOG.info(\"File reader auto configured '{}={}'\", key, maxVal);\n conf.set(key, Integer.toString(maxVal));\n }\n\n // Update the conf iff the new value is greater than the existing val. This handles values that\n // may have suffixes (K, M, G, T, P, E) indicating well known bytes size suffixes\n private void setS3ConfIfGreater(Configuration conf, String key, long newVal) {\n long maxVal = conf.getLongBytes(key, newVal);\n maxVal = Math.max(maxVal, newVal);\n LOG.info(\"File reader auto configured '{}={}'\", key, maxVal);\n conf.set(key, Long.toString(maxVal));\n }\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/RowGroupFilter.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.util.ArrayList;\nimport java.util.List;\n\nimport org.apache.parquet.filter2.compat.FilterCompat;\nimport org.apache.parquet.filter2.compat.FilterCompat.Filter;\nimport org.apache.parquet.filter2.compat.FilterCompat.NoOpFilter;\nimport org.apache.parquet.filter2.compat.FilterCompat.Visitor;\nimport org.apache.parquet.filter2.dictionarylevel.DictionaryFilter;\nimport org.apache.parquet.filter2.predicate.FilterPredicate;\nimport org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator;\nimport org.apache.parquet.filter2.statisticslevel.StatisticsFilter;\nimport org.apache.parquet.hadoop.metadata.BlockMetaData;\nimport org.apache.parquet.schema.MessageType;\n\npublic class RowGroupFilter implements Visitor> {\n private final List blocks;\n private final MessageType schema;\n private final List levels;\n private final FileReader reader;\n\n public enum FilterLevel {\n STATISTICS,\n DICTIONARY,\n BLOOMFILTER\n }\n\n public static List filterRowGroups(\n List levels, Filter filter, List blocks, FileReader reader) {\n return filter.accept(new RowGroupFilter(levels, blocks, reader));\n }\n\n public static List filterRowGroups(\n List levels, Filter filter, List blocks, MessageType schema) {\n return filter.accept(new RowGroupFilter(levels, blocks, schema));\n }\n\n private RowGroupFilter(List levels, List blocks, FileReader reader) {\n this.levels = levels;\n this.blocks = blocks;\n this.reader = reader;\n this.schema = reader.getFileMetaData().getSchema();\n }\n\n private RowGroupFilter(List levels, List blocks, MessageType schema) {\n this.levels = levels;\n this.blocks = blocks;\n this.reader = null;\n this.schema = schema;\n }\n\n @Override\n public List visit(FilterCompat.FilterPredicateCompat filterPredicateCompat) {\n FilterPredicate filterPredicate = filterPredicateCompat.getFilterPredicate();\n\n // check that the schema of the filter matches the schema of the file\n SchemaCompatibilityValidator.validate(filterPredicate, schema);\n\n List filteredBlocks = new ArrayList<>();\n\n for (BlockMetaData block : blocks) {\n boolean drop = false;\n\n if (levels.contains(FilterLevel.STATISTICS)) {\n drop = StatisticsFilter.canDrop(filterPredicate, block.getColumns());\n }\n\n if (!drop && levels.contains(FilterLevel.DICTIONARY)) {\n drop =\n DictionaryFilter.canDrop(\n filterPredicate,\n block.getColumns(),\n new DictionaryPageReader(\n block,\n reader.getFileMetaData().getFileDecryptor(),\n reader.getInputStream(),\n reader.getOptions()));\n }\n\n if (!drop && levels.contains(FilterLevel.BLOOMFILTER)) {\n drop =\n filterPredicate.accept(\n new BloomFilterReader(\n block, reader.getFileMetaData().getFileDecryptor(), reader.getInputStream()));\n }\n\n if (!drop) {\n filteredBlocks.add(block);\n }\n }\n\n return filteredBlocks;\n }\n\n @Override\n public List visit(\n FilterCompat.UnboundRecordFilterCompat unboundRecordFilterCompat) {\n return blocks;\n }\n\n @Override\n public List visit(NoOpFilter noOpFilter) {\n return blocks;\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/RowGroupReader.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.util.HashMap;\nimport java.util.Map;\nimport java.util.Optional;\nimport java.util.PrimitiveIterator;\n\nimport org.apache.parquet.column.ColumnDescriptor;\nimport org.apache.parquet.column.page.PageReadStore;\nimport org.apache.parquet.column.page.PageReader;\nimport org.apache.parquet.internal.filter2.columnindex.RowRanges;\n\nimport org.apache.comet.IcebergApi;\n\n@IcebergApi\npublic class RowGroupReader implements PageReadStore {\n private final Map readers = new HashMap<>();\n private final long rowCount;\n private final RowRanges rowRanges;\n private final long rowIndexOffset;\n\n public RowGroupReader(long rowCount, long rowIndexOffset) {\n this.rowCount = rowCount;\n this.rowRanges = null;\n this.rowIndexOffset = rowIndexOffset;\n }\n\n RowGroupReader(RowRanges rowRanges) {\n this.rowRanges = rowRanges;\n this.rowCount = rowRanges.rowCount();\n this.rowIndexOffset = -1;\n }\n\n @IcebergApi\n @Override\n public long getRowCount() {\n return rowCount;\n }\n\n @Override\n public PageReader getPageReader(ColumnDescriptor path) {\n return getPageReader(path.getPath());\n }\n\n public PageReader getPageReader(String[] path) {\n final PageReader pageReader = readers.get(String.join(\".\", path));\n if (pageReader == null) {\n throw new IllegalArgumentException(\n path + \" is not found: \" + readers.keySet() + \" \" + rowCount);\n }\n return pageReader;\n }\n\n @Override\n public Optional getRowIndexes() {\n return rowRanges == null ? Optional.empty() : Optional.of(rowRanges.iterator());\n }\n\n @Override\n public Optional getRowIndexOffset() {\n return this.rowIndexOffset < 0L ? Optional.empty() : Optional.of(this.rowIndexOffset);\n }\n\n void addColumn(ColumnDescriptor path, ColumnPageReader reader) {\n if (readers.put(String.join(\".\", path.getPath()), reader) != null) {\n throw new IllegalStateException(path + \" was already added\");\n }\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/TypeUtil.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.util.Arrays;\n\nimport org.apache.parquet.column.ColumnDescriptor;\nimport org.apache.parquet.schema.*;\nimport org.apache.parquet.schema.LogicalTypeAnnotation.*;\nimport org.apache.spark.package$;\nimport org.apache.spark.sql.execution.datasources.SchemaColumnConvertNotSupportedException;\nimport org.apache.spark.sql.internal.SQLConf;\nimport org.apache.spark.sql.types.*;\n\nimport org.apache.comet.CometConf;\nimport org.apache.comet.IcebergApi;\n\nimport static org.apache.comet.parquet.Utils.descriptorToParquetColumnSpec;\n\npublic class TypeUtil {\n\n /**\n * Converts the input Spark 'field' into a Parquet column descriptor.\n *\n * @see Comet Issue #2079\n */\n @IcebergApi\n public static ColumnDescriptor convertToParquet(StructField field) {\n Type.Repetition repetition;\n int maxDefinitionLevel;\n if (field.nullable()) {\n repetition = Type.Repetition.OPTIONAL;\n maxDefinitionLevel = 1;\n } else {\n repetition = Type.Repetition.REQUIRED;\n maxDefinitionLevel = 0;\n }\n String[] path = new String[] {field.name()};\n\n DataType type = field.dataType();\n\n Types.PrimitiveBuilder builder = null;\n // Only partition column can be `NullType`. Here we piggy-back onto Parquet boolean type\n // for constant vector of null values, we don't really care what Parquet type it is.\n if (type == DataTypes.BooleanType || type == DataTypes.NullType) {\n builder = Types.primitive(PrimitiveType.PrimitiveTypeName.BOOLEAN, repetition);\n } else if (type == DataTypes.IntegerType || type instanceof YearMonthIntervalType) {\n builder =\n Types.primitive(PrimitiveType.PrimitiveTypeName.INT32, repetition)\n .as(LogicalTypeAnnotation.intType(32, true));\n } else if (type == DataTypes.DateType) {\n builder =\n Types.primitive(PrimitiveType.PrimitiveTypeName.INT32, repetition)\n .as(LogicalTypeAnnotation.dateType());\n } else if (type == DataTypes.ByteType) {\n builder =\n Types.primitive(PrimitiveType.PrimitiveTypeName.INT32, repetition)\n .as(LogicalTypeAnnotation.intType(8, true));\n } else if (type == DataTypes.ShortType) {\n builder =\n Types.primitive(PrimitiveType.PrimitiveTypeName.INT32, repetition)\n .as(LogicalTypeAnnotation.intType(16, true));\n } else if (type == DataTypes.LongType) {\n builder = Types.primitive(PrimitiveType.PrimitiveTypeName.INT64, repetition);\n } else if (type == DataTypes.BinaryType) {\n builder = Types.primitive(PrimitiveType.PrimitiveTypeName.BINARY, repetition);\n } else if (type == DataTypes.StringType\n || (type.sameType(DataTypes.StringType) && isSpark40Plus())) {\n builder =\n Types.primitive(PrimitiveType.PrimitiveTypeName.BINARY, repetition)\n .as(LogicalTypeAnnotation.stringType());\n } else if (type == DataTypes.FloatType) {\n builder = Types.primitive(PrimitiveType.PrimitiveTypeName.FLOAT, repetition);\n } else if (type == DataTypes.DoubleType) {\n builder = Types.primitive(PrimitiveType.PrimitiveTypeName.DOUBLE, repetition);\n } else if (type == DataTypes.TimestampType) {\n builder =\n Types.primitive(PrimitiveType.PrimitiveTypeName.INT64, repetition)\n .as(LogicalTypeAnnotation.timestampType(true, TimeUnit.MICROS));\n } else if (type == TimestampNTZType$.MODULE$) {\n builder =\n Types.primitive(PrimitiveType.PrimitiveTypeName.INT64, repetition)\n .as(LogicalTypeAnnotation.timestampType(false, TimeUnit.MICROS));\n } else if (type instanceof DecimalType) {\n DecimalType decimalType = (DecimalType) type;\n builder =\n Types.primitive(PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY, repetition)\n .length(16) // always store as Decimal128\n .as(LogicalTypeAnnotation.decimalType(decimalType.scale(), decimalType.precision()));\n }\n if (builder == null) {\n throw new UnsupportedOperationException(\"Unsupported input Spark type: \" + type);\n }\n\n return new ColumnDescriptor(path, builder.named(field.name()), 0, maxDefinitionLevel);\n }\n\n public static ParquetColumnSpec convertToParquetSpec(StructField field) {\n return descriptorToParquetColumnSpec(convertToParquet(field));\n }\n\n /**\n * Check whether the Parquet 'descriptor' and Spark read type 'sparkType' are compatible. If not,\n * throw exception.\n *\n *

This mostly follows the logic in Spark's\n * ParquetVectorUpdaterFactory#getUpdater(ColumnDescriptor, DataType)\n *\n * @param descriptor descriptor for a Parquet primitive column\n * @param sparkType Spark read type\n */\n public static void checkParquetType(ColumnDescriptor descriptor, DataType sparkType) {\n PrimitiveType.PrimitiveTypeName typeName = descriptor.getPrimitiveType().getPrimitiveTypeName();\n LogicalTypeAnnotation logicalTypeAnnotation =\n descriptor.getPrimitiveType().getLogicalTypeAnnotation();\n boolean allowTypePromotion = (boolean) CometConf.COMET_SCHEMA_EVOLUTION_ENABLED().get();\n\n if (sparkType instanceof NullType) {\n return;\n }\n\n switch (typeName) {\n case BOOLEAN:\n if (sparkType == DataTypes.BooleanType) return;\n break;\n case INT32:\n if (sparkType == DataTypes.IntegerType || canReadAsIntDecimal(descriptor, sparkType)) {\n return;\n } else if (sparkType == DataTypes.LongType\n && isUnsignedIntTypeMatched(logicalTypeAnnotation, 32)) {\n // In `ParquetToSparkSchemaConverter`, we map parquet UINT32 to our LongType.\n // For unsigned int32, it stores as plain signed int32 in Parquet when dictionary\n // fallbacks. We read them as long values.\n return;\n } else if (sparkType == DataTypes.LongType && allowTypePromotion) {\n // In Comet we allow schema evolution from int to long, if\n // `spark.comet.schemaEvolution.enabled` is enabled.\n return;\n } else if (sparkType == DataTypes.ByteType || sparkType == DataTypes.ShortType) {\n return;\n } else if (sparkType == DataTypes.DateType) {\n // TODO: use dateTimeRebaseMode from Spark side\n return;\n } else if (sparkType instanceof YearMonthIntervalType) {\n return;\n } else if (sparkType == DataTypes.DoubleType && isSpark40Plus()) {\n return;\n } else if (sparkType == TimestampNTZType$.MODULE$\n && isSpark40Plus()\n && logicalTypeAnnotation instanceof DateLogicalTypeAnnotation) {\n return;\n }\n break;\n case INT64:\n if (sparkType == DataTypes.LongType || canReadAsLongDecimal(descriptor, sparkType)) {\n return;\n } else if (isLongDecimal(sparkType)\n && isUnsignedIntTypeMatched(logicalTypeAnnotation, 64)) {\n // In `ParquetToSparkSchemaConverter`, we map parquet UINT64 to our Decimal(20, 0).\n // For unsigned int64, it stores as plain signed int64 in Parquet when dictionary\n // fallbacks. We read them as decimal values.\n return;\n } else if (isTimestampTypeMatched(logicalTypeAnnotation, TimeUnit.MICROS)\n && (sparkType == TimestampNTZType$.MODULE$ || sparkType == DataTypes.TimestampType)) {\n validateTimestampType(logicalTypeAnnotation, sparkType);\n // TODO: use dateTimeRebaseMode from Spark side\n return;\n } else if (isTimestampTypeMatched(logicalTypeAnnotation, TimeUnit.MILLIS)\n && (sparkType == TimestampNTZType$.MODULE$ || sparkType == DataTypes.TimestampType)) {\n validateTimestampType(logicalTypeAnnotation, sparkType);\n return;\n }\n break;\n case INT96:\n if (sparkType == TimestampNTZType$.MODULE$) {\n if (isSpark40Plus()) return; // Spark 4.0+ supports Timestamp NTZ with INT96\n convertErrorForTimestampNTZ(typeName.name());\n } else if (sparkType == DataTypes.TimestampType) {\n return;\n }\n break;\n case FLOAT:\n if (sparkType == DataTypes.FloatType) return;\n // In Comet we allow schema evolution from float to double, if\n // `spark.comet.schemaEvolution.enabled` is enabled.\n if (sparkType == DataTypes.DoubleType && allowTypePromotion) return;\n break;\n case DOUBLE:\n if (sparkType == DataTypes.DoubleType) return;\n break;\n case BINARY:\n if (sparkType == DataTypes.StringType\n || sparkType == DataTypes.BinaryType\n || canReadAsBinaryDecimal(descriptor, sparkType)) {\n return;\n }\n\n if (sparkType.sameType(DataTypes.StringType) && isSpark40Plus()) {\n LogicalTypeAnnotation lta = descriptor.getPrimitiveType().getLogicalTypeAnnotation();\n if (lta instanceof LogicalTypeAnnotation.StringLogicalTypeAnnotation) {\n return;\n }\n }\n break;\n case FIXED_LEN_BYTE_ARRAY:\n if (canReadAsIntDecimal(descriptor, sparkType)\n || canReadAsLongDecimal(descriptor, sparkType)\n || canReadAsBinaryDecimal(descriptor, sparkType)\n || sparkType == DataTypes.BinaryType\n // for uuid, since iceberg maps uuid to StringType\n || sparkType == DataTypes.StringType\n && logicalTypeAnnotation\n instanceof LogicalTypeAnnotation.UUIDLogicalTypeAnnotation) {\n return;\n }\n break;\n default:\n break;\n }\n\n throw new SchemaColumnConvertNotSupportedException(\n Arrays.toString(descriptor.getPath()),\n descriptor.getPrimitiveType().getPrimitiveTypeName().toString(),\n sparkType.catalogString());\n }\n\n private static void validateTimestampType(\n LogicalTypeAnnotation logicalTypeAnnotation, DataType sparkType) {\n assert (logicalTypeAnnotation instanceof TimestampLogicalTypeAnnotation);\n // Throw an exception if the Parquet type is TimestampLTZ and the Catalyst type is TimestampNTZ.\n // This is to avoid mistakes in reading the timestamp values.\n if (((TimestampLogicalTypeAnnotation) logicalTypeAnnotation).isAdjustedToUTC()\n && sparkType == TimestampNTZType$.MODULE$\n && !isSpark40Plus()) {\n convertErrorForTimestampNTZ(\"int64 time(\" + logicalTypeAnnotation + \")\");\n }\n }\n\n private static void convertErrorForTimestampNTZ(String parquetType) {\n throw new RuntimeException(\n \"Unable to create Parquet converter for data type \"\n + TimestampNTZType$.MODULE$.json()\n + \" whose Parquet type is \"\n + parquetType);\n }\n\n private static boolean canReadAsIntDecimal(ColumnDescriptor descriptor, DataType dt) {\n if (!DecimalType.is32BitDecimalType(dt) && !(isSpark40Plus() && dt instanceof DecimalType))\n return false;\n return isDecimalTypeMatched(descriptor, dt);\n }\n\n private static boolean canReadAsLongDecimal(ColumnDescriptor descriptor, DataType dt) {\n if (!DecimalType.is64BitDecimalType(dt) && !(isSpark40Plus() && dt instanceof DecimalType))\n return false;\n return isDecimalTypeMatched(descriptor, dt);\n }\n\n private static boolean canReadAsBinaryDecimal(ColumnDescriptor descriptor, DataType dt) {\n if (!DecimalType.isByteArrayDecimalType(dt)) return false;\n return isDecimalTypeMatched(descriptor, dt);\n }\n\n private static boolean isLongDecimal(DataType dt) {\n if (dt instanceof DecimalType) {\n DecimalType d = (DecimalType) dt;\n return d.precision() == 20 && d.scale() == 0;\n }\n return false;\n }\n\n private static boolean isDecimalTypeMatched(ColumnDescriptor descriptor, DataType dt) {\n DecimalType d = (DecimalType) dt;\n LogicalTypeAnnotation typeAnnotation = descriptor.getPrimitiveType().getLogicalTypeAnnotation();\n if (typeAnnotation instanceof DecimalLogicalTypeAnnotation) {\n DecimalLogicalTypeAnnotation decimalType = (DecimalLogicalTypeAnnotation) typeAnnotation;\n // It's OK if the required decimal precision is larger than or equal to the physical decimal\n // precision in the Parquet metadata, as long as the decimal scale is the same.\n return (decimalType.getPrecision() <= d.precision() && decimalType.getScale() == d.scale())\n || (isSpark40Plus()\n && (!SQLConf.get().parquetVectorizedReaderEnabled()\n || (decimalType.getScale() <= d.scale()\n && decimalType.getPrecision() - decimalType.getScale()\n <= d.precision() - d.scale())));\n } else if (isSpark40Plus()) {\n boolean isNullTypeAnnotation = typeAnnotation == null;\n boolean isIntTypeAnnotation = typeAnnotation instanceof IntLogicalTypeAnnotation;\n if (!SQLConf.get().parquetVectorizedReaderEnabled()) {\n return isNullTypeAnnotation || isIntTypeAnnotation;\n } else if (isNullTypeAnnotation\n || (isIntTypeAnnotation && ((IntLogicalTypeAnnotation) typeAnnotation).isSigned())) {\n PrimitiveType.PrimitiveTypeName typeName =\n descriptor.getPrimitiveType().getPrimitiveTypeName();\n int integerPrecision = d.precision() - d.scale();\n switch (typeName) {\n case INT32:\n return integerPrecision >= DecimalType$.MODULE$.IntDecimal().precision();\n case INT64:\n return integerPrecision >= DecimalType$.MODULE$.LongDecimal().precision();\n }\n }\n }\n return false;\n }\n\n private static boolean isTimestampTypeMatched(\n LogicalTypeAnnotation logicalTypeAnnotation, LogicalTypeAnnotation.TimeUnit unit) {\n return logicalTypeAnnotation instanceof TimestampLogicalTypeAnnotation\n && ((TimestampLogicalTypeAnnotation) logicalTypeAnnotation).getUnit() == unit;\n }\n\n private static boolean isUnsignedIntTypeMatched(\n LogicalTypeAnnotation logicalTypeAnnotation, int bitWidth) {\n return logicalTypeAnnotation instanceof IntLogicalTypeAnnotation\n && !((IntLogicalTypeAnnotation) logicalTypeAnnotation).isSigned()\n && ((IntLogicalTypeAnnotation) logicalTypeAnnotation).getBitWidth() == bitWidth;\n }\n\n static boolean isSpark40Plus() {\n return package$.MODULE$.SPARK_VERSION().compareTo(\"4.0\") >= 0;\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/Utils.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.util.HashMap;\nimport java.util.Map;\n\nimport org.apache.parquet.column.ColumnDescriptor;\nimport org.apache.parquet.schema.LogicalTypeAnnotation;\nimport org.apache.parquet.schema.PrimitiveType;\nimport org.apache.parquet.schema.Type;\nimport org.apache.parquet.schema.Types;\nimport org.apache.spark.sql.types.*;\n\nimport org.apache.comet.CometSchemaImporter;\nimport org.apache.comet.IcebergApi;\n\npublic class Utils {\n\n /** This method is called from Apache Iceberg. */\n @IcebergApi\n public static ColumnReader getColumnReader(\n DataType type,\n ParquetColumnSpec columnSpec,\n CometSchemaImporter importer,\n int batchSize,\n boolean useDecimal128,\n boolean useLazyMaterialization,\n boolean useLegacyTimestamp) {\n\n ColumnDescriptor descriptor = buildColumnDescriptor(columnSpec);\n return getColumnReader(\n type,\n descriptor,\n importer,\n batchSize,\n useDecimal128,\n useLazyMaterialization,\n useLegacyTimestamp);\n }\n\n /**\n * This method is called from Apache Iceberg.\n *\n * @see Comet Issue #2079\n */\n @IcebergApi\n public static ColumnReader getColumnReader(\n DataType type,\n ColumnDescriptor descriptor,\n CometSchemaImporter importer,\n int batchSize,\n boolean useDecimal128,\n boolean useLazyMaterialization) {\n // TODO: support `useLegacyDateTimestamp` for Iceberg\n return getColumnReader(\n type, descriptor, importer, batchSize, useDecimal128, useLazyMaterialization, true);\n }\n\n public static ColumnReader getColumnReader(\n DataType type,\n ColumnDescriptor descriptor,\n CometSchemaImporter importer,\n int batchSize,\n boolean useDecimal128,\n boolean useLazyMaterialization,\n boolean useLegacyDateTimestamp) {\n if (useLazyMaterialization && supportLazyMaterialization(type)) {\n return new LazyColumnReader(\n type, descriptor, importer, batchSize, useDecimal128, useLegacyDateTimestamp);\n } else {\n return new ColumnReader(\n type, descriptor, importer, batchSize, useDecimal128, useLegacyDateTimestamp);\n }\n }\n\n private static boolean supportLazyMaterialization(DataType type) {\n return (type instanceof StringType || type instanceof BinaryType);\n }\n\n /**\n * Initialize the Comet native Parquet reader.\n *\n * @param descriptor the Parquet column descriptor for the column to be read\n * @param readType the Spark read type used for type promotion. Null if promotion is not enabled.\n * @param batchSize the batch size, i.e., maximum number of elements per record batch\n * @param useDecimal128 whether to always represent decimals using 128 bits. If false, the native\n * reader may represent decimals using 32 or 64 bits, depending on the precision.\n * @param useLegacyDateTimestampOrNTZ whether to read dates/timestamps that were written in the\n * legacy hybrid Julian + Gregorian calendar as it is. If false, throw exceptions instead. If\n * the spark type is TimestampNTZ, this should be true.\n */\n public static long initColumnReader(\n ColumnDescriptor descriptor,\n DataType readType,\n int batchSize,\n boolean useDecimal128,\n boolean useLegacyDateTimestampOrNTZ) {\n PrimitiveType primitiveType = descriptor.getPrimitiveType();\n int primitiveTypeId = getPhysicalTypeId(primitiveType.getPrimitiveTypeName());\n LogicalTypeAnnotation annotation = primitiveType.getLogicalTypeAnnotation();\n\n // Process logical type information\n\n int bitWidth = -1;\n boolean isSigned = false;\n if (annotation instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation) {\n LogicalTypeAnnotation.IntLogicalTypeAnnotation intAnnotation =\n (LogicalTypeAnnotation.IntLogicalTypeAnnotation) annotation;\n bitWidth = intAnnotation.getBitWidth();\n isSigned = intAnnotation.isSigned();\n }\n\n int precision, scale;\n precision = scale = -1;\n if (annotation instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) {\n LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimalAnnotation =\n (LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) annotation;\n precision = decimalAnnotation.getPrecision();\n scale = decimalAnnotation.getScale();\n }\n\n int tu = -1;\n boolean isAdjustedUtc = false;\n if (annotation instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {\n LogicalTypeAnnotation.TimestampLogicalTypeAnnotation timestampAnnotation =\n (LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) annotation;\n tu = getTimeUnitId(timestampAnnotation.getUnit());\n isAdjustedUtc = timestampAnnotation.isAdjustedToUTC();\n }\n\n TypePromotionInfo promotionInfo;\n if (readType != null) {\n promotionInfo = new TypePromotionInfo(readType);\n } else {\n // If type promotion is not enable, we'll just use the Parquet primitive type and precision.\n promotionInfo = new TypePromotionInfo(primitiveTypeId, precision, scale, bitWidth);\n }\n\n return Native.initColumnReader(\n primitiveTypeId,\n getLogicalTypeId(annotation),\n promotionInfo.physicalTypeId,\n descriptor.getPath(),\n descriptor.getMaxDefinitionLevel(),\n descriptor.getMaxRepetitionLevel(),\n bitWidth,\n promotionInfo.bitWidth,\n isSigned,\n primitiveType.getTypeLength(),\n precision,\n promotionInfo.precision,\n scale,\n promotionInfo.scale,\n tu,\n isAdjustedUtc,\n batchSize,\n useDecimal128,\n useLegacyDateTimestampOrNTZ);\n }\n\n static class TypePromotionInfo {\n // The Parquet physical type ID converted from the Spark read schema, or the original Parquet\n // physical type ID if type promotion is not enabled.\n int physicalTypeId;\n // Decimal precision from the Spark read schema, or -1 if it's not decimal type.\n int precision;\n // Decimal scale from the Spark read schema, or -1 if it's not decimal type.\n int scale;\n // Integer bit width from the Spark read schema, or -1 if it's not integer type.\n int bitWidth;\n\n TypePromotionInfo(int physicalTypeId, int precision, int scale, int bitWidth) {\n this.physicalTypeId = physicalTypeId;\n this.precision = precision;\n this.scale = scale;\n this.bitWidth = bitWidth;\n }\n\n TypePromotionInfo(DataType sparkReadType) {\n // Create a dummy `StructField` from the input Spark type. We don't care about\n // field name, nullability and metadata.\n StructField f = new StructField(\"f\", sparkReadType, false, Metadata.empty());\n ColumnDescriptor descriptor = TypeUtil.convertToParquet(f);\n PrimitiveType primitiveType = descriptor.getPrimitiveType();\n int physicalTypeId = getPhysicalTypeId(primitiveType.getPrimitiveTypeName());\n LogicalTypeAnnotation annotation = primitiveType.getLogicalTypeAnnotation();\n int precision = -1;\n int scale = -1;\n int bitWidth = -1;\n if (annotation instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) {\n LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimalAnnotation =\n (LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) annotation;\n precision = decimalAnnotation.getPrecision();\n scale = decimalAnnotation.getScale();\n }\n if (annotation instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation) {\n LogicalTypeAnnotation.IntLogicalTypeAnnotation intAnnotation =\n (LogicalTypeAnnotation.IntLogicalTypeAnnotation) annotation;\n bitWidth = intAnnotation.getBitWidth();\n }\n this.physicalTypeId = physicalTypeId;\n this.precision = precision;\n this.scale = scale;\n this.bitWidth = bitWidth;\n }\n }\n\n /**\n * Maps the input Parquet physical type 'typeName' to an integer representing it. This is used for\n * serialization between the Java and native side.\n *\n * @param typeName enum for the Parquet physical type\n * @return an integer representing the input physical type\n */\n static int getPhysicalTypeId(PrimitiveType.PrimitiveTypeName typeName) {\n switch (typeName) {\n case BOOLEAN:\n return 0;\n case INT32:\n return 1;\n case INT64:\n return 2;\n case INT96:\n return 3;\n case FLOAT:\n return 4;\n case DOUBLE:\n return 5;\n case BINARY:\n return 6;\n case FIXED_LEN_BYTE_ARRAY:\n return 7;\n }\n throw new IllegalArgumentException(\"Invalid Parquet physical type: \" + typeName);\n }\n\n /**\n * Maps the input Parquet logical type 'annotation' to an integer representing it. This is used\n * for serialization between the Java and native side.\n *\n * @param annotation the Parquet logical type annotation\n * @return an integer representing the input logical type\n */\n static int getLogicalTypeId(LogicalTypeAnnotation annotation) {\n if (annotation == null) {\n return -1; // No logical type associated\n } else if (annotation instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation) {\n return 0;\n } else if (annotation instanceof LogicalTypeAnnotation.StringLogicalTypeAnnotation) {\n return 1;\n } else if (annotation instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) {\n return 2;\n } else if (annotation instanceof LogicalTypeAnnotation.DateLogicalTypeAnnotation) {\n return 3;\n } else if (annotation instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {\n return 4;\n } else if (annotation instanceof LogicalTypeAnnotation.EnumLogicalTypeAnnotation) {\n return 5;\n } else if (annotation instanceof LogicalTypeAnnotation.UUIDLogicalTypeAnnotation) {\n return 6;\n }\n\n throw new UnsupportedOperationException(\"Unsupported Parquet logical type \" + annotation);\n }\n\n static int getTimeUnitId(LogicalTypeAnnotation.TimeUnit tu) {\n switch (tu) {\n case MILLIS:\n return 0;\n case MICROS:\n return 1;\n case NANOS:\n return 2;\n default:\n throw new UnsupportedOperationException(\"Unsupported TimeUnit \" + tu);\n }\n }\n\n @IcebergApi\n public static ColumnDescriptor buildColumnDescriptor(ParquetColumnSpec columnSpec) {\n PrimitiveType.PrimitiveTypeName primType =\n PrimitiveType.PrimitiveTypeName.valueOf(columnSpec.getPhysicalType());\n\n Type.Repetition repetition;\n if (columnSpec.getMaxRepetitionLevel() > 0) {\n repetition = Type.Repetition.REPEATED;\n } else if (columnSpec.getMaxDefinitionLevel() > 0) {\n repetition = Type.Repetition.OPTIONAL;\n } else {\n repetition = Type.Repetition.REQUIRED;\n }\n\n String name = columnSpec.getPath()[columnSpec.getPath().length - 1];\n // Reconstruct the logical type from parameters\n LogicalTypeAnnotation logicalType = null;\n if (columnSpec.getLogicalTypeName() != null) {\n logicalType =\n reconstructLogicalType(\n columnSpec.getLogicalTypeName(), columnSpec.getLogicalTypeParams());\n }\n\n PrimitiveType primitiveType;\n if (primType == PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY) {\n primitiveType =\n Types.primitive(primType, repetition)\n .length(columnSpec.getTypeLength())\n .as(logicalType)\n .id(columnSpec.getFieldId())\n .named(name);\n } else {\n primitiveType =\n Types.primitive(primType, repetition)\n .as(logicalType)\n .id(columnSpec.getFieldId())\n .named(name);\n }\n\n return new ColumnDescriptor(\n columnSpec.getPath(),\n primitiveType,\n columnSpec.getMaxRepetitionLevel(),\n columnSpec.getMaxDefinitionLevel());\n }\n\n private static LogicalTypeAnnotation reconstructLogicalType(\n String logicalTypeName, java.util.Map params) {\n\n switch (logicalTypeName) {\n // MAP\n case \"MapLogicalTypeAnnotation\":\n return LogicalTypeAnnotation.mapType();\n\n // LIST\n case \"ListLogicalTypeAnnotation\":\n return LogicalTypeAnnotation.listType();\n\n // STRING\n case \"StringLogicalTypeAnnotation\":\n return LogicalTypeAnnotation.stringType();\n\n // MAP_KEY_VALUE\n case \"MapKeyValueLogicalTypeAnnotation\":\n return LogicalTypeAnnotation.MapKeyValueTypeAnnotation.getInstance();\n\n // ENUM\n case \"EnumLogicalTypeAnnotation\":\n return LogicalTypeAnnotation.enumType();\n\n // DECIMAL\n case \"DecimalLogicalTypeAnnotation\":\n if (!params.containsKey(\"scale\") || !params.containsKey(\"precision\")) {\n throw new IllegalArgumentException(\n \"Missing required parameters for DecimalLogicalTypeAnnotation: \" + params);\n }\n int scale = Integer.parseInt(params.get(\"scale\"));\n int precision = Integer.parseInt(params.get(\"precision\"));\n return LogicalTypeAnnotation.decimalType(scale, precision);\n\n // DATE\n case \"DateLogicalTypeAnnotation\":\n return LogicalTypeAnnotation.dateType();\n\n // TIME\n case \"TimeLogicalTypeAnnotation\":\n if (!params.containsKey(\"isAdjustedToUTC\") || !params.containsKey(\"unit\")) {\n throw new IllegalArgumentException(\n \"Missing required parameters for TimeLogicalTypeAnnotation: \" + params);\n }\n\n boolean isUTC = Boolean.parseBoolean(params.get(\"isAdjustedToUTC\"));\n String timeUnitStr = params.get(\"unit\");\n\n LogicalTypeAnnotation.TimeUnit timeUnit;\n switch (timeUnitStr) {\n case \"MILLIS\":\n timeUnit = LogicalTypeAnnotation.TimeUnit.MILLIS;\n break;\n case \"MICROS\":\n timeUnit = LogicalTypeAnnotation.TimeUnit.MICROS;\n break;\n case \"NANOS\":\n timeUnit = LogicalTypeAnnotation.TimeUnit.NANOS;\n break;\n default:\n throw new IllegalArgumentException(\"Unknown time unit: \" + timeUnitStr);\n }\n return LogicalTypeAnnotation.timeType(isUTC, timeUnit);\n\n // TIMESTAMP\n case \"TimestampLogicalTypeAnnotation\":\n if (!params.containsKey(\"isAdjustedToUTC\") || !params.containsKey(\"unit\")) {\n throw new IllegalArgumentException(\n \"Missing required parameters for TimestampLogicalTypeAnnotation: \" + params);\n }\n boolean isAdjustedToUTC = Boolean.parseBoolean(params.get(\"isAdjustedToUTC\"));\n String unitStr = params.get(\"unit\");\n\n LogicalTypeAnnotation.TimeUnit unit;\n switch (unitStr) {\n case \"MILLIS\":\n unit = LogicalTypeAnnotation.TimeUnit.MILLIS;\n break;\n case \"MICROS\":\n unit = LogicalTypeAnnotation.TimeUnit.MICROS;\n break;\n case \"NANOS\":\n unit = LogicalTypeAnnotation.TimeUnit.NANOS;\n break;\n default:\n throw new IllegalArgumentException(\"Unknown timestamp unit: \" + unitStr);\n }\n return LogicalTypeAnnotation.timestampType(isAdjustedToUTC, unit);\n\n // INTEGER\n case \"IntLogicalTypeAnnotation\":\n if (!params.containsKey(\"isSigned\") || !params.containsKey(\"bitWidth\")) {\n throw new IllegalArgumentException(\n \"Missing required parameters for IntLogicalTypeAnnotation: \" + params);\n }\n boolean isSigned = Boolean.parseBoolean(params.get(\"isSigned\"));\n int bitWidth = Integer.parseInt(params.get(\"bitWidth\"));\n return LogicalTypeAnnotation.intType(bitWidth, isSigned);\n\n // JSON\n case \"JsonLogicalTypeAnnotation\":\n return LogicalTypeAnnotation.jsonType();\n\n // BSON\n case \"BsonLogicalTypeAnnotation\":\n return LogicalTypeAnnotation.bsonType();\n\n // UUID\n case \"UUIDLogicalTypeAnnotation\":\n return LogicalTypeAnnotation.uuidType();\n\n // INTERVAL\n case \"IntervalLogicalTypeAnnotation\":\n return LogicalTypeAnnotation.IntervalLogicalTypeAnnotation.getInstance();\n\n default:\n throw new IllegalArgumentException(\"Unknown logical type: \" + logicalTypeName);\n }\n }\n\n @IcebergApi\n public static ParquetColumnSpec descriptorToParquetColumnSpec(ColumnDescriptor descriptor) {\n\n String[] path = descriptor.getPath();\n PrimitiveType primitiveType = descriptor.getPrimitiveType();\n String physicalType = primitiveType.getPrimitiveTypeName().name();\n\n int typeLength =\n primitiveType.getPrimitiveTypeName() == PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY\n ? primitiveType.getTypeLength()\n : 0;\n\n boolean isRepeated = primitiveType.getRepetition() == Type.Repetition.REPEATED;\n\n String logicalTypeName = null;\n Map logicalTypeParams = new HashMap<>();\n LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();\n\n if (logicalType != null) {\n logicalTypeName = logicalType.getClass().getSimpleName();\n\n // Handle specific logical types\n if (logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) {\n LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimal =\n (LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) logicalType;\n logicalTypeParams.put(\"precision\", String.valueOf(decimal.getPrecision()));\n logicalTypeParams.put(\"scale\", String.valueOf(decimal.getScale()));\n } else if (logicalType instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {\n LogicalTypeAnnotation.TimestampLogicalTypeAnnotation timestamp =\n (LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) logicalType;\n logicalTypeParams.put(\"isAdjustedToUTC\", String.valueOf(timestamp.isAdjustedToUTC()));\n logicalTypeParams.put(\"unit\", timestamp.getUnit().name());\n } else if (logicalType instanceof LogicalTypeAnnotation.TimeLogicalTypeAnnotation) {\n LogicalTypeAnnotation.TimeLogicalTypeAnnotation time =\n (LogicalTypeAnnotation.TimeLogicalTypeAnnotation) logicalType;\n logicalTypeParams.put(\"isAdjustedToUTC\", String.valueOf(time.isAdjustedToUTC()));\n logicalTypeParams.put(\"unit\", time.getUnit().name());\n } else if (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation) {\n LogicalTypeAnnotation.IntLogicalTypeAnnotation intType =\n (LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType;\n logicalTypeParams.put(\"isSigned\", String.valueOf(intType.isSigned()));\n logicalTypeParams.put(\"bitWidth\", String.valueOf(intType.getBitWidth()));\n }\n }\n\n int id = -1;\n Type type = descriptor.getPrimitiveType();\n if (type != null && type.getId() != null) {\n id = type.getId().intValue();\n }\n\n return new ParquetColumnSpec(\n id,\n path,\n physicalType,\n typeLength,\n isRepeated,\n descriptor.getMaxDefinitionLevel(),\n descriptor.getMaxRepetitionLevel(),\n logicalTypeName,\n logicalTypeParams);\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/WrappedInputFile.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.io.IOException;\nimport java.io.InputStream;\nimport java.lang.reflect.Method;\n\nimport org.apache.parquet.io.InputFile;\nimport org.apache.parquet.io.SeekableInputStream;\n\nimport org.apache.comet.IcebergApi;\n\n/**\n * Wraps an Object that possibly implements the methods of a Parquet InputFile (but is not a Parquet\n * InputFile). Such an object` exists, for instance, in Iceberg's InputFile\n */\n@IcebergApi\npublic class WrappedInputFile implements InputFile {\n Object wrapped;\n\n @IcebergApi\n public WrappedInputFile(Object inputFile) {\n this.wrapped = inputFile;\n }\n\n @Override\n public long getLength() throws IOException {\n try {\n Method targetMethod = wrapped.getClass().getDeclaredMethod(\"getLength\"); //\n targetMethod.setAccessible(true);\n return (long) targetMethod.invoke(wrapped);\n } catch (Exception e) {\n throw new IOException(e);\n }\n }\n\n @Override\n public SeekableInputStream newStream() throws IOException {\n try {\n Method targetMethod = wrapped.getClass().getDeclaredMethod(\"newStream\"); //\n targetMethod.setAccessible(true);\n InputStream stream = (InputStream) targetMethod.invoke(wrapped);\n return new WrappedSeekableInputStream(stream);\n } catch (Exception e) {\n throw new IOException(e);\n }\n }\n\n @Override\n public String toString() {\n return wrapped.toString();\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/parquet/WrappedSeekableInputStream.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.io.IOException;\nimport java.io.InputStream;\nimport java.lang.reflect.Method;\nimport java.util.Objects;\n\nimport org.apache.parquet.io.DelegatingSeekableInputStream;\n\n/**\n * Wraps an InputStream that possibly implements the methods of a Parquet SeekableInputStream (but\n * is not a Parquet SeekableInputStream). Such an InputStream exists, for instance, in Iceberg's\n * SeekableInputStream\n */\npublic class WrappedSeekableInputStream extends DelegatingSeekableInputStream {\n\n private final InputStream wrappedInputStream; // The InputStream we are wrapping\n\n public WrappedSeekableInputStream(InputStream inputStream) {\n super(inputStream);\n this.wrappedInputStream = Objects.requireNonNull(inputStream, \"InputStream cannot be null\");\n }\n\n @Override\n public long getPos() throws IOException {\n try {\n Method targetMethod = wrappedInputStream.getClass().getDeclaredMethod(\"getPos\"); //\n targetMethod.setAccessible(true);\n return (long) targetMethod.invoke(wrappedInputStream);\n } catch (Exception e) {\n throw new IOException(e);\n }\n }\n\n @Override\n public void seek(long newPos) throws IOException {\n try {\n Method targetMethod = wrappedInputStream.getClass().getDeclaredMethod(\"seek\", long.class);\n targetMethod.setAccessible(true);\n targetMethod.invoke(wrappedInputStream, newPos);\n } catch (Exception e) {\n throw new IOException(e);\n }\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/vector/CometDecodedVector.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.vector;\n\nimport org.apache.arrow.vector.BaseVariableWidthVector;\nimport org.apache.arrow.vector.ValueVector;\nimport org.apache.arrow.vector.types.pojo.Field;\nimport org.apache.spark.sql.comet.util.Utils;\nimport org.apache.spark.unsafe.Platform;\n\n/** A Comet vector whose elements are already decoded (i.e., materialized). */\npublic abstract class CometDecodedVector extends CometVector {\n /**\n * The vector that stores all the values. For dictionary-backed vector, this is the vector of\n * indices.\n */\n protected final ValueVector valueVector;\n\n private boolean hasNull;\n private int numNulls;\n private int numValues;\n private int validityByteCacheIndex = -1;\n private byte validityByteCache;\n protected boolean isUuid;\n\n protected CometDecodedVector(ValueVector vector, Field valueField, boolean useDecimal128) {\n this(vector, valueField, useDecimal128, false);\n }\n\n protected CometDecodedVector(\n ValueVector vector, Field valueField, boolean useDecimal128, boolean isUuid) {\n super(Utils.fromArrowField(valueField), useDecimal128);\n this.valueVector = vector;\n this.numNulls = valueVector.getNullCount();\n this.numValues = valueVector.getValueCount();\n this.hasNull = numNulls != 0;\n this.isUuid = isUuid;\n }\n\n @Override\n public ValueVector getValueVector() {\n return valueVector;\n }\n\n @Override\n public void setNumNulls(int numNulls) {\n // We don't need to update null count in 'valueVector' since 'ValueVector.getNullCount' will\n // re-compute the null count from validity buffer.\n this.numNulls = numNulls;\n this.hasNull = numNulls != 0;\n this.validityByteCacheIndex = -1;\n }\n\n @Override\n public void setNumValues(int numValues) {\n this.numValues = numValues;\n if (valueVector instanceof BaseVariableWidthVector) {\n BaseVariableWidthVector bv = (BaseVariableWidthVector) valueVector;\n // In case `lastSet` is smaller than `numValues`, `setValueCount` will set all the offsets\n // within `[lastSet + 1, numValues)` to be empty, which is incorrect in our case.\n //\n // For instance, this can happen if one first call `setNumValues` with input 100, and then\n // again `setNumValues` with 200. The first call will set `lastSet` to 99, while the second\n // call will set all strings between indices `[100, 200)` to be empty.\n bv.setLastSet(numValues);\n }\n valueVector.setValueCount(numValues);\n }\n\n public int numValues() {\n return numValues;\n }\n\n @Override\n public boolean hasNull() {\n return hasNull;\n }\n\n @Override\n public int numNulls() {\n return numNulls;\n }\n\n @Override\n public boolean isNullAt(int rowId) {\n if (!hasNull) return false;\n\n int byteIndex = rowId >> 3;\n if (byteIndex != validityByteCacheIndex) {\n long validityBufferAddress = valueVector.getValidityBuffer().memoryAddress();\n validityByteCache = Platform.getByte(null, validityBufferAddress + byteIndex);\n validityByteCacheIndex = byteIndex;\n }\n return ((validityByteCache >> (rowId & 7)) & 1) == 0;\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/vector/CometDelegateVector.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.vector;\n\nimport org.apache.arrow.vector.ValueVector;\nimport org.apache.arrow.vector.dictionary.DictionaryProvider;\nimport org.apache.spark.sql.types.DataType;\nimport org.apache.spark.sql.types.Decimal;\nimport org.apache.spark.sql.vectorized.ColumnVector;\nimport org.apache.spark.sql.vectorized.ColumnarArray;\nimport org.apache.spark.sql.vectorized.ColumnarMap;\nimport org.apache.spark.unsafe.types.UTF8String;\n\n/** A special Comet vector that just delegate all calls */\npublic class CometDelegateVector extends CometVector {\n protected CometVector delegate;\n\n public CometDelegateVector(DataType dataType) {\n this(dataType, null, false);\n }\n\n public CometDelegateVector(DataType dataType, boolean useDecimal128) {\n this(dataType, null, useDecimal128);\n }\n\n public CometDelegateVector(DataType dataType, CometVector delegate, boolean useDecimal128) {\n super(dataType, useDecimal128);\n if (delegate instanceof CometDelegateVector) {\n throw new IllegalArgumentException(\"cannot have nested delegation\");\n }\n this.delegate = delegate;\n }\n\n protected void setDelegate(CometVector delegate) {\n this.delegate = delegate;\n }\n\n @Override\n public void setNumNulls(int numNulls) {\n delegate.setNumNulls(numNulls);\n }\n\n @Override\n public void setNumValues(int numValues) {\n delegate.setNumValues(numValues);\n }\n\n @Override\n public int numValues() {\n return delegate.numValues();\n }\n\n @Override\n public boolean hasNull() {\n return delegate.hasNull();\n }\n\n @Override\n public int numNulls() {\n return delegate.numNulls();\n }\n\n @Override\n public boolean isNullAt(int rowId) {\n return delegate.isNullAt(rowId);\n }\n\n @Override\n public boolean getBoolean(int rowId) {\n return delegate.getBoolean(rowId);\n }\n\n @Override\n public byte getByte(int rowId) {\n return delegate.getByte(rowId);\n }\n\n @Override\n public short getShort(int rowId) {\n return delegate.getShort(rowId);\n }\n\n @Override\n public int getInt(int rowId) {\n return delegate.getInt(rowId);\n }\n\n @Override\n public long getLong(int rowId) {\n return delegate.getLong(rowId);\n }\n\n @Override\n public long getLongDecimal(int rowId) {\n return delegate.getLongDecimal(rowId);\n }\n\n @Override\n public float getFloat(int rowId) {\n return delegate.getFloat(rowId);\n }\n\n @Override\n public double getDouble(int rowId) {\n return delegate.getDouble(rowId);\n }\n\n @Override\n public Decimal getDecimal(int i, int precision, int scale) {\n return delegate.getDecimal(i, precision, scale);\n }\n\n @Override\n byte[] getBinaryDecimal(int i) {\n return delegate.getBinaryDecimal(i);\n }\n\n @Override\n public UTF8String getUTF8String(int rowId) {\n return delegate.getUTF8String(rowId);\n }\n\n @Override\n public byte[] getBinary(int rowId) {\n return delegate.getBinary(rowId);\n }\n\n @Override\n public ColumnarArray getArray(int i) {\n return delegate.getArray(i);\n }\n\n @Override\n public ColumnarMap getMap(int i) {\n return delegate.getMap(i);\n }\n\n @Override\n public ColumnVector getChild(int i) {\n return delegate.getChild(i);\n }\n\n @Override\n public ValueVector getValueVector() {\n return delegate.getValueVector();\n }\n\n @Override\n public CometVector slice(int offset, int length) {\n return delegate.slice(offset, length);\n }\n\n @Override\n public DictionaryProvider getDictionaryProvider() {\n return delegate.getDictionaryProvider();\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/vector/CometDictionary.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.vector;\n\nimport org.apache.arrow.vector.ValueVector;\nimport org.apache.spark.unsafe.types.UTF8String;\n\n/** A dictionary which maps indices (integers) to values. */\npublic class CometDictionary implements AutoCloseable {\n private static final int DECIMAL_BYTE_WIDTH = 16;\n\n private CometPlainVector values;\n private final int numValues;\n\n /** Decoded dictionary values. We only need to copy values for decimal type. */\n private volatile ByteArrayWrapper[] binaries;\n\n public CometDictionary(CometPlainVector values) {\n this.values = values;\n this.numValues = values.numValues();\n }\n\n public void setDictionaryVector(CometPlainVector values) {\n this.values = values;\n if (values.numValues() != numValues) {\n throw new IllegalArgumentException(\"Mismatched dictionary size\");\n }\n }\n\n public ValueVector getValueVector() {\n return values.getValueVector();\n }\n\n public boolean decodeToBoolean(int index) {\n return values.getBoolean(index);\n }\n\n public byte decodeToByte(int index) {\n return values.getByte(index);\n }\n\n public short decodeToShort(int index) {\n return values.getShort(index);\n }\n\n public int decodeToInt(int index) {\n return values.getInt(index);\n }\n\n public long decodeToLong(int index) {\n return values.getLong(index);\n }\n\n public long decodeToLongDecimal(int index) {\n return values.getLongDecimal(index);\n }\n\n public float decodeToFloat(int index) {\n return values.getFloat(index);\n }\n\n public double decodeToDouble(int index) {\n return values.getDouble(index);\n }\n\n public byte[] decodeToBinary(int index) {\n switch (values.getValueVector().getMinorType()) {\n case VARBINARY:\n case FIXEDSIZEBINARY:\n return values.getBinary(index);\n case DECIMAL:\n if (binaries == null) {\n // We only need to copy values for decimal 128 type as random access\n // to the dictionary is not efficient for decimal (it needs to copy\n // the value to a new byte array everytime).\n ByteArrayWrapper[] binaries = new ByteArrayWrapper[numValues];\n for (int i = 0; i < numValues; i++) {\n // Need copying here since we re-use byte array for decimal\n byte[] bytes = new byte[DECIMAL_BYTE_WIDTH];\n bytes = values.copyBinaryDecimal(i, bytes);\n binaries[i] = new ByteArrayWrapper(bytes);\n }\n this.binaries = binaries;\n }\n return binaries[index].bytes;\n default:\n throw new IllegalArgumentException(\n \"Invalid Arrow minor type: \" + values.getValueVector().getMinorType());\n }\n }\n\n public UTF8String decodeToUTF8String(int index) {\n return values.getUTF8String(index);\n }\n\n @Override\n public void close() {\n values.close();\n }\n\n private static class ByteArrayWrapper {\n private final byte[] bytes;\n\n ByteArrayWrapper(byte[] bytes) {\n this.bytes = bytes;\n }\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/vector/CometDictionaryVector.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.vector;\n\nimport org.apache.arrow.vector.IntVector;\nimport org.apache.arrow.vector.dictionary.DictionaryProvider;\nimport org.apache.arrow.vector.util.TransferPair;\nimport org.apache.parquet.Preconditions;\nimport org.apache.spark.unsafe.types.UTF8String;\n\n/** A column vector whose elements are dictionary-encoded. */\npublic class CometDictionaryVector extends CometDecodedVector {\n public final CometPlainVector indices;\n public final CometDictionary values;\n public final DictionaryProvider provider;\n\n /** Whether this vector is an alias sliced from another vector. */\n private final boolean isAlias;\n\n public CometDictionaryVector(\n CometPlainVector indices,\n CometDictionary values,\n DictionaryProvider provider,\n boolean useDecimal128) {\n this(indices, values, provider, useDecimal128, false, false);\n }\n\n public CometDictionaryVector(\n CometPlainVector indices,\n CometDictionary values,\n DictionaryProvider provider,\n boolean useDecimal128,\n boolean isAlias,\n boolean isUuid) {\n super(indices.valueVector, values.getValueVector().getField(), useDecimal128, isUuid);\n Preconditions.checkArgument(\n indices.valueVector instanceof IntVector, \"'indices' should be a IntVector\");\n this.values = values;\n this.indices = indices;\n this.provider = provider;\n this.isAlias = isAlias;\n }\n\n @Override\n public DictionaryProvider getDictionaryProvider() {\n return this.provider;\n }\n\n @Override\n public void close() {\n super.close();\n // Only close the values vector if this is not a sliced vector.\n if (!isAlias) {\n values.close();\n }\n }\n\n @Override\n public boolean getBoolean(int i) {\n return values.decodeToBoolean(indices.getInt(i));\n }\n\n @Override\n public byte getByte(int i) {\n return values.decodeToByte(indices.getInt(i));\n }\n\n @Override\n public short getShort(int i) {\n return values.decodeToShort(indices.getInt(i));\n }\n\n @Override\n public int getInt(int i) {\n return values.decodeToInt(indices.getInt(i));\n }\n\n @Override\n public long getLong(int i) {\n return values.decodeToLong(indices.getInt(i));\n }\n\n @Override\n public long getLongDecimal(int i) {\n return values.decodeToLongDecimal(indices.getInt(i));\n }\n\n @Override\n public float getFloat(int i) {\n return values.decodeToFloat(indices.getInt(i));\n }\n\n @Override\n public double getDouble(int i) {\n return values.decodeToDouble(indices.getInt(i));\n }\n\n @Override\n public UTF8String getUTF8String(int i) {\n return values.decodeToUTF8String(indices.getInt(i));\n }\n\n @Override\n public byte[] getBinary(int i) {\n return values.decodeToBinary(indices.getInt(i));\n }\n\n @Override\n byte[] getBinaryDecimal(int i) {\n return values.decodeToBinary(indices.getInt(i));\n }\n\n @Override\n public CometVector slice(int offset, int length) {\n TransferPair tp = indices.valueVector.getTransferPair(indices.valueVector.getAllocator());\n tp.splitAndTransfer(offset, length);\n CometPlainVector sliced = new CometPlainVector(tp.getTo(), useDecimal128);\n\n // Set the alias flag to true so that the sliced vector will not close the dictionary vector.\n // Otherwise, if the dictionary is closed, the sliced vector will not be able to access the\n // dictionary.\n return new CometDictionaryVector(sliced, values, provider, useDecimal128, true, isUuid);\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/vector/CometLazyVector.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.vector;\n\nimport org.apache.arrow.vector.ValueVector;\nimport org.apache.spark.sql.types.DataType;\n\nimport org.apache.comet.parquet.LazyColumnReader;\n\npublic class CometLazyVector extends CometDelegateVector {\n private final LazyColumnReader columnReader;\n\n public CometLazyVector(DataType type, LazyColumnReader columnReader, boolean useDecimal128) {\n super(type, useDecimal128);\n this.columnReader = columnReader;\n }\n\n public CometDecodedVector getDecodedVector() {\n return (CometDecodedVector) delegate;\n }\n\n @Override\n public ValueVector getValueVector() {\n columnReader.readAllBatch();\n setDelegate(columnReader.loadVector());\n return super.getValueVector();\n }\n\n @Override\n public void setNumNulls(int numNulls) {\n throw new UnsupportedOperationException(\"CometLazyVector doesn't support 'setNumNulls'\");\n }\n\n @Override\n public void setNumValues(int numValues) {\n throw new UnsupportedOperationException(\"CometLazyVector doesn't support 'setNumValues'\");\n }\n\n @Override\n public void close() {\n // Do nothing. 'vector' is closed by 'columnReader' which owns it.\n }\n\n @Override\n public boolean hasNull() {\n columnReader.readAllBatch();\n setDelegate(columnReader.loadVector());\n return super.hasNull();\n }\n\n @Override\n public int numNulls() {\n columnReader.readAllBatch();\n setDelegate(columnReader.loadVector());\n return super.numNulls();\n }\n\n @Override\n public boolean isNullAt(int rowId) {\n if (columnReader.materializeUpToIfNecessary(rowId)) {\n setDelegate(columnReader.loadVector());\n }\n return super.isNullAt(rowId);\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/vector/CometListVector.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.vector;\n\nimport org.apache.arrow.vector.*;\nimport org.apache.arrow.vector.complex.ListVector;\nimport org.apache.arrow.vector.dictionary.DictionaryProvider;\nimport org.apache.arrow.vector.util.TransferPair;\nimport org.apache.spark.sql.vectorized.ColumnVector;\nimport org.apache.spark.sql.vectorized.ColumnarArray;\n\n/** A Comet column vector for list type. */\npublic class CometListVector extends CometDecodedVector {\n final ListVector listVector;\n final ValueVector dataVector;\n final ColumnVector dataColumnVector;\n final DictionaryProvider dictionaryProvider;\n\n public CometListVector(\n ValueVector vector, boolean useDecimal128, DictionaryProvider dictionaryProvider) {\n super(vector, vector.getField(), useDecimal128);\n\n this.listVector = ((ListVector) vector);\n this.dataVector = listVector.getDataVector();\n this.dictionaryProvider = dictionaryProvider;\n this.dataColumnVector = getVector(dataVector, useDecimal128, dictionaryProvider);\n }\n\n @Override\n public ColumnarArray getArray(int i) {\n if (isNullAt(i)) return null;\n int start = listVector.getOffsetBuffer().getInt(i * ListVector.OFFSET_WIDTH);\n int end = listVector.getOffsetBuffer().getInt((i + 1) * ListVector.OFFSET_WIDTH);\n\n return new ColumnarArray(dataColumnVector, start, end - start);\n }\n\n @Override\n public CometVector slice(int offset, int length) {\n TransferPair tp = this.valueVector.getTransferPair(this.valueVector.getAllocator());\n tp.splitAndTransfer(offset, length);\n\n return new CometListVector(tp.getTo(), useDecimal128, dictionaryProvider);\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/vector/CometMapVector.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.vector;\n\nimport org.apache.arrow.vector.*;\nimport org.apache.arrow.vector.complex.MapVector;\nimport org.apache.arrow.vector.complex.StructVector;\nimport org.apache.arrow.vector.dictionary.DictionaryProvider;\nimport org.apache.arrow.vector.util.TransferPair;\nimport org.apache.spark.sql.vectorized.ColumnVector;\nimport org.apache.spark.sql.vectorized.ColumnarMap;\n\n/** A Comet column vector for map type. */\npublic class CometMapVector extends CometDecodedVector {\n final MapVector mapVector;\n final ValueVector dataVector;\n final CometStructVector dataColumnVector;\n final DictionaryProvider dictionaryProvider;\n\n final ColumnVector keys;\n final ColumnVector values;\n\n public CometMapVector(\n ValueVector vector, boolean useDecimal128, DictionaryProvider dictionaryProvider) {\n super(vector, vector.getField(), useDecimal128);\n\n this.mapVector = ((MapVector) vector);\n this.dataVector = mapVector.getDataVector();\n this.dictionaryProvider = dictionaryProvider;\n\n if (dataVector instanceof StructVector) {\n this.dataColumnVector = new CometStructVector(dataVector, useDecimal128, dictionaryProvider);\n\n if (dataColumnVector.children.size() != 2) {\n throw new RuntimeException(\n \"MapVector's dataVector should have 2 children, but got: \"\n + dataColumnVector.children.size());\n }\n\n this.keys = dataColumnVector.getChild(0);\n this.values = dataColumnVector.getChild(1);\n } else {\n throw new RuntimeException(\n \"MapVector's dataVector should be StructVector, but got: \"\n + dataVector.getClass().getSimpleName());\n }\n }\n\n @Override\n public ColumnarMap getMap(int i) {\n if (isNullAt(i)) return null;\n int start = mapVector.getOffsetBuffer().getInt(i * MapVector.OFFSET_WIDTH);\n int end = mapVector.getOffsetBuffer().getInt((i + 1) * MapVector.OFFSET_WIDTH);\n\n return new ColumnarMap(keys, values, start, end - start);\n }\n\n @Override\n public CometVector slice(int offset, int length) {\n TransferPair tp = this.valueVector.getTransferPair(this.valueVector.getAllocator());\n tp.splitAndTransfer(offset, length);\n\n return new CometMapVector(tp.getTo(), useDecimal128, dictionaryProvider);\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/vector/CometPlainVector.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.vector;\n\nimport java.nio.ByteBuffer;\nimport java.nio.ByteOrder;\nimport java.util.UUID;\n\nimport org.apache.arrow.c.CDataDictionaryProvider;\nimport org.apache.arrow.vector.*;\nimport org.apache.arrow.vector.util.TransferPair;\nimport org.apache.parquet.Preconditions;\nimport org.apache.spark.unsafe.Platform;\nimport org.apache.spark.unsafe.types.UTF8String;\n\n/** A column vector whose elements are plainly decoded. */\npublic class CometPlainVector extends CometDecodedVector {\n private final long valueBufferAddress;\n private final boolean isBaseFixedWidthVector;\n\n private byte booleanByteCache;\n private int booleanByteCacheIndex = -1;\n\n private boolean isReused;\n\n public CometPlainVector(ValueVector vector, boolean useDecimal128) {\n this(vector, useDecimal128, false);\n }\n\n public CometPlainVector(ValueVector vector, boolean useDecimal128, boolean isUuid) {\n this(vector, useDecimal128, isUuid, false);\n }\n\n public CometPlainVector(\n ValueVector vector, boolean useDecimal128, boolean isUuid, boolean isReused) {\n super(vector, vector.getField(), useDecimal128, isUuid);\n // NullType doesn't have data buffer.\n if (vector instanceof NullVector) {\n this.valueBufferAddress = -1;\n } else {\n this.valueBufferAddress = vector.getDataBuffer().memoryAddress();\n }\n\n isBaseFixedWidthVector = valueVector instanceof BaseFixedWidthVector;\n this.isReused = isReused;\n }\n\n public boolean isReused() {\n return isReused;\n }\n\n public void setReused(boolean isReused) {\n this.isReused = isReused;\n }\n\n @Override\n public void setNumNulls(int numNulls) {\n super.setNumNulls(numNulls);\n this.booleanByteCacheIndex = -1;\n }\n\n @Override\n public boolean getBoolean(int rowId) {\n int byteIndex = rowId >> 3;\n if (byteIndex != booleanByteCacheIndex) {\n booleanByteCache = getByte(byteIndex);\n booleanByteCacheIndex = byteIndex;\n }\n return ((booleanByteCache >> (rowId & 7)) & 1) == 1;\n }\n\n @Override\n public byte getByte(int rowId) {\n return Platform.getByte(null, valueBufferAddress + rowId);\n }\n\n @Override\n public short getShort(int rowId) {\n return Platform.getShort(null, valueBufferAddress + rowId * 2L);\n }\n\n @Override\n public int getInt(int rowId) {\n return Platform.getInt(null, valueBufferAddress + rowId * 4L);\n }\n\n @Override\n public long getLong(int rowId) {\n return Platform.getLong(null, valueBufferAddress + rowId * 8L);\n }\n\n @Override\n public long getLongDecimal(int rowId) {\n return Platform.getLong(null, valueBufferAddress + rowId * 16L);\n }\n\n @Override\n public float getFloat(int rowId) {\n return Platform.getFloat(null, valueBufferAddress + rowId * 4L);\n }\n\n @Override\n public double getDouble(int rowId) {\n return Platform.getDouble(null, valueBufferAddress + rowId * 8L);\n }\n\n @Override\n public UTF8String getUTF8String(int rowId) {\n if (isNullAt(rowId)) return null;\n if (!isBaseFixedWidthVector) {\n BaseVariableWidthVector varWidthVector = (BaseVariableWidthVector) valueVector;\n long offsetBufferAddress = varWidthVector.getOffsetBuffer().memoryAddress();\n int offset = Platform.getInt(null, offsetBufferAddress + rowId * 4L);\n int length = Platform.getInt(null, offsetBufferAddress + (rowId + 1L) * 4L) - offset;\n return UTF8String.fromAddress(null, valueBufferAddress + offset, length);\n } else {\n BaseFixedWidthVector fixedWidthVector = (BaseFixedWidthVector) valueVector;\n int length = fixedWidthVector.getTypeWidth();\n int offset = rowId * length;\n byte[] result = new byte[length];\n Platform.copyMemory(\n null, valueBufferAddress + offset, result, Platform.BYTE_ARRAY_OFFSET, length);\n\n if (!isUuid) {\n return UTF8String.fromBytes(result);\n } else {\n return UTF8String.fromString(convertToUuid(result).toString());\n }\n }\n }\n\n @Override\n public byte[] getBinary(int rowId) {\n if (isNullAt(rowId)) return null;\n int offset;\n int length;\n if (valueVector instanceof BaseVariableWidthVector) {\n BaseVariableWidthVector varWidthVector = (BaseVariableWidthVector) valueVector;\n long offsetBufferAddress = varWidthVector.getOffsetBuffer().memoryAddress();\n offset = Platform.getInt(null, offsetBufferAddress + rowId * 4L);\n length = Platform.getInt(null, offsetBufferAddress + (rowId + 1L) * 4L) - offset;\n } else if (valueVector instanceof BaseFixedWidthVector) {\n BaseFixedWidthVector fixedWidthVector = (BaseFixedWidthVector) valueVector;\n length = fixedWidthVector.getTypeWidth();\n offset = rowId * length;\n } else {\n throw new RuntimeException(\"Unsupported binary vector type: \" + valueVector.getName());\n }\n byte[] result = new byte[length];\n Platform.copyMemory(\n null, valueBufferAddress + offset, result, Platform.BYTE_ARRAY_OFFSET, length);\n return result;\n }\n\n @Override\n public CDataDictionaryProvider getDictionaryProvider() {\n return null;\n }\n\n @Override\n public boolean isNullAt(int rowId) {\n return this.valueBufferAddress == -1 || super.isNullAt(rowId);\n }\n\n @Override\n public CometVector slice(int offset, int length) {\n TransferPair tp = this.valueVector.getTransferPair(this.valueVector.getAllocator());\n tp.splitAndTransfer(offset, length);\n\n return new CometPlainVector(tp.getTo(), useDecimal128);\n }\n\n private static UUID convertToUuid(byte[] buf) {\n Preconditions.checkArgument(buf.length == 16, \"UUID require 16 bytes\");\n ByteBuffer bb = ByteBuffer.wrap(buf);\n bb.order(ByteOrder.BIG_ENDIAN);\n long mostSigBits = bb.getLong();\n long leastSigBits = bb.getLong();\n return new UUID(mostSigBits, leastSigBits);\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/vector/CometSelectionVector.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.vector;\n\nimport org.apache.arrow.memory.BufferAllocator;\nimport org.apache.arrow.vector.IntVector;\nimport org.apache.arrow.vector.dictionary.DictionaryProvider;\nimport org.apache.spark.sql.vectorized.ColumnVector;\nimport org.apache.spark.sql.vectorized.ColumnarArray;\nimport org.apache.spark.sql.vectorized.ColumnarMap;\nimport org.apache.spark.unsafe.types.UTF8String;\n\n/**\n * A zero-copy selection vector that extends CometVector. This implementation stores the original\n * data vector and selection indices as separate CometVectors, providing zero copy access to the the\n * underlying data.\n *\n *

If the original vector has values [v0, v1, v2, v3, v4, v5, v6, v7] and the selection indices\n * are [0, 1, 3, 4, 5, 7], then this selection vector will logically represent [v0, v1, v3, v4, v5,\n * v7] without actually copying the data.\n *\n *

Most of the implementations of CometVector methods are implemented for completeness. We don't\n * use this class except to transfer the original data and the selection indices to the native code.\n */\npublic class CometSelectionVector extends CometVector {\n /** The original vector containing all values */\n private final CometVector values;\n\n /**\n * The valid indices in the values vector. This array is converted into an Arrow vector so we can\n * transfer the data to native in one JNI call. This is used to represent the rowid mapping used\n * by Iceberg\n */\n private final int[] selectionIndices;\n\n /**\n * The indices vector containing selection indices. This is currently allocated by the JVM side\n * unlike the values vector which is allocated on the native side\n */\n private final CometVector indices;\n\n /**\n * Number of selected elements. The indices array may have a length greater than this but only\n * numValues elements in the array are valid\n */\n private final int numValues;\n\n /**\n * Creates a new selection vector from the given vector and indices.\n *\n * @param values The original vector to select from\n * @param indices The indices to select from the original vector\n * @param numValues The number of valid values in the indices array\n * @throws IllegalArgumentException if any index is out of bounds\n */\n public CometSelectionVector(CometVector values, int[] indices, int numValues) {\n // Use the values vector's datatype, useDecimal128, and dictionary provider\n super(values.dataType(), values.useDecimal128);\n\n this.values = values;\n this.selectionIndices = indices;\n this.numValues = numValues;\n\n // Validate indices\n int originalLength = values.numValues();\n for (int idx : indices) {\n if (idx < 0 || idx >= originalLength) {\n throw new IllegalArgumentException(\n String.format(\n \"Index %d is out of bounds for vector of length %d\", idx, originalLength));\n }\n }\n\n // Create indices vector\n BufferAllocator allocator = values.getValueVector().getAllocator();\n IntVector indicesVector = new IntVector(\"selection_indices\", allocator);\n indicesVector.allocateNew(numValues);\n for (int i = 0; i < numValues; i++) {\n indicesVector.set(i, indices[i]);\n }\n indicesVector.setValueCount(numValues);\n\n this.indices =\n CometVector.getVector(indicesVector, values.useDecimal128, values.getDictionaryProvider());\n }\n\n /**\n * Returns the index in the values vector for the given selection vector index.\n *\n * @param selectionIndex Index in the selection vector\n * @return The corresponding index in the original vector\n * @throws IndexOutOfBoundsException if selectionIndex is out of bounds\n */\n private int getValuesIndex(int selectionIndex) {\n if (selectionIndex < 0 || selectionIndex >= numValues) {\n throw new IndexOutOfBoundsException(\n String.format(\n \"Selection index %d is out of bounds for selection vector of length %d\",\n selectionIndex, numValues));\n }\n return indices.getInt(selectionIndex);\n }\n\n /**\n * Returns a reference to the values vector.\n *\n * @return The CometVector containing the values\n */\n public CometVector getValues() {\n return values;\n }\n\n /**\n * Returns the indices vector.\n *\n * @return The CometVector containing the indices\n */\n public CometVector getIndices() {\n return indices;\n }\n\n /**\n * Returns the selected indices.\n *\n * @return Array of selected indices\n */\n private int[] getSelectedIndices() {\n return selectionIndices;\n }\n\n @Override\n public int numValues() {\n return numValues;\n }\n\n @Override\n public void setNumValues(int numValues) {\n // For selection vectors, we don't allow changing the number of values\n // as it would break the mapping between selection indices and values\n throw new UnsupportedOperationException(\"CometSelectionVector doesn't support setNumValues\");\n }\n\n @Override\n public void setNumNulls(int numNulls) {\n // For selection vectors, null count should be delegated to the underlying values vector\n // The selection doesn't change the null semantics\n values.setNumNulls(numNulls);\n }\n\n @Override\n public boolean hasNull() {\n return values.hasNull();\n }\n\n @Override\n public int numNulls() {\n return values.numNulls();\n }\n\n // ColumnVector method implementations - delegate to original vector with index mapping\n @Override\n public boolean isNullAt(int rowId) {\n return values.isNullAt(getValuesIndex(rowId));\n }\n\n @Override\n public boolean getBoolean(int rowId) {\n return values.getBoolean(getValuesIndex(rowId));\n }\n\n @Override\n public byte getByte(int rowId) {\n return values.getByte(getValuesIndex(rowId));\n }\n\n @Override\n public short getShort(int rowId) {\n return values.getShort(getValuesIndex(rowId));\n }\n\n @Override\n public int getInt(int rowId) {\n return values.getInt(getValuesIndex(rowId));\n }\n\n @Override\n public long getLong(int rowId) {\n return values.getLong(getValuesIndex(rowId));\n }\n\n @Override\n public long getLongDecimal(int rowId) {\n return values.getLongDecimal(getValuesIndex(rowId));\n }\n\n @Override\n public float getFloat(int rowId) {\n return values.getFloat(getValuesIndex(rowId));\n }\n\n @Override\n public double getDouble(int rowId) {\n return values.getDouble(getValuesIndex(rowId));\n }\n\n @Override\n public UTF8String getUTF8String(int rowId) {\n return values.getUTF8String(getValuesIndex(rowId));\n }\n\n @Override\n public byte[] getBinary(int rowId) {\n return values.getBinary(getValuesIndex(rowId));\n }\n\n @Override\n public ColumnarArray getArray(int rowId) {\n return values.getArray(getValuesIndex(rowId));\n }\n\n @Override\n public ColumnarMap getMap(int rowId) {\n return values.getMap(getValuesIndex(rowId));\n }\n\n @Override\n public ColumnVector getChild(int ordinal) {\n // Return the child from the original vector\n return values.getChild(ordinal);\n }\n\n @Override\n public DictionaryProvider getDictionaryProvider() {\n return values.getDictionaryProvider();\n }\n\n @Override\n public CometVector slice(int offset, int length) {\n if (offset < 0 || length < 0 || offset + length > numValues) {\n throw new IllegalArgumentException(\"Invalid slice parameters\");\n }\n // Get the current indices and slice them\n int[] currentIndices = getSelectedIndices();\n int[] slicedIndices = new int[length];\n // This is not a very efficient version of slicing, but that is\n // not important because we are not likely to use it.\n System.arraycopy(currentIndices, offset, slicedIndices, 0, length);\n return new CometSelectionVector(values, slicedIndices, length);\n }\n\n @Override\n public org.apache.arrow.vector.ValueVector getValueVector() {\n return values.getValueVector();\n }\n\n @Override\n public void close() {\n // Close both the values and indices vectors\n values.close();\n indices.close();\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/vector/CometStructVector.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.vector;\n\nimport java.util.ArrayList;\nimport java.util.List;\n\nimport org.apache.arrow.vector.*;\nimport org.apache.arrow.vector.complex.StructVector;\nimport org.apache.arrow.vector.dictionary.DictionaryProvider;\nimport org.apache.arrow.vector.util.TransferPair;\nimport org.apache.spark.sql.vectorized.ColumnVector;\n\n/** A Comet column vector for struct type. */\npublic class CometStructVector extends CometDecodedVector {\n final List children;\n final DictionaryProvider dictionaryProvider;\n\n public CometStructVector(\n ValueVector vector, boolean useDecimal128, DictionaryProvider dictionaryProvider) {\n super(vector, vector.getField(), useDecimal128);\n\n StructVector structVector = ((StructVector) vector);\n\n int size = structVector.size();\n List children = new ArrayList<>();\n\n for (int i = 0; i < size; ++i) {\n ValueVector value = structVector.getVectorById(i);\n children.add(getVector(value, useDecimal128, dictionaryProvider));\n }\n this.children = children;\n this.dictionaryProvider = dictionaryProvider;\n }\n\n @Override\n public ColumnVector getChild(int i) {\n return children.get(i);\n }\n\n @Override\n public CometVector slice(int offset, int length) {\n TransferPair tp = this.valueVector.getTransferPair(this.valueVector.getAllocator());\n tp.splitAndTransfer(offset, length);\n\n return new CometStructVector(tp.getTo(), useDecimal128, dictionaryProvider);\n }\n}\n" }, { "path": "common/src/main/java/org/apache/comet/vector/CometVector.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.vector;\n\nimport java.math.BigDecimal;\nimport java.math.BigInteger;\n\nimport org.apache.arrow.vector.FixedWidthVector;\nimport org.apache.arrow.vector.ValueVector;\nimport org.apache.arrow.vector.complex.ListVector;\nimport org.apache.arrow.vector.complex.MapVector;\nimport org.apache.arrow.vector.complex.StructVector;\nimport org.apache.arrow.vector.dictionary.Dictionary;\nimport org.apache.arrow.vector.dictionary.DictionaryProvider;\nimport org.apache.arrow.vector.types.pojo.DictionaryEncoding;\nimport org.apache.spark.sql.types.DataType;\nimport org.apache.spark.sql.types.Decimal;\nimport org.apache.spark.sql.types.IntegerType;\nimport org.apache.spark.sql.vectorized.ColumnVector;\nimport org.apache.spark.sql.vectorized.ColumnarArray;\nimport org.apache.spark.sql.vectorized.ColumnarMap;\nimport org.apache.spark.unsafe.Platform;\nimport org.apache.spark.unsafe.types.UTF8String;\n\nimport org.apache.comet.IcebergApi;\n\n/** Base class for all Comet column vector implementations. */\n@IcebergApi\npublic abstract class CometVector extends ColumnVector {\n private static final int DECIMAL_BYTE_WIDTH = 16;\n private final byte[] DECIMAL_BYTES = new byte[DECIMAL_BYTE_WIDTH];\n protected final boolean useDecimal128;\n\n private static final long decimalValOffset;\n\n static {\n try {\n java.lang.reflect.Field unsafeField = sun.misc.Unsafe.class.getDeclaredField(\"theUnsafe\");\n unsafeField.setAccessible(true);\n final sun.misc.Unsafe unsafe = (sun.misc.Unsafe) unsafeField.get(null);\n decimalValOffset = unsafe.objectFieldOffset(Decimal.class.getDeclaredField(\"decimalVal\"));\n } catch (Throwable e) {\n throw new RuntimeException(e);\n }\n }\n\n @IcebergApi\n public CometVector(DataType type, boolean useDecimal128) {\n super(type);\n this.useDecimal128 = useDecimal128;\n }\n\n /**\n * Sets the number of nulls in this vector to be 'numNulls'. This is used when the vector is\n * reused across batches.\n */\n @IcebergApi\n public abstract void setNumNulls(int numNulls);\n\n /**\n * Sets the number of values (including both nulls and non-nulls) in this vector to be\n * 'numValues'. This is used when the vector is reused across batches.\n */\n @IcebergApi\n public abstract void setNumValues(int numValues);\n\n /** Returns the number of values in this vector. */\n @IcebergApi\n public abstract int numValues();\n\n /** Whether the elements of this vector are of fixed length. */\n public boolean isFixedLength() {\n return getValueVector() instanceof FixedWidthVector;\n }\n\n @Override\n public Decimal getDecimal(int i, int precision, int scale) {\n if (isNullAt(i)) return null;\n if (!useDecimal128 && precision <= Decimal.MAX_INT_DIGITS() && type instanceof IntegerType) {\n return createDecimal(getInt(i), precision, scale);\n } else if (precision <= Decimal.MAX_LONG_DIGITS()) {\n return createDecimal(useDecimal128 ? getLongDecimal(i) : getLong(i), precision, scale);\n } else {\n byte[] bytes = getBinaryDecimal(i);\n BigInteger bigInteger = new BigInteger(bytes);\n BigDecimal javaDecimal = new BigDecimal(bigInteger, scale);\n return createDecimal(javaDecimal, precision, scale);\n }\n }\n\n /** This method skips the negative scale check, otherwise the same as Decimal.createUnsafe(). */\n private Decimal createDecimal(long unscaled, int precision, int scale) {\n Decimal dec = new Decimal();\n dec.org$apache$spark$sql$types$Decimal$$longVal_$eq(unscaled);\n dec.org$apache$spark$sql$types$Decimal$$_precision_$eq(precision);\n dec.org$apache$spark$sql$types$Decimal$$_scale_$eq(scale);\n return dec;\n }\n\n /** This method skips a few checks, otherwise the same as Decimal.apply(). */\n private Decimal createDecimal(BigDecimal value, int precision, int scale) {\n Decimal dec = new Decimal();\n Platform.putObjectVolatile(dec, decimalValOffset, new scala.math.BigDecimal(value));\n dec.org$apache$spark$sql$types$Decimal$$_precision_$eq(precision);\n dec.org$apache$spark$sql$types$Decimal$$_scale_$eq(scale);\n return dec;\n }\n\n /**\n * Reads a 16-byte byte array which are encoded big-endian for decimal128 into internal byte\n * array.\n */\n byte[] getBinaryDecimal(int i) {\n return copyBinaryDecimal(i, DECIMAL_BYTES);\n }\n\n /** Reads a 16-byte byte array which are encoded big-endian for decimal128. */\n public byte[] copyBinaryDecimal(int i, byte[] dest) {\n long valueBufferAddress = getValueVector().getDataBuffer().memoryAddress();\n Platform.copyMemory(\n null,\n valueBufferAddress + (long) i * DECIMAL_BYTE_WIDTH,\n dest,\n Platform.BYTE_ARRAY_OFFSET,\n DECIMAL_BYTE_WIDTH);\n // Decimal is stored little-endian in Arrow, so we need to reverse the bytes here\n for (int j = 0, k = DECIMAL_BYTE_WIDTH - 1; j < DECIMAL_BYTE_WIDTH / 2; j++, k--) {\n byte tmp = dest[j];\n dest[j] = dest[k];\n dest[k] = tmp;\n }\n return dest;\n }\n\n @Override\n public boolean getBoolean(int rowId) {\n throw notImplementedException();\n }\n\n @Override\n public byte getByte(int rowId) {\n throw notImplementedException();\n }\n\n @Override\n public short getShort(int rowId) {\n throw notImplementedException();\n }\n\n @Override\n public int getInt(int rowId) {\n throw notImplementedException();\n }\n\n @Override\n public long getLong(int rowId) {\n throw notImplementedException();\n }\n\n public long getLongDecimal(int rowId) {\n throw notImplementedException();\n }\n\n @Override\n public float getFloat(int rowId) {\n throw notImplementedException();\n }\n\n @Override\n public double getDouble(int rowId) {\n throw notImplementedException();\n }\n\n @Override\n public UTF8String getUTF8String(int rowId) {\n throw notImplementedException();\n }\n\n @Override\n public byte[] getBinary(int rowId) {\n throw notImplementedException();\n }\n\n @Override\n public ColumnarArray getArray(int i) {\n throw notImplementedException();\n }\n\n @Override\n public ColumnarMap getMap(int i) {\n throw notImplementedException();\n }\n\n @Override\n public ColumnVector getChild(int i) {\n throw notImplementedException();\n }\n\n @Override\n public void close() {\n getValueVector().close();\n }\n\n public DictionaryProvider getDictionaryProvider() {\n throw new UnsupportedOperationException(\"Not implemented\");\n }\n\n @IcebergApi\n public abstract ValueVector getValueVector();\n\n /**\n * Returns a zero-copying new vector that contains the values from [offset, offset + length).\n *\n * @param offset the offset of the new vector\n * @param length the length of the new vector\n * @return the new vector\n */\n @IcebergApi\n public abstract CometVector slice(int offset, int length);\n\n /**\n * Returns a corresponding `CometVector` implementation based on the given Arrow `ValueVector`.\n *\n * @param vector Arrow `ValueVector`\n * @param useDecimal128 Whether to use Decimal128 for decimal column\n * @return `CometVector` implementation\n */\n public static CometVector getVector(\n ValueVector vector, boolean useDecimal128, DictionaryProvider dictionaryProvider) {\n if (vector instanceof StructVector) {\n return new CometStructVector(vector, useDecimal128, dictionaryProvider);\n } else if (vector instanceof MapVector) {\n return new CometMapVector(vector, useDecimal128, dictionaryProvider);\n } else if (vector instanceof ListVector) {\n return new CometListVector(vector, useDecimal128, dictionaryProvider);\n } else {\n DictionaryEncoding dictionaryEncoding = vector.getField().getDictionary();\n CometPlainVector cometVector = new CometPlainVector(vector, useDecimal128);\n\n if (dictionaryEncoding == null) {\n return cometVector;\n } else {\n Dictionary dictionary = dictionaryProvider.lookup(dictionaryEncoding.getId());\n CometPlainVector dictionaryVector =\n new CometPlainVector(dictionary.getVector(), useDecimal128);\n CometDictionary cometDictionary = new CometDictionary(dictionaryVector);\n\n return new CometDictionaryVector(\n cometVector, cometDictionary, dictionaryProvider, useDecimal128);\n }\n }\n }\n\n protected static CometVector getVector(ValueVector vector, boolean useDecimal128) {\n return getVector(vector, useDecimal128, null);\n }\n\n private UnsupportedOperationException notImplementedException() {\n return new UnsupportedOperationException(\n \"CometVector subclass \" + this.getClass().getName() + \" does not implement this method\");\n }\n}\n" }, { "path": "common/src/main/resources/log4j2.properties", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n# \n# http://www.apache.org/licenses/LICENSE-2.0\n# \n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n# Set everything to be logged to the file target/unit-tests.log\nrootLogger.level = info\nrootLogger.appenderRef.file.ref = ${sys:test.appender:-File}\n\nappender.file.type = File\nappender.file.name = File\nappender.file.fileName = target/unit-tests.log\nappender.file.layout.type = PatternLayout\nappender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n\n\n# Tests that launch java subprocesses can set the \"test.appender\" system property to\n# \"console\" to avoid having the child process's logs overwrite the unit test's\n# log file.\nappender.console.type = Console\nappender.console.name = console\nappender.console.target = SYSTEM_ERR\nappender.console.layout.type = PatternLayout\nappender.console.layout.pattern = %t: %m%n\n\n# Ignore messages below warning level from Jetty, because it's a bit verbose\nlogger.jetty.name = org.sparkproject.jetty\nlogger.jetty.level = warn\n\n" }, { "path": "common/src/main/scala/org/apache/comet/CometConf.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet\n\nimport java.util.Locale\nimport java.util.concurrent.TimeUnit\n\nimport scala.collection.mutable.ListBuffer\n\nimport org.apache.spark.network.util.ByteUnit\nimport org.apache.spark.network.util.JavaUtils\nimport org.apache.spark.sql.comet.util.Utils\nimport org.apache.spark.sql.internal.SQLConf\n\nimport org.apache.comet.shims.ShimCometConf\n\n/**\n * Configurations for a Comet application. Mostly inspired by [[SQLConf]] in Spark.\n *\n * To get the value of a Comet config key from a [[SQLConf]], you can do the following:\n *\n * {{{\n * CometConf.COMET_ENABLED.get\n * }}}\n *\n * which retrieves the config value from the thread-local [[SQLConf]] object. Alternatively, you\n * can also explicitly pass a [[SQLConf]] object to the `get` method.\n */\nobject CometConf extends ShimCometConf {\n\n val COMPAT_GUIDE: String = \"For more information, refer to the Comet Compatibility \" +\n \"Guide (https://datafusion.apache.org/comet/user-guide/compatibility.html)\"\n\n private val TUNING_GUIDE = \"For more information, refer to the Comet Tuning \" +\n \"Guide (https://datafusion.apache.org/comet/user-guide/tuning.html)\"\n\n private val TRACING_GUIDE = \"For more information, refer to the Comet Tracing \" +\n \"Guide (https://datafusion.apache.org/comet/contributor-guide/tracing.html)\"\n\n private val DEBUGGING_GUIDE = \"For more information, refer to the Comet Debugging \" +\n \"Guide (https://datafusion.apache.org/comet/contributor-guide/debugging.html)\"\n\n /** List of all configs that is used for generating documentation */\n val allConfs = new ListBuffer[ConfigEntry[_]]\n\n private val CATEGORY_SCAN = \"scan\"\n private val CATEGORY_PARQUET = \"parquet\"\n private val CATEGORY_EXEC = \"exec\"\n private val CATEGORY_EXEC_EXPLAIN = \"exec_explain\"\n private val CATEGORY_ENABLE_EXEC = \"enable_exec\"\n private val CATEGORY_SHUFFLE = \"shuffle\"\n private val CATEGORY_TUNING = \"tuning\"\n private val CATEGORY_TESTING = \"testing\"\n\n def register(conf: ConfigEntry[_]): Unit = {\n assert(conf.category.nonEmpty, s\"${conf.key} does not have a category defined\")\n allConfs.append(conf)\n }\n\n def conf(key: String): ConfigBuilder = ConfigBuilder(key)\n\n val COMET_PREFIX = \"spark.comet\";\n\n val COMET_EXEC_CONFIG_PREFIX: String = s\"$COMET_PREFIX.exec\"\n\n val COMET_EXPR_CONFIG_PREFIX: String = s\"$COMET_PREFIX.expression\"\n\n val COMET_OPERATOR_CONFIG_PREFIX: String = s\"$COMET_PREFIX.operator\"\n\n val COMET_ENABLED: ConfigEntry[Boolean] = conf(\"spark.comet.enabled\")\n .category(CATEGORY_EXEC)\n .doc(\n \"Whether to enable Comet extension for Spark. When this is turned on, Spark will use \" +\n \"Comet to read Parquet data source. Note that to enable native vectorized execution, \" +\n \"both this config and `spark.comet.exec.enabled` need to be enabled.\")\n .booleanConf\n .createWithEnvVarOrDefault(\"ENABLE_COMET\", true)\n\n val COMET_NATIVE_SCAN_ENABLED: ConfigEntry[Boolean] = conf(\"spark.comet.scan.enabled\")\n .category(CATEGORY_SCAN)\n .doc(\n \"Whether to enable native scans. When this is turned on, Spark will use Comet to \" +\n \"read supported data sources (currently only Parquet is supported natively). Note \" +\n \"that to enable native vectorized execution, both this config and \" +\n \"`spark.comet.exec.enabled` need to be enabled.\")\n .booleanConf\n .createWithDefault(true)\n\n val COMET_NATIVE_PARQUET_WRITE_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.parquet.write.enabled\")\n .category(CATEGORY_TESTING)\n .doc(\n \"Whether to enable native Parquet write through Comet. When enabled, \" +\n \"Comet will intercept Parquet write operations and execute them natively. This \" +\n \"feature is highly experimental and only partially implemented. It should not \" +\n \"be used in production.\")\n .booleanConf\n .createWithEnvVarOrDefault(\"ENABLE_COMET_WRITE\", false)\n\n val SCAN_NATIVE_DATAFUSION = \"native_datafusion\"\n val SCAN_NATIVE_ICEBERG_COMPAT = \"native_iceberg_compat\"\n val SCAN_AUTO = \"auto\"\n\n val COMET_NATIVE_SCAN_IMPL: ConfigEntry[String] = conf(\"spark.comet.scan.impl\")\n .category(CATEGORY_PARQUET)\n .doc(\n \"The implementation of Comet's Parquet scan to use. Available scans are \" +\n s\"`$SCAN_NATIVE_DATAFUSION`, and `$SCAN_NATIVE_ICEBERG_COMPAT`. \" +\n s\"`$SCAN_NATIVE_DATAFUSION` is a fully native implementation, and \" +\n s\"`$SCAN_NATIVE_ICEBERG_COMPAT` is a hybrid implementation that supports some \" +\n \"additional features, such as row indexes and field ids. \" +\n s\"`$SCAN_AUTO` (default) chooses the best available scan based on the scan schema.\")\n .stringConf\n .transform(_.toLowerCase(Locale.ROOT))\n .checkValues(Set(SCAN_NATIVE_DATAFUSION, SCAN_NATIVE_ICEBERG_COMPAT, SCAN_AUTO))\n .createWithEnvVarOrDefault(\"COMET_PARQUET_SCAN_IMPL\", SCAN_AUTO)\n\n val COMET_ICEBERG_NATIVE_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.scan.icebergNative.enabled\")\n .category(CATEGORY_SCAN)\n .doc(\n \"Whether to enable native Iceberg table scan using iceberg-rust. When enabled, \" +\n \"Iceberg tables are read directly through native execution, bypassing Spark's \" +\n \"DataSource V2 API for better performance.\")\n .booleanConf\n .createWithDefault(true)\n\n val COMET_ICEBERG_DATA_FILE_CONCURRENCY_LIMIT: ConfigEntry[Int] =\n conf(\"spark.comet.scan.icebergNative.dataFileConcurrencyLimit\")\n .category(CATEGORY_SCAN)\n .doc(\n \"The number of Iceberg data files to read concurrently within a single task. \" +\n \"Higher values improve throughput for tables with many small files by overlapping \" +\n \"I/O latency, but increase memory usage. Values between 2 and 8 are suggested.\")\n .intConf\n .checkValue(v => v > 0, \"Data file concurrency limit must be positive\")\n .createWithDefault(1)\n\n val COMET_CSV_V2_NATIVE_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.scan.csv.v2.enabled\")\n .category(CATEGORY_TESTING)\n .doc(\n \"Whether to use the native Comet V2 CSV reader for improved performance. \" +\n \"Default: false (uses standard Spark CSV reader) \" +\n \"Experimental: Performance benefits are workload-dependent.\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_RESPECT_PARQUET_FILTER_PUSHDOWN: ConfigEntry[Boolean] =\n conf(\"spark.comet.parquet.respectFilterPushdown\")\n .category(CATEGORY_PARQUET)\n .doc(\n \"Whether to respect Spark's PARQUET_FILTER_PUSHDOWN_ENABLED config. This needs to be \" +\n \"respected when running the Spark SQL test suite but the default setting \" +\n \"results in poor performance in Comet when using the new native scans, \" +\n \"disabled by default\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_PARQUET_PARALLEL_IO_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.parquet.read.parallel.io.enabled\")\n .category(CATEGORY_PARQUET)\n .doc(\n \"Whether to enable Comet's parallel reader for Parquet files. The parallel reader reads \" +\n \"ranges of consecutive data in a file in parallel. It is faster for large files and \" +\n \"row groups but uses more resources.\")\n .booleanConf\n .createWithDefault(true)\n\n val COMET_PARQUET_PARALLEL_IO_THREADS: ConfigEntry[Int] =\n conf(\"spark.comet.parquet.read.parallel.io.thread-pool.size\")\n .category(CATEGORY_PARQUET)\n .doc(\"The maximum number of parallel threads the parallel reader will use in a single \" +\n \"executor. For executors configured with a smaller number of cores, use a smaller number.\")\n .intConf\n .createWithDefault(16)\n\n val COMET_IO_MERGE_RANGES: ConfigEntry[Boolean] =\n conf(\"spark.comet.parquet.read.io.mergeRanges\")\n .category(CATEGORY_PARQUET)\n .doc(\n \"When enabled the parallel reader will try to merge ranges of data that are separated \" +\n \"by less than `comet.parquet.read.io.mergeRanges.delta` bytes. Longer continuous reads \" +\n \"are faster on cloud storage.\")\n .booleanConf\n .createWithDefault(true)\n\n val COMET_IO_MERGE_RANGES_DELTA: ConfigEntry[Int] =\n conf(\"spark.comet.parquet.read.io.mergeRanges.delta\")\n .category(CATEGORY_PARQUET)\n .doc(\"The delta in bytes between consecutive read ranges below which the parallel reader \" +\n \"will try to merge the ranges. The default is 8MB.\")\n .intConf\n .createWithDefault(1 << 23) // 8 MB\n\n val COMET_IO_ADJUST_READRANGE_SKEW: ConfigEntry[Boolean] =\n conf(\"spark.comet.parquet.read.io.adjust.readRange.skew\")\n .category(CATEGORY_PARQUET)\n .doc(\"In the parallel reader, if the read ranges submitted are skewed in sizes, this \" +\n \"option will cause the reader to break up larger read ranges into smaller ranges to \" +\n \"reduce the skew. This will result in a slightly larger number of connections opened to \" +\n \"the file system but may give improved performance.\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_CONVERT_FROM_PARQUET_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.convert.parquet.enabled\")\n .category(CATEGORY_EXEC)\n .doc(\n \"When enabled, data from Spark (non-native) Parquet v1 and v2 scans will be converted to \" +\n \"Arrow format.\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_CONVERT_FROM_JSON_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.convert.json.enabled\")\n .category(CATEGORY_EXEC)\n .doc(\n \"When enabled, data from Spark (non-native) JSON v1 and v2 scans will be converted to \" +\n \"Arrow format.\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_CONVERT_FROM_CSV_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.convert.csv.enabled\")\n .category(CATEGORY_EXEC)\n .doc(\n \"When enabled, data from Spark (non-native) CSV v1 and v2 scans will be converted to \" +\n \"Arrow format.\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_EXEC_ENABLED: ConfigEntry[Boolean] = conf(s\"$COMET_EXEC_CONFIG_PREFIX.enabled\")\n .category(CATEGORY_EXEC)\n .doc(\n \"Whether to enable Comet native vectorized execution for Spark. This controls whether \" +\n \"Spark should convert operators into their Comet counterparts and execute them in \" +\n \"native space. Note: each operator is associated with a separate config in the \" +\n \"format of `spark.comet.exec..enabled` at the moment, and both the \" +\n \"config and this need to be turned on, in order for the operator to be executed in \" +\n \"native.\")\n .booleanConf\n .createWithDefault(true)\n\n val COMET_EXEC_PROJECT_ENABLED: ConfigEntry[Boolean] =\n createExecEnabledConfig(\"project\", defaultValue = true)\n val COMET_EXEC_FILTER_ENABLED: ConfigEntry[Boolean] =\n createExecEnabledConfig(\"filter\", defaultValue = true)\n val COMET_EXEC_SORT_ENABLED: ConfigEntry[Boolean] =\n createExecEnabledConfig(\"sort\", defaultValue = true)\n val COMET_EXEC_LOCAL_LIMIT_ENABLED: ConfigEntry[Boolean] =\n createExecEnabledConfig(\"localLimit\", defaultValue = true)\n val COMET_EXEC_GLOBAL_LIMIT_ENABLED: ConfigEntry[Boolean] =\n createExecEnabledConfig(\"globalLimit\", defaultValue = true)\n val COMET_EXEC_BROADCAST_HASH_JOIN_ENABLED: ConfigEntry[Boolean] =\n createExecEnabledConfig(\"broadcastHashJoin\", defaultValue = true)\n val COMET_EXEC_BROADCAST_EXCHANGE_ENABLED: ConfigEntry[Boolean] =\n createExecEnabledConfig(\"broadcastExchange\", defaultValue = true)\n val COMET_EXEC_HASH_JOIN_ENABLED: ConfigEntry[Boolean] =\n createExecEnabledConfig(\"hashJoin\", defaultValue = true)\n val COMET_EXEC_SORT_MERGE_JOIN_ENABLED: ConfigEntry[Boolean] =\n createExecEnabledConfig(\"sortMergeJoin\", defaultValue = true)\n val COMET_EXEC_AGGREGATE_ENABLED: ConfigEntry[Boolean] =\n createExecEnabledConfig(\"aggregate\", defaultValue = true)\n val COMET_EXEC_COLLECT_LIMIT_ENABLED: ConfigEntry[Boolean] =\n createExecEnabledConfig(\"collectLimit\", defaultValue = true)\n val COMET_EXEC_COALESCE_ENABLED: ConfigEntry[Boolean] =\n createExecEnabledConfig(\"coalesce\", defaultValue = true)\n val COMET_EXEC_UNION_ENABLED: ConfigEntry[Boolean] =\n createExecEnabledConfig(\"union\", defaultValue = true)\n val COMET_EXEC_EXPAND_ENABLED: ConfigEntry[Boolean] =\n createExecEnabledConfig(\"expand\", defaultValue = true)\n val COMET_EXEC_EXPLODE_ENABLED: ConfigEntry[Boolean] =\n createExecEnabledConfig(\"explode\", defaultValue = true)\n val COMET_EXEC_WINDOW_ENABLED: ConfigEntry[Boolean] =\n createExecEnabledConfig(\"window\", defaultValue = true)\n val COMET_EXEC_TAKE_ORDERED_AND_PROJECT_ENABLED: ConfigEntry[Boolean] =\n createExecEnabledConfig(\"takeOrderedAndProject\", defaultValue = true)\n val COMET_EXEC_LOCAL_TABLE_SCAN_ENABLED: ConfigEntry[Boolean] =\n createExecEnabledConfig(\"localTableScan\", defaultValue = false)\n\n val COMET_NATIVE_COLUMNAR_TO_ROW_ENABLED: ConfigEntry[Boolean] =\n conf(s\"$COMET_EXEC_CONFIG_PREFIX.columnarToRow.native.enabled\")\n .category(CATEGORY_EXEC)\n .doc(\n \"Whether to enable native columnar to row conversion. When enabled, Comet will use \" +\n \"native Rust code to convert Arrow columnar data to Spark UnsafeRow format instead \" +\n \"of the JVM implementation. This can improve performance for queries that need to \" +\n \"convert between columnar and row formats.\")\n .booleanConf\n .createWithDefault(true)\n\n val COMET_EXEC_SORT_MERGE_JOIN_WITH_JOIN_FILTER_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.exec.sortMergeJoinWithJoinFilter.enabled\")\n .category(CATEGORY_ENABLE_EXEC)\n .doc(\"Experimental support for Sort Merge Join with filter\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_TRACING_ENABLED: ConfigEntry[Boolean] = conf(\"spark.comet.tracing.enabled\")\n .category(CATEGORY_TUNING)\n .doc(s\"Enable fine-grained tracing of events and memory usage. $TRACING_GUIDE.\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_ONHEAP_MEMORY_OVERHEAD: ConfigEntry[Long] = conf(\"spark.comet.memoryOverhead\")\n .category(CATEGORY_TESTING)\n .doc(\n \"The amount of additional memory to be allocated per executor process for Comet, in MiB, \" +\n \"when running Spark in on-heap mode.\")\n .bytesConf(ByteUnit.MiB)\n .createWithDefault(1024)\n\n val COMET_EXEC_SHUFFLE_ENABLED: ConfigEntry[Boolean] =\n conf(s\"$COMET_EXEC_CONFIG_PREFIX.shuffle.enabled\")\n .category(CATEGORY_SHUFFLE)\n .doc(\n \"Whether to enable Comet native shuffle. \" +\n \"Note that this requires setting `spark.shuffle.manager` to \" +\n \"`org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager`. \" +\n \"`spark.shuffle.manager` must be set before starting the Spark application and \" +\n \"cannot be changed during the application.\")\n .booleanConf\n .createWithDefault(true)\n\n val COMET_SHUFFLE_DIRECT_READ_ENABLED: ConfigEntry[Boolean] =\n conf(s\"$COMET_EXEC_CONFIG_PREFIX.shuffle.directRead.enabled\")\n .category(CATEGORY_SHUFFLE)\n .doc(\n \"When enabled, native operators that consume shuffle output will read \" +\n \"compressed shuffle blocks directly in native code, bypassing Arrow FFI. \" +\n \"Applies to both native shuffle and JVM columnar shuffle. \" +\n \"Requires spark.comet.exec.shuffle.enabled to be true.\")\n .booleanConf\n .createWithDefault(true)\n\n val COMET_SHUFFLE_MODE: ConfigEntry[String] = conf(s\"$COMET_EXEC_CONFIG_PREFIX.shuffle.mode\")\n .category(CATEGORY_SHUFFLE)\n .doc(\n \"This is test config to allow tests to force a particular shuffle implementation to be \" +\n \"used. Valid values are `jvm` for Columnar Shuffle, `native` for Native Shuffle, \" +\n s\"and `auto` to pick the best supported option (`native` has priority). $TUNING_GUIDE.\")\n .internal()\n .stringConf\n .transform(_.toLowerCase(Locale.ROOT))\n .checkValues(Set(\"native\", \"jvm\", \"auto\"))\n .createWithDefault(\"auto\")\n\n val COMET_EXEC_BROADCAST_FORCE_ENABLED: ConfigEntry[Boolean] =\n conf(s\"$COMET_EXEC_CONFIG_PREFIX.broadcast.enabled\")\n .category(CATEGORY_EXEC)\n .doc(\n \"Whether to force enabling broadcasting for Comet native operators. \" +\n \"Comet broadcast feature will be enabled automatically by \" +\n \"Comet extension. But for unit tests, we need this feature to force enabling it \" +\n \"for invalid cases. So this config is only used for unit test.\")\n .internal()\n .booleanConf\n .createWithDefault(false)\n\n val COMET_REPLACE_SMJ: ConfigEntry[Boolean] =\n conf(s\"$COMET_EXEC_CONFIG_PREFIX.replaceSortMergeJoin\")\n .category(CATEGORY_EXEC)\n .doc(\"Experimental feature to force Spark to replace SortMergeJoin with ShuffledHashJoin \" +\n s\"for improved performance. This feature is not stable yet. $TUNING_GUIDE.\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_EXEC_SHUFFLE_WITH_HASH_PARTITIONING_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.native.shuffle.partitioning.hash.enabled\")\n .category(CATEGORY_SHUFFLE)\n .doc(\"Whether to enable hash partitioning for Comet native shuffle.\")\n .booleanConf\n .createWithDefault(true)\n\n val COMET_EXEC_SHUFFLE_WITH_RANGE_PARTITIONING_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.native.shuffle.partitioning.range.enabled\")\n .category(CATEGORY_SHUFFLE)\n .doc(\"Whether to enable range partitioning for Comet native shuffle.\")\n .booleanConf\n .createWithDefault(true)\n\n val COMET_EXEC_SHUFFLE_WITH_ROUND_ROBIN_PARTITIONING_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.native.shuffle.partitioning.roundrobin.enabled\")\n .category(CATEGORY_SHUFFLE)\n .doc(\n \"Whether to enable round robin partitioning for Comet native shuffle. \" +\n \"This is disabled by default because Comet's round-robin produces different \" +\n \"partition assignments than Spark. Spark sorts rows by their binary UnsafeRow \" +\n \"representation before assigning partitions, but Comet uses Arrow format which \" +\n \"has a different binary layout. Instead, Comet implements round-robin as hash \" +\n \"partitioning on all columns, which achieves the same goals: even distribution, \" +\n \"deterministic output (for fault tolerance), and no semantic grouping. \" +\n \"Sorted output will be identical to Spark, but unsorted row ordering may differ.\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_EXEC_SHUFFLE_WITH_ROUND_ROBIN_PARTITIONING_MAX_HASH_COLUMNS: ConfigEntry[Int] =\n conf(\"spark.comet.native.shuffle.partitioning.roundrobin.maxHashColumns\")\n .category(CATEGORY_SHUFFLE)\n .doc(\n \"The maximum number of columns to hash for round robin partitioning. \" +\n \"When set to 0 (the default), all columns are hashed. \" +\n \"When set to a positive value, only the first N columns are used for hashing, \" +\n \"which can improve performance for wide tables while still providing \" +\n \"reasonable distribution.\")\n .intConf\n .checkValue(\n v => v >= 0,\n \"The maximum number of columns to hash for round robin partitioning must be non-negative.\")\n .createWithDefault(0)\n\n val COMET_EXEC_SHUFFLE_COMPRESSION_CODEC: ConfigEntry[String] =\n conf(s\"$COMET_EXEC_CONFIG_PREFIX.shuffle.compression.codec\")\n .category(CATEGORY_SHUFFLE)\n .doc(\n \"The codec of Comet native shuffle used to compress shuffle data. lz4, zstd, and \" +\n \"snappy are supported. Compression can be disabled by setting \" +\n \"spark.shuffle.compress=false.\")\n .stringConf\n .checkValues(Set(\"zstd\", \"lz4\", \"snappy\"))\n .createWithDefault(\"lz4\")\n\n val COMET_EXEC_SHUFFLE_COMPRESSION_ZSTD_LEVEL: ConfigEntry[Int] =\n conf(s\"$COMET_EXEC_CONFIG_PREFIX.shuffle.compression.zstd.level\")\n .category(CATEGORY_SHUFFLE)\n .doc(\"The compression level to use when compressing shuffle files with zstd.\")\n .intConf\n .createWithDefault(1)\n\n val COMET_COLUMNAR_SHUFFLE_ASYNC_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.columnar.shuffle.async.enabled\")\n .category(CATEGORY_SHUFFLE)\n .doc(\"Whether to enable asynchronous shuffle for Arrow-based shuffle.\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_COLUMNAR_SHUFFLE_ASYNC_THREAD_NUM: ConfigEntry[Int] =\n conf(\"spark.comet.columnar.shuffle.async.thread.num\")\n .category(CATEGORY_SHUFFLE)\n .doc(\n \"Number of threads used for Comet async columnar shuffle per shuffle task. \" +\n \"Note that more threads means more memory requirement to \" +\n \"buffer shuffle data before flushing to disk. Also, more threads may not always \" +\n \"improve performance, and should be set based on the number of cores available.\")\n .intConf\n .createWithDefault(3)\n\n val COMET_COLUMNAR_SHUFFLE_ASYNC_MAX_THREAD_NUM: ConfigEntry[Int] = {\n conf(\"spark.comet.columnar.shuffle.async.max.thread.num\")\n .category(CATEGORY_SHUFFLE)\n .doc(\"Maximum number of threads on an executor used for Comet async columnar shuffle. \" +\n \"This is the upper bound of total number of shuffle \" +\n \"threads per executor. In other words, if the number of cores * the number of shuffle \" +\n \"threads per task `spark.comet.columnar.shuffle.async.thread.num` is larger than \" +\n \"this config. Comet will use this config as the number of shuffle threads per \" +\n \"executor instead.\")\n .intConf\n .createWithDefault(100)\n }\n\n val COMET_COLUMNAR_SHUFFLE_SPILL_THRESHOLD: ConfigEntry[Int] =\n conf(\"spark.comet.columnar.shuffle.spill.threshold\")\n .category(CATEGORY_SHUFFLE)\n .doc(\n \"Number of rows to be spilled used for Comet columnar shuffle. \" +\n \"For every configured number of rows, a new spill file will be created. \" +\n \"Higher value means more memory requirement to buffer shuffle data before \" +\n \"flushing to disk. As Comet uses columnar shuffle which is columnar format, \" +\n \"higher value usually helps to improve shuffle data compression ratio. This is \" +\n \"internal config for testing purpose or advanced tuning.\")\n .internal()\n .intConf\n .createWithDefault(Int.MaxValue)\n\n val COMET_ONHEAP_SHUFFLE_MEMORY_FACTOR: ConfigEntry[Double] =\n conf(\"spark.comet.columnar.shuffle.memory.factor\")\n .category(CATEGORY_TESTING)\n .doc(\"Fraction of Comet memory to be allocated per executor process for columnar shuffle \" +\n s\"when running in on-heap mode. $TUNING_GUIDE.\")\n .doubleConf\n .checkValue(\n factor => factor > 0,\n \"Ensure that Comet shuffle memory overhead factor is a double greater than 0\")\n .createWithDefault(1.0)\n\n val COMET_BATCH_SIZE: ConfigEntry[Int] = conf(\"spark.comet.batchSize\")\n .category(CATEGORY_TUNING)\n .doc(\"The columnar batch size, i.e., the maximum number of rows that a batch can contain.\")\n .intConf\n .checkValue(v => v > 0, \"Batch size must be positive\")\n .createWithDefault(8192)\n\n val COMET_COLUMNAR_SHUFFLE_BATCH_SIZE: ConfigEntry[Int] =\n conf(\"spark.comet.columnar.shuffle.batch.size\")\n .category(CATEGORY_SHUFFLE)\n .doc(\"Batch size when writing out sorted spill files on the native side. Note that \" +\n \"this should not be larger than batch size (i.e., `spark.comet.batchSize`). Otherwise \" +\n \"it will produce larger batches than expected in the native operator after shuffle.\")\n .intConf\n .checkValue(\n v => v <= COMET_BATCH_SIZE.get(),\n \"Should not be larger than batch size `spark.comet.batchSize`\")\n .createWithDefault(8192)\n\n val COMET_SHUFFLE_WRITE_BUFFER_SIZE: ConfigEntry[Long] =\n conf(s\"$COMET_EXEC_CONFIG_PREFIX.shuffle.writeBufferSize\")\n .category(CATEGORY_SHUFFLE)\n .doc(\"Size of the write buffer in bytes used by the native shuffle writer when writing \" +\n \"shuffle data to disk. Larger values may improve write performance by reducing \" +\n \"the number of system calls, but will use more memory. \" +\n \"The default is 1MB which provides a good balance between performance and memory usage.\")\n .bytesConf(ByteUnit.MiB)\n .checkValue(v => v > 0, \"Write buffer size must be positive\")\n .createWithDefault(1)\n\n val COMET_SHUFFLE_PREFER_DICTIONARY_RATIO: ConfigEntry[Double] = conf(\n \"spark.comet.shuffle.preferDictionary.ratio\")\n .category(CATEGORY_SHUFFLE)\n .doc(\n \"The ratio of total values to distinct values in a string column to decide whether to \" +\n \"prefer dictionary encoding when shuffling the column. If the ratio is higher than \" +\n \"this config, dictionary encoding will be used on shuffling string column. This config \" +\n \"is effective if it is higher than 1.0. Note that this \" +\n \"config is only used when `spark.comet.exec.shuffle.mode` is `jvm`.\")\n .doubleConf\n .createWithDefault(10.0)\n\n val COMET_EXCHANGE_SIZE_MULTIPLIER: ConfigEntry[Double] = conf(\n \"spark.comet.shuffle.sizeInBytesMultiplier\")\n .category(CATEGORY_SHUFFLE)\n .doc(\n \"Comet reports smaller sizes for shuffle due to using Arrow's columnar memory format \" +\n \"and this can result in Spark choosing a different join strategy due to the estimated \" +\n \"size of the exchange being smaller. Comet will multiple sizeInBytes by this amount to \" +\n \"avoid regressions in join strategy.\")\n .doubleConf\n .createWithDefault(1.0)\n\n val COMET_DPP_FALLBACK_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.dppFallback.enabled\")\n .category(CATEGORY_EXEC)\n .doc(\"Whether to fall back to Spark for queries that use DPP.\")\n .booleanConf\n .createWithDefault(true)\n\n val COMET_DEBUG_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.debug.enabled\")\n .category(CATEGORY_EXEC)\n .doc(\n \"Whether to enable debug mode for Comet. \" +\n \"When enabled, Comet will do additional checks for debugging purpose. For example, \" +\n \"validating array when importing arrays from JVM at native side. Note that these \" +\n \"checks may be expensive in performance and should only be enabled for debugging \" +\n \"purpose.\")\n .booleanConf\n .createWithDefault(false)\n\n // Used on native side. Check spark_config.rs how the config is used\n val COMET_DEBUG_MEMORY_ENABLED: ConfigEntry[Boolean] =\n conf(s\"$COMET_PREFIX.debug.memory\")\n .category(CATEGORY_TESTING)\n .doc(s\"When enabled, log all native memory pool interactions. $DEBUGGING_GUIDE.\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_EXTENDED_EXPLAIN_FORMAT_VERBOSE = \"verbose\"\n val COMET_EXTENDED_EXPLAIN_FORMAT_FALLBACK = \"fallback\"\n\n val COMET_EXTENDED_EXPLAIN_FORMAT: ConfigEntry[String] =\n conf(\"spark.comet.explain.format\")\n .category(CATEGORY_EXEC_EXPLAIN)\n .doc(\"Choose extended explain output. The default format of \" +\n s\"'$COMET_EXTENDED_EXPLAIN_FORMAT_VERBOSE' will provide the full query plan annotated \" +\n \"with fallback reasons as well as a summary of how much of the plan was accelerated \" +\n s\"by Comet. The format '$COMET_EXTENDED_EXPLAIN_FORMAT_FALLBACK' provides a list of \" +\n \"fallback reasons instead.\")\n .stringConf\n .checkValues(\n Set(COMET_EXTENDED_EXPLAIN_FORMAT_VERBOSE, COMET_EXTENDED_EXPLAIN_FORMAT_FALLBACK))\n .createWithDefault(COMET_EXTENDED_EXPLAIN_FORMAT_VERBOSE)\n\n val COMET_EXPLAIN_NATIVE_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.explain.native.enabled\")\n .category(CATEGORY_EXEC_EXPLAIN)\n .doc(\n \"When this setting is enabled, Comet will provide a tree representation of \" +\n \"the native query plan before execution and again after execution, with \" +\n \"metrics.\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_EXPLAIN_TRANSFORMATIONS: ConfigEntry[Boolean] =\n conf(\"spark.comet.explain.rules\")\n .category(CATEGORY_EXEC_EXPLAIN)\n .doc(\"When this setting is enabled, Comet will log all plan transformations performed \" +\n \"in physical optimizer rules. Default: false\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_LOG_FALLBACK_REASONS: ConfigEntry[Boolean] =\n conf(\"spark.comet.logFallbackReasons.enabled\")\n .category(CATEGORY_EXEC_EXPLAIN)\n .doc(\"When this setting is enabled, Comet will log warnings for all fallback reasons.\")\n .booleanConf\n .createWithEnvVarOrDefault(\"ENABLE_COMET_LOG_FALLBACK_REASONS\", false)\n\n val COMET_EXPLAIN_FALLBACK_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.explainFallback.enabled\")\n .category(CATEGORY_EXEC_EXPLAIN)\n .doc(\n \"When this setting is enabled, Comet will provide logging explaining the reason(s) \" +\n \"why a query stage cannot be executed natively. Set this to false to \" +\n \"reduce the amount of logging.\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_ONHEAP_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.exec.onHeap.enabled\")\n .category(CATEGORY_TESTING)\n .doc(\"Whether to allow Comet to run in on-heap mode. Required for running Spark SQL tests.\")\n .booleanConf\n .createWithEnvVarOrDefault(\"ENABLE_COMET_ONHEAP\", false)\n\n val COMET_OFFHEAP_MEMORY_POOL_TYPE: ConfigEntry[String] =\n conf(\"spark.comet.exec.memoryPool\")\n .category(CATEGORY_TUNING)\n .doc(\n \"The type of memory pool to be used for Comet native execution when running Spark in \" +\n \"off-heap mode. Available pool types are `greedy_unified` and `fair_unified`. \" +\n s\"$TUNING_GUIDE.\")\n .stringConf\n .createWithDefault(\"fair_unified\")\n\n val COMET_ONHEAP_MEMORY_POOL_TYPE: ConfigEntry[String] = conf(\n \"spark.comet.exec.onHeap.memoryPool\")\n .category(CATEGORY_TESTING)\n .doc(\n \"The type of memory pool to be used for Comet native execution \" +\n \"when running Spark in on-heap mode. Available pool types are `greedy`, `fair_spill`, \" +\n \"`greedy_task_shared`, `fair_spill_task_shared`, `greedy_global`, `fair_spill_global`, \" +\n \"and `unbounded`.\")\n .stringConf\n .createWithDefault(\"greedy_task_shared\")\n\n val COMET_OFFHEAP_MEMORY_POOL_FRACTION: ConfigEntry[Double] =\n conf(\"spark.comet.exec.memoryPool.fraction\")\n .category(CATEGORY_TUNING)\n .doc(\n \"Fraction of off-heap memory pool that is available to Comet. \" +\n \"Only applies to off-heap mode. \" +\n s\"$TUNING_GUIDE.\")\n .doubleConf\n .createWithDefault(1.0)\n\n val COMET_NATIVE_LOAD_REQUIRED: ConfigEntry[Boolean] = conf(\"spark.comet.nativeLoadRequired\")\n .category(CATEGORY_EXEC)\n .doc(\n \"Whether to require Comet native library to load successfully when Comet is enabled. \" +\n \"If not, Comet will silently fallback to Spark when it fails to load the native lib. \" +\n \"Otherwise, an error will be thrown and the Spark job will be aborted.\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_EXCEPTION_ON_LEGACY_DATE_TIMESTAMP: ConfigEntry[Boolean] =\n conf(\"spark.comet.exceptionOnDatetimeRebase\")\n .category(CATEGORY_EXEC)\n .doc(\"Whether to throw exception when seeing dates/timestamps from the legacy hybrid \" +\n \"(Julian + Gregorian) calendar. Since Spark 3, dates/timestamps were written according \" +\n \"to the Proleptic Gregorian calendar. When this is true, Comet will \" +\n \"throw exceptions when seeing these dates/timestamps that were written by Spark version \" +\n \"before 3.0. If this is false, these dates/timestamps will be read as if they were \" +\n \"written to the Proleptic Gregorian calendar and will not be rebased.\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_USE_DECIMAL_128: ConfigEntry[Boolean] = conf(\"spark.comet.use.decimal128\")\n .internal()\n .category(CATEGORY_EXEC)\n .doc(\"If true, Comet will always use 128 bits to represent a decimal value, regardless of \" +\n \"its precision. If false, Comet will use 32, 64 and 128 bits respectively depending on \" +\n \"the precision. N.B. this is NOT a user-facing config but should be inferred and set by \" +\n \"Comet itself.\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_USE_LAZY_MATERIALIZATION: ConfigEntry[Boolean] = conf(\n \"spark.comet.use.lazyMaterialization\")\n .internal()\n .category(CATEGORY_PARQUET)\n .doc(\n \"Whether to enable lazy materialization for Comet. When this is turned on, Comet will \" +\n \"read Parquet data source lazily for string and binary columns. For filter operations, \" +\n \"lazy materialization will improve read performance by skipping unused pages.\")\n .booleanConf\n .createWithDefault(true)\n\n val COMET_SCHEMA_EVOLUTION_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.schemaEvolution.enabled\")\n .internal()\n .category(CATEGORY_SCAN)\n .doc(\"Whether to enable schema evolution in Comet. For instance, promoting a integer \" +\n \"column to a long column, a float column to a double column, etc. This is automatically\" +\n \"enabled when reading from Iceberg tables.\")\n .booleanConf\n .createWithDefault(COMET_SCHEMA_EVOLUTION_ENABLED_DEFAULT)\n\n val COMET_ENABLE_PARTIAL_HASH_AGGREGATE: ConfigEntry[Boolean] =\n conf(\"spark.comet.testing.aggregate.partialMode.enabled\")\n .internal()\n .category(CATEGORY_TESTING)\n .doc(\"This setting is used in unit tests\")\n .booleanConf\n .createWithDefault(true)\n\n val COMET_ENABLE_FINAL_HASH_AGGREGATE: ConfigEntry[Boolean] =\n conf(\"spark.comet.testing.aggregate.finalMode.enabled\")\n .internal()\n .category(CATEGORY_TESTING)\n .doc(\"This setting is used in unit tests\")\n .booleanConf\n .createWithDefault(true)\n\n val COMET_SPARK_TO_ARROW_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.sparkToColumnar.enabled\")\n .category(CATEGORY_EXEC)\n .doc(\"Whether to enable Spark to Arrow columnar conversion. When this is turned on, \" +\n \"Comet will convert operators in \" +\n \"`spark.comet.sparkToColumnar.supportedOperatorList` into Arrow columnar format before \" +\n \"processing.\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_SPARK_TO_ARROW_SUPPORTED_OPERATOR_LIST: ConfigEntry[Seq[String]] =\n conf(\"spark.comet.sparkToColumnar.supportedOperatorList\")\n .category(CATEGORY_EXEC)\n .doc(\"A comma-separated list of operators that will be converted to Arrow columnar \" +\n s\"format when `${COMET_SPARK_TO_ARROW_ENABLED.key}` is true.\")\n .stringConf\n .toSequence\n .createWithDefault(Seq(\"Range,InMemoryTableScan,RDDScan\"))\n\n val COMET_CASE_CONVERSION_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.caseConversion.enabled\")\n .category(CATEGORY_EXEC)\n .doc(\"Java uses locale-specific rules when converting strings to upper or lower case and \" +\n \"Rust does not, so we disable upper and lower by default.\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_PARQUET_UNSIGNED_SMALL_INT_CHECK: ConfigEntry[Boolean] =\n conf(\"spark.comet.scan.unsignedSmallIntSafetyCheck\")\n .category(CATEGORY_SCAN)\n .doc(\n \"Parquet files may contain unsigned 8-bit integers (UINT_8) which Spark maps to \" +\n \"ShortType. When this config is true (default), Comet falls back to Spark for \" +\n \"ShortType columns because we cannot distinguish signed INT16 (safe) from unsigned \" +\n \"UINT_8 (may produce different results). Set to false to allow native execution of \" +\n \"ShortType columns if you know your data does not contain unsigned UINT_8 columns \" +\n s\"from improperly encoded Parquet files. $COMPAT_GUIDE.\")\n .booleanConf\n .createWithDefault(true)\n\n val COMET_EXEC_STRICT_FLOATING_POINT: ConfigEntry[Boolean] =\n conf(\"spark.comet.exec.strictFloatingPoint\")\n .category(CATEGORY_EXEC)\n .doc(\n \"When enabled, fall back to Spark for floating-point operations that may differ from \" +\n s\"Spark, such as when comparing or sorting -0.0 and 0.0. $COMPAT_GUIDE.\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_METRICS_UPDATE_INTERVAL: ConfigEntry[Long] =\n conf(\"spark.comet.metrics.updateInterval\")\n .category(CATEGORY_EXEC)\n .doc(\"The interval in milliseconds to update metrics. If interval is negative,\" +\n \" metrics will be updated upon task completion.\")\n .longConf\n .createWithDefault(3000L)\n\n val COMET_METRICS_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.metrics.enabled\")\n .category(CATEGORY_EXEC)\n .doc(\n \"Whether to enable Comet metrics reporting through Spark's external monitoring system. \" +\n \"When enabled, Comet exposes metrics such as native operators, Spark operators, \" +\n \"queries planned, transitions, and acceleration ratio. These metrics can be \" +\n \"visualized through tools like Grafana when a metrics sink (e.g., Prometheus) is \" +\n \"configured. Disabled by default because Spark plan traversal adds overhead and \" +\n \"metrics require a sink to be useful. \" +\n \"This config must be set before the SparkSession is created to take effect.\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_LIBHDFS_SCHEMES_KEY = \"fs.comet.libhdfs.schemes\"\n\n val COMET_LIBHDFS_SCHEMES: OptionalConfigEntry[String] =\n conf(s\"spark.hadoop.$COMET_LIBHDFS_SCHEMES_KEY\")\n .category(CATEGORY_SCAN)\n .doc(\"Defines filesystem schemes (e.g., hdfs, webhdfs) that the native side accesses \" +\n \"via libhdfs, separated by commas. Valid only when built with hdfs feature enabled.\")\n .stringConf\n .createOptional\n\n // Used on native side. Check spark_config.rs how the config is used\n val COMET_MAX_TEMP_DIRECTORY_SIZE: ConfigEntry[Long] =\n conf(\"spark.comet.maxTempDirectorySize\")\n .category(CATEGORY_EXEC)\n .doc(\"The maximum amount of data (in bytes) stored inside the temporary directories.\")\n .bytesConf(ByteUnit.BYTE)\n .createWithDefault(100L * 1024 * 1024 * 1024) // 100 GB\n\n val COMET_RESPECT_DATAFUSION_CONFIGS: ConfigEntry[Boolean] =\n conf(s\"$COMET_EXEC_CONFIG_PREFIX.respectDataFusionConfigs\")\n .category(CATEGORY_TESTING)\n .doc(\n \"Development and testing configuration option to allow DataFusion configs set in \" +\n \"Spark configuration settings starting with `spark.comet.datafusion.` to be passed \" +\n \"into native execution.\")\n .booleanConf\n .createWithDefault(false)\n\n val COMET_STRICT_TESTING: ConfigEntry[Boolean] = conf(s\"$COMET_PREFIX.testing.strict\")\n .category(CATEGORY_TESTING)\n .doc(\"Experimental option to enable strict testing, which will fail tests that could be \" +\n \"more comprehensive, such as checking for a specific fallback reason.\")\n .booleanConf\n .createWithEnvVarOrDefault(\"ENABLE_COMET_STRICT_TESTING\", false)\n\n val COMET_OPERATOR_DATA_WRITING_COMMAND_ALLOW_INCOMPAT: ConfigEntry[Boolean] =\n createOperatorIncompatConfig(\"DataWritingCommandExec\")\n\n /** Create a config to enable a specific operator */\n private def createExecEnabledConfig(\n exec: String,\n defaultValue: Boolean,\n notes: Option[String] = None): ConfigEntry[Boolean] = {\n conf(s\"$COMET_EXEC_CONFIG_PREFIX.$exec.enabled\")\n .category(CATEGORY_ENABLE_EXEC)\n .doc(\n s\"Whether to enable $exec by default.\" + notes\n .map(s => s\" $s.\")\n .getOrElse(\"\"))\n .booleanConf\n .createWithDefault(defaultValue)\n }\n\n /**\n * Converts a config key to a valid environment variable name. Example:\n * \"spark.comet.operator.DataWritingCommandExec.allowIncompatible\" ->\n * \"SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE\"\n */\n private def configKeyToEnvVar(configKey: String): String =\n configKey.toUpperCase(Locale.ROOT).replace('.', '_')\n\n private def createOperatorIncompatConfig(name: String): ConfigEntry[Boolean] = {\n val configKey = getOperatorAllowIncompatConfigKey(name)\n val envVar = configKeyToEnvVar(configKey)\n conf(configKey)\n .category(CATEGORY_EXEC)\n .doc(s\"Whether to allow incompatibility for operator: $name. \" +\n s\"False by default. Can be overridden with $envVar env variable\")\n .booleanConf\n .createWithEnvVarOrDefault(envVar, false)\n }\n\n def isExprEnabled(name: String, conf: SQLConf = SQLConf.get): Boolean = {\n getBooleanConf(getExprEnabledConfigKey(name), defaultValue = true, conf)\n }\n\n def getExprEnabledConfigKey(name: String): String = {\n s\"${CometConf.COMET_EXPR_CONFIG_PREFIX}.$name.enabled\"\n }\n\n def isExprAllowIncompat(name: String, conf: SQLConf = SQLConf.get): Boolean = {\n getBooleanConf(getExprAllowIncompatConfigKey(name), defaultValue = false, conf)\n }\n\n def getExprAllowIncompatConfigKey(name: String): String = {\n s\"${CometConf.COMET_EXPR_CONFIG_PREFIX}.$name.allowIncompatible\"\n }\n\n def getExprAllowIncompatConfigKey(exprClass: Class[_]): String = {\n s\"${CometConf.COMET_EXPR_CONFIG_PREFIX}.${exprClass.getSimpleName}.allowIncompatible\"\n }\n\n def isOperatorAllowIncompat(name: String, conf: SQLConf = SQLConf.get): Boolean = {\n getBooleanConf(getOperatorAllowIncompatConfigKey(name), defaultValue = false, conf)\n }\n\n def getOperatorAllowIncompatConfigKey(name: String): String = {\n s\"${CometConf.COMET_OPERATOR_CONFIG_PREFIX}.$name.allowIncompatible\"\n }\n\n def getOperatorAllowIncompatConfigKey(exprClass: Class[_]): String = {\n s\"${CometConf.COMET_OPERATOR_CONFIG_PREFIX}.${exprClass.getSimpleName}.allowIncompatible\"\n }\n\n def getBooleanConf(name: String, defaultValue: Boolean, conf: SQLConf): Boolean = {\n conf.getConfString(name, defaultValue.toString).toLowerCase(Locale.ROOT) == \"true\"\n }\n}\n\nobject ConfigHelpers {\n def toNumber[T](s: String, converter: String => T, key: String, configType: String): T = {\n try {\n converter(s.trim)\n } catch {\n case _: NumberFormatException =>\n throw new IllegalArgumentException(s\"$key should be $configType, but was $s\")\n }\n }\n\n def toBoolean(s: String, key: String): Boolean = {\n try {\n s.trim.toBoolean\n } catch {\n case _: IllegalArgumentException =>\n throw new IllegalArgumentException(s\"$key should be boolean, but was $s\")\n }\n }\n\n def stringToSeq[T](str: String, converter: String => T): Seq[T] = {\n Utils.stringToSeq(str).map(converter)\n }\n\n def seqToString[T](v: Seq[T], stringConverter: T => String): String = {\n v.map(stringConverter).mkString(\",\")\n }\n\n def timeFromString(str: String, unit: TimeUnit): Long = JavaUtils.timeStringAs(str, unit)\n\n def timeToString(v: Long, unit: TimeUnit): String =\n TimeUnit.MILLISECONDS.convert(v, unit) + \"ms\"\n\n def byteFromString(str: String, unit: ByteUnit): Long = {\n val (input, multiplier) =\n if (str.nonEmpty && str.charAt(0) == '-') {\n (str.substring(1), -1)\n } else {\n (str, 1)\n }\n multiplier * JavaUtils.byteStringAs(input, unit)\n }\n\n def byteToString(v: Long, unit: ByteUnit): String = unit.convertTo(v, ByteUnit.BYTE) + \"b\"\n}\n\nprivate class TypedConfigBuilder[T](\n val parent: ConfigBuilder,\n val converter: String => T,\n val stringConverter: T => String) {\n\n import ConfigHelpers._\n\n def this(parent: ConfigBuilder, converter: String => T) = {\n this(parent, converter, Option(_).map(_.toString).orNull)\n }\n\n /** Apply a transformation to the user-provided values of the config entry. */\n def transform(fn: T => T): TypedConfigBuilder[T] = {\n new TypedConfigBuilder(parent, s => fn(converter(s)), stringConverter)\n }\n\n /** Checks if the user-provided value for the config matches the validator. */\n def checkValue(validator: T => Boolean, errorMsg: String): TypedConfigBuilder[T] = {\n transform { v =>\n if (!validator(v)) {\n throw new IllegalArgumentException(s\"'$v' in ${parent.key} is invalid. $errorMsg\")\n }\n v\n }\n }\n\n /** Check that user-provided values for the config match a pre-defined set. */\n def checkValues(validValues: Set[T]): TypedConfigBuilder[T] = {\n transform { v =>\n if (!validValues.contains(v)) {\n throw new IllegalArgumentException(\n s\"The value of ${parent.key} should be one of ${validValues.mkString(\", \")}, but was $v\")\n }\n v\n }\n }\n\n /** Turns the config entry into a sequence of values of the underlying type. */\n def toSequence: TypedConfigBuilder[Seq[T]] = {\n new TypedConfigBuilder(parent, stringToSeq(_, converter), seqToString(_, stringConverter))\n }\n\n /** Creates a [[ConfigEntry]] that does not have a default value. */\n def createOptional: OptionalConfigEntry[T] = {\n val conf = new OptionalConfigEntry[T](\n parent.key,\n converter,\n stringConverter,\n parent._doc,\n parent._category,\n parent._public,\n parent._version)\n CometConf.register(conf)\n conf\n }\n\n /** Creates a [[ConfigEntry]] that has a default value. */\n def createWithDefault(default: T): ConfigEntry[T] = {\n val transformedDefault = converter(stringConverter(default))\n val conf = new ConfigEntryWithDefault[T](\n parent.key,\n transformedDefault,\n converter,\n stringConverter,\n parent._doc,\n parent._category,\n parent._public,\n parent._version)\n CometConf.register(conf)\n conf\n }\n\n /**\n * Creates a [[ConfigEntry]] that has a default value, with support for environment variable\n * override.\n *\n * The value is resolved in the following priority order:\n * 1. Spark config value (if set) 2. Environment variable value (if set) 3. Default value\n *\n * @param envVar\n * The environment variable name to check for override value\n * @param default\n * The default value to use if neither config nor env var is set\n * @return\n * A ConfigEntry with environment variable support\n */\n def createWithEnvVarOrDefault(envVar: String, default: T): ConfigEntry[T] = {\n val transformedDefault = converter(sys.env.getOrElse(envVar, stringConverter(default)))\n val conf = new ConfigEntryWithDefault[T](\n parent.key,\n transformedDefault,\n converter,\n stringConverter,\n parent._doc,\n parent._category,\n parent._public,\n parent._version,\n Some(envVar))\n CometConf.register(conf)\n conf\n }\n}\n\nabstract class ConfigEntry[T](\n val key: String,\n val valueConverter: String => T,\n val stringConverter: T => String,\n val doc: String,\n val category: String,\n val isPublic: Boolean,\n val version: String) {\n\n /**\n * Retrieves the config value from the given [[SQLConf]].\n */\n def get(conf: SQLConf): T\n\n /**\n * Retrieves the config value from the current thread-local [[SQLConf]]\n *\n * @return\n */\n def get(): T = get(SQLConf.get)\n\n def defaultValue: Option[T] = None\n\n def defaultValueString: String\n\n /**\n * The environment variable name that can override this config's default value, if applicable.\n */\n def envVar: Option[String] = None\n\n override def toString: String = {\n s\"ConfigEntry(key=$key, defaultValue=$defaultValueString, doc=$doc, \" +\n s\"public=$isPublic, version=$version)\"\n }\n}\n\nprivate[comet] class ConfigEntryWithDefault[T](\n key: String,\n _defaultValue: T,\n valueConverter: String => T,\n stringConverter: T => String,\n doc: String,\n category: String,\n isPublic: Boolean,\n version: String,\n _envVar: Option[String] = None)\n extends ConfigEntry(key, valueConverter, stringConverter, doc, category, isPublic, version) {\n override def defaultValue: Option[T] = Some(_defaultValue)\n\n override def defaultValueString: String = stringConverter(_defaultValue)\n\n override def envVar: Option[String] = _envVar\n\n def get(conf: SQLConf): T = {\n val tmp = conf.getConfString(key, null)\n if (tmp == null) {\n _defaultValue\n } else {\n valueConverter(tmp)\n }\n }\n}\n\nprivate[comet] class OptionalConfigEntry[T](\n key: String,\n val rawValueConverter: String => T,\n val rawStringConverter: T => String,\n doc: String,\n category: String,\n isPublic: Boolean,\n version: String)\n extends ConfigEntry[Option[T]](\n key,\n s => Some(rawValueConverter(s)),\n v => v.map(rawStringConverter).orNull,\n doc,\n category,\n isPublic,\n version) {\n\n override def defaultValueString: String = ConfigEntry.UNDEFINED\n\n override def get(conf: SQLConf): Option[T] = {\n Option(conf.getConfString(key, null)).map(rawValueConverter)\n }\n}\n\nprivate[comet] case class ConfigBuilder(key: String) {\n\n import ConfigHelpers._\n\n var _public = true\n var _doc = \"\"\n var _version = \"\"\n var _category = \"\"\n\n def internal(): ConfigBuilder = {\n _public = false\n this\n }\n\n def doc(s: String): ConfigBuilder = {\n _doc = s\n this\n }\n\n def category(s: String): ConfigBuilder = {\n _category = s\n this\n }\n\n def version(v: String): ConfigBuilder = {\n _version = v\n this\n }\n\n def intConf: TypedConfigBuilder[Int] = {\n new TypedConfigBuilder(this, toNumber(_, _.toInt, key, \"int\"))\n }\n\n def longConf: TypedConfigBuilder[Long] = {\n new TypedConfigBuilder(this, toNumber(_, _.toLong, key, \"long\"))\n }\n\n def doubleConf: TypedConfigBuilder[Double] = {\n new TypedConfigBuilder(this, toNumber(_, _.toDouble, key, \"double\"))\n }\n\n def booleanConf: TypedConfigBuilder[Boolean] = {\n new TypedConfigBuilder(this, toBoolean(_, key))\n }\n\n def stringConf: TypedConfigBuilder[String] = {\n new TypedConfigBuilder(this, v => v)\n }\n\n def timeConf(unit: TimeUnit): TypedConfigBuilder[Long] = {\n new TypedConfigBuilder(this, timeFromString(_, unit), timeToString(_, unit))\n }\n\n def bytesConf(unit: ByteUnit): TypedConfigBuilder[Long] = {\n new TypedConfigBuilder(this, byteFromString(_, unit), byteToString(_, unit))\n }\n}\n\nprivate object ConfigEntry {\n val UNDEFINED = \"\"\n}\n" }, { "path": "common/src/main/scala/org/apache/comet/Constants.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet\n\nobject Constants {\n val COMET_CONF_DIR_ENV = \"COMET_CONF_DIR\"\n val LOG_CONF_PATH = \"comet.log.file.path\"\n val LOG_CONF_NAME = \"log4rs.yaml\"\n val LOG_LEVEL_ENV = \"COMET_LOG_LEVEL\"\n}\n" }, { "path": "common/src/main/scala/org/apache/comet/objectstore/NativeConfig.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.objectstore\n\nimport java.net.URI\nimport java.util.Locale\n\nimport org.apache.commons.lang3.StringUtils\nimport org.apache.hadoop.conf.Configuration\n\nimport org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES_KEY\n\nobject NativeConfig {\n\n private val objectStoreConfigPrefixes = Map(\n // Amazon S3 configurations\n \"s3\" -> Seq(\"fs.s3a.\"),\n \"s3a\" -> Seq(\"fs.s3a.\"),\n // Google Cloud Storage configurations\n \"gs\" -> Seq(\"fs.gs.\"),\n // Azure Blob Storage configurations (can use both prefixes)\n \"wasb\" -> Seq(\"fs.azure.\", \"fs.wasb.\"),\n \"wasbs\" -> Seq(\"fs.azure.\", \"fs.wasb.\"),\n // Azure Data Lake Storage Gen2 configurations\n \"abfs\" -> Seq(\"fs.abfs.\"),\n // Azure Data Lake Storage Gen2 secure configurations (can use both prefixes)\n \"abfss\" -> Seq(\"fs.abfss.\", \"fs.abfs.\"))\n\n /**\n * Extract object store configurations from Hadoop configuration for native DataFusion usage.\n * This includes S3, GCS, Azure and other cloud storage configurations.\n *\n * This method extracts all configurations with supported prefixes, automatically capturing both\n * global configurations (e.g., fs.s3a.access.key) and per-bucket configurations (e.g.,\n * fs.s3a.bucket.{bucket-name}.access.key). The native code will prioritize per-bucket\n * configurations over global ones when both are present.\n *\n * The configurations are passed to the native code which uses object_store's parse_url_opts for\n * consistent and standardized cloud storage support across all providers.\n */\n def extractObjectStoreOptions(hadoopConf: Configuration, uri: URI): Map[String, String] = {\n val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse(\"file\")\n\n import scala.jdk.CollectionConverters._\n val options = scala.collection.mutable.Map[String, String]()\n\n // The schemes will use libhdfs\n val libhdfsSchemes = hadoopConf.get(COMET_LIBHDFS_SCHEMES_KEY)\n if (StringUtils.isNotBlank(libhdfsSchemes)) {\n options(COMET_LIBHDFS_SCHEMES_KEY) = libhdfsSchemes\n }\n\n // Get prefixes for this scheme, return early if none found\n val prefixes = objectStoreConfigPrefixes.get(scheme)\n if (prefixes.isEmpty) {\n return options.toMap\n }\n\n // Extract all configurations that match the object store prefixes\n hadoopConf.iterator().asScala.foreach { entry =>\n val key = entry.getKey\n val value = entry.getValue\n // Check if key starts with any of the prefixes for this scheme\n if (prefixes.get.exists(prefix => key.startsWith(prefix))) {\n options(key) = value\n }\n }\n\n options.toMap\n }\n}\n" }, { "path": "common/src/main/scala/org/apache/comet/package.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache\n\nimport java.util.Properties\n\nimport org.apache.arrow.memory.RootAllocator\nimport org.apache.spark.internal.Logging\n\npackage object comet {\n\n /**\n * The root allocator for Comet execution. Because Arrow Java memory management is based on\n * reference counting, exposed arrays increase the reference count of the underlying buffers.\n * Until the reference count is zero, the memory will not be released. If the consumer side is\n * finished later than the close of the allocator, the allocator will think the memory is\n * leaked. To avoid this, we use a single allocator for the whole execution process.\n */\n val CometArrowAllocator = new RootAllocator(Long.MaxValue)\n\n /**\n * Provides access to build information about the Comet libraries. This will be used by the\n * benchmarking software to provide the source revision and repository. In addition, the build\n * information is included to aid in future debugging efforts for releases.\n */\n private object CometBuildInfo extends Logging {\n private val GIT_INFO_PROPS_FILENAME = \"comet-git-info.properties\"\n\n val props: Properties = {\n val props = new Properties()\n val resourceStream = Thread\n .currentThread()\n .getContextClassLoader\n .getResourceAsStream(GIT_INFO_PROPS_FILENAME)\n if (resourceStream != null) {\n try {\n props.load(resourceStream)\n } catch {\n case e: Exception =>\n logError(s\"Error loading properties from $GIT_INFO_PROPS_FILENAME\", e)\n } finally {\n if (resourceStream != null) {\n try {\n resourceStream.close()\n } catch {\n case e: Exception =>\n logError(\"Error closing Comet build info resource stream\", e)\n }\n }\n }\n } else {\n logWarning(s\"Could not find $GIT_INFO_PROPS_FILENAME\")\n }\n props\n }\n }\n\n private def getProp(name: String): String = {\n CometBuildInfo.props.getProperty(name, \"\")\n }\n\n val COMET_VERSION: String = getProp(\"git.build.version\")\n val COMET_BRANCH: String = getProp(\"git.branch\")\n val COMET_REVISION: String = getProp(\"git.commit.id.full\")\n val COMET_BUILD_USER_EMAIL: String = getProp(\"git.build.user.name\")\n val COMET_BUILD_USER_NAME: String = getProp(\"git.build.user.email\")\n val COMET_REPO_URL: String = getProp(\"git.remote.origin.url\")\n val COMET_BUILD_TIMESTAMP: String = getProp(\"git.build.time\")\n}\n" }, { "path": "common/src/main/scala/org/apache/comet/parquet/CometParquetUtils.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet\n\nimport org.apache.hadoop.conf.Configuration\nimport org.apache.parquet.crypto.DecryptionPropertiesFactory\nimport org.apache.parquet.crypto.keytools.{KeyToolkit, PropertiesDrivenCryptoFactory}\nimport org.apache.spark.sql.internal.SQLConf\n\nobject CometParquetUtils {\n private val PARQUET_FIELD_ID_WRITE_ENABLED = \"spark.sql.parquet.fieldId.write.enabled\"\n private val PARQUET_FIELD_ID_READ_ENABLED = \"spark.sql.parquet.fieldId.read.enabled\"\n private val IGNORE_MISSING_PARQUET_FIELD_ID = \"spark.sql.parquet.fieldId.read.ignoreMissing\"\n\n // Map of encryption configuration key-value pairs that, if present, are only supported with\n // these specific values. Generally, these are the default values that won't be present,\n // but if they are present we want to check them.\n private val SUPPORTED_ENCRYPTION_CONFIGS: Map[String, Set[String]] = Map(\n // https://github.com/apache/arrow-rs/blob/main/parquet/src/encryption/ciphers.rs#L21\n KeyToolkit.DATA_KEY_LENGTH_PROPERTY_NAME -> Set(KeyToolkit.DATA_KEY_LENGTH_DEFAULT.toString),\n KeyToolkit.KEK_LENGTH_PROPERTY_NAME -> Set(KeyToolkit.KEK_LENGTH_DEFAULT.toString),\n // https://github.com/apache/arrow-rs/blob/main/parquet/src/file/metadata/parser.rs#L494\n PropertiesDrivenCryptoFactory.ENCRYPTION_ALGORITHM_PROPERTY_NAME -> Set(\"AES_GCM_V1\"))\n\n def writeFieldId(conf: SQLConf): Boolean =\n conf.getConfString(PARQUET_FIELD_ID_WRITE_ENABLED, \"false\").toBoolean\n\n def writeFieldId(conf: Configuration): Boolean =\n conf.getBoolean(PARQUET_FIELD_ID_WRITE_ENABLED, false)\n\n def readFieldId(conf: SQLConf): Boolean =\n conf.getConfString(PARQUET_FIELD_ID_READ_ENABLED, \"false\").toBoolean\n\n def ignoreMissingIds(conf: SQLConf): Boolean =\n conf.getConfString(IGNORE_MISSING_PARQUET_FIELD_ID, \"false\").toBoolean\n\n /**\n * Checks if the given Hadoop configuration contains any unsupported encryption settings.\n *\n * @param hadoopConf\n * The Hadoop configuration to check\n * @return\n * true if all encryption configurations are supported, false if any unsupported config is\n * found\n */\n def isEncryptionConfigSupported(hadoopConf: Configuration): Boolean = {\n // Check configurations that, if present, can only have specific allowed values\n val supportedListCheck = SUPPORTED_ENCRYPTION_CONFIGS.forall {\n case (configKey, supportedValues) =>\n val configValue = Option(hadoopConf.get(configKey))\n configValue match {\n case Some(value) => supportedValues.contains(value)\n case None => true // Config not set, so it's supported\n }\n }\n\n supportedListCheck\n }\n\n def encryptionEnabled(hadoopConf: Configuration): Boolean = {\n // TODO: Are there any other properties to check?\n val encryptionKeys = Seq(\n DecryptionPropertiesFactory.CRYPTO_FACTORY_CLASS_PROPERTY_NAME,\n KeyToolkit.KMS_CLIENT_CLASS_PROPERTY_NAME)\n\n encryptionKeys.exists(key => Option(hadoopConf.get(key)).exists(_.nonEmpty))\n }\n}\n" }, { "path": "common/src/main/scala/org/apache/comet/parquet/CometReaderThreadPool.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet\n\nimport java.util.concurrent.{Executors, ExecutorService, ThreadFactory}\nimport java.util.concurrent.atomic.AtomicLong\n\nabstract class CometReaderThreadPool {\n private var threadPool: Option[ExecutorService] = None\n\n protected def threadNamePrefix: String\n\n private def initThreadPool(maxThreads: Int): ExecutorService = synchronized {\n if (threadPool.isEmpty) {\n val threadFactory: ThreadFactory = new ThreadFactory() {\n private val defaultThreadFactory = Executors.defaultThreadFactory\n val count = new AtomicLong(0)\n\n override def newThread(r: Runnable): Thread = {\n val thread = defaultThreadFactory.newThread(r)\n thread.setName(s\"${threadNamePrefix}_${count.getAndIncrement()}\")\n thread.setDaemon(true)\n thread\n }\n }\n\n val threadPoolExecutor = Executors.newFixedThreadPool(maxThreads, threadFactory)\n threadPool = Some(threadPoolExecutor)\n }\n\n threadPool.get\n }\n\n def getOrCreateThreadPool(numThreads: Int): ExecutorService = {\n threadPool.getOrElse(initThreadPool(numThreads))\n }\n\n}\n\n// Thread pool used by the Parquet parallel reader\nobject CometFileReaderThreadPool extends CometReaderThreadPool {\n override def threadNamePrefix: String = \"file_reader_thread\"\n}\n" }, { "path": "common/src/main/scala/org/apache/comet/vector/NativeUtil.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.vector\n\nimport scala.collection.mutable\n\nimport org.apache.arrow.c.{ArrowArray, ArrowImporter, ArrowSchema, CDataDictionaryProvider, Data}\nimport org.apache.arrow.vector.VectorSchemaRoot\nimport org.apache.arrow.vector.dictionary.DictionaryProvider\nimport org.apache.spark.SparkException\nimport org.apache.spark.sql.comet.util.Utils\nimport org.apache.spark.sql.vectorized.ColumnarBatch\n\nimport org.apache.comet.CometArrowAllocator\n\n/**\n * Provides functionality for importing Arrow vectors from native code and wrapping them as\n * CometVectors.\n *\n * Also provides functionality for exporting Comet columnar batches to native code.\n *\n * Each instance of NativeUtil creates an instance of CDataDictionaryProvider (a\n * DictionaryProvider that is used in C Data Interface for imports).\n *\n * NativeUtil must be closed after use to release resources in the dictionary provider.\n */\nclass NativeUtil {\n import Utils._\n\n /** Use the global allocator */\n private val allocator = CometArrowAllocator\n\n /** ArrowImporter does not hold any state and does not need to be closed */\n private val importer = new ArrowImporter(allocator)\n\n /**\n * Dictionary provider to use for the lifetime of this instance of NativeUtil. The dictionary\n * provider is closed when NativeUtil is closed.\n */\n private val dictionaryProvider: CDataDictionaryProvider = new CDataDictionaryProvider\n\n /**\n * Allocates Arrow structs for the given number of columns.\n *\n * @param numCols\n * the number of columns\n * @return\n * a pair of Arrow arrays and Arrow schemas\n */\n def allocateArrowStructs(numCols: Int): (Array[ArrowArray], Array[ArrowSchema]) = {\n val arrays = new Array[ArrowArray](numCols)\n val schemas = new Array[ArrowSchema](numCols)\n\n (0 until numCols).foreach { index =>\n val arrowSchema = ArrowSchema.allocateNew(allocator)\n val arrowArray = ArrowArray.allocateNew(allocator)\n arrays(index) = arrowArray\n schemas(index) = arrowSchema\n }\n\n (arrays, schemas)\n }\n\n /**\n * Exports a ColumnarBatch to Arrow FFI and returns the memory addresses.\n *\n * This is a convenience method that allocates Arrow structs, exports the batch, and returns\n * just the memory addresses (without exposing the Arrow types).\n *\n * @param batch\n * the columnar batch to export\n * @return\n * a tuple of (array addresses, schema addresses, number of rows)\n */\n def exportBatchToAddresses(batch: ColumnarBatch): (Array[Long], Array[Long], Int) = {\n val numCols = batch.numCols()\n val (arrays, schemas) = allocateArrowStructs(numCols)\n val arrayAddrs = arrays.map(_.memoryAddress())\n val schemaAddrs = schemas.map(_.memoryAddress())\n val numRows = exportBatch(arrayAddrs, schemaAddrs, batch)\n (arrayAddrs, schemaAddrs, numRows)\n }\n\n /**\n * Exports a Comet `ColumnarBatch` into a list of memory addresses that can be consumed by the\n * native execution.\n *\n * @param batch\n * the input Comet columnar batch\n * @return\n * an exported batches object containing an array containing number of rows + pairs of memory\n * addresses in the format of (address of Arrow array, address of Arrow schema)\n */\n def exportBatch(\n arrayAddrs: Array[Long],\n schemaAddrs: Array[Long],\n batch: ColumnarBatch): Int = {\n val numRows = mutable.ArrayBuffer.empty[Int]\n\n (0 until batch.numCols()).foreach { index =>\n batch.column(index) match {\n case selectionVector: CometSelectionVector =>\n // For CometSelectionVector, export only the values vector\n val valuesVector = selectionVector.getValues\n val valueVector = valuesVector.getValueVector\n\n // Use the selection vector's logical row count\n numRows += selectionVector.numValues()\n\n val provider = if (valueVector.getField.getDictionary != null) {\n valuesVector.getDictionaryProvider\n } else {\n null\n }\n\n // The array and schema structures are allocated by native side.\n // Don't need to deallocate them here.\n val arrowSchema = ArrowSchema.wrap(schemaAddrs(index))\n val arrowArray = ArrowArray.wrap(arrayAddrs(index))\n Data.exportVector(\n allocator,\n getFieldVector(valueVector, \"export\"),\n provider,\n arrowArray,\n arrowSchema)\n case a: CometVector =>\n val valueVector = a.getValueVector\n\n numRows += valueVector.getValueCount\n\n val provider = if (valueVector.getField.getDictionary != null) {\n a.getDictionaryProvider\n } else {\n null\n }\n\n // The array and schema structures are allocated by native side.\n // Don't need to deallocate them here.\n val arrowSchema = ArrowSchema.wrap(schemaAddrs(index))\n val arrowArray = ArrowArray.wrap(arrayAddrs(index))\n Data.exportVector(\n allocator,\n getFieldVector(valueVector, \"export\"),\n provider,\n arrowArray,\n arrowSchema)\n case c =>\n throw new SparkException(\n \"Comet execution only takes Arrow Arrays, but got \" +\n s\"${c.getClass}\")\n }\n }\n\n if (numRows.distinct.length > 1) {\n throw new SparkException(\n s\"Number of rows in each column should be the same, but got [${numRows.distinct}]\")\n }\n\n // `ColumnarBatch.numRows` might return a different number than the actual number of rows in\n // the Arrow arrays. For example, Iceberg column reader will skip deleted rows internally in\n // its `CometVector` implementation. The `ColumnarBatch` returned by the reader will report\n // logical number of rows which is less than actual number of rows due to row deletion.\n // Similarly, CometSelectionVector represents a different number of logical rows than the\n // underlying vector.\n numRows.headOption.getOrElse(batch.numRows())\n }\n\n /**\n * Exports a single CometVector to native side.\n *\n * @param vector\n * The CometVector to export\n * @param arrayAddr\n * The address of the ArrowArray structure\n * @param schemaAddr\n * The address of the ArrowSchema structure\n */\n def exportSingleVector(vector: CometVector, arrayAddr: Long, schemaAddr: Long): Unit = {\n val valueVector = vector.getValueVector\n\n val provider = if (valueVector.getField.getDictionary != null) {\n vector.getDictionaryProvider\n } else {\n null\n }\n\n val arrowSchema = ArrowSchema.wrap(schemaAddr)\n val arrowArray = ArrowArray.wrap(arrayAddr)\n Data.exportVector(\n allocator,\n getFieldVector(valueVector, \"export\"),\n provider,\n arrowArray,\n arrowSchema)\n }\n\n /**\n * Gets the next batch from native execution.\n *\n * @param numOutputCols\n * The number of output columns\n * @param func\n * The function to call to get the next batch\n * @return\n * The number of row of the next batch, or None if there are no more batches\n */\n def getNextBatch(\n numOutputCols: Int,\n func: (Array[Long], Array[Long]) => Long): Option[ColumnarBatch] = {\n val (arrays, schemas) = allocateArrowStructs(numOutputCols)\n\n val arrayAddrs = arrays.map(_.memoryAddress())\n val schemaAddrs = schemas.map(_.memoryAddress())\n\n val result = func(arrayAddrs, schemaAddrs)\n\n result match {\n case -1 =>\n // EOF\n None\n case numRows =>\n val cometVectors = importVector(arrays, schemas)\n Some(new ColumnarBatch(cometVectors.toArray, numRows.toInt))\n }\n }\n\n /**\n * Imports a list of Arrow addresses from native execution, and return a list of Comet vectors.\n *\n * @param arrays\n * a list of Arrow array\n * @param schemas\n * a list of Arrow schema\n * @return\n * a list of Comet vectors\n */\n def importVector(arrays: Array[ArrowArray], schemas: Array[ArrowSchema]): Seq[CometVector] = {\n val arrayVectors = mutable.ArrayBuffer.empty[CometVector]\n\n (0 until arrays.length).foreach { i =>\n val arrowSchema = schemas(i)\n val arrowArray = arrays(i)\n\n // Native execution should always have 'useDecimal128' set to true since it doesn't support\n // other cases.\n arrayVectors += CometVector.getVector(\n importer.importVector(arrowArray, arrowSchema, dictionaryProvider),\n true,\n dictionaryProvider)\n }\n arrayVectors.toSeq\n }\n\n /**\n * Takes zero-copy slices of the input batch with given start index and maximum number of rows.\n *\n * @param batch\n * Input batch\n * @param startIndex\n * Start index of the slice\n * @param maxNumRows\n * Maximum number of rows in the slice\n * @return\n * A new batch with the sliced vectors\n */\n def takeRows(batch: ColumnarBatch, startIndex: Int, maxNumRows: Int): ColumnarBatch = {\n val arrayVectors = mutable.ArrayBuffer.empty[CometVector]\n\n for (i <- 0 until batch.numCols()) {\n val column = batch.column(i).asInstanceOf[CometVector]\n arrayVectors += column.slice(startIndex, maxNumRows)\n }\n\n new ColumnarBatch(arrayVectors.toArray, maxNumRows)\n }\n\n def close(): Unit = {\n // closing the dictionary provider also closes the dictionary arrays\n dictionaryProvider.close()\n }\n}\n\nobject NativeUtil {\n def rootAsBatch(arrowRoot: VectorSchemaRoot): ColumnarBatch = {\n rootAsBatch(arrowRoot, null)\n }\n\n def rootAsBatch(arrowRoot: VectorSchemaRoot, provider: DictionaryProvider): ColumnarBatch = {\n val vectors = (0 until arrowRoot.getFieldVectors.size()).map { i =>\n val vector = arrowRoot.getFieldVectors.get(i)\n // Native shuffle always uses decimal128.\n CometVector.getVector(vector, true, provider)\n }\n new ColumnarBatch(vectors.toArray, arrowRoot.getRowCount)\n }\n}\n" }, { "path": "common/src/main/scala/org/apache/comet/vector/StreamReader.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.vector\n\nimport java.nio.channels.ReadableByteChannel\n\nimport org.apache.arrow.vector.VectorSchemaRoot\nimport org.apache.arrow.vector.ipc.{ArrowStreamReader, ReadChannel}\nimport org.apache.arrow.vector.ipc.message.MessageChannelReader\nimport org.apache.spark.sql.vectorized.ColumnarBatch\n\nimport org.apache.comet.CometArrowAllocator\n\n/**\n * A reader that consumes Arrow data from an input channel, and produces Comet batches.\n */\ncase class StreamReader(channel: ReadableByteChannel, source: String) extends AutoCloseable {\n private val channelReader =\n new MessageChannelReader(new ReadChannel(channel), CometArrowAllocator)\n private var arrowReader = new ArrowStreamReader(channelReader, CometArrowAllocator)\n private var root = arrowReader.getVectorSchemaRoot\n\n def nextBatch(): Option[ColumnarBatch] = {\n if (arrowReader.loadNextBatch()) {\n Some(rootAsBatch(root))\n } else {\n None\n }\n }\n\n private def rootAsBatch(root: VectorSchemaRoot): ColumnarBatch = {\n NativeUtil.rootAsBatch(root, arrowReader)\n }\n\n override def close(): Unit = {\n if (root != null) {\n arrowReader.close()\n root.close()\n\n arrowReader = null\n root = null\n }\n }\n}\n" }, { "path": "common/src/main/scala/org/apache/spark/sql/comet/CastOverflowException.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.spark.sql.comet\n\nimport org.apache.spark.SparkArithmeticException\nimport org.apache.spark.sql.errors.QueryExecutionErrors.toSQLConf\nimport org.apache.spark.sql.internal.SQLConf\n\nclass CastOverflowException(t: String, from: String, to: String)\n extends SparkArithmeticException(\n \"CAST_OVERFLOW\",\n Map(\n \"value\" -> t,\n \"sourceType\" -> s\"\"\"\"$from\"\"\"\",\n \"targetType\" -> s\"\"\"\"$to\"\"\"\",\n \"ansiConfig\" -> toSQLConf(SQLConf.ANSI_ENABLED.key)),\n Array.empty,\n \"\") {}\n" }, { "path": "common/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowReaderIterator.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.spark.sql.comet.execution.arrow\n\nimport java.nio.channels.ReadableByteChannel\n\nimport org.apache.spark.sql.vectorized.ColumnarBatch\n\nimport org.apache.comet.vector._\n\nclass ArrowReaderIterator(channel: ReadableByteChannel, source: String)\n extends Iterator[ColumnarBatch] {\n\n private val reader = StreamReader(channel, source)\n private var batch = nextBatch()\n private var currentBatch: ColumnarBatch = null\n private var isClosed: Boolean = false\n\n override def hasNext: Boolean = {\n if (isClosed) {\n return false\n }\n if (batch.isDefined) {\n return true\n }\n\n // Release the previous batch.\n // If it is not released, when closing the reader, arrow library will complain about\n // memory leak.\n if (currentBatch != null) {\n currentBatch.close()\n currentBatch = null\n }\n\n batch = nextBatch()\n if (batch.isEmpty) {\n close()\n return false\n }\n true\n }\n\n override def next(): ColumnarBatch = {\n if (!hasNext) {\n throw new NoSuchElementException\n }\n\n val nextBatch = batch.get\n\n currentBatch = nextBatch\n batch = None\n currentBatch\n }\n\n private def nextBatch(): Option[ColumnarBatch] = {\n reader.nextBatch()\n }\n\n def close(): Unit =\n synchronized {\n if (!isClosed) {\n if (currentBatch != null) {\n currentBatch.close()\n currentBatch = null\n }\n reader.close()\n isClosed = true\n }\n }\n}\n" }, { "path": "common/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowWriters.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.spark.sql.comet.execution.arrow\n\nimport scala.jdk.CollectionConverters._\n\nimport org.apache.arrow.vector._\nimport org.apache.arrow.vector.complex._\nimport org.apache.spark.sql.catalyst.InternalRow\nimport org.apache.spark.sql.catalyst.expressions.SpecializedGetters\nimport org.apache.spark.sql.comet.util.Utils\nimport org.apache.spark.sql.errors.QueryExecutionErrors\nimport org.apache.spark.sql.types._\nimport org.apache.spark.sql.vectorized.ColumnarArray\n\n/**\n * This file is mostly copied from Spark SQL's\n * org.apache.spark.sql.execution.arrow.ArrowWriter.scala. Comet shadows Arrow classes to avoid\n * potential conflicts with Spark's Arrow dependencies, hence we cannot reuse Spark's ArrowWriter\n * directly.\n *\n * Performance enhancement: https://github.com/apache/datafusion-comet/issues/888\n */\nprivate[arrow] object ArrowWriter {\n def create(root: VectorSchemaRoot): ArrowWriter = {\n val children = root.getFieldVectors().asScala.map { vector =>\n vector.allocateNew()\n createFieldWriter(vector)\n }\n new ArrowWriter(root, children.toArray)\n }\n\n private[sql] def createFieldWriter(vector: ValueVector): ArrowFieldWriter = {\n val field = vector.getField()\n (Utils.fromArrowField(field), vector) match {\n case (BooleanType, vector: BitVector) => new BooleanWriter(vector)\n case (ByteType, vector: TinyIntVector) => new ByteWriter(vector)\n case (ShortType, vector: SmallIntVector) => new ShortWriter(vector)\n case (IntegerType, vector: IntVector) => new IntegerWriter(vector)\n case (LongType, vector: BigIntVector) => new LongWriter(vector)\n case (FloatType, vector: Float4Vector) => new FloatWriter(vector)\n case (DoubleType, vector: Float8Vector) => new DoubleWriter(vector)\n case (DecimalType.Fixed(precision, scale), vector: DecimalVector) =>\n new DecimalWriter(vector, precision, scale)\n case (StringType, vector: VarCharVector) => new StringWriter(vector)\n case (StringType, vector: LargeVarCharVector) => new LargeStringWriter(vector)\n case (BinaryType, vector: VarBinaryVector) => new BinaryWriter(vector)\n case (BinaryType, vector: LargeVarBinaryVector) => new LargeBinaryWriter(vector)\n case (DateType, vector: DateDayVector) => new DateWriter(vector)\n case (TimestampType, vector: TimeStampMicroTZVector) => new TimestampWriter(vector)\n case (TimestampNTZType, vector: TimeStampMicroVector) => new TimestampNTZWriter(vector)\n case (ArrayType(_, _), vector: ListVector) =>\n val elementVector = createFieldWriter(vector.getDataVector())\n new ArrayWriter(vector, elementVector)\n case (MapType(_, _, _), vector: MapVector) =>\n val structVector = vector.getDataVector.asInstanceOf[StructVector]\n val keyWriter = createFieldWriter(structVector.getChild(MapVector.KEY_NAME))\n val valueWriter = createFieldWriter(structVector.getChild(MapVector.VALUE_NAME))\n new MapWriter(vector, structVector, keyWriter, valueWriter)\n case (StructType(_), vector: StructVector) =>\n val children = (0 until vector.size()).map { ordinal =>\n createFieldWriter(vector.getChildByOrdinal(ordinal))\n }\n new StructWriter(vector, children.toArray)\n case (NullType, vector: NullVector) => new NullWriter(vector)\n case (_: YearMonthIntervalType, vector: IntervalYearVector) =>\n new IntervalYearWriter(vector)\n case (_: DayTimeIntervalType, vector: DurationVector) => new DurationWriter(vector)\n// case (CalendarIntervalType, vector: IntervalMonthDayNanoVector) =>\n// new IntervalMonthDayNanoWriter(vector)\n case (dt, _) =>\n throw QueryExecutionErrors.notSupportTypeError(dt)\n }\n }\n}\n\nclass ArrowWriter(val root: VectorSchemaRoot, fields: Array[ArrowFieldWriter]) {\n\n def schema: StructType = Utils.fromArrowSchema(root.getSchema())\n\n private var count: Int = 0\n\n def write(row: InternalRow): Unit = {\n var i = 0\n while (i < fields.length) {\n fields(i).write(row, i)\n i += 1\n }\n count += 1\n }\n\n def writeCol(input: ColumnarArray, columnIndex: Int): Unit = {\n fields(columnIndex).writeCol(input)\n count = input.numElements()\n }\n\n def writeColNoNull(input: ColumnarArray, columnIndex: Int): Unit = {\n fields(columnIndex).writeColNoNull(input)\n count = input.numElements()\n }\n\n def finish(): Unit = {\n root.setRowCount(count)\n fields.foreach(_.finish())\n }\n\n def reset(): Unit = {\n root.setRowCount(0)\n count = 0\n fields.foreach(_.reset())\n }\n}\n\nprivate[arrow] abstract class ArrowFieldWriter {\n\n def valueVector: ValueVector\n\n def name: String = valueVector.getField().getName()\n def dataType: DataType = Utils.fromArrowField(valueVector.getField())\n def nullable: Boolean = valueVector.getField().isNullable()\n\n def setNull(): Unit\n def setValue(input: SpecializedGetters, ordinal: Int): Unit\n\n private[arrow] var count: Int = 0\n\n def write(input: SpecializedGetters, ordinal: Int): Unit = {\n if (input.isNullAt(ordinal)) {\n setNull()\n } else {\n setValue(input, ordinal)\n }\n count += 1\n }\n\n def writeCol(input: ColumnarArray): Unit = {\n val inputNumElements = input.numElements()\n valueVector.setInitialCapacity(inputNumElements)\n while (count < inputNumElements) {\n if (input.isNullAt(count)) {\n setNull()\n } else {\n setValue(input, count)\n }\n count += 1\n }\n }\n\n def writeColNoNull(input: ColumnarArray): Unit = {\n val inputNumElements = input.numElements()\n valueVector.setInitialCapacity(inputNumElements)\n while (count < inputNumElements) {\n setValue(input, count)\n count += 1\n }\n }\n\n def finish(): Unit = {\n valueVector.setValueCount(count)\n }\n\n def reset(): Unit = {\n valueVector.reset()\n count = 0\n }\n}\n\nprivate[arrow] class BooleanWriter(val valueVector: BitVector) extends ArrowFieldWriter {\n\n override def setNull(): Unit = {\n valueVector.setNull(count)\n }\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {\n valueVector.setSafe(count, if (input.getBoolean(ordinal)) 1 else 0)\n }\n}\n\nprivate[arrow] class ByteWriter(val valueVector: TinyIntVector) extends ArrowFieldWriter {\n\n override def setNull(): Unit = {\n valueVector.setNull(count)\n }\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {\n valueVector.setSafe(count, input.getByte(ordinal))\n }\n}\n\nprivate[arrow] class ShortWriter(val valueVector: SmallIntVector) extends ArrowFieldWriter {\n\n override def setNull(): Unit = {\n valueVector.setNull(count)\n }\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {\n valueVector.setSafe(count, input.getShort(ordinal))\n }\n}\n\nprivate[arrow] class IntegerWriter(val valueVector: IntVector) extends ArrowFieldWriter {\n\n override def setNull(): Unit = {\n valueVector.setNull(count)\n }\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {\n valueVector.setSafe(count, input.getInt(ordinal))\n }\n}\n\nprivate[arrow] class LongWriter(val valueVector: BigIntVector) extends ArrowFieldWriter {\n\n override def setNull(): Unit = {\n valueVector.setNull(count)\n }\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {\n valueVector.setSafe(count, input.getLong(ordinal))\n }\n}\n\nprivate[arrow] class FloatWriter(val valueVector: Float4Vector) extends ArrowFieldWriter {\n\n override def setNull(): Unit = {\n valueVector.setNull(count)\n }\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {\n valueVector.setSafe(count, input.getFloat(ordinal))\n }\n}\n\nprivate[arrow] class DoubleWriter(val valueVector: Float8Vector) extends ArrowFieldWriter {\n\n override def setNull(): Unit = {\n valueVector.setNull(count)\n }\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {\n valueVector.setSafe(count, input.getDouble(ordinal))\n }\n}\n\nprivate[arrow] class DecimalWriter(val valueVector: DecimalVector, precision: Int, scale: Int)\n extends ArrowFieldWriter {\n\n override def setNull(): Unit = {\n valueVector.setNull(count)\n }\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {\n val decimal = input.getDecimal(ordinal, precision, scale)\n if (decimal.changePrecision(precision, scale)) {\n valueVector.setSafe(count, decimal.toJavaBigDecimal)\n } else {\n setNull()\n }\n }\n}\n\nprivate[arrow] class StringWriter(val valueVector: VarCharVector) extends ArrowFieldWriter {\n\n override def setNull(): Unit = {\n valueVector.setNull(count)\n }\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {\n val utf8 = input.getUTF8String(ordinal)\n val utf8ByteBuffer = utf8.getByteBuffer\n // todo: for off-heap UTF8String, how to pass in to arrow without copy?\n valueVector.setSafe(count, utf8ByteBuffer, utf8ByteBuffer.position(), utf8.numBytes())\n }\n}\n\nprivate[arrow] class LargeStringWriter(val valueVector: LargeVarCharVector)\n extends ArrowFieldWriter {\n\n override def setNull(): Unit = {\n valueVector.setNull(count)\n }\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {\n val utf8 = input.getUTF8String(ordinal)\n val utf8ByteBuffer = utf8.getByteBuffer\n // todo: for off-heap UTF8String, how to pass in to arrow without copy?\n valueVector.setSafe(count, utf8ByteBuffer, utf8ByteBuffer.position(), utf8.numBytes())\n }\n}\n\nprivate[arrow] class BinaryWriter(val valueVector: VarBinaryVector) extends ArrowFieldWriter {\n\n override def setNull(): Unit = {\n valueVector.setNull(count)\n }\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {\n val bytes = input.getBinary(ordinal)\n valueVector.setSafe(count, bytes, 0, bytes.length)\n }\n}\n\nprivate[arrow] class LargeBinaryWriter(val valueVector: LargeVarBinaryVector)\n extends ArrowFieldWriter {\n\n override def setNull(): Unit = {\n valueVector.setNull(count)\n }\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {\n val bytes = input.getBinary(ordinal)\n valueVector.setSafe(count, bytes, 0, bytes.length)\n }\n}\n\nprivate[arrow] class DateWriter(val valueVector: DateDayVector) extends ArrowFieldWriter {\n\n override def setNull(): Unit = {\n valueVector.setNull(count)\n }\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {\n valueVector.setSafe(count, input.getInt(ordinal))\n }\n}\n\nprivate[arrow] class TimestampWriter(val valueVector: TimeStampMicroTZVector)\n extends ArrowFieldWriter {\n\n override def setNull(): Unit = {\n valueVector.setNull(count)\n }\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {\n valueVector.setSafe(count, input.getLong(ordinal))\n }\n}\n\nprivate[arrow] class TimestampNTZWriter(val valueVector: TimeStampMicroVector)\n extends ArrowFieldWriter {\n\n override def setNull(): Unit = {\n valueVector.setNull(count)\n }\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {\n valueVector.setSafe(count, input.getLong(ordinal))\n }\n}\n\nprivate[arrow] class ArrayWriter(val valueVector: ListVector, val elementWriter: ArrowFieldWriter)\n extends ArrowFieldWriter {\n\n override def setNull(): Unit = {}\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {\n val array = input.getArray(ordinal)\n var i = 0\n valueVector.startNewValue(count)\n while (i < array.numElements()) {\n elementWriter.write(array, i)\n i += 1\n }\n valueVector.endValue(count, array.numElements())\n }\n\n override def finish(): Unit = {\n super.finish()\n elementWriter.finish()\n }\n\n override def reset(): Unit = {\n super.reset()\n elementWriter.reset()\n }\n}\n\nprivate[arrow] class StructWriter(\n val valueVector: StructVector,\n children: Array[ArrowFieldWriter])\n extends ArrowFieldWriter {\n\n override def setNull(): Unit = {\n var i = 0\n while (i < children.length) {\n children(i).setNull()\n children(i).count += 1\n i += 1\n }\n valueVector.setNull(count)\n }\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {\n val struct = input.getStruct(ordinal, children.length)\n var i = 0\n valueVector.setIndexDefined(count)\n while (i < struct.numFields) {\n children(i).write(struct, i)\n i += 1\n }\n }\n\n override def finish(): Unit = {\n super.finish()\n children.foreach(_.finish())\n }\n\n override def reset(): Unit = {\n super.reset()\n children.foreach(_.reset())\n }\n}\n\nprivate[arrow] class MapWriter(\n val valueVector: MapVector,\n val structVector: StructVector,\n val keyWriter: ArrowFieldWriter,\n val valueWriter: ArrowFieldWriter)\n extends ArrowFieldWriter {\n\n override def setNull(): Unit = {}\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {\n val map = input.getMap(ordinal)\n valueVector.startNewValue(count)\n val keys = map.keyArray()\n val values = map.valueArray()\n var i = 0\n while (i < map.numElements()) {\n structVector.setIndexDefined(keyWriter.count)\n keyWriter.write(keys, i)\n valueWriter.write(values, i)\n i += 1\n }\n\n valueVector.endValue(count, map.numElements())\n }\n\n override def finish(): Unit = {\n super.finish()\n keyWriter.finish()\n valueWriter.finish()\n }\n\n override def reset(): Unit = {\n super.reset()\n keyWriter.reset()\n valueWriter.reset()\n }\n}\n\nprivate[arrow] class NullWriter(val valueVector: NullVector) extends ArrowFieldWriter {\n\n override def setNull(): Unit = {}\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {}\n}\n\nprivate[arrow] class IntervalYearWriter(val valueVector: IntervalYearVector)\n extends ArrowFieldWriter {\n override def setNull(): Unit = {\n valueVector.setNull(count)\n }\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {\n valueVector.setSafe(count, input.getInt(ordinal));\n }\n}\n\nprivate[arrow] class DurationWriter(val valueVector: DurationVector) extends ArrowFieldWriter {\n override def setNull(): Unit = {\n valueVector.setNull(count)\n }\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {\n valueVector.setSafe(count, input.getLong(ordinal))\n }\n}\n\nprivate[arrow] class IntervalMonthDayNanoWriter(val valueVector: IntervalMonthDayNanoVector)\n extends ArrowFieldWriter {\n override def setNull(): Unit = {\n valueVector.setNull(count)\n }\n\n override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {\n val ci = input.getInterval(ordinal)\n valueVector.setSafe(count, ci.months, ci.days, Math.multiplyExact(ci.microseconds, 1000L))\n }\n}\n" }, { "path": "common/src/main/scala/org/apache/spark/sql/comet/execution/arrow/CometArrowConverters.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.spark.sql.comet.execution.arrow\n\nimport org.apache.arrow.memory.BufferAllocator\nimport org.apache.arrow.vector.VectorSchemaRoot\nimport org.apache.arrow.vector.types.pojo.Schema\nimport org.apache.spark.TaskContext\nimport org.apache.spark.internal.Logging\nimport org.apache.spark.sql.catalyst.InternalRow\nimport org.apache.spark.sql.comet.util.Utils\nimport org.apache.spark.sql.types.StructType\nimport org.apache.spark.sql.vectorized.{ColumnarArray, ColumnarBatch}\n\nimport org.apache.comet.CometArrowAllocator\nimport org.apache.comet.vector.NativeUtil\n\nobject CometArrowConverters extends Logging {\n // This is similar how Spark converts internal row to Arrow format except that it is transforming\n // the result batch to Comet's ColumnarBatch instead of serialized bytes.\n // There's another big difference that Comet may consume the ColumnarBatch by exporting it to\n // the native side. Hence, we need to:\n // 1. reset the Arrow writer after the ColumnarBatch is consumed\n // 2. close the allocator when the task is finished but not when the iterator is all consumed\n // The reason for the second point is that when ColumnarBatch is exported to the native side, the\n // exported process increases the reference count of the Arrow vectors. The reference count is\n // only decreased when the native plan is done with the vectors, which is usually longer than\n // all the ColumnarBatches are consumed.\n\n abstract private[sql] class ArrowBatchIterBase(\n schema: StructType,\n timeZoneId: String,\n context: TaskContext)\n extends Iterator[ColumnarBatch]\n with AutoCloseable {\n\n protected val arrowSchema: Schema = Utils.toArrowSchema(schema, timeZoneId)\n // Reuse the same root allocator here.\n protected val allocator: BufferAllocator =\n CometArrowAllocator.newChildAllocator(s\"to${this.getClass.getSimpleName}\", 0, Long.MaxValue)\n protected val root: VectorSchemaRoot = VectorSchemaRoot.create(arrowSchema, allocator)\n protected val arrowWriter: ArrowWriter = ArrowWriter.create(root)\n\n protected var currentBatch: ColumnarBatch = null\n protected var closed: Boolean = false\n\n Option(context).foreach {\n _.addTaskCompletionListener[Unit] { _ =>\n close(true)\n }\n }\n\n override def close(): Unit = {\n close(false)\n }\n\n protected def close(closeAllocator: Boolean): Unit = {\n try {\n if (!closed) {\n if (currentBatch != null) {\n arrowWriter.reset()\n currentBatch.close()\n currentBatch = null\n }\n root.close()\n closed = true\n }\n } finally {\n // the allocator shall be closed when the task is finished\n if (closeAllocator) {\n allocator.close()\n }\n }\n }\n\n override def next(): ColumnarBatch = {\n currentBatch = nextBatch()\n currentBatch\n }\n\n protected def nextBatch(): ColumnarBatch\n\n }\n\n private[sql] class RowToArrowBatchIter(\n rowIter: Iterator[InternalRow],\n schema: StructType,\n maxRecordsPerBatch: Long,\n timeZoneId: String,\n context: TaskContext)\n extends ArrowBatchIterBase(schema, timeZoneId, context)\n with AutoCloseable {\n\n override def hasNext: Boolean = rowIter.hasNext || {\n close(false)\n false\n }\n\n override protected def nextBatch(): ColumnarBatch = {\n if (rowIter.hasNext) {\n // the arrow writer shall be reset before writing the next batch\n arrowWriter.reset()\n var rowCount = 0L\n while (rowIter.hasNext && (maxRecordsPerBatch <= 0 || rowCount < maxRecordsPerBatch)) {\n val row = rowIter.next()\n arrowWriter.write(row)\n rowCount += 1\n }\n arrowWriter.finish()\n NativeUtil.rootAsBatch(root)\n } else {\n null\n }\n }\n }\n\n def rowToArrowBatchIter(\n rowIter: Iterator[InternalRow],\n schema: StructType,\n maxRecordsPerBatch: Long,\n timeZoneId: String,\n context: TaskContext): Iterator[ColumnarBatch] = {\n new RowToArrowBatchIter(rowIter, schema, maxRecordsPerBatch, timeZoneId, context)\n }\n\n private[sql] class ColumnBatchToArrowBatchIter(\n colBatch: ColumnarBatch,\n schema: StructType,\n maxRecordsPerBatch: Int,\n timeZoneId: String,\n context: TaskContext)\n extends ArrowBatchIterBase(schema, timeZoneId, context)\n with AutoCloseable {\n\n private var rowsProduced: Int = 0\n\n override def hasNext: Boolean = rowsProduced < colBatch.numRows() || {\n close(false)\n false\n }\n\n override protected def nextBatch(): ColumnarBatch = {\n val rowsInBatch = colBatch.numRows()\n if (rowsProduced < rowsInBatch) {\n // the arrow writer shall be reset before writing the next batch\n arrowWriter.reset()\n val rowsToProduce =\n if (maxRecordsPerBatch <= 0) rowsInBatch - rowsProduced\n else Math.min(maxRecordsPerBatch, rowsInBatch - rowsProduced)\n\n for (columnIndex <- 0 until colBatch.numCols()) {\n val column = colBatch.column(columnIndex)\n val columnArray = new ColumnarArray(column, rowsProduced, rowsToProduce)\n if (column.hasNull) {\n arrowWriter.writeCol(columnArray, columnIndex)\n } else {\n arrowWriter.writeColNoNull(columnArray, columnIndex)\n }\n }\n\n rowsProduced += rowsToProduce\n\n arrowWriter.finish()\n NativeUtil.rootAsBatch(root)\n } else {\n null\n }\n }\n }\n\n def columnarBatchToArrowBatchIter(\n colBatch: ColumnarBatch,\n schema: StructType,\n maxRecordsPerBatch: Int,\n timeZoneId: String,\n context: TaskContext): Iterator[ColumnarBatch] = {\n new ColumnBatchToArrowBatchIter(colBatch, schema, maxRecordsPerBatch, timeZoneId, context)\n }\n}\n" }, { "path": "common/src/main/scala/org/apache/spark/sql/comet/parquet/CometParquetReadSupport.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.spark.sql.comet.parquet\n\nimport java.util.{Locale, UUID}\n\nimport scala.jdk.CollectionConverters._\n\nimport org.apache.parquet.schema._\nimport org.apache.parquet.schema.LogicalTypeAnnotation.ListLogicalTypeAnnotation\nimport org.apache.parquet.schema.Type.Repetition\nimport org.apache.spark.sql.errors.QueryExecutionErrors\nimport org.apache.spark.sql.execution.datasources.parquet.ParquetUtils\nimport org.apache.spark.sql.types._\n\n/**\n * This class is copied & slightly modified from [[ParquetReadSupport]] in Spark. Changes:\n * - This doesn't extend from Parquet's `ReadSupport` class since that is used for row-based\n * Parquet reader. Therefore, there is no `init`, `prepareForRead` as well as other methods\n * that are unused.\n */\nobject CometParquetReadSupport {\n val SPARK_PARQUET_SCHEMA_NAME = \"spark_schema\"\n\n val EMPTY_MESSAGE: MessageType =\n Types.buildMessage().named(SPARK_PARQUET_SCHEMA_NAME)\n\n def generateFakeColumnName: String = s\"_fake_name_${UUID.randomUUID()}\"\n\n def clipParquetSchema(\n parquetSchema: MessageType,\n catalystSchema: StructType,\n caseSensitive: Boolean,\n useFieldId: Boolean,\n ignoreMissingIds: Boolean): MessageType = {\n if (!ignoreMissingIds &&\n !containsFieldIds(parquetSchema) &&\n ParquetUtils.hasFieldIds(catalystSchema)) {\n throw new RuntimeException(\n \"Spark read schema expects field Ids, \" +\n \"but Parquet file schema doesn't contain any field Ids.\\n\" +\n \"Please remove the field ids from Spark schema or ignore missing ids by \" +\n \"setting `spark.sql.parquet.fieldId.read.ignoreMissing = true`\\n\" +\n s\"\"\"\n |Spark read schema:\n |${catalystSchema.prettyJson}\n |\n |Parquet file schema:\n |${parquetSchema.toString}\n |\"\"\".stripMargin)\n }\n clipParquetSchema(parquetSchema, catalystSchema, caseSensitive, useFieldId)\n }\n\n /**\n * Tailors `parquetSchema` according to `catalystSchema` by removing column paths don't exist in\n * `catalystSchema`, and adding those only exist in `catalystSchema`.\n */\n def clipParquetSchema(\n parquetSchema: MessageType,\n catalystSchema: StructType,\n caseSensitive: Boolean,\n useFieldId: Boolean): MessageType = {\n val clippedParquetFields = clipParquetGroupFields(\n parquetSchema.asGroupType(),\n catalystSchema,\n caseSensitive,\n useFieldId)\n if (clippedParquetFields.isEmpty) {\n EMPTY_MESSAGE\n } else {\n Types\n .buildMessage()\n .addFields(clippedParquetFields: _*)\n .named(SPARK_PARQUET_SCHEMA_NAME)\n }\n }\n\n private def clipParquetType(\n parquetType: Type,\n catalystType: DataType,\n caseSensitive: Boolean,\n useFieldId: Boolean): Type = {\n val newParquetType = catalystType match {\n case t: ArrayType if !isPrimitiveCatalystType(t.elementType) =>\n // Only clips array types with nested type as element type.\n clipParquetListType(parquetType.asGroupType(), t.elementType, caseSensitive, useFieldId)\n\n case t: MapType\n if !isPrimitiveCatalystType(t.keyType) ||\n !isPrimitiveCatalystType(t.valueType) =>\n // Only clips map types with nested key type or value type\n clipParquetMapType(\n parquetType.asGroupType(),\n t.keyType,\n t.valueType,\n caseSensitive,\n useFieldId)\n\n case t: StructType =>\n clipParquetGroup(parquetType.asGroupType(), t, caseSensitive, useFieldId)\n\n case _ =>\n // UDTs and primitive types are not clipped. For UDTs, a clipped version might not be able\n // to be mapped to desired user-space types. So UDTs shouldn't participate schema merging.\n parquetType\n }\n\n if (useFieldId && parquetType.getId != null) {\n newParquetType.withId(parquetType.getId.intValue())\n } else {\n newParquetType\n }\n }\n\n /**\n * Whether a Catalyst [[DataType]] is primitive. Primitive [[DataType]] is not equivalent to\n * [[AtomicType]]. For example, [[CalendarIntervalType]] is primitive, but it's not an\n * [[AtomicType]].\n */\n private def isPrimitiveCatalystType(dataType: DataType): Boolean = {\n dataType match {\n case _: ArrayType | _: MapType | _: StructType => false\n case _ => true\n }\n }\n\n /**\n * Clips a Parquet [[GroupType]] which corresponds to a Catalyst [[ArrayType]]. The element type\n * of the [[ArrayType]] should also be a nested type, namely an [[ArrayType]], a [[MapType]], or\n * a [[StructType]].\n */\n private def clipParquetListType(\n parquetList: GroupType,\n elementType: DataType,\n caseSensitive: Boolean,\n useFieldId: Boolean): Type = {\n // Precondition of this method, should only be called for lists with nested element types.\n assert(!isPrimitiveCatalystType(elementType))\n\n // Unannotated repeated group should be interpreted as required list of required element, so\n // list element type is just the group itself. Clip it.\n if (parquetList.getLogicalTypeAnnotation == null &&\n parquetList.isRepetition(Repetition.REPEATED)) {\n clipParquetType(parquetList, elementType, caseSensitive, useFieldId)\n } else {\n assert(\n parquetList.getLogicalTypeAnnotation.isInstanceOf[ListLogicalTypeAnnotation],\n \"Invalid Parquet schema. \" +\n \"Logical type annotation of annotated Parquet lists must be ListLogicalTypeAnnotation: \" +\n parquetList.toString)\n\n assert(\n parquetList.getFieldCount == 1 && parquetList\n .getType(0)\n .isRepetition(Repetition.REPEATED),\n \"Invalid Parquet schema. \" +\n \"LIST-annotated group should only have exactly one repeated field: \" +\n parquetList)\n\n // Precondition of this method, should only be called for lists with nested element types.\n assert(!parquetList.getType(0).isPrimitive)\n\n val repeatedGroup = parquetList.getType(0).asGroupType()\n\n // If the repeated field is a group with multiple fields, or the repeated field is a group\n // with one field and is named either \"array\" or uses the LIST-annotated group's name with\n // \"_tuple\" appended then the repeated type is the element type and elements are required.\n // Build a new LIST-annotated group with clipped `repeatedGroup` as element type and the\n // only field.\n if (repeatedGroup.getFieldCount > 1 ||\n repeatedGroup.getName == \"array\" ||\n repeatedGroup.getName == parquetList.getName + \"_tuple\") {\n Types\n .buildGroup(parquetList.getRepetition)\n .as(LogicalTypeAnnotation.listType())\n .addField(clipParquetType(repeatedGroup, elementType, caseSensitive, useFieldId))\n .named(parquetList.getName)\n } else {\n val newRepeatedGroup = Types\n .repeatedGroup()\n .addField(\n clipParquetType(repeatedGroup.getType(0), elementType, caseSensitive, useFieldId))\n .named(repeatedGroup.getName)\n\n val newElementType = if (useFieldId && repeatedGroup.getId != null) {\n newRepeatedGroup.withId(repeatedGroup.getId.intValue())\n } else {\n newRepeatedGroup\n }\n\n // Otherwise, the repeated field's type is the element type with the repeated field's\n // repetition.\n Types\n .buildGroup(parquetList.getRepetition)\n .as(LogicalTypeAnnotation.listType())\n .addField(newElementType)\n .named(parquetList.getName)\n }\n }\n }\n\n /**\n * Clips a Parquet [[GroupType]] which corresponds to a Catalyst [[MapType]]. Either key type or\n * value type of the [[MapType]] must be a nested type, namely an [[ArrayType]], a [[MapType]],\n * or a [[StructType]].\n */\n private def clipParquetMapType(\n parquetMap: GroupType,\n keyType: DataType,\n valueType: DataType,\n caseSensitive: Boolean,\n useFieldId: Boolean): GroupType = {\n // Precondition of this method, only handles maps with nested key types or value types.\n assert(!isPrimitiveCatalystType(keyType) || !isPrimitiveCatalystType(valueType))\n\n val repeatedGroup = parquetMap.getType(0).asGroupType()\n val parquetKeyType = repeatedGroup.getType(0)\n val parquetValueType = repeatedGroup.getType(1)\n\n val clippedRepeatedGroup = {\n val newRepeatedGroup = Types\n .repeatedGroup()\n .as(repeatedGroup.getLogicalTypeAnnotation)\n .addField(clipParquetType(parquetKeyType, keyType, caseSensitive, useFieldId))\n .addField(clipParquetType(parquetValueType, valueType, caseSensitive, useFieldId))\n .named(repeatedGroup.getName)\n if (useFieldId && repeatedGroup.getId != null) {\n newRepeatedGroup.withId(repeatedGroup.getId.intValue())\n } else {\n newRepeatedGroup\n }\n }\n\n Types\n .buildGroup(parquetMap.getRepetition)\n .as(parquetMap.getLogicalTypeAnnotation)\n .addField(clippedRepeatedGroup)\n .named(parquetMap.getName)\n }\n\n /**\n * Clips a Parquet [[GroupType]] which corresponds to a Catalyst [[StructType]].\n *\n * @return\n * A clipped [[GroupType]], which has at least one field.\n * @note\n * Parquet doesn't allow creating empty [[GroupType]] instances except for empty\n * [[MessageType]]. Because it's legal to construct an empty requested schema for column\n * pruning.\n */\n private def clipParquetGroup(\n parquetRecord: GroupType,\n structType: StructType,\n caseSensitive: Boolean,\n useFieldId: Boolean): GroupType = {\n val clippedParquetFields =\n clipParquetGroupFields(parquetRecord, structType, caseSensitive, useFieldId)\n Types\n .buildGroup(parquetRecord.getRepetition)\n .as(parquetRecord.getLogicalTypeAnnotation)\n .addFields(clippedParquetFields: _*)\n .named(parquetRecord.getName)\n }\n\n /**\n * Clips a Parquet [[GroupType]] which corresponds to a Catalyst [[StructType]].\n *\n * @return\n * A list of clipped [[GroupType]] fields, which can be empty.\n */\n private def clipParquetGroupFields(\n parquetRecord: GroupType,\n structType: StructType,\n caseSensitive: Boolean,\n useFieldId: Boolean): Seq[Type] = {\n val toParquet = new CometSparkToParquetSchemaConverter(\n writeLegacyParquetFormat = false,\n useFieldId = useFieldId)\n lazy val caseSensitiveParquetFieldMap =\n parquetRecord.getFields.asScala.map(f => f.getName -> f).toMap\n lazy val caseInsensitiveParquetFieldMap =\n parquetRecord.getFields.asScala.groupBy(_.getName.toLowerCase(Locale.ROOT))\n lazy val idToParquetFieldMap =\n parquetRecord.getFields.asScala.filter(_.getId != null).groupBy(f => f.getId.intValue())\n\n def matchCaseSensitiveField(f: StructField): Type = {\n caseSensitiveParquetFieldMap\n .get(f.name)\n .map(clipParquetType(_, f.dataType, caseSensitive, useFieldId))\n .getOrElse(toParquet.convertField(f))\n }\n\n def matchCaseInsensitiveField(f: StructField): Type = {\n // Do case-insensitive resolution only if in case-insensitive mode\n caseInsensitiveParquetFieldMap\n .get(f.name.toLowerCase(Locale.ROOT))\n .map { parquetTypes =>\n if (parquetTypes.size > 1) {\n // Need to fail if there is ambiguity, i.e. more than one field is matched\n val parquetTypesString = parquetTypes.map(_.getName).mkString(\"[\", \", \", \"]\")\n throw QueryExecutionErrors.foundDuplicateFieldInCaseInsensitiveModeError(\n f.name,\n parquetTypesString)\n } else {\n clipParquetType(parquetTypes.head, f.dataType, caseSensitive, useFieldId)\n }\n }\n .getOrElse(toParquet.convertField(f))\n }\n\n def matchIdField(f: StructField): Type = {\n val fieldId = ParquetUtils.getFieldId(f)\n idToParquetFieldMap\n .get(fieldId)\n .map { parquetTypes =>\n if (parquetTypes.size > 1) {\n // Need to fail if there is ambiguity, i.e. more than one field is matched\n val parquetTypesString = parquetTypes.map(_.getName).mkString(\"[\", \", \", \"]\")\n throw QueryExecutionErrors.foundDuplicateFieldInFieldIdLookupModeError(\n fieldId,\n parquetTypesString)\n } else {\n clipParquetType(parquetTypes.head, f.dataType, caseSensitive, useFieldId)\n }\n }\n .getOrElse {\n // When there is no ID match, we use a fake name to avoid a name match by accident\n // We need this name to be unique as well, otherwise there will be type conflicts\n toParquet.convertField(f.copy(name = generateFakeColumnName))\n }\n }\n\n val shouldMatchById = useFieldId && ParquetUtils.hasFieldIds(structType)\n structType.map { f =>\n if (shouldMatchById && ParquetUtils.hasFieldId(f)) {\n matchIdField(f)\n } else if (caseSensitive) {\n matchCaseSensitiveField(f)\n } else {\n matchCaseInsensitiveField(f)\n }\n }\n }\n\n /**\n * Whether the parquet schema contains any field IDs.\n */\n private def containsFieldIds(schema: Type): Boolean = schema match {\n case p: PrimitiveType => p.getId != null\n // We don't require all fields to have IDs, so we use `exists` here.\n case g: GroupType => g.getId != null || g.getFields.asScala.exists(containsFieldIds)\n }\n}\n" }, { "path": "common/src/main/scala/org/apache/spark/sql/comet/parquet/CometSparkToParquetSchemaConverter.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.spark.sql.comet.parquet\n\nimport org.apache.hadoop.conf.Configuration\nimport org.apache.parquet.schema._\nimport org.apache.parquet.schema.LogicalTypeAnnotation._\nimport org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName._\nimport org.apache.parquet.schema.Type.Repetition._\nimport org.apache.spark.sql.errors.QueryCompilationErrors\nimport org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter\nimport org.apache.spark.sql.execution.datasources.parquet.ParquetUtils\nimport org.apache.spark.sql.internal.SQLConf\nimport org.apache.spark.sql.types._\n\nimport org.apache.comet.parquet.CometParquetUtils\n\n/**\n * This class is copied & modified from Spark's [[SparkToParquetSchemaConverter]] class.\n */\nclass CometSparkToParquetSchemaConverter(\n writeLegacyParquetFormat: Boolean = SQLConf.PARQUET_WRITE_LEGACY_FORMAT.defaultValue.get,\n outputTimestampType: SQLConf.ParquetOutputTimestampType.Value =\n SQLConf.ParquetOutputTimestampType.INT96,\n useFieldId: Boolean = CometParquetUtils.writeFieldId(new SQLConf)) {\n\n def this(conf: SQLConf) = this(\n writeLegacyParquetFormat = conf.writeLegacyParquetFormat,\n outputTimestampType = conf.parquetOutputTimestampType,\n useFieldId = CometParquetUtils.writeFieldId(conf))\n\n def this(conf: Configuration) = this(\n writeLegacyParquetFormat = conf.get(SQLConf.PARQUET_WRITE_LEGACY_FORMAT.key).toBoolean,\n outputTimestampType = SQLConf.ParquetOutputTimestampType.withName(\n conf.get(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key)),\n useFieldId = CometParquetUtils.writeFieldId(conf))\n\n /**\n * Converts a Spark SQL [[StructType]] to a Parquet [[MessageType]].\n */\n def convert(catalystSchema: StructType): MessageType = {\n Types\n .buildMessage()\n .addFields(catalystSchema.map(convertField): _*)\n .named(ParquetSchemaConverter.SPARK_PARQUET_SCHEMA_NAME)\n }\n\n /**\n * Converts a Spark SQL [[StructField]] to a Parquet [[Type]].\n */\n def convertField(field: StructField): Type = {\n val converted = convertField(field, if (field.nullable) OPTIONAL else REQUIRED)\n if (useFieldId && ParquetUtils.hasFieldId(field)) {\n converted.withId(ParquetUtils.getFieldId(field))\n } else {\n converted\n }\n }\n\n private def convertField(field: StructField, repetition: Type.Repetition): Type = {\n\n field.dataType match {\n // ===================\n // Simple atomic types\n // ===================\n\n case BooleanType =>\n Types.primitive(BOOLEAN, repetition).named(field.name)\n\n case ByteType =>\n Types\n .primitive(INT32, repetition)\n .as(LogicalTypeAnnotation.intType(8, true))\n .named(field.name)\n\n case ShortType =>\n Types\n .primitive(INT32, repetition)\n .as(LogicalTypeAnnotation.intType(16, true))\n .named(field.name)\n\n case IntegerType =>\n Types.primitive(INT32, repetition).named(field.name)\n\n case LongType =>\n Types.primitive(INT64, repetition).named(field.name)\n\n case FloatType =>\n Types.primitive(FLOAT, repetition).named(field.name)\n\n case DoubleType =>\n Types.primitive(DOUBLE, repetition).named(field.name)\n\n case StringType =>\n Types\n .primitive(BINARY, repetition)\n .as(LogicalTypeAnnotation.stringType())\n .named(field.name)\n\n case DateType =>\n Types\n .primitive(INT32, repetition)\n .as(LogicalTypeAnnotation.dateType())\n .named(field.name)\n\n // NOTE: Spark SQL can write timestamp values to Parquet using INT96, TIMESTAMP_MICROS or\n // TIMESTAMP_MILLIS. TIMESTAMP_MICROS is recommended but INT96 is the default to keep the\n // behavior same as before.\n //\n // As stated in PARQUET-323, Parquet `INT96` was originally introduced to represent nanosecond\n // timestamp in Impala for some historical reasons. It's not recommended to be used for any\n // other types and will probably be deprecated in some future version of parquet-format spec.\n // That's the reason why parquet-format spec only defines `TIMESTAMP_MILLIS` and\n // `TIMESTAMP_MICROS` which are both logical types annotating `INT64`.\n //\n // Originally, Spark SQL uses the same nanosecond timestamp type as Impala and Hive. Starting\n // from Spark 1.5.0, we resort to a timestamp type with microsecond precision so that we can\n // store a timestamp into a `Long`. This design decision is subject to change though, for\n // example, we may resort to nanosecond precision in the future.\n case TimestampType =>\n outputTimestampType match {\n case SQLConf.ParquetOutputTimestampType.INT96 =>\n Types.primitive(INT96, repetition).named(field.name)\n case SQLConf.ParquetOutputTimestampType.TIMESTAMP_MICROS =>\n Types\n .primitive(INT64, repetition)\n .as(LogicalTypeAnnotation.timestampType(true, TimeUnit.MICROS))\n .named(field.name)\n case SQLConf.ParquetOutputTimestampType.TIMESTAMP_MILLIS =>\n Types\n .primitive(INT64, repetition)\n .as(LogicalTypeAnnotation.timestampType(true, TimeUnit.MILLIS))\n .named(field.name)\n }\n\n case TimestampNTZType =>\n Types\n .primitive(INT64, repetition)\n .as(LogicalTypeAnnotation.timestampType(false, TimeUnit.MICROS))\n .named(field.name)\n\n case BinaryType =>\n Types.primitive(BINARY, repetition).named(field.name)\n\n // ======================\n // Decimals (legacy mode)\n // ======================\n\n // Spark 1.4.x and prior versions only support decimals with a maximum precision of 18 and\n // always store decimals in fixed-length byte arrays. To keep compatibility with these older\n // versions, here we convert decimals with all precisions to `FIXED_LEN_BYTE_ARRAY` annotated\n // by `DECIMAL`.\n case DecimalType.Fixed(precision, scale) if writeLegacyParquetFormat =>\n Types\n .primitive(FIXED_LEN_BYTE_ARRAY, repetition)\n .as(LogicalTypeAnnotation.decimalType(scale, precision))\n .length(Decimal.minBytesForPrecision(precision))\n .named(field.name)\n\n // ========================\n // Decimals (standard mode)\n // ========================\n\n // Uses INT32 for 1 <= precision <= 9\n case DecimalType.Fixed(precision, scale)\n if precision <= Decimal.MAX_INT_DIGITS && !writeLegacyParquetFormat =>\n Types\n .primitive(INT32, repetition)\n .as(LogicalTypeAnnotation.decimalType(scale, precision))\n .named(field.name)\n\n // Uses INT64 for 1 <= precision <= 18\n case DecimalType.Fixed(precision, scale)\n if precision <= Decimal.MAX_LONG_DIGITS && !writeLegacyParquetFormat =>\n Types\n .primitive(INT64, repetition)\n .as(LogicalTypeAnnotation.decimalType(scale, precision))\n .named(field.name)\n\n // Uses FIXED_LEN_BYTE_ARRAY for all other precisions\n case DecimalType.Fixed(precision, scale) if !writeLegacyParquetFormat =>\n Types\n .primitive(FIXED_LEN_BYTE_ARRAY, repetition)\n .as(LogicalTypeAnnotation.decimalType(scale, precision))\n .length(Decimal.minBytesForPrecision(precision))\n .named(field.name)\n\n // ===================================\n // ArrayType and MapType (legacy mode)\n // ===================================\n\n // Spark 1.4.x and prior versions convert `ArrayType` with nullable elements into a 3-level\n // `LIST` structure. This behavior is somewhat a hybrid of parquet-hive and parquet-avro\n // (1.6.0rc3): the 3-level structure is similar to parquet-hive while the 3rd level element\n // field name \"array\" is borrowed from parquet-avro.\n case ArrayType(elementType, nullable @ true) if writeLegacyParquetFormat =>\n // group (LIST) {\n // optional group bag {\n // repeated array;\n // }\n // }\n\n // This should not use `listOfElements` here because this new method checks if the\n // element name is `element` in the `GroupType` and throws an exception if not.\n // As mentioned above, Spark prior to 1.4.x writes `ArrayType` as `LIST` but with\n // `array` as its element name as below. Therefore, we build manually\n // the correct group type here via the builder. (See SPARK-16777)\n Types\n .buildGroup(repetition)\n .as(LogicalTypeAnnotation.listType())\n .addField(\n Types\n .buildGroup(REPEATED)\n // \"array\" is the name chosen by parquet-hive (1.7.0 and prior version)\n .addField(convertField(StructField(\"array\", elementType, nullable)))\n .named(\"bag\"))\n .named(field.name)\n\n // Spark 1.4.x and prior versions convert ArrayType with non-nullable elements into a 2-level\n // LIST structure. This behavior mimics parquet-avro (1.6.0rc3). Note that this case is\n // covered by the backwards-compatibility rules implemented in `isElementType()`.\n case ArrayType(elementType, nullable @ false) if writeLegacyParquetFormat =>\n // group (LIST) {\n // repeated element;\n // }\n\n // Here too, we should not use `listOfElements`. (See SPARK-16777)\n Types\n .buildGroup(repetition)\n .as(LogicalTypeAnnotation.listType())\n // \"array\" is the name chosen by parquet-avro (1.7.0 and prior version)\n .addField(convertField(StructField(\"array\", elementType, nullable), REPEATED))\n .named(field.name)\n\n // Spark 1.4.x and prior versions convert MapType into a 3-level group annotated by\n // MAP_KEY_VALUE. This is covered by `convertGroupField(field: GroupType): DataType`.\n case MapType(keyType, valueType, valueContainsNull) if writeLegacyParquetFormat =>\n // group (MAP) {\n // repeated group map (MAP_KEY_VALUE) {\n // required key;\n // value;\n // }\n // }\n ConversionPatterns.mapType(\n repetition,\n field.name,\n convertField(StructField(\"key\", keyType, nullable = false)),\n convertField(StructField(\"value\", valueType, valueContainsNull)))\n\n // =====================================\n // ArrayType and MapType (standard mode)\n // =====================================\n\n case ArrayType(elementType, containsNull) if !writeLegacyParquetFormat =>\n // group (LIST) {\n // repeated group list {\n // element;\n // }\n // }\n Types\n .buildGroup(repetition)\n .as(LogicalTypeAnnotation.listType())\n .addField(\n Types\n .repeatedGroup()\n .addField(convertField(StructField(\"element\", elementType, containsNull)))\n .named(\"list\"))\n .named(field.name)\n\n case MapType(keyType, valueType, valueContainsNull) =>\n // group (MAP) {\n // repeated group key_value {\n // required key;\n // value;\n // }\n // }\n Types\n .buildGroup(repetition)\n .as(LogicalTypeAnnotation.mapType())\n .addField(\n Types\n .repeatedGroup()\n .addField(convertField(StructField(\"key\", keyType, nullable = false)))\n .addField(convertField(StructField(\"value\", valueType, valueContainsNull)))\n .named(\"key_value\"))\n .named(field.name)\n\n // ===========\n // Other types\n // ===========\n\n case StructType(fields) =>\n fields\n .foldLeft(Types.buildGroup(repetition)) { (builder, field) =>\n builder.addField(convertField(field))\n }\n .named(field.name)\n\n case udt: UserDefinedType[_] =>\n convertField(field.copy(dataType = udt.sqlType))\n\n case _ =>\n throw QueryCompilationErrors.cannotConvertDataTypeToParquetTypeError(field)\n }\n }\n}\n" }, { "path": "common/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.spark.sql.comet.util\n\nimport java.io.{DataInputStream, DataOutputStream, File}\nimport java.nio.ByteBuffer\nimport java.nio.channels.Channels\n\nimport scala.jdk.CollectionConverters._\n\nimport org.apache.arrow.c.CDataDictionaryProvider\nimport org.apache.arrow.vector._\nimport org.apache.arrow.vector.complex.{ListVector, MapVector, StructVector}\nimport org.apache.arrow.vector.dictionary.DictionaryProvider\nimport org.apache.arrow.vector.ipc.{ArrowStreamReader, ArrowStreamWriter}\nimport org.apache.arrow.vector.types._\nimport org.apache.arrow.vector.types.pojo.{ArrowType, Field, FieldType, Schema}\nimport org.apache.arrow.vector.util.VectorSchemaRootAppender\nimport org.apache.spark.{SparkEnv, SparkException}\nimport org.apache.spark.internal.Logging\nimport org.apache.spark.io.CompressionCodec\nimport org.apache.spark.sql.comet.execution.arrow.ArrowReaderIterator\nimport org.apache.spark.sql.types._\nimport org.apache.spark.sql.vectorized.ColumnarBatch\nimport org.apache.spark.util.io.{ChunkedByteBuffer, ChunkedByteBufferOutputStream}\n\nimport org.apache.comet.Constants.COMET_CONF_DIR_ENV\nimport org.apache.comet.shims.CometTypeShim\nimport org.apache.comet.vector.CometVector\n\nobject Utils extends CometTypeShim with Logging {\n def getConfPath(confFileName: String): String = {\n sys.env\n .get(COMET_CONF_DIR_ENV)\n .map { t => new File(s\"$t${File.separator}$confFileName\") }\n .filter(_.isFile)\n .map(_.getAbsolutePath)\n .orNull\n }\n\n def stringToSeq(str: String): Seq[String] = {\n str.split(\",\").map(_.trim()).filter(_.nonEmpty)\n }\n\n /** bridges the function call to Spark's Util */\n def getSimpleName(cls: Class[_]): String = {\n org.apache.spark.util.Utils.getSimpleName(cls)\n }\n\n def fromArrowField(field: Field): DataType = {\n field.getType match {\n case _: ArrowType.Map =>\n val elementField = field.getChildren.get(0)\n val keyType = fromArrowField(elementField.getChildren.get(0))\n val valueType = fromArrowField(elementField.getChildren.get(1))\n MapType(keyType, valueType, elementField.getChildren.get(1).isNullable)\n case ArrowType.List.INSTANCE =>\n val elementField = field.getChildren().get(0)\n val elementType = fromArrowField(elementField)\n ArrayType(elementType, containsNull = elementField.isNullable)\n case ArrowType.Struct.INSTANCE =>\n val fields = field.getChildren().asScala.map { child =>\n val dt = fromArrowField(child)\n StructField(child.getName, dt, child.isNullable)\n }\n StructType(fields.toSeq)\n case arrowType => fromArrowType(arrowType)\n }\n }\n\n def fromArrowType(dt: ArrowType): DataType = dt match {\n case ArrowType.Bool.INSTANCE => BooleanType\n case int: ArrowType.Int if int.getIsSigned && int.getBitWidth == 8 => ByteType\n case int: ArrowType.Int if int.getIsSigned && int.getBitWidth == 8 * 2 => ShortType\n case int: ArrowType.Int if int.getIsSigned && int.getBitWidth == 8 * 4 => IntegerType\n case int: ArrowType.Int if int.getIsSigned && int.getBitWidth == 8 * 8 => LongType\n case float: ArrowType.FloatingPoint if float.getPrecision == FloatingPointPrecision.SINGLE =>\n FloatType\n case float: ArrowType.FloatingPoint if float.getPrecision == FloatingPointPrecision.DOUBLE =>\n DoubleType\n case ArrowType.Utf8.INSTANCE => StringType\n case ArrowType.Binary.INSTANCE => BinaryType\n case _: ArrowType.FixedSizeBinary => BinaryType\n case d: ArrowType.Decimal => DecimalType(d.getPrecision, d.getScale)\n case date: ArrowType.Date if date.getUnit == DateUnit.DAY => DateType\n case ts: ArrowType.Timestamp\n if ts.getUnit == TimeUnit.MICROSECOND && ts.getTimezone == null =>\n TimestampNTZType\n case ts: ArrowType.Timestamp if ts.getUnit == TimeUnit.MICROSECOND => TimestampType\n case ArrowType.Null.INSTANCE => NullType\n case yi: ArrowType.Interval if yi.getUnit == IntervalUnit.YEAR_MONTH =>\n YearMonthIntervalType()\n case di: ArrowType.Interval if di.getUnit == IntervalUnit.DAY_TIME => DayTimeIntervalType()\n case _ => throw new UnsupportedOperationException(s\"Unsupported data type: ${dt.toString}\")\n }\n\n def fromArrowSchema(schema: Schema): StructType = {\n StructType(schema.getFields.asScala.map { field =>\n val dt = fromArrowField(field)\n StructField(field.getName, dt, field.isNullable)\n }.toArray)\n }\n\n /** Maps data type from Spark to Arrow. NOTE: timeZoneId required for TimestampTypes */\n def toArrowType(dt: DataType, timeZoneId: String): ArrowType =\n dt match {\n case BooleanType => ArrowType.Bool.INSTANCE\n case ByteType => new ArrowType.Int(8, true)\n case ShortType => new ArrowType.Int(8 * 2, true)\n case IntegerType => new ArrowType.Int(8 * 4, true)\n case LongType => new ArrowType.Int(8 * 8, true)\n case FloatType => new ArrowType.FloatingPoint(FloatingPointPrecision.SINGLE)\n case DoubleType => new ArrowType.FloatingPoint(FloatingPointPrecision.DOUBLE)\n case _: StringType => ArrowType.Utf8.INSTANCE\n case dt if isStringCollationType(dt) => ArrowType.Utf8.INSTANCE\n case BinaryType => ArrowType.Binary.INSTANCE\n case DecimalType.Fixed(precision, scale) => new ArrowType.Decimal(precision, scale, 128)\n case DateType => new ArrowType.Date(DateUnit.DAY)\n case TimestampType =>\n if (timeZoneId == null) {\n throw new UnsupportedOperationException(\n s\"${TimestampType.catalogString} must supply timeZoneId parameter\")\n } else {\n new ArrowType.Timestamp(TimeUnit.MICROSECOND, timeZoneId)\n }\n case TimestampNTZType =>\n new ArrowType.Timestamp(TimeUnit.MICROSECOND, null)\n case _ =>\n throw new UnsupportedOperationException(\n s\"Unsupported data type: [${dt.getClass.getName}] ${dt.catalogString}\")\n }\n\n /** Maps field from Spark to Arrow. NOTE: timeZoneId required for TimestampType */\n def toArrowField(name: String, dt: DataType, nullable: Boolean, timeZoneId: String): Field = {\n dt match {\n case ArrayType(elementType, containsNull) =>\n val fieldType = new FieldType(nullable, ArrowType.List.INSTANCE, null)\n new Field(\n name,\n fieldType,\n Seq(toArrowField(\"element\", elementType, containsNull, timeZoneId)).asJava)\n case StructType(fields) =>\n val fieldType = new FieldType(nullable, ArrowType.Struct.INSTANCE, null)\n new Field(\n name,\n fieldType,\n fields\n .map { field =>\n toArrowField(field.name, field.dataType, field.nullable, timeZoneId)\n }\n .toSeq\n .asJava)\n case MapType(keyType, valueType, valueContainsNull) =>\n val mapType = new FieldType(nullable, new ArrowType.Map(false), null)\n // Note: Map Type struct can not be null, Struct Type key field can not be null\n new Field(\n name,\n mapType,\n Seq(\n toArrowField(\n MapVector.DATA_VECTOR_NAME,\n new StructType()\n .add(MapVector.KEY_NAME, keyType, nullable = false)\n .add(MapVector.VALUE_NAME, valueType, nullable = valueContainsNull),\n nullable = false,\n timeZoneId)).asJava)\n case dataType =>\n val fieldType = new FieldType(nullable, toArrowType(dataType, timeZoneId), null)\n new Field(name, fieldType, Seq.empty[Field].asJava)\n }\n }\n\n /**\n * Maps schema from Spark to Arrow. NOTE: timeZoneId required for TimestampType in StructType\n */\n def toArrowSchema(schema: StructType, timeZoneId: String): Schema = {\n new Schema(schema.map { field =>\n toArrowField(field.name, field.dataType, field.nullable, timeZoneId)\n }.asJava)\n }\n\n /**\n * Serializes a list of `ColumnarBatch` into an output stream. This method must be in `spark`\n * package because `ChunkedByteBufferOutputStream` is spark private class. As it uses Arrow\n * classes, it must be in `common` module.\n *\n * @param batches\n * the output batches, each batch is a list of Arrow vectors wrapped in `CometVector`\n * @param out\n * the output stream\n */\n def serializeBatches(batches: Iterator[ColumnarBatch]): Iterator[(Long, ChunkedByteBuffer)] = {\n batches.map { batch =>\n val dictionaryProvider: CDataDictionaryProvider = new CDataDictionaryProvider\n\n val codec = CompressionCodec.createCodec(SparkEnv.get.conf)\n val cbbos = new ChunkedByteBufferOutputStream(1024 * 1024, ByteBuffer.allocate)\n val out = new DataOutputStream(codec.compressedOutputStream(cbbos))\n\n val (fieldVectors, batchProviderOpt) = getBatchFieldVectors(batch)\n val root = new VectorSchemaRoot(fieldVectors.asJava)\n val provider = batchProviderOpt.getOrElse(dictionaryProvider)\n\n val writer = new ArrowStreamWriter(root, provider, Channels.newChannel(out))\n writer.start()\n writer.writeBatch()\n root.clear()\n writer.close()\n\n if (out.size() > 0) {\n (batch.numRows().toLong, cbbos.toChunkedByteBuffer)\n } else {\n (batch.numRows().toLong, new ChunkedByteBuffer(Array.empty[ByteBuffer]))\n }\n }\n }\n\n /**\n * Decodes the byte arrays back to ColumnarBatchs and put them into buffer.\n *\n * @param bytes\n * the serialized batches\n * @param source\n * the class that calls this method\n * @return\n * an iterator of ColumnarBatch\n */\n def decodeBatches(bytes: ChunkedByteBuffer, source: String): Iterator[ColumnarBatch] = {\n if (bytes.size == 0) {\n return Iterator.empty\n }\n\n // use Spark's compression codec (LZ4 by default) and not Comet's compression\n val codec = CompressionCodec.createCodec(SparkEnv.get.conf)\n val cbbis = bytes.toInputStream()\n val ins = new DataInputStream(codec.compressedInputStream(cbbis))\n // batches are in Arrow IPC format\n new ArrowReaderIterator(Channels.newChannel(ins), source)\n }\n\n /**\n * Coalesces many small Arrow IPC batches into a single batch for broadcasting.\n *\n * Why this is necessary: The broadcast exchange collects shuffle output by calling\n * getByteArrayRdd, which serializes each ColumnarBatch independently into its own\n * ChunkedByteBuffer. The shuffle reader (CometBlockStoreShuffleReader) produces one\n * ColumnarBatch per shuffle block, and there is one block per writer task per output partition.\n * So with W writer tasks and P output partitions, the broadcast collects up to W * P tiny\n * batches. For example, with 400 writer tasks and 500 partitions, 1M rows would arrive as ~200K\n * batches of ~5 rows each.\n *\n * Without coalescing, every consumer task in the broadcast join would independently deserialize\n * all of these tiny Arrow IPC streams, paying per-stream overhead (schema parsing, buffer\n * allocation) for each one. With coalescing, we decode and append all batches into one\n * VectorSchemaRoot on the driver, then re-serialize once. Each consumer task then deserializes\n * a single Arrow IPC stream.\n */\n def coalesceBroadcastBatches(\n input: Iterator[ChunkedByteBuffer]): (Array[ChunkedByteBuffer], Long, Long) = {\n val buffers = input.filterNot(_.size == 0).toArray\n if (buffers.isEmpty) {\n return (Array.empty, 0L, 0L)\n }\n\n val allocator = org.apache.comet.CometArrowAllocator\n .newChildAllocator(\"broadcast-coalesce\", 0, Long.MaxValue)\n try {\n var targetRoot: VectorSchemaRoot = null\n var totalRows = 0L\n var batchCount = 0\n\n val codec = CompressionCodec.createCodec(SparkEnv.get.conf)\n try {\n for (bytes <- buffers) {\n val compressedInputStream =\n new DataInputStream(codec.compressedInputStream(bytes.toInputStream()))\n val reader =\n new ArrowStreamReader(Channels.newChannel(compressedInputStream), allocator)\n try {\n // Comet decodes dictionaries during execution, so this shouldn't happen.\n // If it does, fall back to the original uncoalesced buffers because each\n // partition can have a different dictionary, and appending index vectors\n // would silently mix indices from incompatible dictionaries.\n if (!reader.getDictionaryVectors.isEmpty) {\n logWarning(\n \"Unexpected dictionary-encoded column during BroadcastExchange coalescing; \" +\n \"skipping coalesce\")\n reader.close()\n if (targetRoot != null) {\n targetRoot.close()\n targetRoot = null\n }\n return (buffers, 0L, 0L)\n }\n while (reader.loadNextBatch()) {\n val sourceRoot = reader.getVectorSchemaRoot\n if (targetRoot == null) {\n targetRoot = VectorSchemaRoot.create(sourceRoot.getSchema, allocator)\n targetRoot.allocateNew()\n }\n VectorSchemaRootAppender.append(targetRoot, sourceRoot)\n totalRows += sourceRoot.getRowCount\n batchCount += 1\n }\n } finally {\n reader.close()\n }\n }\n\n if (targetRoot == null) {\n return (Array.empty, 0L, 0L)\n }\n\n assert(\n targetRoot.getRowCount.toLong == totalRows,\n s\"Row count mismatch after coalesce: ${targetRoot.getRowCount} != $totalRows\")\n\n logInfo(s\"Coalesced $batchCount broadcast batches into 1 ($totalRows rows)\")\n\n val outputStream = new ChunkedByteBufferOutputStream(1024 * 1024, ByteBuffer.allocate)\n val compressedOutputStream =\n new DataOutputStream(codec.compressedOutputStream(outputStream))\n val writer =\n new ArrowStreamWriter(targetRoot, null, Channels.newChannel(compressedOutputStream))\n try {\n writer.start()\n writer.writeBatch()\n } finally {\n writer.close()\n }\n\n (Array(outputStream.toChunkedByteBuffer), batchCount.toLong, totalRows)\n } finally {\n if (targetRoot != null) {\n targetRoot.close()\n }\n }\n } finally {\n allocator.close()\n }\n }\n\n def getBatchFieldVectors(\n batch: ColumnarBatch): (Seq[FieldVector], Option[DictionaryProvider]) = {\n var provider: Option[DictionaryProvider] = None\n val fieldVectors = (0 until batch.numCols()).map { index =>\n batch.column(index) match {\n case a: CometVector =>\n val valueVector = a.getValueVector\n if (valueVector.getField.getDictionary != null) {\n if (provider.isEmpty) {\n provider = Some(a.getDictionaryProvider)\n }\n }\n\n getFieldVector(valueVector, \"serialize\")\n\n case c =>\n throw new SparkException(\n s\"Comet execution only takes Arrow Arrays, but got ${c.getClass}. \" +\n \"This typically happens when a Comet scan falls back to Spark due to unsupported \" +\n \"data types (e.g., complex types like structs, arrays, or maps). \" +\n \"To resolve this, you can: \" +\n \"(1) enable spark.comet.scan.allowIncompatible=true to use a compatible native \" +\n \"scan variant, or \" +\n \"(2) enable spark.comet.convert.parquet.enabled=true to convert Spark Parquet \" +\n \"data to Arrow format automatically.\")\n }\n }\n (fieldVectors, provider)\n }\n\n def getFieldVector(valueVector: ValueVector, reason: String): FieldVector = {\n valueVector match {\n case v @ (_: BitVector | _: TinyIntVector | _: SmallIntVector | _: IntVector |\n _: BigIntVector | _: Float4Vector | _: Float8Vector | _: VarCharVector |\n _: DecimalVector | _: DateDayVector | _: TimeStampMicroTZVector | _: VarBinaryVector |\n _: FixedSizeBinaryVector | _: TimeStampMicroVector | _: StructVector | _: ListVector |\n _: MapVector | _: NullVector) =>\n v.asInstanceOf[FieldVector]\n case _ =>\n throw new SparkException(s\"Unsupported Arrow Vector for $reason: ${valueVector.getClass}\")\n }\n }\n}\n" }, { "path": "common/src/main/spark-3.4/org/apache/comet/shims/ShimBatchReader.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.shims\n\nimport org.apache.spark.paths.SparkPath\nimport org.apache.spark.sql.catalyst.InternalRow\nimport org.apache.spark.sql.execution.datasources.PartitionedFile\n\nobject ShimBatchReader {\n\n def newPartitionedFile(partitionValues: InternalRow, file: String): PartitionedFile =\n PartitionedFile(\n partitionValues,\n SparkPath.fromPathString(file),\n -1, // -1 means we read the entire file\n -1,\n Array.empty[String],\n 0,\n 0)\n}\n" }, { "path": "common/src/main/spark-3.4/org/apache/comet/shims/ShimFileFormat.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.shims\n\nimport org.apache.spark.sql.execution.datasources.{FileFormat, RowIndexUtil}\nimport org.apache.spark.sql.types.StructType\n\nobject ShimFileFormat {\n\n // A name for a temporary column that holds row indexes computed by the file format reader\n // until they can be placed in the _metadata struct.\n val ROW_INDEX_TEMPORARY_COLUMN_NAME: String = FileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME\n\n def findRowIndexColumnIndexInSchema(sparkSchema: StructType): Int =\n RowIndexUtil.findRowIndexColumnIndexInSchema(sparkSchema)\n}\n" }, { "path": "common/src/main/spark-3.4/org/apache/spark/sql/comet/shims/ShimTaskMetrics.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.spark.sql.comet.shims\n\nimport org.apache.spark.executor.TaskMetrics\nimport org.apache.spark.util.AccumulatorV2\n\nobject ShimTaskMetrics {\n\n def getTaskAccumulator(taskMetrics: TaskMetrics): Option[AccumulatorV2[_, _]] =\n taskMetrics.externalAccums.lastOption\n}\n" }, { "path": "common/src/main/spark-3.5/org/apache/comet/shims/ShimBatchReader.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.shims\n\nimport org.apache.spark.paths.SparkPath\nimport org.apache.spark.sql.catalyst.InternalRow\nimport org.apache.spark.sql.execution.datasources.PartitionedFile\n\nobject ShimBatchReader {\n\n def newPartitionedFile(partitionValues: InternalRow, file: String): PartitionedFile =\n PartitionedFile(\n partitionValues,\n SparkPath.fromPathString(file),\n -1, // -1 means we read the entire file\n -1,\n Array.empty[String],\n 0,\n 0,\n Map.empty)\n}\n" }, { "path": "common/src/main/spark-3.5/org/apache/comet/shims/ShimFileFormat.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.shims\n\nimport org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat\nimport org.apache.spark.sql.execution.datasources.parquet.ParquetRowIndexUtil\nimport org.apache.spark.sql.types.StructType\n\nobject ShimFileFormat {\n // A name for a temporary column that holds row indexes computed by the file format reader\n // until they can be placed in the _metadata struct.\n val ROW_INDEX_TEMPORARY_COLUMN_NAME = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME\n\n def findRowIndexColumnIndexInSchema(sparkSchema: StructType): Int =\n ParquetRowIndexUtil.findRowIndexColumnIndexInSchema(sparkSchema)\n}\n" }, { "path": "common/src/main/spark-3.5/org/apache/spark/sql/comet/shims/ShimTaskMetrics.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.spark.sql.comet.shims\n\nimport scala.collection.mutable.ArrayBuffer\n\nimport org.apache.spark.executor.TaskMetrics\nimport org.apache.spark.util.AccumulatorV2\n\nobject ShimTaskMetrics {\n\n def getTaskAccumulator(taskMetrics: TaskMetrics): Option[AccumulatorV2[_, _]] =\n taskMetrics.withExternalAccums(identity[ArrayBuffer[AccumulatorV2[_, _]]](_)).lastOption\n}\n" }, { "path": "common/src/main/spark-3.x/org/apache/comet/shims/CometTypeShim.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.shims\n\nimport scala.annotation.nowarn\n\nimport org.apache.spark.sql.types.DataType\n\ntrait CometTypeShim {\n @nowarn // Spark 4 feature; stubbed to false in Spark 3.x for compatibility.\n def isStringCollationType(dt: DataType): Boolean = false\n}\n" }, { "path": "common/src/main/spark-3.x/org/apache/comet/shims/ShimCometConf.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.shims\n\ntrait ShimCometConf {\n protected val COMET_SCHEMA_EVOLUTION_ENABLED_DEFAULT = false\n}\n" }, { "path": "common/src/main/spark-4.0/org/apache/comet/shims/CometTypeShim.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.shims\n\nimport org.apache.spark.sql.internal.types.StringTypeWithCollation\nimport org.apache.spark.sql.types.DataType\n\ntrait CometTypeShim {\n def isStringCollationType(dt: DataType): Boolean = dt.isInstanceOf[StringTypeWithCollation]\n}\n" }, { "path": "common/src/main/spark-4.0/org/apache/comet/shims/ShimBatchReader.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.shims\n\nimport org.apache.spark.paths.SparkPath\nimport org.apache.spark.sql.catalyst.InternalRow\nimport org.apache.spark.sql.execution.datasources.PartitionedFile\n\nobject ShimBatchReader {\n def newPartitionedFile(partitionValues: InternalRow, file: String): PartitionedFile =\n PartitionedFile(\n partitionValues,\n SparkPath.fromUrlString(file),\n -1, // -1 means we read the entire file\n -1,\n Array.empty[String],\n 0,\n 0)\n}\n" }, { "path": "common/src/main/spark-4.0/org/apache/comet/shims/ShimCometConf.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.shims\n\ntrait ShimCometConf {\n protected val COMET_SCHEMA_EVOLUTION_ENABLED_DEFAULT = true\n}\n" }, { "path": "common/src/main/spark-4.0/org/apache/comet/shims/ShimFileFormat.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.shims\n\nimport org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat\nimport org.apache.spark.sql.execution.datasources.parquet.ParquetRowIndexUtil\nimport org.apache.spark.sql.types.StructType\n\nobject ShimFileFormat {\n // A name for a temporary column that holds row indexes computed by the file format reader\n // until they can be placed in the _metadata struct.\n val ROW_INDEX_TEMPORARY_COLUMN_NAME = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME\n\n def findRowIndexColumnIndexInSchema(sparkSchema: StructType): Int =\n ParquetRowIndexUtil.findRowIndexColumnIndexInSchema(sparkSchema)\n}\n" }, { "path": "common/src/main/spark-4.0/org/apache/spark/sql/comet/shims/ShimTaskMetrics.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.spark.sql.comet.shims\n\nimport org.apache.spark.executor.TaskMetrics\nimport org.apache.spark.util.AccumulatorV2\n\nobject ShimTaskMetrics {\n\n def getTaskAccumulator(taskMetrics: TaskMetrics): Option[AccumulatorV2[_, _]] =\n taskMetrics._externalAccums.lastOption\n}\n" }, { "path": "common/src/test/java/org/apache/comet/parquet/TestColumnReader.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport org.junit.Test;\n\nimport org.apache.arrow.memory.BufferAllocator;\nimport org.apache.arrow.memory.RootAllocator;\nimport org.apache.arrow.vector.FixedSizeBinaryVector;\nimport org.apache.arrow.vector.IntVector;\nimport org.apache.arrow.vector.ValueVector;\nimport org.apache.arrow.vector.VarBinaryVector;\n\nimport org.apache.comet.vector.CometPlainVector;\nimport org.apache.comet.vector.CometVector;\n\nimport static org.junit.Assert.*;\n\npublic class TestColumnReader {\n @Test\n public void testIsFixedLength() {\n BufferAllocator allocator = new RootAllocator(Integer.MAX_VALUE);\n\n ValueVector vv = new IntVector(\"v1\", allocator);\n CometVector vector = new CometPlainVector(vv, false);\n assertTrue(vector.isFixedLength());\n\n vv = new FixedSizeBinaryVector(\"v2\", allocator, 12);\n vector = new CometPlainVector(vv, false);\n assertTrue(vector.isFixedLength());\n\n vv = new VarBinaryVector(\"v3\", allocator);\n vector = new CometPlainVector(vv, false);\n assertFalse(vector.isFixedLength());\n }\n}\n" }, { "path": "common/src/test/java/org/apache/comet/parquet/TestCometInputFile.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport org.junit.Assert;\nimport org.junit.Test;\n\npublic class TestCometInputFile {\n @Test\n public void testIsAtLeastHadoop33() {\n Assert.assertTrue(CometInputFile.isAtLeastHadoop33(\"3.3.0\"));\n Assert.assertTrue(CometInputFile.isAtLeastHadoop33(\"3.4.0-SNAPSHOT\"));\n Assert.assertTrue(CometInputFile.isAtLeastHadoop33(\"3.12.5\"));\n Assert.assertTrue(CometInputFile.isAtLeastHadoop33(\"3.20.6.4-xyz\"));\n\n Assert.assertFalse(CometInputFile.isAtLeastHadoop33(\"2.7.2\"));\n Assert.assertFalse(CometInputFile.isAtLeastHadoop33(\"2.7.3-SNAPSHOT\"));\n Assert.assertFalse(CometInputFile.isAtLeastHadoop33(\"2.7\"));\n Assert.assertFalse(CometInputFile.isAtLeastHadoop33(\"2\"));\n Assert.assertFalse(CometInputFile.isAtLeastHadoop33(\"3\"));\n Assert.assertFalse(CometInputFile.isAtLeastHadoop33(\"3.2\"));\n Assert.assertFalse(CometInputFile.isAtLeastHadoop33(\"3.0.2.5-abc\"));\n Assert.assertFalse(CometInputFile.isAtLeastHadoop33(\"3.1.2-test\"));\n Assert.assertFalse(CometInputFile.isAtLeastHadoop33(\"3-SNAPSHOT\"));\n Assert.assertFalse(CometInputFile.isAtLeastHadoop33(\"3.2-SNAPSHOT\"));\n }\n}\n" }, { "path": "common/src/test/java/org/apache/comet/parquet/TestFileReader.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.io.ByteArrayOutputStream;\nimport java.io.File;\nimport java.io.IOException;\nimport java.lang.reflect.Method;\nimport java.nio.ByteBuffer;\nimport java.nio.charset.StandardCharsets;\nimport java.util.*;\n\nimport org.junit.Assert;\nimport org.junit.Rule;\nimport org.junit.Test;\nimport org.junit.rules.TemporaryFolder;\n\nimport org.apache.hadoop.conf.Configuration;\nimport org.apache.hadoop.fs.Path;\nimport org.apache.parquet.HadoopReadOptions;\nimport org.apache.parquet.ParquetReadOptions;\nimport org.apache.parquet.bytes.BytesInput;\nimport org.apache.parquet.bytes.BytesUtils;\nimport org.apache.parquet.column.ColumnDescriptor;\nimport org.apache.parquet.column.Encoding;\nimport org.apache.parquet.column.page.DataPage;\nimport org.apache.parquet.column.page.DataPageV1;\nimport org.apache.parquet.column.page.DataPageV2;\nimport org.apache.parquet.column.page.DictionaryPage;\nimport org.apache.parquet.column.page.PageReadStore;\nimport org.apache.parquet.column.page.PageReader;\nimport org.apache.parquet.column.statistics.BinaryStatistics;\nimport org.apache.parquet.column.statistics.Statistics;\nimport org.apache.parquet.column.values.bloomfilter.BlockSplitBloomFilter;\nimport org.apache.parquet.column.values.bloomfilter.BloomFilter;\nimport org.apache.parquet.filter2.predicate.FilterApi;\nimport org.apache.parquet.filter2.predicate.FilterPredicate;\nimport org.apache.parquet.filter2.predicate.Operators;\nimport org.apache.parquet.hadoop.ParquetFileWriter;\nimport org.apache.parquet.hadoop.ParquetInputFormat;\nimport org.apache.parquet.hadoop.metadata.*;\nimport org.apache.parquet.hadoop.util.HadoopInputFile;\nimport org.apache.parquet.internal.column.columnindex.BoundaryOrder;\nimport org.apache.parquet.internal.column.columnindex.ColumnIndex;\nimport org.apache.parquet.internal.column.columnindex.OffsetIndex;\nimport org.apache.parquet.io.InputFile;\nimport org.apache.parquet.io.api.Binary;\nimport org.apache.parquet.schema.MessageType;\nimport org.apache.parquet.schema.MessageTypeParser;\nimport org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;\nimport org.apache.parquet.schema.Types;\n\nimport org.apache.comet.CometConf;\n\nimport static org.apache.parquet.column.Encoding.*;\nimport static org.apache.parquet.format.converter.ParquetMetadataConverter.MAX_STATS_SIZE;\nimport static org.junit.Assert.*;\nimport static org.junit.Assert.assertEquals;\n\nimport static org.apache.comet.parquet.TypeUtil.isSpark40Plus;\n\n@SuppressWarnings(\"deprecation\")\npublic class TestFileReader {\n private static final MessageType SCHEMA =\n MessageTypeParser.parseMessageType(\n \"\"\n + \"message m {\"\n + \" required group a {\"\n + \" required binary b;\"\n + \" }\"\n + \" required group c {\"\n + \" required int64 d;\"\n + \" }\"\n + \"}\");\n\n private static final MessageType SCHEMA2 =\n MessageTypeParser.parseMessageType(\n \"\"\n + \"message root { \"\n + \"required int32 id;\"\n + \"required binary name(UTF8); \"\n + \"required int32 num; \"\n + \"required binary comment(UTF8);\"\n + \"}\");\n\n private static final MessageType PROJECTED_SCHEMA2 =\n MessageTypeParser.parseMessageType(\n \"\"\n + \"message root { \"\n + \"required int32 id;\"\n + \"required binary name(UTF8); \"\n + \"required binary comment(UTF8);\"\n + \"}\");\n\n private static final String[] PATH1 = {\"a\", \"b\"};\n private static final ColumnDescriptor C1 = SCHEMA.getColumnDescription(PATH1);\n private static final String[] PATH2 = {\"c\", \"d\"};\n private static final ColumnDescriptor C2 = SCHEMA.getColumnDescription(PATH2);\n\n private static final byte[] BYTES1 = {0, 1, 2, 3};\n private static final byte[] BYTES2 = {1, 2, 3, 4};\n private static final byte[] BYTES3 = {2, 3, 4, 5};\n private static final byte[] BYTES4 = {3, 4, 5, 6};\n private static final CompressionCodecName CODEC = CompressionCodecName.UNCOMPRESSED;\n\n private static final org.apache.parquet.column.statistics.Statistics EMPTY_STATS =\n org.apache.parquet.column.statistics.Statistics.getBuilderForReading(\n Types.required(PrimitiveTypeName.BINARY).named(\"test_binary\"))\n .build();\n\n @Rule public final TemporaryFolder temp = new TemporaryFolder();\n\n @Test\n public void testEnableReadParallel() {\n Configuration configuration = new Configuration();\n ReadOptions options = ReadOptions.builder(configuration).build();\n\n assertFalse(FileReader.shouldReadParallel(options, \"hdfs\"));\n assertFalse(FileReader.shouldReadParallel(options, \"file\"));\n assertFalse(FileReader.shouldReadParallel(options, null));\n assertTrue(FileReader.shouldReadParallel(options, \"s3a\"));\n\n options = ReadOptions.builder(configuration).enableParallelIO(false).build();\n assertFalse(FileReader.shouldReadParallel(options, \"s3a\"));\n }\n\n @Test\n public void testReadWrite() throws Exception {\n File testFile = temp.newFile();\n testFile.delete();\n\n Path path = new Path(testFile.toURI());\n Configuration configuration = new Configuration();\n\n // Start a Parquet file with 2 row groups, each with 2 column chunks\n ParquetFileWriter w = new ParquetFileWriter(configuration, SCHEMA, path);\n w.start();\n w.startBlock(3);\n w.startColumn(C1, 5, CODEC);\n long c1Starts = w.getPos();\n long c1p1Starts = w.getPos();\n w.writeDataPage(2, 4, BytesInput.from(BYTES1), EMPTY_STATS, 2, RLE, RLE, PLAIN);\n w.writeDataPage(3, 4, BytesInput.from(BYTES1), EMPTY_STATS, 3, RLE, RLE, PLAIN);\n w.endColumn();\n long c1Ends = w.getPos();\n w.startColumn(C2, 6, CODEC);\n long c2Starts = w.getPos();\n w.writeDictionaryPage(new DictionaryPage(BytesInput.from(BYTES2), 4, RLE_DICTIONARY));\n long c2p1Starts = w.getPos();\n w.writeDataPage(2, 4, BytesInput.from(BYTES2), EMPTY_STATS, 2, RLE, RLE, PLAIN);\n w.writeDataPage(3, 4, BytesInput.from(BYTES2), EMPTY_STATS, 3, RLE, RLE, PLAIN);\n w.writeDataPage(1, 4, BytesInput.from(BYTES2), EMPTY_STATS, 1, RLE, RLE, PLAIN);\n w.endColumn();\n long c2Ends = w.getPos();\n w.endBlock();\n w.startBlock(4);\n w.startColumn(C1, 7, CODEC);\n w.writeDataPage(7, 4, BytesInput.from(BYTES3), EMPTY_STATS, 7, RLE, RLE, PLAIN);\n w.endColumn();\n w.startColumn(C2, 8, CODEC);\n w.writeDataPage(8, 4, BytesInput.from(BYTES4), EMPTY_STATS, 8, RLE, RLE, PLAIN);\n w.endColumn();\n w.endBlock();\n w.end(new HashMap<>());\n\n InputFile file = HadoopInputFile.fromPath(path, configuration);\n ParquetReadOptions options = ParquetReadOptions.builder().build();\n ReadOptions cometOptions = ReadOptions.builder(configuration).build();\n\n try (FileReader reader = new FileReader(file, options, cometOptions)) {\n ParquetMetadata readFooter = reader.getFooter();\n assertEquals(\"footer: \" + readFooter, 2, readFooter.getBlocks().size());\n BlockMetaData rowGroup = readFooter.getBlocks().get(0);\n assertEquals(c1Ends - c1Starts, rowGroup.getColumns().get(0).getTotalSize());\n assertEquals(c2Ends - c2Starts, rowGroup.getColumns().get(1).getTotalSize());\n assertEquals(c2Ends - c1Starts, rowGroup.getTotalByteSize());\n\n assertEquals(c1Starts, rowGroup.getColumns().get(0).getStartingPos());\n assertEquals(0, rowGroup.getColumns().get(0).getDictionaryPageOffset());\n assertEquals(c1p1Starts, rowGroup.getColumns().get(0).getFirstDataPageOffset());\n assertEquals(c2Starts, rowGroup.getColumns().get(1).getStartingPos());\n assertEquals(c2Starts, rowGroup.getColumns().get(1).getDictionaryPageOffset());\n assertEquals(c2p1Starts, rowGroup.getColumns().get(1).getFirstDataPageOffset());\n\n HashSet expectedEncoding = new HashSet<>();\n expectedEncoding.add(PLAIN);\n expectedEncoding.add(RLE);\n assertEquals(expectedEncoding, rowGroup.getColumns().get(0).getEncodings());\n }\n\n // read first block of col #1\n try (FileReader r = new FileReader(file, options, cometOptions)) {\n r.setRequestedSchema(Arrays.asList(SCHEMA.getColumnDescription(PATH1)));\n PageReadStore pages = r.readNextRowGroup();\n assertEquals(3, pages.getRowCount());\n validateContains(pages, PATH1, 2, BytesInput.from(BYTES1));\n validateContains(pages, PATH1, 3, BytesInput.from(BYTES1));\n assertTrue(r.skipNextRowGroup());\n assertNull(r.readNextRowGroup());\n }\n\n // read all blocks of col #1 and #2\n try (FileReader r = new FileReader(file, options, cometOptions)) {\n r.setRequestedSchema(\n Arrays.asList(SCHEMA.getColumnDescription(PATH1), SCHEMA.getColumnDescription(PATH2)));\n PageReadStore pages = r.readNextRowGroup();\n assertEquals(3, pages.getRowCount());\n validateContains(pages, PATH1, 2, BytesInput.from(BYTES1));\n validateContains(pages, PATH1, 3, BytesInput.from(BYTES1));\n validateContains(pages, PATH2, 2, BytesInput.from(BYTES2));\n validateContains(pages, PATH2, 3, BytesInput.from(BYTES2));\n validateContains(pages, PATH2, 1, BytesInput.from(BYTES2));\n\n pages = r.readNextRowGroup();\n assertEquals(4, pages.getRowCount());\n\n validateContains(pages, PATH1, 7, BytesInput.from(BYTES3));\n validateContains(pages, PATH2, 8, BytesInput.from(BYTES4));\n\n assertNull(r.readNextRowGroup());\n }\n }\n\n @Test\n public void testBloomFilterReadWrite() throws Exception {\n MessageType schema =\n MessageTypeParser.parseMessageType(\"message test { required binary foo; }\");\n File testFile = temp.newFile();\n testFile.delete();\n Path path = new Path(testFile.toURI());\n Configuration configuration = new Configuration();\n configuration.set(\"parquet.bloom.filter.column.names\", \"foo\");\n String[] colPath = {\"foo\"};\n\n ColumnDescriptor col = schema.getColumnDescription(colPath);\n BinaryStatistics stats1 = new BinaryStatistics();\n ParquetFileWriter w = new ParquetFileWriter(configuration, schema, path);\n w.start();\n w.startBlock(3);\n w.startColumn(col, 5, CODEC);\n w.writeDataPage(2, 4, BytesInput.from(BYTES1), stats1, 2, RLE, RLE, PLAIN);\n w.writeDataPage(3, 4, BytesInput.from(BYTES1), stats1, 2, RLE, RLE, PLAIN);\n w.endColumn();\n BloomFilter blockSplitBloomFilter = new BlockSplitBloomFilter(0);\n blockSplitBloomFilter.insertHash(blockSplitBloomFilter.hash(Binary.fromString(\"hello\")));\n blockSplitBloomFilter.insertHash(blockSplitBloomFilter.hash(Binary.fromString(\"world\")));\n addBloomFilter(w, \"foo\", blockSplitBloomFilter);\n w.endBlock();\n w.end(new HashMap<>());\n\n InputFile file = HadoopInputFile.fromPath(path, configuration);\n ParquetReadOptions options = ParquetReadOptions.builder().build();\n ReadOptions cometOptions = ReadOptions.builder(configuration).build();\n\n try (FileReader r = new FileReader(file, options, cometOptions)) {\n ParquetMetadata footer = r.getFooter();\n r.setRequestedSchema(Arrays.asList(schema.getColumnDescription(colPath)));\n BloomFilterReader bloomFilterReader =\n new BloomFilterReader(\n footer.getBlocks().get(0),\n r.getFileMetaData().getFileDecryptor(),\n r.getInputStream());\n BloomFilter bloomFilter =\n bloomFilterReader.readBloomFilter(footer.getBlocks().get(0).getColumns().get(0));\n assertTrue(bloomFilter.findHash(blockSplitBloomFilter.hash(Binary.fromString(\"hello\"))));\n assertTrue(bloomFilter.findHash(blockSplitBloomFilter.hash(Binary.fromString(\"world\"))));\n }\n }\n\n @Test\n public void testReadWriteDataPageV2() throws Exception {\n File testFile = temp.newFile();\n testFile.delete();\n\n Path path = new Path(testFile.toURI());\n Configuration configuration = new Configuration();\n\n ParquetFileWriter w = new ParquetFileWriter(configuration, SCHEMA, path);\n w.start();\n w.startBlock(14);\n\n BytesInput repLevels = BytesInput.fromInt(2);\n BytesInput defLevels = BytesInput.fromInt(1);\n BytesInput data = BytesInput.fromInt(3);\n BytesInput data2 = BytesInput.fromInt(10);\n\n org.apache.parquet.column.statistics.Statistics statsC1P1 = createStatistics(\"s\", \"z\", C1);\n org.apache.parquet.column.statistics.Statistics statsC1P2 = createStatistics(\"b\", \"d\", C1);\n\n w.startColumn(C1, 6, CODEC);\n long c1Starts = w.getPos();\n w.writeDataPageV2(4, 1, 3, repLevels, defLevels, PLAIN, data, 4, statsC1P1);\n w.writeDataPageV2(3, 0, 3, repLevels, defLevels, PLAIN, data, 4, statsC1P2);\n w.endColumn();\n long c1Ends = w.getPos();\n\n w.startColumn(C2, 5, CODEC);\n long c2Starts = w.getPos();\n w.writeDataPageV2(5, 2, 3, repLevels, defLevels, PLAIN, data2, 4, EMPTY_STATS);\n w.writeDataPageV2(2, 0, 2, repLevels, defLevels, PLAIN, data2, 4, EMPTY_STATS);\n w.endColumn();\n long c2Ends = w.getPos();\n\n w.endBlock();\n w.end(new HashMap<>());\n\n InputFile file = HadoopInputFile.fromPath(path, configuration);\n ParquetReadOptions options = ParquetReadOptions.builder().build();\n ReadOptions cometOptions = ReadOptions.builder(configuration).build();\n\n try (FileReader reader = new FileReader(file, options, cometOptions)) {\n ParquetMetadata footer = reader.getFooter();\n assertEquals(\"footer: \" + footer, 1, footer.getBlocks().size());\n assertEquals(c1Ends - c1Starts, footer.getBlocks().get(0).getColumns().get(0).getTotalSize());\n assertEquals(c2Ends - c2Starts, footer.getBlocks().get(0).getColumns().get(1).getTotalSize());\n assertEquals(c2Ends - c1Starts, footer.getBlocks().get(0).getTotalByteSize());\n\n // check for stats\n org.apache.parquet.column.statistics.Statistics expectedStats =\n createStatistics(\"b\", \"z\", C1);\n assertStatsValuesEqual(\n expectedStats, footer.getBlocks().get(0).getColumns().get(0).getStatistics());\n\n HashSet expectedEncoding = new HashSet<>();\n expectedEncoding.add(PLAIN);\n assertEquals(expectedEncoding, footer.getBlocks().get(0).getColumns().get(0).getEncodings());\n }\n\n try (FileReader r = new FileReader(file, options, cometOptions)) {\n r.setRequestedSchema(\n Arrays.asList(SCHEMA.getColumnDescription(PATH1), SCHEMA.getColumnDescription(PATH2)));\n PageReadStore pages = r.readNextRowGroup();\n assertEquals(14, pages.getRowCount());\n validateV2Page(\n pages,\n PATH1,\n 3,\n 4,\n 1,\n repLevels.toByteArray(),\n defLevels.toByteArray(),\n data.toByteArray(),\n 12);\n validateV2Page(\n pages,\n PATH1,\n 3,\n 3,\n 0,\n repLevels.toByteArray(),\n defLevels.toByteArray(),\n data.toByteArray(),\n 12);\n validateV2Page(\n pages,\n PATH2,\n 3,\n 5,\n 2,\n repLevels.toByteArray(),\n defLevels.toByteArray(),\n data2.toByteArray(),\n 12);\n validateV2Page(\n pages,\n PATH2,\n 2,\n 2,\n 0,\n repLevels.toByteArray(),\n defLevels.toByteArray(),\n data2.toByteArray(),\n 12);\n assertNull(r.readNextRowGroup());\n }\n }\n\n @Test\n public void testColumnIndexFilter() throws Exception {\n File testFile = temp.newFile();\n testFile.delete();\n\n Path path = new Path(testFile.toURI());\n Configuration configuration = new Configuration();\n\n ParquetFileWriter w = new ParquetFileWriter(configuration, SCHEMA, path);\n\n w.start();\n w.startBlock(4);\n w.startColumn(C1, 7, CODEC);\n w.writeDataPage(2, 4, BytesInput.from(BYTES1), EMPTY_STATS, 2, RLE, RLE, PLAIN);\n w.writeDataPage(2, 4, BytesInput.from(BYTES2), EMPTY_STATS, 2, RLE, RLE, PLAIN);\n w.endColumn();\n w.startColumn(C2, 8, CODEC);\n // the first page contains one matching record\n w.writeDataPage(1, 4, BytesInput.from(BYTES3), statsC2(2L), 1, RLE, RLE, PLAIN);\n // all the records of the second page are larger than 2, so should be filtered out\n w.writeDataPage(3, 4, BytesInput.from(BYTES4), statsC2(3L, 4L, 5L), 3, RLE, RLE, PLAIN);\n w.endColumn();\n w.endBlock();\n\n w.startBlock(4);\n w.startColumn(C1, 7, CODEC);\n w.writeDataPage(2, 4, BytesInput.from(BYTES1), EMPTY_STATS, 2, RLE, RLE, PLAIN);\n w.writeDataPage(2, 4, BytesInput.from(BYTES2), EMPTY_STATS, 2, RLE, RLE, PLAIN);\n w.endColumn();\n w.startColumn(C2, 8, CODEC);\n // the first page should be filtered out\n w.writeDataPage(1, 4, BytesInput.from(BYTES3), statsC2(4L), 1, RLE, RLE, PLAIN);\n // the second page will be read since it contains matching record\n w.writeDataPage(3, 4, BytesInput.from(BYTES4), statsC2(0L, 1L, 3L), 3, RLE, RLE, PLAIN);\n w.endColumn();\n w.endBlock();\n\n w.end(new HashMap<>());\n\n // set a simple equality filter in the ParquetInputFormat\n Operators.LongColumn c2 = FilterApi.longColumn(\"c.d\");\n FilterPredicate p = FilterApi.eq(c2, 2L);\n ParquetInputFormat.setFilterPredicate(configuration, p);\n InputFile file = HadoopInputFile.fromPath(path, configuration);\n ParquetReadOptions options = HadoopReadOptions.builder(configuration).build();\n ReadOptions cometOptions = ReadOptions.builder(configuration).build();\n\n try (FileReader r = new FileReader(file, options, cometOptions)) {\n assertEquals(4, r.getFilteredRecordCount());\n PageReadStore readStore = r.readNextFilteredRowGroup();\n\n PageReader c1Reader = readStore.getPageReader(C1);\n List c1Pages = new ArrayList<>();\n DataPage page;\n while ((page = c1Reader.readPage()) != null) {\n c1Pages.add(page);\n }\n // second page of c1 should be filtered out\n assertEquals(1, c1Pages.size());\n validatePage(c1Pages.get(0), 2, BytesInput.from(BYTES1));\n\n PageReader c2Reader = readStore.getPageReader(C2);\n List c2Pages = new ArrayList<>();\n while ((page = c2Reader.readPage()) != null) {\n c2Pages.add(page);\n }\n assertEquals(1, c2Pages.size());\n validatePage(c2Pages.get(0), 1, BytesInput.from(BYTES3));\n\n // test the second row group\n readStore = r.readNextFilteredRowGroup();\n assertNotNull(readStore);\n\n c1Reader = readStore.getPageReader(C1);\n c1Pages.clear();\n while ((page = c1Reader.readPage()) != null) {\n c1Pages.add(page);\n }\n // all pages of c1 should be retained\n assertEquals(2, c1Pages.size());\n validatePage(c1Pages.get(0), 2, BytesInput.from(BYTES1));\n validatePage(c1Pages.get(1), 2, BytesInput.from(BYTES2));\n\n c2Reader = readStore.getPageReader(C2);\n c2Pages.clear();\n while ((page = c2Reader.readPage()) != null) {\n c2Pages.add(page);\n }\n assertEquals(1, c2Pages.size());\n validatePage(c2Pages.get(0), 3, BytesInput.from(BYTES4));\n }\n }\n\n @Test\n public void testColumnIndexReadWrite() throws Exception {\n File testFile = temp.newFile();\n testFile.delete();\n\n Path path = new Path(testFile.toURI());\n Configuration configuration = new Configuration();\n\n ParquetFileWriter w = new ParquetFileWriter(configuration, SCHEMA, path);\n w.start();\n w.startBlock(4);\n w.startColumn(C1, 7, CODEC);\n w.writeDataPage(7, 4, BytesInput.from(BYTES3), EMPTY_STATS, RLE, RLE, PLAIN);\n w.endColumn();\n w.startColumn(C2, 8, CODEC);\n w.writeDataPage(8, 4, BytesInput.from(BYTES4), EMPTY_STATS, RLE, RLE, PLAIN);\n w.endColumn();\n w.endBlock();\n w.startBlock(4);\n w.startColumn(C1, 5, CODEC);\n long c1p1Starts = w.getPos();\n w.writeDataPage(\n 2, 4, BytesInput.from(BYTES1), statsC1(null, Binary.fromString(\"aaa\")), 1, RLE, RLE, PLAIN);\n long c1p2Starts = w.getPos();\n w.writeDataPage(\n 3,\n 4,\n BytesInput.from(BYTES1),\n statsC1(Binary.fromString(\"bbb\"), Binary.fromString(\"ccc\")),\n 3,\n RLE,\n RLE,\n PLAIN);\n w.endColumn();\n long c1Ends = w.getPos();\n w.startColumn(C2, 6, CODEC);\n long c2p1Starts = w.getPos();\n w.writeDataPage(2, 4, BytesInput.from(BYTES2), statsC2(117L, 100L), 1, RLE, RLE, PLAIN);\n long c2p2Starts = w.getPos();\n w.writeDataPage(3, 4, BytesInput.from(BYTES2), statsC2(null, null, null), 2, RLE, RLE, PLAIN);\n long c2p3Starts = w.getPos();\n w.writeDataPage(1, 4, BytesInput.from(BYTES2), statsC2(0L), 1, RLE, RLE, PLAIN);\n w.endColumn();\n long c2Ends = w.getPos();\n w.endBlock();\n w.startBlock(4);\n w.startColumn(C1, 7, CODEC);\n w.writeDataPage(\n 7,\n 4,\n BytesInput.from(BYTES3),\n // Creating huge stats so the column index will reach the limit and won't be written\n statsC1(\n Binary.fromConstantByteArray(new byte[(int) MAX_STATS_SIZE]),\n Binary.fromConstantByteArray(new byte[1])),\n 4,\n RLE,\n RLE,\n PLAIN);\n w.endColumn();\n w.startColumn(C2, 8, CODEC);\n w.writeDataPage(8, 4, BytesInput.from(BYTES4), EMPTY_STATS, RLE, RLE, PLAIN);\n w.endColumn();\n w.endBlock();\n w.end(new HashMap<>());\n\n InputFile file = HadoopInputFile.fromPath(path, configuration);\n ParquetReadOptions options = ParquetReadOptions.builder().build();\n ReadOptions cometOptions = ReadOptions.builder(configuration).build();\n\n try (FileReader reader = new FileReader(file, options, cometOptions)) {\n ParquetMetadata footer = reader.getFooter();\n assertEquals(3, footer.getBlocks().size());\n BlockMetaData blockMeta = footer.getBlocks().get(1);\n assertEquals(2, blockMeta.getColumns().size());\n\n ColumnIndexReader indexReader = reader.getColumnIndexReader(1);\n ColumnIndex columnIndex = indexReader.readColumnIndex(blockMeta.getColumns().get(0));\n assertEquals(BoundaryOrder.ASCENDING, columnIndex.getBoundaryOrder());\n assertEquals(Arrays.asList(1L, 0L), columnIndex.getNullCounts());\n assertEquals(Arrays.asList(false, false), columnIndex.getNullPages());\n List minValues = columnIndex.getMinValues();\n assertEquals(2, minValues.size());\n List maxValues = columnIndex.getMaxValues();\n assertEquals(2, maxValues.size());\n assertEquals(\"aaa\", new String(minValues.get(0).array(), StandardCharsets.UTF_8));\n assertEquals(\"aaa\", new String(maxValues.get(0).array(), StandardCharsets.UTF_8));\n assertEquals(\"bbb\", new String(minValues.get(1).array(), StandardCharsets.UTF_8));\n assertEquals(\"ccc\", new String(maxValues.get(1).array(), StandardCharsets.UTF_8));\n\n columnIndex = indexReader.readColumnIndex(blockMeta.getColumns().get(1));\n assertEquals(BoundaryOrder.DESCENDING, columnIndex.getBoundaryOrder());\n assertEquals(Arrays.asList(0L, 3L, 0L), columnIndex.getNullCounts());\n assertEquals(Arrays.asList(false, true, false), columnIndex.getNullPages());\n minValues = columnIndex.getMinValues();\n assertEquals(3, minValues.size());\n maxValues = columnIndex.getMaxValues();\n assertEquals(3, maxValues.size());\n assertEquals(100, BytesUtils.bytesToLong(minValues.get(0).array()));\n assertEquals(117, BytesUtils.bytesToLong(maxValues.get(0).array()));\n assertEquals(0, minValues.get(1).array().length);\n assertEquals(0, maxValues.get(1).array().length);\n assertEquals(0, BytesUtils.bytesToLong(minValues.get(2).array()));\n assertEquals(0, BytesUtils.bytesToLong(maxValues.get(2).array()));\n\n OffsetIndex offsetIndex = indexReader.readOffsetIndex(blockMeta.getColumns().get(0));\n assertEquals(2, offsetIndex.getPageCount());\n assertEquals(c1p1Starts, offsetIndex.getOffset(0));\n assertEquals(c1p2Starts, offsetIndex.getOffset(1));\n assertEquals(c1p2Starts - c1p1Starts, offsetIndex.getCompressedPageSize(0));\n assertEquals(c1Ends - c1p2Starts, offsetIndex.getCompressedPageSize(1));\n assertEquals(0, offsetIndex.getFirstRowIndex(0));\n assertEquals(1, offsetIndex.getFirstRowIndex(1));\n\n offsetIndex = indexReader.readOffsetIndex(blockMeta.getColumns().get(1));\n assertEquals(3, offsetIndex.getPageCount());\n assertEquals(c2p1Starts, offsetIndex.getOffset(0));\n assertEquals(c2p2Starts, offsetIndex.getOffset(1));\n assertEquals(c2p3Starts, offsetIndex.getOffset(2));\n assertEquals(c2p2Starts - c2p1Starts, offsetIndex.getCompressedPageSize(0));\n assertEquals(c2p3Starts - c2p2Starts, offsetIndex.getCompressedPageSize(1));\n assertEquals(c2Ends - c2p3Starts, offsetIndex.getCompressedPageSize(2));\n assertEquals(0, offsetIndex.getFirstRowIndex(0));\n assertEquals(1, offsetIndex.getFirstRowIndex(1));\n assertEquals(3, offsetIndex.getFirstRowIndex(2));\n\n if (!isSpark40Plus()) { // TODO: https://github.com/apache/datafusion-comet/issues/1948\n assertNull(indexReader.readColumnIndex(footer.getBlocks().get(2).getColumns().get(0)));\n }\n }\n }\n\n // Test reader with merging of scan ranges enabled\n @Test\n public void testWriteReadMergeScanRange() throws Throwable {\n Configuration conf = new Configuration();\n conf.set(CometConf.COMET_IO_MERGE_RANGES().key(), Boolean.toString(true));\n // Set the merge range delta so small that ranges do not get merged\n conf.set(CometConf.COMET_IO_MERGE_RANGES_DELTA().key(), Integer.toString(1024));\n testReadWrite(conf, 2, 1024);\n // Set the merge range delta so large that all ranges get merged\n conf.set(CometConf.COMET_IO_MERGE_RANGES_DELTA().key(), Integer.toString(1024 * 1024));\n testReadWrite(conf, 2, 1024);\n }\n\n // `addBloomFilter` is package-private in Parquet, so this uses reflection to access it\n private void addBloomFilter(ParquetFileWriter w, String s, BloomFilter filter) throws Exception {\n Method method =\n ParquetFileWriter.class.getDeclaredMethod(\n \"addBloomFilter\", String.class, BloomFilter.class);\n method.setAccessible(true);\n method.invoke(w, s, filter);\n }\n\n private void validateContains(PageReadStore pages, String[] path, int values, BytesInput bytes)\n throws IOException {\n PageReader pageReader = pages.getPageReader(SCHEMA.getColumnDescription(path));\n DataPage page = pageReader.readPage();\n validatePage(page, values, bytes);\n }\n\n private void validatePage(DataPage page, int values, BytesInput bytes) throws IOException {\n assertEquals(values, page.getValueCount());\n assertArrayEquals(bytes.toByteArray(), ((DataPageV1) page).getBytes().toByteArray());\n }\n\n private void validateV2Page(\n PageReadStore pages,\n String[] path,\n int values,\n int rows,\n int nullCount,\n byte[] repetition,\n byte[] definition,\n byte[] data,\n int uncompressedSize)\n throws IOException {\n PageReader pageReader = pages.getPageReader(SCHEMA.getColumnDescription(path));\n DataPageV2 page = (DataPageV2) pageReader.readPage();\n assertEquals(values, page.getValueCount());\n assertEquals(rows, page.getRowCount());\n assertEquals(nullCount, page.getNullCount());\n assertEquals(uncompressedSize, page.getUncompressedSize());\n assertArrayEquals(repetition, page.getRepetitionLevels().toByteArray());\n assertArrayEquals(definition, page.getDefinitionLevels().toByteArray());\n assertArrayEquals(data, page.getData().toByteArray());\n }\n\n private Statistics createStatistics(String min, String max, ColumnDescriptor col) {\n return Statistics.getBuilderForReading(col.getPrimitiveType())\n .withMin(Binary.fromString(min).getBytes())\n .withMax(Binary.fromString(max).getBytes())\n .withNumNulls(0)\n .build();\n }\n\n public static void assertStatsValuesEqual(Statistics expected, Statistics actual) {\n if (expected == actual) {\n return;\n }\n if (expected == null || actual == null) {\n assertEquals(expected, actual);\n }\n Assert.assertArrayEquals(expected.getMaxBytes(), actual.getMaxBytes());\n Assert.assertArrayEquals(expected.getMinBytes(), actual.getMinBytes());\n Assert.assertEquals(expected.getNumNulls(), actual.getNumNulls());\n }\n\n private Statistics statsC1(Binary... values) {\n Statistics stats = Statistics.createStats(C1.getPrimitiveType());\n for (Binary value : values) {\n if (value == null) {\n stats.incrementNumNulls();\n } else {\n stats.updateStats(value);\n }\n }\n return stats;\n }\n\n /**\n * Generates arbitrary data for simple schemas, writes the data to a file and also returns the\n * data.\n *\n * @return array of data pages for each column\n */\n private HashMap generateAndWriteData(\n Configuration configuration,\n Path path,\n MessageType schema,\n int numPages,\n int numRecordsPerPage)\n throws IOException {\n\n HashMap dataPages = new HashMap<>();\n\n Generator generator = new Generator();\n ParquetFileWriter writer = new ParquetFileWriter(configuration, schema, path);\n writer.start();\n writer.startBlock((long) numPages * numRecordsPerPage);\n for (ColumnDescriptor colDesc : schema.getColumns()) {\n writer.startColumn(colDesc, (long) numPages * numRecordsPerPage, CODEC);\n String type = colDesc.getPrimitiveType().getName();\n byte[][] allPages = new byte[numPages][];\n byte[] data;\n for (int i = 0; i < numPages; i++) {\n data = generator.generateValues(numRecordsPerPage, type);\n writer.writeDataPage(\n numRecordsPerPage,\n data.length,\n BytesInput.from(data),\n EMPTY_STATS,\n numRecordsPerPage,\n RLE,\n RLE,\n PLAIN);\n allPages[i] = data;\n }\n dataPages.put(String.join(\".\", colDesc.getPath()), allPages);\n writer.endColumn();\n }\n writer.endBlock();\n writer.end(new HashMap<>());\n return dataPages;\n }\n\n private void readAndValidatePageData(\n InputFile inputFile,\n ParquetReadOptions options,\n ReadOptions cometOptions,\n MessageType schema,\n HashMap expected,\n int expectedValuesPerPage)\n throws IOException {\n try (FileReader fileReader = new FileReader(inputFile, options, cometOptions)) {\n fileReader.setRequestedSchema(schema.getColumns());\n PageReadStore pages = fileReader.readNextRowGroup();\n for (ColumnDescriptor colDesc : schema.getColumns()) {\n byte[][] allExpectedPages = expected.get(String.join(\".\", colDesc.getPath()));\n PageReader pageReader = pages.getPageReader(colDesc);\n for (byte[] expectedPage : allExpectedPages) {\n DataPage page = pageReader.readPage();\n validatePage(page, expectedValuesPerPage, BytesInput.from(expectedPage));\n }\n }\n }\n }\n\n public void testReadWrite(Configuration configuration, int numPages, int numRecordsPerPage)\n throws Exception {\n File testFile = temp.newFile();\n testFile.delete();\n\n Path path = new Path(testFile.toURI());\n HashMap dataPages =\n generateAndWriteData(configuration, path, SCHEMA2, numPages, numRecordsPerPage);\n InputFile file = HadoopInputFile.fromPath(path, configuration);\n ParquetReadOptions options = ParquetReadOptions.builder().build();\n ReadOptions cometOptions = ReadOptions.builder(configuration).build();\n\n readAndValidatePageData(\n file, options, cometOptions, PROJECTED_SCHEMA2, dataPages, numRecordsPerPage);\n }\n\n static class Generator {\n\n static Random random = new Random(1729);\n private static final String ALPHABET = \"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz -\";\n private static final int STR_MIN_SIZE = 5;\n private static final int STR_MAX_SIZE = 30;\n\n private byte[] getString(int minSize, int maxSize) {\n int size = random.nextInt(maxSize - minSize) + minSize;\n byte[] str = new byte[size];\n for (int i = 0; i < size; ++i) {\n str[i] = (byte) ALPHABET.charAt(random.nextInt(ALPHABET.length()));\n }\n return str;\n }\n\n private byte[] generateValues(int numValues, String type) throws IOException {\n\n if (type.equals(\"int32\")) {\n byte[] data = new byte[4 * numValues];\n random.nextBytes(data);\n return data;\n } else {\n ByteArrayOutputStream outputStream = new ByteArrayOutputStream();\n for (int i = 0; i < numValues; i++) {\n outputStream.write(getString(STR_MIN_SIZE, STR_MAX_SIZE));\n }\n return outputStream.toByteArray();\n }\n }\n }\n\n private Statistics statsC2(Long... values) {\n Statistics stats = Statistics.createStats(C2.getPrimitiveType());\n for (Long value : values) {\n if (value == null) {\n stats.incrementNumNulls();\n } else {\n stats.updateStats(value);\n }\n }\n return stats;\n }\n}\n" }, { "path": "common/src/test/java/org/apache/comet/parquet/TestUtils.java", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.parquet;\n\nimport java.util.Collections;\nimport java.util.HashMap;\nimport java.util.Map;\n\nimport org.junit.Test;\n\nimport org.apache.parquet.column.ColumnDescriptor;\nimport org.apache.parquet.schema.LogicalTypeAnnotation;\nimport org.apache.parquet.schema.PrimitiveType;\n\nimport static org.junit.Assert.*;\n\npublic class TestUtils {\n\n @Test\n public void testBuildColumnDescriptorWithTimestamp() {\n Map params = new HashMap<>();\n params.put(\"isAdjustedToUTC\", \"true\");\n params.put(\"unit\", \"MICROS\");\n\n ParquetColumnSpec spec =\n new ParquetColumnSpec(\n 10,\n new String[] {\"event_time\"},\n \"INT64\",\n 0,\n false,\n 0,\n 0,\n \"TimestampLogicalTypeAnnotation\",\n params);\n\n ColumnDescriptor descriptor = Utils.buildColumnDescriptor(spec);\n assertNotNull(descriptor);\n\n PrimitiveType primitiveType = descriptor.getPrimitiveType();\n assertEquals(PrimitiveType.PrimitiveTypeName.INT64, primitiveType.getPrimitiveTypeName());\n assertTrue(\n primitiveType.getLogicalTypeAnnotation()\n instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation);\n\n LogicalTypeAnnotation.TimestampLogicalTypeAnnotation ts =\n (LogicalTypeAnnotation.TimestampLogicalTypeAnnotation)\n primitiveType.getLogicalTypeAnnotation();\n assertTrue(ts.isAdjustedToUTC());\n assertEquals(LogicalTypeAnnotation.TimeUnit.MICROS, ts.getUnit());\n }\n\n @Test\n public void testBuildColumnDescriptorWithDecimal() {\n Map params = new HashMap<>();\n params.put(\"precision\", \"10\");\n params.put(\"scale\", \"2\");\n\n ParquetColumnSpec spec =\n new ParquetColumnSpec(\n 11,\n new String[] {\"price\"},\n \"FIXED_LEN_BYTE_ARRAY\",\n 5,\n false,\n 0,\n 0,\n \"DecimalLogicalTypeAnnotation\",\n params);\n\n ColumnDescriptor descriptor = Utils.buildColumnDescriptor(spec);\n PrimitiveType primitiveType = descriptor.getPrimitiveType();\n assertEquals(\n PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY, primitiveType.getPrimitiveTypeName());\n\n LogicalTypeAnnotation.DecimalLogicalTypeAnnotation dec =\n (LogicalTypeAnnotation.DecimalLogicalTypeAnnotation)\n primitiveType.getLogicalTypeAnnotation();\n assertEquals(10, dec.getPrecision());\n assertEquals(2, dec.getScale());\n }\n\n @Test\n public void testBuildColumnDescriptorWithIntLogicalType() {\n Map params = new HashMap<>();\n params.put(\"bitWidth\", \"32\");\n params.put(\"isSigned\", \"true\");\n\n ParquetColumnSpec spec =\n new ParquetColumnSpec(\n 12,\n new String[] {\"count\"},\n \"INT32\",\n 0,\n false,\n 0,\n 0,\n \"IntLogicalTypeAnnotation\",\n params);\n\n ColumnDescriptor descriptor = Utils.buildColumnDescriptor(spec);\n PrimitiveType primitiveType = descriptor.getPrimitiveType();\n assertEquals(PrimitiveType.PrimitiveTypeName.INT32, primitiveType.getPrimitiveTypeName());\n\n LogicalTypeAnnotation.IntLogicalTypeAnnotation ann =\n (LogicalTypeAnnotation.IntLogicalTypeAnnotation) primitiveType.getLogicalTypeAnnotation();\n assertEquals(32, ann.getBitWidth());\n assertTrue(ann.isSigned());\n }\n\n @Test\n public void testBuildColumnDescriptorWithStringLogicalType() {\n ParquetColumnSpec spec =\n new ParquetColumnSpec(\n 13,\n new String[] {\"name\"},\n \"BINARY\",\n 0,\n false,\n 0,\n 0,\n \"StringLogicalTypeAnnotation\",\n Collections.emptyMap());\n\n ColumnDescriptor descriptor = Utils.buildColumnDescriptor(spec);\n PrimitiveType primitiveType = descriptor.getPrimitiveType();\n assertEquals(PrimitiveType.PrimitiveTypeName.BINARY, primitiveType.getPrimitiveTypeName());\n assertTrue(\n primitiveType.getLogicalTypeAnnotation()\n instanceof LogicalTypeAnnotation.StringLogicalTypeAnnotation);\n }\n}\n" }, { "path": "common/src/test/resources/log4j.properties", "content": "#\n# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n\n# Set everything to be logged to the file target/unit-tests.log\ntest.appender=file\nlog4j.rootCategory=INFO, ${test.appender}\nlog4j.appender.file=org.apache.log4j.FileAppender\nlog4j.appender.file.append=true\nlog4j.appender.file.file=target/unit-tests.log\nlog4j.appender.file.layout=org.apache.log4j.PatternLayout\nlog4j.appender.file.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n\n\n# Tests that launch java subprocesses can set the \"test.appender\" system property to\n# \"console\" to avoid having the child process's logs overwrite the unit test's\n# log file.\nlog4j.appender.console=org.apache.log4j.ConsoleAppender\nlog4j.appender.console.target=System.err\nlog4j.appender.console.layout=org.apache.log4j.PatternLayout\nlog4j.appender.console.layout.ConversionPattern=%t: %m%n\n\n# Ignore messages below warning level from Jetty, because it's a bit verbose\nlog4j.logger.org.sparkproject.jetty=WARN\n" }, { "path": "common/src/test/resources/log4j2.properties", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n# \n# http://www.apache.org/licenses/LICENSE-2.0\n# \n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n# Set everything to be logged to the file target/unit-tests.log\nrootLogger.level = info\nrootLogger.appenderRef.file.ref = ${sys:test.appender:-File}\n\nappender.file.type = File\nappender.file.name = File\nappender.file.fileName = target/unit-tests.log\nappender.file.layout.type = PatternLayout\nappender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n\n\n# Tests that launch java subprocesses can set the \"test.appender\" system property to\n# \"console\" to avoid having the child process's logs overwrite the unit test's\n# log file.\nappender.console.type = Console\nappender.console.name = console\nappender.console.target = SYSTEM_ERR\nappender.console.layout.type = PatternLayout\nappender.console.layout.pattern = %t: %m%n\n\n# Ignore messages below warning level from Jetty, because it's a bit verbose\nlogger.jetty.name = org.sparkproject.jetty\nlogger.jetty.level = warn\n\n" }, { "path": "conf/log4rs.yaml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n# \n# http://www.apache.org/licenses/LICENSE-2.0\n# \n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nappenders:\n unittest:\n kind: file\n path: \"target/unit-tests.log\"\n\nroot:\n level: info\n appenders:\n - unittest\n" }, { "path": "dev/cargo.config", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n# \n# http://www.apache.org/licenses/LICENSE-2.0\n# \n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n[target.x86_64-apple-darwin]\nlinker = \"x86_64-apple-darwin21.4-clang\"\nar = \"x86_64-apple-darwin21.4-ar\"\n\n[target.aarch64-apple-darwin]\nlinker = \"aarch64-apple-darwin21.4-clang\"\nar = \"aarch64-apple-darwin21.4-ar\"\n" }, { "path": "dev/changelog/0.1.0.md", "content": "\n\n# DataFusion Comet 0.1.0 Changelog\n\nThis release consists of 343 commits from 41 contributors. See credits at the end of this changelog for more information.\n\n**Implemented enhancements:**\n\n- feat: Add native shuffle and columnar shuffle [#30](https://github.com/apache/datafusion-comet/pull/30) (viirya)\n- feat: Support Emit::First for SumDecimalGroupsAccumulator [#47](https://github.com/apache/datafusion-comet/pull/47) (viirya)\n- feat: Nested map support for columnar shuffle [#51](https://github.com/apache/datafusion-comet/pull/51) (viirya)\n- feat: Support Count(Distinct) and similar aggregation functions [#42](https://github.com/apache/datafusion-comet/pull/42) (huaxingao)\n- feat: Upgrade to `jni-rs` 0.21 [#50](https://github.com/apache/datafusion-comet/pull/50) (sunchao)\n- feat: Handle exception thrown from native side [#61](https://github.com/apache/datafusion-comet/pull/61) (sunchao)\n- feat: Support InSet expression in Comet [#59](https://github.com/apache/datafusion-comet/pull/59) (viirya)\n- feat: Add `CometNativeException` for exceptions thrown from the native side [#62](https://github.com/apache/datafusion-comet/pull/62) (sunchao)\n- feat: Add cause to native exception [#63](https://github.com/apache/datafusion-comet/pull/63) (viirya)\n- feat: Pull based native execution [#69](https://github.com/apache/datafusion-comet/pull/69) (viirya)\n- feat: Add executeColumnarCollectIterator to CometExec to collect Comet operator result [#71](https://github.com/apache/datafusion-comet/pull/71) (viirya)\n- feat: Add CometBroadcastExchangeExec to support broadcasting the result of Comet native operator [#80](https://github.com/apache/datafusion-comet/pull/80) (viirya)\n- feat: Reduce memory consumption when writing sorted shuffle files [#82](https://github.com/apache/datafusion-comet/pull/82) (sunchao)\n- feat: Add struct/map as unsupported map key/value for columnar shuffle [#84](https://github.com/apache/datafusion-comet/pull/84) (viirya)\n- feat: Support multiple input sources for CometNativeExec [#87](https://github.com/apache/datafusion-comet/pull/87) (viirya)\n- feat: Date and timestamp trunc with format array [#94](https://github.com/apache/datafusion-comet/pull/94) (parthchandra)\n- feat: Support `First`/`Last` aggregate functions [#97](https://github.com/apache/datafusion-comet/pull/97) (huaxingao)\n- feat: Add support of TakeOrderedAndProjectExec in Comet [#88](https://github.com/apache/datafusion-comet/pull/88) (viirya)\n- feat: Support Binary in shuffle writer [#106](https://github.com/apache/datafusion-comet/pull/106) (advancedxy)\n- feat: Add license header by spotless:apply automatically [#110](https://github.com/apache/datafusion-comet/pull/110) (advancedxy)\n- feat: Add dictionary binary to shuffle writer [#111](https://github.com/apache/datafusion-comet/pull/111) (viirya)\n- feat: Minimize number of connections used by parallel reader [#126](https://github.com/apache/datafusion-comet/pull/126) (parthchandra)\n- feat: Support CollectLimit operator [#100](https://github.com/apache/datafusion-comet/pull/100) (advancedxy)\n- feat: Enable min/max for boolean type [#165](https://github.com/apache/datafusion-comet/pull/165) (huaxingao)\n- feat: Introduce `CometTaskMemoryManager` and native side memory pool [#83](https://github.com/apache/datafusion-comet/pull/83) (sunchao)\n- feat: Fix old style names [#201](https://github.com/apache/datafusion-comet/pull/201) (comphead)\n- feat: enable comet shuffle manager for comet shell [#204](https://github.com/apache/datafusion-comet/pull/204) (zuston)\n- feat: Support bitwise aggregate functions [#197](https://github.com/apache/datafusion-comet/pull/197) (huaxingao)\n- feat: Support BloomFilterMightContain expr [#179](https://github.com/apache/datafusion-comet/pull/179) (advancedxy)\n- feat: Support sort merge join [#178](https://github.com/apache/datafusion-comet/pull/178) (viirya)\n- feat: Support HashJoin operator [#194](https://github.com/apache/datafusion-comet/pull/194) (viirya)\n- feat: Remove use of nightly int_roundings feature [#228](https://github.com/apache/datafusion-comet/pull/228) (psvri)\n- feat: Support Broadcast HashJoin [#211](https://github.com/apache/datafusion-comet/pull/211) (viirya)\n- feat: Enable Comet broadcast by default [#213](https://github.com/apache/datafusion-comet/pull/213) (viirya)\n- feat: Add CometRowToColumnar operator [#206](https://github.com/apache/datafusion-comet/pull/206) (advancedxy)\n- feat: Document the class path / classloader issue with the shuffle manager [#256](https://github.com/apache/datafusion-comet/pull/256) (holdenk)\n- feat: Port Datafusion Covariance to Comet [#234](https://github.com/apache/datafusion-comet/pull/234) (huaxingao)\n- feat: Add manual test to calculate spark builtin functions coverage [#263](https://github.com/apache/datafusion-comet/pull/263) (comphead)\n- feat: Support ANSI mode in CAST from String to Bool [#290](https://github.com/apache/datafusion-comet/pull/290) (andygrove)\n- feat: Add extended explain info to Comet plan [#255](https://github.com/apache/datafusion-comet/pull/255) (parthchandra)\n- feat: Improve CometSortMergeJoin statistics [#304](https://github.com/apache/datafusion-comet/pull/304) (planga82)\n- feat: Add compatibility guide [#316](https://github.com/apache/datafusion-comet/pull/316) (andygrove)\n- feat: Improve CometHashJoin statistics [#309](https://github.com/apache/datafusion-comet/pull/309) (planga82)\n- feat: Support Variance [#297](https://github.com/apache/datafusion-comet/pull/297) (huaxingao)\n- feat: Support murmur3_hash and sha2 family hash functions [#226](https://github.com/apache/datafusion-comet/pull/226) (advancedxy)\n- feat: Disable cast string to timestamp by default [#337](https://github.com/apache/datafusion-comet/pull/337) (andygrove)\n- feat: Improve CometBroadcastHashJoin statistics [#339](https://github.com/apache/datafusion-comet/pull/339) (planga82)\n- feat: Implement Spark-compatible CAST from string to integral types [#307](https://github.com/apache/datafusion-comet/pull/307) (andygrove)\n- feat: Implement Spark-compatible CAST from string to timestamp types [#335](https://github.com/apache/datafusion-comet/pull/335) (vaibhawvipul)\n- feat: Implement Spark-compatible CAST float/double to string [#346](https://github.com/apache/datafusion-comet/pull/346) (mattharder91)\n- feat: Only allow incompatible cast expressions to run in comet if a config is enabled [#362](https://github.com/apache/datafusion-comet/pull/362) (andygrove)\n- feat: Implement Spark-compatible CAST between integer types [#340](https://github.com/apache/datafusion-comet/pull/340) (ganeshkumar269)\n- feat: Supports Stddev [#348](https://github.com/apache/datafusion-comet/pull/348) (huaxingao)\n- feat: Improve cast compatibility tests and docs [#379](https://github.com/apache/datafusion-comet/pull/379) (andygrove)\n- feat: Implement Spark-compatible CAST from non-integral numeric types to integral types [#399](https://github.com/apache/datafusion-comet/pull/399) (rohitrastogi)\n- feat: Implement Spark unhex [#342](https://github.com/apache/datafusion-comet/pull/342) (tshauck)\n- feat: Enable columnar shuffle by default [#250](https://github.com/apache/datafusion-comet/pull/250) (viirya)\n- feat: Implement Spark-compatible CAST from floating-point/double to decimal [#384](https://github.com/apache/datafusion-comet/pull/384) (vaibhawvipul)\n- feat: Add logging to explain reasons for Comet not being able to run a query stage natively [#397](https://github.com/apache/datafusion-comet/pull/397) (andygrove)\n- feat: Add support for TryCast expression in Spark 3.2 and 3.3 [#416](https://github.com/apache/datafusion-comet/pull/416) (vaibhawvipul)\n- feat: Supports UUID column [#395](https://github.com/apache/datafusion-comet/pull/395) (huaxingao)\n- feat: correlation support [#456](https://github.com/apache/datafusion-comet/pull/456) (huaxingao)\n- feat: Implement Spark-compatible CAST from String to Date [#383](https://github.com/apache/datafusion-comet/pull/383) (vidyasankarv)\n- feat: Add COMET_SHUFFLE_MODE config to control Comet shuffle mode [#460](https://github.com/apache/datafusion-comet/pull/460) (viirya)\n- feat: Add random row generator in data generator [#451](https://github.com/apache/datafusion-comet/pull/451) (advancedxy)\n- feat: Add xxhash64 function support [#424](https://github.com/apache/datafusion-comet/pull/424) (advancedxy)\n- feat: add hex scalar function [#449](https://github.com/apache/datafusion-comet/pull/449) (tshauck)\n- feat: Add \"Comet Fuzz\" fuzz-testing utility [#472](https://github.com/apache/datafusion-comet/pull/472) (andygrove)\n- feat: Use enum to represent CAST eval_mode in expr.proto [#415](https://github.com/apache/datafusion-comet/pull/415) (prashantksharma)\n- feat: Implement ANSI support for UnaryMinus [#471](https://github.com/apache/datafusion-comet/pull/471) (vaibhawvipul)\n- feat: Add specific fuzz tests for cast and try_cast and fix NPE found during fuzz testing [#514](https://github.com/apache/datafusion-comet/pull/514) (andygrove)\n- feat: Add fuzz testing for arithmetic expressions [#519](https://github.com/apache/datafusion-comet/pull/519) (andygrove)\n- feat: Add HashJoin support for BuildRight [#437](https://github.com/apache/datafusion-comet/pull/437) (viirya)\n- feat: Fix Comet error message [#544](https://github.com/apache/datafusion-comet/pull/544) (comphead)\n- feat: Support Ansi mode in abs function [#500](https://github.com/apache/datafusion-comet/pull/500) (planga82)\n- feat: Enable xxhash64 by default [#583](https://github.com/apache/datafusion-comet/pull/583) (andygrove)\n- feat: Add experimental support for Apache Spark 3.5.1 [#587](https://github.com/apache/datafusion-comet/pull/587) (andygrove)\n- feat: add nullOnDivideByZero for Covariance [#564](https://github.com/apache/datafusion-comet/pull/564) (huaxingao)\n- feat: Implement more efficient version of xxhash64 [#575](https://github.com/apache/datafusion-comet/pull/575) (andygrove)\n- feat: Enable Spark SQL tests for Spark 3.5.1 [#603](https://github.com/apache/datafusion-comet/pull/603) (andygrove)\n- feat: Initial support for Window function [#599](https://github.com/apache/datafusion-comet/pull/599) (huaxingao)\n- feat: IsNaN expression in Comet [#612](https://github.com/apache/datafusion-comet/pull/612) (eejbyfeldt)\n- feat: Add support for CreateNamedStruct [#620](https://github.com/apache/datafusion-comet/pull/620) (eejbyfeldt)\n- feat: add cargo machete to remove udeps [#641](https://github.com/apache/datafusion-comet/pull/641) (vaibhawvipul)\n- feat: Upgrade to DataFusion 40.0.0-rc1 [#644](https://github.com/apache/datafusion-comet/pull/644) (andygrove)\n- feat: Use unified allocator for execution iterators [#613](https://github.com/apache/datafusion-comet/pull/613) (viirya)\n- feat: Create new `datafusion-comet-spark-expr` crate containing Spark-compatible DataFusion expressions [#638](https://github.com/apache/datafusion-comet/pull/638) (andygrove)\n- feat: Move `IfExpr` to `spark-expr` crate [#653](https://github.com/apache/datafusion-comet/pull/653) (andygrove)\n- feat: Upgrade to DataFusion 40 [#657](https://github.com/apache/datafusion-comet/pull/657) (andygrove)\n- feat: Show user a more intuitive message when queries fall back to Spark [#656](https://github.com/apache/datafusion-comet/pull/656) (andygrove)\n- feat: Enable remaining Spark 3.5.1 tests [#676](https://github.com/apache/datafusion-comet/pull/676) (andygrove)\n- feat: Spark-4.0 widening type support [#604](https://github.com/apache/datafusion-comet/pull/604) (kazuyukitanimura)\n- feat: add scalar subquery pushdown to scan [#678](https://github.com/apache/datafusion-comet/pull/678) (parthchandra)\n\n**Fixed bugs:**\n\n- fix: Comet sink operator should not have children operators [#26](https://github.com/apache/datafusion-comet/pull/26) (viirya)\n- fix: Fix the UnionExec match branches in CometExecRule [#68](https://github.com/apache/datafusion-comet/pull/68) (wankunde)\n- fix: Appending null values to element array builders of StructBuilder for null row in a StructArray [#78](https://github.com/apache/datafusion-comet/pull/78) (viirya)\n- fix: Fix compilation error for CometBroadcastExchangeExec [#86](https://github.com/apache/datafusion-comet/pull/86) (viirya)\n- fix: Avoid exception caused by broadcasting empty result [#92](https://github.com/apache/datafusion-comet/pull/92) (wForget)\n- fix: Add num_rows when building RecordBatch [#103](https://github.com/apache/datafusion-comet/pull/103) (advancedxy)\n- fix: Cast string to boolean not compatible with Spark [#107](https://github.com/apache/datafusion-comet/pull/107) (erenavsarogullari)\n- fix: Another attempt to fix libcrypto.dylib loading issue [#112](https://github.com/apache/datafusion-comet/pull/112) (advancedxy)\n- fix: Fix compilation error for Spark 3.2 & 3.3 [#117](https://github.com/apache/datafusion-comet/pull/117) (sunchao)\n- fix: Fix corrupted AggregateMode when transforming plan parameters [#118](https://github.com/apache/datafusion-comet/pull/118) (viirya)\n- fix: bitwise shift with different left/right types [#135](https://github.com/apache/datafusion-comet/pull/135) (viirya)\n- fix: Avoid null exception in removeSubquery [#147](https://github.com/apache/datafusion-comet/pull/147) (viirya)\n- fix: rat check error in vscode ide [#161](https://github.com/apache/datafusion-comet/pull/161) (thexiay)\n- fix: Final aggregation should not bind to the input of partial aggregation [#155](https://github.com/apache/datafusion-comet/pull/155) (viirya)\n- fix: coalesce should return correct datatype [#168](https://github.com/apache/datafusion-comet/pull/168) (viirya)\n- fix: attempt to divide by zero error on decimal division [#172](https://github.com/apache/datafusion-comet/pull/172) (viirya)\n- fix: Aggregation without aggregation expressions should use correct result expressions [#175](https://github.com/apache/datafusion-comet/pull/175) (viirya)\n- fix: Comet native operator can be executed after ReusedExchange [#187](https://github.com/apache/datafusion-comet/pull/187) (viirya)\n- fix: Try to convert a static list into a set in Rust [#184](https://github.com/apache/datafusion-comet/pull/184) (advancedxy)\n- fix: Include active spiller when computing peak shuffle memory [#196](https://github.com/apache/datafusion-comet/pull/196) (sunchao)\n- fix: CometExecRule should handle ShuffleQueryStage and ReusedExchange [#186](https://github.com/apache/datafusion-comet/pull/186) (viirya)\n- fix: Use `makeCopy` to change relation in `FileSourceScanExec` [#207](https://github.com/apache/datafusion-comet/pull/207) (viirya)\n- fix: Remove duplicate byte array allocation for CometDictionary [#224](https://github.com/apache/datafusion-comet/pull/224) (viirya)\n- fix: Remove redundant data copy in columnar shuffle [#233](https://github.com/apache/datafusion-comet/pull/233) (viirya)\n- fix: Only maps FIXED_LEN_BYTE_ARRAY to String for uuid type [#238](https://github.com/apache/datafusion-comet/pull/238) (huaxingao)\n- fix: Reduce RowPartition memory allocation [#244](https://github.com/apache/datafusion-comet/pull/244) (viirya)\n- fix: Remove wrong calculation for Murmur3Hash for float with null input [#245](https://github.com/apache/datafusion-comet/pull/245) (advancedxy)\n- fix: Deallocate row addresses and size arrays after exporting [#246](https://github.com/apache/datafusion-comet/pull/246) (viirya)\n- fix: Fix wrong children expression order in IfExpr [#249](https://github.com/apache/datafusion-comet/pull/249) (viirya)\n- fix: Average expression in Comet Final should handle all null inputs from partial Spark aggregation [#261](https://github.com/apache/datafusion-comet/pull/261) (viirya)\n- fix: Only trigger Comet Final aggregation on Comet partial aggregation [#264](https://github.com/apache/datafusion-comet/pull/264) (viirya)\n- fix: incorrect result on Comet multiple column distinct count [#268](https://github.com/apache/datafusion-comet/pull/268) (viirya)\n- fix: Avoid using CometConf [#266](https://github.com/apache/datafusion-comet/pull/266) (snmvaughan)\n- fix: Fix arrow error when sorting on empty batch [#271](https://github.com/apache/datafusion-comet/pull/271) (viirya)\n- fix: Include license using `#` instead of using XML comment [#274](https://github.com/apache/datafusion-comet/pull/274) (snmvaughan)\n- fix: Comet should not translate try_sum to native sum expression [#277](https://github.com/apache/datafusion-comet/pull/277) (viirya)\n- fix: incorrect result with aggregate expression with filter [#284](https://github.com/apache/datafusion-comet/pull/284) (viirya)\n- fix: Comet should not fail on negative limit parameter [#288](https://github.com/apache/datafusion-comet/pull/288) (viirya)\n- fix: Comet columnar shuffle should not be on top of another Comet shuffle operator [#296](https://github.com/apache/datafusion-comet/pull/296) (viirya)\n- fix: Iceberg scan transition should be in front of other data source v2 [#302](https://github.com/apache/datafusion-comet/pull/302) (viirya)\n- fix: CometExec's outputPartitioning might not be same as Spark expects after AQE interferes [#299](https://github.com/apache/datafusion-comet/pull/299) (viirya)\n- fix: CometShuffleExchangeExec logical link should be correct [#324](https://github.com/apache/datafusion-comet/pull/324) (viirya)\n- fix: SortMergeJoin with unsupported key type should fall back to Spark [#355](https://github.com/apache/datafusion-comet/pull/355) (viirya)\n- fix: limit with offset should return correct results [#359](https://github.com/apache/datafusion-comet/pull/359) (viirya)\n- fix: Disable Comet shuffle with AQE coalesce partitions enabled [#380](https://github.com/apache/datafusion-comet/pull/380) (viirya)\n- fix: Unknown operator id when explain with formatted mode [#410](https://github.com/apache/datafusion-comet/pull/410) (leoluan2009)\n- fix: Reuse CometBroadcastExchangeExec with Spark ReuseExchangeAndSubquery rule [#441](https://github.com/apache/datafusion-comet/pull/441) (viirya)\n- fix: newFileScanRDD should not take constructor from custom Spark versions [#412](https://github.com/apache/datafusion-comet/pull/412) (ceppelli)\n- fix: fix CometNativeExec.doCanonicalize for ReusedExchangeExec [#447](https://github.com/apache/datafusion-comet/pull/447) (viirya)\n- fix: Enable cast string to int tests and fix compatibility issue [#453](https://github.com/apache/datafusion-comet/pull/453) (andygrove)\n- fix: Compute murmur3 hash with dictionary input correctly [#433](https://github.com/apache/datafusion-comet/pull/433) (advancedxy)\n- fix: Only delegate to DataFusion cast when we know that it is compatible with Spark [#461](https://github.com/apache/datafusion-comet/pull/461) (andygrove)\n- fix: `ColumnReader.loadVector` should initiate `CometDictionary` after re-import arrays [#473](https://github.com/apache/datafusion-comet/pull/473) (viirya)\n- fix: substring with negative indices should produce correct result [#470](https://github.com/apache/datafusion-comet/pull/470) (sonhmai)\n- fix: CometReader.loadVector should not overwrite dictionary ids [#476](https://github.com/apache/datafusion-comet/pull/476) (viirya)\n- fix: Reuse previous CometDictionary Java arrays [#489](https://github.com/apache/datafusion-comet/pull/489) (viirya)\n- fix: Fallback to Spark for LIKE with custom escape character [#478](https://github.com/apache/datafusion-comet/pull/478) (sujithjay)\n- fix: Incorrect input schema when preparing result expressions for HashAggregation [#501](https://github.com/apache/datafusion-comet/pull/501) (viirya)\n- fix: Input batch to ShuffleRepartitioner.insert_batch should not be larger than configured batch size [#523](https://github.com/apache/datafusion-comet/pull/523) (viirya)\n- fix: Fix integer overflow in date_parser [#529](https://github.com/apache/datafusion-comet/pull/529) (eejbyfeldt)\n- fix: null character not permitted in chr function [#513](https://github.com/apache/datafusion-comet/pull/513) (vaibhawvipul)\n- fix: Overflow when reading Timestamp from parquet file [#542](https://github.com/apache/datafusion-comet/pull/542) (eejbyfeldt)\n- fix: Re-implement some Parquet decode methods without `copy_nonoverlapping` [#558](https://github.com/apache/datafusion-comet/pull/558) (andygrove)\n- fix: requested character too large for encoding in chr function [#552](https://github.com/apache/datafusion-comet/pull/552) (vaibhawvipul)\n- fix: Running cargo build always triggers rebuild [#579](https://github.com/apache/datafusion-comet/pull/579) (eejbyfeldt)\n- fix: Avoid recursive call to `canonicalizePlans` [#582](https://github.com/apache/datafusion-comet/pull/582) (viirya)\n- fix: Return error in pre_timestamp_cast instead of panic [#543](https://github.com/apache/datafusion-comet/pull/543) (eejbyfeldt)\n- perf: Add criterion benchmark for xxhash64 function [#560](https://github.com/apache/datafusion-comet/pull/560) (andygrove)\n- fix: Fix range out of index error with a temporary workaround [#584](https://github.com/apache/datafusion-comet/pull/584) (viirya)\n- fix: Improve error \"BroadcastExchange is not supported\" [#577](https://github.com/apache/datafusion-comet/pull/577) (parthchandra)\n- fix: Avoid creating huge duplicate of canonicalized plans for CometNativeExec [#639](https://github.com/apache/datafusion-comet/pull/639) (viirya)\n- fix: Tag ignored tests that require SubqueryBroadcastExec [#647](https://github.com/apache/datafusion-comet/pull/647) (parthchandra)\n- fix: Optimize some functions to rewrite dictionary-encoded strings [#627](https://github.com/apache/datafusion-comet/pull/627) (vaibhawvipul)\n- fix: Remove nightly flag in release-nogit target in Makefile [#667](https://github.com/apache/datafusion-comet/pull/667) (andygrove)\n- fix: change the not exists base image apache/spark:3.4.3 to 3.4.2 [#686](https://github.com/apache/datafusion-comet/pull/686) (haoxins)\n- fix: Spark 4.0 SparkArithmeticException test [#688](https://github.com/apache/datafusion-comet/pull/688) (kazuyukitanimura)\n- fix: address failure caused by method signature change in SPARK-48791 [#693](https://github.com/apache/datafusion-comet/pull/693) (parthchandra)\n\n**Documentation updates:**\n\n- doc: Add Quickstart Comet doc section [#125](https://github.com/apache/datafusion-comet/pull/125) (comphead)\n- doc: Minor fix Getting started reformatting [#128](https://github.com/apache/datafusion-comet/pull/128) (comphead)\n- doc: Add initial doc how to expand Comet exceptions [#170](https://github.com/apache/datafusion-comet/pull/170) (comphead)\n- doc: Update README.md with shuffle configs [#208](https://github.com/apache/datafusion-comet/pull/208) (viirya)\n- doc: Update supported expressions [#237](https://github.com/apache/datafusion-comet/pull/237) (viirya)\n- doc: Fix a small typo in README.md [#272](https://github.com/apache/datafusion-comet/pull/272) (rz-vastdata)\n- doc: Update DataFusion project name and url [#300](https://github.com/apache/datafusion-comet/pull/300) (viirya)\n- docs: Move existing documentation into new Contributor Guide and add Getting Started section [#334](https://github.com/apache/datafusion-comet/pull/334) (andygrove)\n- docs: Add more content to the user guide [#347](https://github.com/apache/datafusion-comet/pull/347) (andygrove)\n- docs: Generate configuration guide in mvn build [#349](https://github.com/apache/datafusion-comet/pull/349) (andygrove)\n- docs: Add a plugin overview page to the contributors guide [#345](https://github.com/apache/datafusion-comet/pull/345) (andygrove)\n- doc: Fix target typo in development.md [#364](https://github.com/apache/datafusion-comet/pull/364) (jc4x4)\n- doc: Clean up supported JDKs in README [#366](https://github.com/apache/datafusion-comet/pull/366) (edmondop)\n- doc: add contributing in README.md [#382](https://github.com/apache/datafusion-comet/pull/382) (caicancai)\n- docs: fix the docs url of installation instructions [#393](https://github.com/apache/datafusion-comet/pull/393) (haoxins)\n- docs: Running ScalaTest suites from the CLI [#404](https://github.com/apache/datafusion-comet/pull/404) (edmondop)\n- docs: Remove spark.comet.exec.broadcast.enabled from config docs [#421](https://github.com/apache/datafusion-comet/pull/421) (andygrove)\n- docs: fix various sphinx warnings [#428](https://github.com/apache/datafusion-comet/pull/428) (tshauck)\n- doc: Add Plan Stability Testing to development guide [#432](https://github.com/apache/datafusion-comet/pull/432) (viirya)\n- docs: Update Spark shell command to include setting additional class path [#435](https://github.com/apache/datafusion-comet/pull/435) (andygrove)\n- doc: Add Tuning Guide with shuffle configs [#443](https://github.com/apache/datafusion-comet/pull/443) (viirya)\n- docs: Add benchmarking guide [#444](https://github.com/apache/datafusion-comet/pull/444) (andygrove)\n- docs: add guide to adding a new expression [#422](https://github.com/apache/datafusion-comet/pull/422) (tshauck)\n- docs: changes in documentation [#512](https://github.com/apache/datafusion-comet/pull/512) (SemyonSinchenko)\n- docs: Improve user documentation for supported operators and expressions [#520](https://github.com/apache/datafusion-comet/pull/520) (andygrove)\n- docs: Proposal for source release process [#556](https://github.com/apache/datafusion-comet/pull/556) (andygrove)\n- docs: Update benchmark results [#687](https://github.com/apache/datafusion-comet/pull/687) (andygrove)\n- docs: Update percentage speedups in benchmarking guide [#691](https://github.com/apache/datafusion-comet/pull/691) (andygrove)\n- doc: Add memory tuning section to user guide [#684](https://github.com/apache/datafusion-comet/pull/684) (viirya)\n\n**Other:**\n\n- Initial PR [#1](https://github.com/apache/datafusion-comet/pull/1) (sunchao)\n- build: Add Maven wrapper to the project [#13](https://github.com/apache/datafusion-comet/pull/13) (sunchao)\n- build: Add basic CI test pipelines [#18](https://github.com/apache/datafusion-comet/pull/18) (sunchao)\n- Bump com.google.protobuf:protobuf-java from 3.17.3 to 3.19.6 [#5](https://github.com/apache/datafusion-comet/pull/5) (dependabot[bot])\n- build: Add PR template [#23](https://github.com/apache/datafusion-comet/pull/23) (sunchao)\n- build: Create ticket templates [#24](https://github.com/apache/datafusion-comet/pull/24) (comphead)\n- build: Re-enable Scala style checker and spotless [#21](https://github.com/apache/datafusion-comet/pull/21) (sunchao)\n- build: Remove license header from pull request template [#28](https://github.com/apache/datafusion-comet/pull/28) (viirya)\n- build: Exclude .github from apache-rat-plugin check [#32](https://github.com/apache/datafusion-comet/pull/32) (viirya)\n- build: Add CI for MacOS (x64 and aarch64) [#35](https://github.com/apache/datafusion-comet/pull/35) (sunchao)\n- fix broken link in README.md [#39](https://github.com/apache/datafusion-comet/pull/39) (nairbv)\n- test: Add some fuzz testing for cast operations [#16](https://github.com/apache/datafusion-comet/pull/16) (andygrove)\n- test: Fix CI failure on libcrypto [#41](https://github.com/apache/datafusion-comet/pull/41) (sunchao)\n- test: Reduce test time spent in `CometShuffleSuite` [#40](https://github.com/apache/datafusion-comet/pull/40) (sunchao)\n- test: Add test for RoundRobinPartitioning [#54](https://github.com/apache/datafusion-comet/pull/54) (viirya)\n- build: Fix potential libcrypto lib loading issue for X86 mac runners [#55](https://github.com/apache/datafusion-comet/pull/55) (advancedxy)\n- refactor: Remove a few duplicated occurrences [#53](https://github.com/apache/datafusion-comet/pull/53) (sunchao)\n- build: Fix mvn cache for containerized runners [#48](https://github.com/apache/datafusion-comet/pull/48) (advancedxy)\n- test: Ensure traversed operators during finding first partial aggregaion are all native [#58](https://github.com/apache/datafusion-comet/pull/58) (viirya)\n- build: Upgrade arrow-rs to 50.0.0 and DataFusion to 35.0.0 [#65](https://github.com/apache/datafusion-comet/pull/65) (viirya)\n- build: Support built with java 1.8 [#45](https://github.com/apache/datafusion-comet/pull/45) (advancedxy)\n- test: Add golden files for TPCDSPlanStabilitySuite [#73](https://github.com/apache/datafusion-comet/pull/73) (sunchao)\n- test: Add TPC-DS test results [#77](https://github.com/apache/datafusion-comet/pull/77) (sunchao)\n- build: Upgrade spotless version to 2.43.0 [#85](https://github.com/apache/datafusion-comet/pull/85) (viirya)\n- test: Expose thrown exception when executing query in CometTPCHQuerySuite [#96](https://github.com/apache/datafusion-comet/pull/96) (viirya)\n- test: Enable TPCDS q41 in CometTPCDSQuerySuite [#98](https://github.com/apache/datafusion-comet/pull/98) (viirya)\n- build: Add CI for TPCDS queries [#99](https://github.com/apache/datafusion-comet/pull/99) (viirya)\n- build: Add tpcds-sf-1 to license header excluded list [#108](https://github.com/apache/datafusion-comet/pull/108) (viirya)\n- build: Show time duration for scala test [#116](https://github.com/apache/datafusion-comet/pull/116) (advancedxy)\n- test: Move MacOS (x86) pipelines to post-commit [#122](https://github.com/apache/datafusion-comet/pull/122) (sunchao)\n- build: Upgrade DF to 36.0.0 and arrow-rs 50.0.0 [#66](https://github.com/apache/datafusion-comet/pull/66) (comphead)\n- test: Reduce end-to-end test time [#109](https://github.com/apache/datafusion-comet/pull/109) (sunchao)\n- build: Separate and speedup TPC-DS benchmark [#130](https://github.com/apache/datafusion-comet/pull/130) (advancedxy)\n- build: Re-enable TPCDS queries q34 and q64 in `CometTPCDSQuerySuite` [#133](https://github.com/apache/datafusion-comet/pull/133) (viirya)\n- build: Refine names in benchmark.yml [#132](https://github.com/apache/datafusion-comet/pull/132) (advancedxy)\n- build: Make the build system work out of box [#136](https://github.com/apache/datafusion-comet/pull/136) (advancedxy)\n- minor: Update README.md with system diagram [#148](https://github.com/apache/datafusion-comet/pull/148) (alamb)\n- test: Add golden files for test [#150](https://github.com/apache/datafusion-comet/pull/150) (snmvaughan)\n- build: Add checker for PR title [#151](https://github.com/apache/datafusion-comet/pull/151) (sunchao)\n- build: Support CI pipelines for Spark 3.2, 3.3 and 3.4 [#153](https://github.com/apache/datafusion-comet/pull/153) (advancedxy)\n- minor: Only trigger PR title checker on pull requests [#154](https://github.com/apache/datafusion-comet/pull/154) (sunchao)\n- chore: Fix warnings in both compiler and test environments [#164](https://github.com/apache/datafusion-comet/pull/164) (advancedxy)\n- build: Upload test reports and coverage [#163](https://github.com/apache/datafusion-comet/pull/163) (advancedxy)\n- minor: Remove unnecessary logic [#169](https://github.com/apache/datafusion-comet/pull/169) (sunchao)\n- minor: Make `QueryPlanSerde` warning log less confusing [#181](https://github.com/apache/datafusion-comet/pull/181) (viirya)\n- refactor: Skipping slicing on shuffle arrays in shuffle reader [#189](https://github.com/apache/datafusion-comet/pull/189) (viirya)\n- build: Run Spark SQL tests for 3.4 [#166](https://github.com/apache/datafusion-comet/pull/166) (sunchao)\n- build: Enforce scalafix check in CI [#203](https://github.com/apache/datafusion-comet/pull/203) (advancedxy)\n- test: Follow up on Spark 3.4 diff [#209](https://github.com/apache/datafusion-comet/pull/209) (sunchao)\n- build: Avoid confusion by using profile with clean [#215](https://github.com/apache/datafusion-comet/pull/215) (snmvaughan)\n- test: Add TPC-H test results [#218](https://github.com/apache/datafusion-comet/pull/218) (viirya)\n- build: Add CI for TPC-H queries [#220](https://github.com/apache/datafusion-comet/pull/220) (viirya)\n- test: Enable Comet shuffle in Spark SQL tests [#210](https://github.com/apache/datafusion-comet/pull/210) (sunchao)\n- test: Disable spark ui in unit test by default [#235](https://github.com/apache/datafusion-comet/pull/235) (beryllw)\n- chore: Replace deprecated temporal methods [#229](https://github.com/apache/datafusion-comet/pull/229) (snmvaughan)\n- build: Use specified branch of arrow-rs with workaround to invalid offset buffers from Java Arrow [#239](https://github.com/apache/datafusion-comet/pull/239) (viirya)\n- test: Enable string-to-bool cast test [#251](https://github.com/apache/datafusion-comet/pull/251) (andygrove)\n- test: Restore tests in CometTPCDSQuerySuite [#252](https://github.com/apache/datafusion-comet/pull/252) (viirya)\n- test: Enable all remaining TPCDS queries [#254](https://github.com/apache/datafusion-comet/pull/254) (viirya)\n- test: Enable all remaining TPCH queries [#257](https://github.com/apache/datafusion-comet/pull/257) (viirya)\n- chore: Remove some calls to unwrap when calling create_expr in planner.rs [#269](https://github.com/apache/datafusion-comet/pull/269) (andygrove)\n- chore: Fix typo in info message [#279](https://github.com/apache/datafusion-comet/pull/279) (andygrove)\n- chore: Fix NPE when running CometTPCHQueriesList directly [#285](https://github.com/apache/datafusion-comet/pull/285) (advancedxy)\n- chore: Update Comet repo description [#291](https://github.com/apache/datafusion-comet/pull/291) (viirya)\n- Chore: Cleanup how datafusion session config is created [#289](https://github.com/apache/datafusion-comet/pull/289) (psvri)\n- build: Update asf.yaml to use `@datafusion.apache.org` [#294](https://github.com/apache/datafusion-comet/pull/294) (sunchao)\n- chore: Remove unused functions [#301](https://github.com/apache/datafusion-comet/pull/301) (kazuyukitanimura)\n- chore: Ignore unused variables [#306](https://github.com/apache/datafusion-comet/pull/306) (snmvaughan)\n- chore: Update documentation publishing domain and path [#310](https://github.com/apache/datafusion-comet/pull/310) (andygrove)\n- chore: Add documentation publishing infrastructure [#314](https://github.com/apache/datafusion-comet/pull/314) (andygrove)\n- build: Move shim directories [#318](https://github.com/apache/datafusion-comet/pull/318) (kazuyukitanimura)\n- test: Suppress decimal random number tests for 3.2 and 3.3 [#319](https://github.com/apache/datafusion-comet/pull/319) (kazuyukitanimura)\n- chore: Add allocation source to StreamReader [#332](https://github.com/apache/datafusion-comet/pull/332) (viirya)\n- chore: Add more cast tests and improve test framework [#351](https://github.com/apache/datafusion-comet/pull/351) (andygrove)\n- chore: Implement remaining CAST tests [#356](https://github.com/apache/datafusion-comet/pull/356) (andygrove)\n- build: Add Spark SQL test pipeline with ANSI mode enabled [#321](https://github.com/apache/datafusion-comet/pull/321) (parthchandra)\n- chore: Store EXTENSION_INFO as Set[String] instead of newline-delimited String [#386](https://github.com/apache/datafusion-comet/pull/386) (andygrove)\n- build: Add scala-version to matrix [#396](https://github.com/apache/datafusion-comet/pull/396) (snmvaughan)\n- chore: Add criterion benchmarks for casting between integer types [#401](https://github.com/apache/datafusion-comet/pull/401) (andygrove)\n- chore: Make COMET_EXEC_BROADCAST_FORCE_ENABLED internal config [#413](https://github.com/apache/datafusion-comet/pull/413) (viirya)\n- chore: Rename some columnar shuffle configs for code consistently [#418](https://github.com/apache/datafusion-comet/pull/418) (leoluan2009)\n- chore: Remove an unused config [#430](https://github.com/apache/datafusion-comet/pull/430) (andygrove)\n- tests: Move random data generation methods from CometCastSuite to new DataGenerator class [#426](https://github.com/apache/datafusion-comet/pull/426) (andygrove)\n- test: Fix explain with exteded info comet test [#436](https://github.com/apache/datafusion-comet/pull/436) (kazuyukitanimura)\n- chore: Add cargo bench for shuffle writer [#438](https://github.com/apache/datafusion-comet/pull/438) (andygrove)\n- chore: improve fallback message when comet native shuffle is not enabled [#445](https://github.com/apache/datafusion-comet/pull/445) (andygrove)\n- Coverage: Add a manual test to show what Spark built in expression the DF can support directly [#331](https://github.com/apache/datafusion-comet/pull/331) (comphead)\n- build: Add spark-4.0 profile and shims [#407](https://github.com/apache/datafusion-comet/pull/407) (kazuyukitanimura)\n- build: bump spark version to 3.4.3 [#292](https://github.com/apache/datafusion-comet/pull/292) (huaxingao)\n- chore: Removing copying data from dictionary values into CometDictionary [#490](https://github.com/apache/datafusion-comet/pull/490) (viirya)\n- chore: Update README to highlight Comet benefits [#497](https://github.com/apache/datafusion-comet/pull/497) (andygrove)\n- test: fix ClassNotFoundException for Hive tests [#499](https://github.com/apache/datafusion-comet/pull/499) (kazuyukitanimura)\n- build: Enable comet tests with spark-4.0 profile [#493](https://github.com/apache/datafusion-comet/pull/493) (kazuyukitanimura)\n- chore: Switch to stable Rust [#505](https://github.com/apache/datafusion-comet/pull/505) (andygrove)\n- Minor: Generate the supported Spark builtin expression list into MD file [#455](https://github.com/apache/datafusion-comet/pull/455) (comphead)\n- chore: Simplify code in CometExecIterator and avoid some small overhead [#522](https://github.com/apache/datafusion-comet/pull/522) (andygrove)\n- chore: Upgrade spark to 4.0.0-preview1 [#526](https://github.com/apache/datafusion-comet/pull/526) (advancedxy)\n- chore: Add UnboundColumn to carry datatype for unbound reference [#518](https://github.com/apache/datafusion-comet/pull/518) (viirya)\n- chore: Remove 3.4.2.diff [#528](https://github.com/apache/datafusion-comet/pull/528) (kazuyukitanimura)\n- build: Switch back to official DataFusion repo and arrow-rs after Arrow Java 16 is released [#403](https://github.com/apache/datafusion-comet/pull/403) (viirya)\n- chore: Add CometEvalMode enum to replace string literals [#539](https://github.com/apache/datafusion-comet/pull/539) (andygrove)\n- chore: Create initial release process scripts for official ASF source release [#429](https://github.com/apache/datafusion-comet/pull/429) (andygrove)\n- build: Use DataFusion 39.0.0 release [#550](https://github.com/apache/datafusion-comet/pull/550) (viirya)\n- chore: disable xxhash64 by default [#548](https://github.com/apache/datafusion-comet/pull/548) (andygrove)\n- chore: Remove unsafe use of from_raw_parts in Parquet decoder [#549](https://github.com/apache/datafusion-comet/pull/549) (andygrove)\n- test: Add tests for Scalar and Inverval values for UnaryMinus [#538](https://github.com/apache/datafusion-comet/pull/538) (vaibhawvipul)\n- chore: Add changelog generator [#545](https://github.com/apache/datafusion-comet/pull/545) (andygrove)\n- chore: Remove unused hash_utils.rs [#561](https://github.com/apache/datafusion-comet/pull/561) (andygrove)\n- chore: Use in_list func directly [#559](https://github.com/apache/datafusion-comet/pull/559) (advancedxy)\n- chore: Fix most of the scala/java build warnings [#562](https://github.com/apache/datafusion-comet/pull/562) (andygrove)\n- chore: Upgrade to Rust 1.78 and fix UB issues in unsafe code [#546](https://github.com/apache/datafusion-comet/pull/546) (andygrove)\n- chore: Remove `spark.comet.xxhash64.enabled` from the config document [#586](https://github.com/apache/datafusion-comet/pull/586) (viirya)\n- build: Drop Spark 3.2 support [#581](https://github.com/apache/datafusion-comet/pull/581) (huaxingao)\n- test: Enable Spark 4.0 tests [#537](https://github.com/apache/datafusion-comet/pull/537) (kazuyukitanimura)\n- refactor: Remove method get_global_jclass [#580](https://github.com/apache/datafusion-comet/pull/580) (eejbyfeldt)\n- chore: Move some utility methods to submodules of scalar_funcs [#590](https://github.com/apache/datafusion-comet/pull/590) (advancedxy)\n- chore: Upgrade to Rust 1.79 [#570](https://github.com/apache/datafusion-comet/pull/570) (andygrove)\n- chore: Remove some calls to `unwrap` [#598](https://github.com/apache/datafusion-comet/pull/598) (andygrove)\n- chore: Improve JNI safety [#600](https://github.com/apache/datafusion-comet/pull/600) (andygrove)\n- chore: remove some unwraps from shuffle module [#601](https://github.com/apache/datafusion-comet/pull/601) (andygrove)\n- chore: Use proper constructor of IndexShuffleBlockResolver [#610](https://github.com/apache/datafusion-comet/pull/610) (viirya)\n- chore: Update benchmark results [#614](https://github.com/apache/datafusion-comet/pull/614) (andygrove)\n- build: Upgrade to 2.13.14 for scala-2.13 profile [#626](https://github.com/apache/datafusion-comet/pull/626) (viirya)\n- chore: Rename shuffle write metric [#624](https://github.com/apache/datafusion-comet/pull/624) (andygrove)\n- minor: replace .downcast_ref::().is_some() with .is::() [#635](https://github.com/apache/datafusion-comet/pull/635) (andygrove)\n- test: Add CometTPCDSQueryTestSuite [#628](https://github.com/apache/datafusion-comet/pull/628) (viirya)\n- chore: Convert Rust project into a workspace [#637](https://github.com/apache/datafusion-comet/pull/637) (andygrove)\n- chore: Add Miri workflow [#636](https://github.com/apache/datafusion-comet/pull/636) (andygrove)\n- test: Run optimized version of q72 derived from TPC-DS [#652](https://github.com/apache/datafusion-comet/pull/652) (viirya)\n- chore: Refactoring of CometError/SparkError [#655](https://github.com/apache/datafusion-comet/pull/655) (andygrove)\n- chore: Move `cast` to `spark-expr` crate [#654](https://github.com/apache/datafusion-comet/pull/654) (andygrove)\n- chore: Remove utils crate and move utils into spark-expr crate [#658](https://github.com/apache/datafusion-comet/pull/658) (andygrove)\n- chore: Move temporal kernels and expressions to spark-expr crate [#660](https://github.com/apache/datafusion-comet/pull/660) (andygrove)\n- chore: Move protobuf files to separate crate [#661](https://github.com/apache/datafusion-comet/pull/661) (andygrove)\n- Use IfExpr to check when input to log2 is <=0 and return null [#506](https://github.com/apache/datafusion-comet/pull/506) (PedroMDuarte)\n- chore: Change suffix on some expressions from Exec to Expr [#673](https://github.com/apache/datafusion-comet/pull/673) (andygrove)\n- chore: Fix some regressions with Spark 3.5.1 [#674](https://github.com/apache/datafusion-comet/pull/674) (andygrove)\n- chore: Improve fuzz testing coverage [#668](https://github.com/apache/datafusion-comet/pull/668) (andygrove)\n- Create Comet docker file [#675](https://github.com/apache/datafusion-comet/pull/675) (comphead)\n- chore: Add microbenchmarks [#671](https://github.com/apache/datafusion-comet/pull/671) (andygrove)\n- build: Exclude protobug generated codes from apache-rat check [#683](https://github.com/apache/datafusion-comet/pull/683) (viirya)\n- chore: Disable abs and signum because they return incorrect results [#695](https://github.com/apache/datafusion-comet/pull/695) (andygrove)\n\n## Credits\n\nThank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.\n\n```\n 100\tLiang-Chi Hsieh\n 82\tAndy Grove\n 28\tadvancedxy\n 27\tChao Sun\n 14\tHuaxin Gao\n 11\tKAZUYUKI TANIMURA\n 9\tVipul Vaibhaw\n 8\tParth Chandra\n 7\tEmil Ejbyfeldt\n 7\tSteve Vaughan\n 7\tcomphead\n 4\tOleks V\n 4\tPablo Langa\n 4\tTrent Hauck\n 2\tEdmondo Porcu\n 2\tVrishabh\n 2\tXin Hao\n 2\tXuedong Luan\n 1\tAndrew Lamb\n 1\tBrian Vaughan\n 1\tCancai Cai\n 1\tEren Avsarogullari\n 1\tHolden Karau\n 1\tJC\n 1\tJunbo wang\n 1\tJunfan Zhang\n 1\tPedro M Duarte\n 1\tPrashant K. Sharma\n 1\tRickestCode\n 1\tRohit Rastogi\n 1\tRoman Zeyde\n 1\tSemyon\n 1\tSon\n 1\tSujith Jay Nair\n 1\tZhen Wang\n 1\tceppelli\n 1\tdependabot[bot]\n 1\tthexia\n 1\tvidyasankarv\n 1\twankun\n 1\tగణేష్\n```\n\nThank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.\n" }, { "path": "dev/changelog/0.10.0.md", "content": "\n\n# DataFusion Comet 0.10.0 Changelog\n\nThis release consists of 183 commits from 26 contributors. See credits at the end of this changelog for more information.\n\n**Fixed bugs:**\n\n- fix: [Iceberg] Fix decimal corruption [#1985](https://github.com/apache/datafusion-comet/pull/1985) (andygrove)\n- fix: broken link in development.md [#2024](https://github.com/apache/datafusion-comet/pull/2024) (petern48)\n- fix: [iceberg] Add LogicalTypeAnnotation in ParquetColumnSpec [#2000](https://github.com/apache/datafusion-comet/pull/2000) (huaxingao)\n- fix: hdfs read into buffer fully [#2031](https://github.com/apache/datafusion-comet/pull/2031) (parthchandra)\n- fix: Refactor arithmetic serde and fix correctness issues with EvalMode::TRY [#2018](https://github.com/apache/datafusion-comet/pull/2018) (andygrove)\n- fix: clean up [iceberg] integration APIs [#2032](https://github.com/apache/datafusion-comet/pull/2032) (huaxingao)\n- fix: zero Arrow Array offset before sending across FFI [#2052](https://github.com/apache/datafusion-comet/pull/2052) (mbutrovich)\n- fix: [iceberg] more fixes for Iceberg integration APIs. [#2078](https://github.com/apache/datafusion-comet/pull/2078) (parthchandra)\n- fix: Add support for StringDecode in Spark 4.0.0 [#2075](https://github.com/apache/datafusion-comet/pull/2075) (peter-toth)\n- fix: Avoid double free in CometUnifiedShuffleMemoryAllocator [#2122](https://github.com/apache/datafusion-comet/pull/2122) (andygrove)\n- fix: Remove duplicate serde code [#2098](https://github.com/apache/datafusion-comet/pull/2098) (andygrove)\n- fix: Improve logic for determining when an UnpackOrDeepCopy is needed [#2142](https://github.com/apache/datafusion-comet/pull/2142) (andygrove)\n- fix: Add CopyExec to inputs to SortMergeJoinExec [#2155](https://github.com/apache/datafusion-comet/pull/2155) (andygrove)\n- fix: Fix repeatedly url-decode path when reading parquet from s3 using native parquet reader [#2138](https://github.com/apache/datafusion-comet/pull/2138) (Kontinuation)\n- fix: [iceberg] Switch to OSS Spark and run Iceberg Spark tests in parallel [#1987](https://github.com/apache/datafusion-comet/pull/1987) (hsiang-c)\n- fix: [iceberg] Fall back to spark for schemas with empty structs [#2204](https://github.com/apache/datafusion-comet/pull/2204) (andygrove)\n- fix: Fix failing TPC-DS workflow in PR CI runs [#2207](https://github.com/apache/datafusion-comet/pull/2207) (andygrove)\n- fix: [iceberg] order query result deterministically [#2208](https://github.com/apache/datafusion-comet/pull/2208) (hsiang-c)\n- fix: use `spark.comet.batchSize` instead of `conf.arrowMaxRecordsPerBatch` for data that is coming from Java [#2196](https://github.com/apache/datafusion-comet/pull/2196) (rluvaton)\n- fix: if expr nullable [#2217](https://github.com/apache/datafusion-comet/pull/2217) (Asura7969)\n- fix: Support `auto` scan mode with Spark 4.0.0 [#1975](https://github.com/apache/datafusion-comet/pull/1975) (andygrove)\n- fix: Make Sha2 fallback message more user-friendly [#2213](https://github.com/apache/datafusion-comet/pull/2213) (rishvin)\n- fix: separate type checking for CometExchange and CometColumnarExchange [#2241](https://github.com/apache/datafusion-comet/pull/2241) (mbutrovich)\n- fix: Fix potential resource leak in native shuffle block reader [#2247](https://github.com/apache/datafusion-comet/pull/2247) (andygrove)\n- fix: Remove unreachable code in `CometScanRule` [#2252](https://github.com/apache/datafusion-comet/pull/2252) (andygrove)\n- fix: Fall back to `native_comet` for encrypted Parquet scans [#2250](https://github.com/apache/datafusion-comet/pull/2250) (andygrove)\n- fix: Fall back to `native_comet` when object store not supported by `native_iceberg_compat` [#2251](https://github.com/apache/datafusion-comet/pull/2251) (andygrove)\n- fix: split expr.proto file (new) [#2267](https://github.com/apache/datafusion-comet/pull/2267) (kination)\n- fix: handle cast to dictionary vector introduced by case when [#2044](https://github.com/apache/datafusion-comet/pull/2044) (parthchandra)\n- fix: Remove check for custom S3 endpoints [#2288](https://github.com/apache/datafusion-comet/pull/2288) (andygrove)\n- fix: implement lazy evaluation in Coalesce function [#2270](https://github.com/apache/datafusion-comet/pull/2270) (coderfender)\n- fix: Update benchmarking scripts [#2293](https://github.com/apache/datafusion-comet/pull/2293) (andygrove)\n- fix: Fix regression in NativeConfigSuite [#2299](https://github.com/apache/datafusion-comet/pull/2299) (andygrove)\n- fix: Validating object store configs should not throw exception [#2308](https://github.com/apache/datafusion-comet/pull/2308) (andygrove)\n- fix: TakeOrderedAndProjectExec is not reporting all fallback reasons [#2323](https://github.com/apache/datafusion-comet/pull/2323) (kazuyukitanimura)\n- fix: Fallback length function with binary input [#2349](https://github.com/apache/datafusion-comet/pull/2349) (wForget)\n\n**Performance related:**\n\n- perf: Optimize `AvgDecimalGroupsAccumulator` [#1893](https://github.com/apache/datafusion-comet/pull/1893) (leung-ming)\n- perf: Optimize `SumDecimalGroupsAccumulator::update_single` [#2069](https://github.com/apache/datafusion-comet/pull/2069) (leung-ming)\n- perf: Avoid FFI copy in `ScanExec` when reading data from exchanges [#2268](https://github.com/apache/datafusion-comet/pull/2268) (andygrove)\n\n**Implemented enhancements:**\n\n- feat: Add from_unixtime support [#1943](https://github.com/apache/datafusion-comet/pull/1943) (kazuyukitanimura)\n- feat: randn expression support [#2010](https://github.com/apache/datafusion-comet/pull/2010) (akupchinskiy)\n- feat: monotonically_increasing_id and spark_partition_id implementation [#2037](https://github.com/apache/datafusion-comet/pull/2037) (akupchinskiy)\n- feat: support `map_entries` [#2059](https://github.com/apache/datafusion-comet/pull/2059) (comphead)\n- feat: Support Array Literal [#2057](https://github.com/apache/datafusion-comet/pull/2057) (comphead)\n- feat: Add new trait for operator serde [#2115](https://github.com/apache/datafusion-comet/pull/2115) (andygrove)\n- feat: limit with offset support [#2070](https://github.com/apache/datafusion-comet/pull/2070) (akupchinskiy)\n- feat: Include scan implementation name in CometScan nodeName [#2141](https://github.com/apache/datafusion-comet/pull/2141) (andygrove)\n- feat: Add config option to log fallback reasons [#2154](https://github.com/apache/datafusion-comet/pull/2154) (andygrove)\n- feat: [iceberg] Enable Comet shuffle in Iceberg diff [#2205](https://github.com/apache/datafusion-comet/pull/2205) (andygrove)\n- feat: Improve shuffle fallback reporting [#2194](https://github.com/apache/datafusion-comet/pull/2194) (andygrove)\n- feat: Reset data buf of NativeBatchDecoderIterator on close [#2235](https://github.com/apache/datafusion-comet/pull/2235) (wForget)\n- feat: Improve fallback mechanism for ANSI mode [#2211](https://github.com/apache/datafusion-comet/pull/2211) (andygrove)\n- feat: Support hdfs with OpenDAL [#2244](https://github.com/apache/datafusion-comet/pull/2244) (wForget)\n- feat: Ignore fallback info for command execs [#2297](https://github.com/apache/datafusion-comet/pull/2297) (wForget)\n- feat: Improve some confusing fallback reasons [#2301](https://github.com/apache/datafusion-comet/pull/2301) (wForget)\n- feat: Make supported hadoop filesystem schemes configurable [#2272](https://github.com/apache/datafusion-comet/pull/2272) (wForget)\n- feat: [1941-Part1]: Introduce map-sort scalar function [#2262](https://github.com/apache/datafusion-comet/pull/2262) (rishvin)\n- feat: [iceberg] delete rows support using selection vectors [#2346](https://github.com/apache/datafusion-comet/pull/2346) (parthchandra)\n\n**Documentation updates:**\n\n- docs: Update benchmark results for 0.9.0 [#1959](https://github.com/apache/datafusion-comet/pull/1959) (andygrove)\n- doc: Add comment about local clippy run before submitting a pull request [#1961](https://github.com/apache/datafusion-comet/pull/1961) (akupchinskiy)\n- docs: Minor improvements to Spark SQL test docs [#1980](https://github.com/apache/datafusion-comet/pull/1980) (andygrove)\n- docs: Update Maven links for 0.9.0 release [#1988](https://github.com/apache/datafusion-comet/pull/1988) (andygrove)\n- docs: Documentation updates for 0.9.0 release [#1981](https://github.com/apache/datafusion-comet/pull/1981) (andygrove)\n- docs: Add guide showing comparison between Comet and Gluten [#2012](https://github.com/apache/datafusion-comet/pull/2012) (andygrove)\n- docs: Remove legacy comment in docs [#2022](https://github.com/apache/datafusion-comet/pull/2022) (andygrove)\n- docs: Update Gluten comparision to clarify that Velox is open-source [#2043](https://github.com/apache/datafusion-comet/pull/2043) (andygrove)\n- docs: Improve Gluten comparison based on feedback from the community [#2048](https://github.com/apache/datafusion-comet/pull/2048) (andygrove)\n- docs: added a missing export into the plan stability section [#2071](https://github.com/apache/datafusion-comet/pull/2071) (akupchinskiy)\n- doc: Added documentation for supported map functions [#2074](https://github.com/apache/datafusion-comet/pull/2074) (codetyri0n)\n- doc: Alternative way to start Spark Master to run benchmarks [#2072](https://github.com/apache/datafusion-comet/pull/2072) (comphead)\n- docs: Update to support try arithmetic functions [#2143](https://github.com/apache/datafusion-comet/pull/2143) (coderfender)\n- doc: update macos standalone spark start instructions [#2103](https://github.com/apache/datafusion-comet/pull/2103) (comphead)\n- docs: Update confs to bypass Iceberg Spark issues [#2166](https://github.com/apache/datafusion-comet/pull/2166) (hsiang-c)\n- docs: Add Roadmap [#2191](https://github.com/apache/datafusion-comet/pull/2191) (andygrove)\n- docs: Update installation guide for 0.9.1 [#2230](https://github.com/apache/datafusion-comet/pull/2230) (andygrove)\n- docs: Publish version-specific user guides [#2269](https://github.com/apache/datafusion-comet/pull/2269) (andygrove)\n- docs: Fix issues with publishing user guide for older Comet versions [#2284](https://github.com/apache/datafusion-comet/pull/2284) (andygrove)\n- docs: Move user guide docs into /user-guide/latest [#2318](https://github.com/apache/datafusion-comet/pull/2318) (andygrove)\n- docs: Add manual redirects from old pages that no longer exist [#2317](https://github.com/apache/datafusion-comet/pull/2317) (andygrove)\n- docs: Fix broken links and other Sphinx warnings [#2320](https://github.com/apache/datafusion-comet/pull/2320) (andygrove)\n- docs: Use `sphinx-reredirects` for redirects [#2324](https://github.com/apache/datafusion-comet/pull/2324) (andygrove)\n- docs: Add note about Root CA Certificate location with native scans [#2325](https://github.com/apache/datafusion-comet/pull/2325) (andygrove)\n- docs: Stop hard-coding Comet version in docs [#2326](https://github.com/apache/datafusion-comet/pull/2326) (andygrove)\n- docs: Update supported expressions and operators in user guide [#2327](https://github.com/apache/datafusion-comet/pull/2327) (andygrove)\n- docs: Update Iceberg docs for 0.10.0 release [#2355](https://github.com/apache/datafusion-comet/pull/2355) (hsiang-c)\n\n**Other:**\n\n- chore: Start 0.10.0 development [#1958](https://github.com/apache/datafusion-comet/pull/1958) (andygrove)\n- build: Fix release dockerfile [#1960](https://github.com/apache/datafusion-comet/pull/1960) (andygrove)\n- test: Run Iceberg Spark tests only when PR title contains [iceberg] [#1976](https://github.com/apache/datafusion-comet/pull/1976) (hsiang-c)\n- chore: Reuse comet allocator [#1973](https://github.com/apache/datafusion-comet/pull/1973) (EmilyMatt)\n- chore: update `CopyExec` with `maintains_input_order`, `supports_limit_pushdown` and `cardinality_effect` [#1979](https://github.com/apache/datafusion-comet/pull/1979) (rluvaton)\n- chore: extract CreateArray from QueryPlanSerde [#1991](https://github.com/apache/datafusion-comet/pull/1991) (tglanz)\n- chore: use DF scalar functions for StartsWith, EndsWith, Contains, DF LikeExpr [#1887](https://github.com/apache/datafusion-comet/pull/1887) (mbutrovich)\n- refactor: standardize div_ceil [#1999](https://github.com/apache/datafusion-comet/pull/1999) (tglanz)\n- Feat: support map_from_arrays [#1932](https://github.com/apache/datafusion-comet/pull/1932) (kazantsev-maksim)\n- chore: Implement BloomFilterMightContain as a ScalarUDFImpl [#1954](https://github.com/apache/datafusion-comet/pull/1954) (tglanz)\n- chore: Drop support for RightSemi and RightAnti join types [#1935](https://github.com/apache/datafusion-comet/pull/1935) (dharanad)\n- minor: Refactor to reduce duplicate serde code [#2011](https://github.com/apache/datafusion-comet/pull/2011) (andygrove)\n- chore: Introduce ANSI support for remainder operation [#1971](https://github.com/apache/datafusion-comet/pull/1971) (rishvin)\n- chore: Improve process for generating dynamic content into documentation [#2017](https://github.com/apache/datafusion-comet/pull/2017) (andygrove)\n- minor: Refactor to move some shuffle-related logic from `QueryPlanSerde` to `CometExecRule` [#2015](https://github.com/apache/datafusion-comet/pull/2015) (andygrove)\n- chore: Add benchmarking scripts [#2025](https://github.com/apache/datafusion-comet/pull/2025) (andygrove)\n- chore: Add scripts for running benchmark based on TPC-DS [#2042](https://github.com/apache/datafusion-comet/pull/2042) (andygrove)\n- Chore: Improve array contains test coverage [#2030](https://github.com/apache/datafusion-comet/pull/2030) (kazantsev-maksim)\n- fix : cast_operands_to_decimal_type_to_fix_arithmetic_overflow [#1996](https://github.com/apache/datafusion-comet/pull/1996) (coderfender)\n- chore: Add scripts for running benchmarks with Blaze [#2050](https://github.com/apache/datafusion-comet/pull/2050) (andygrove)\n- chore: migrate to DF 49.0.0 [#2040](https://github.com/apache/datafusion-comet/pull/2040) (comphead)\n- chore: Refactor aggregate serde to be consistent with other expression serde [#2055](https://github.com/apache/datafusion-comet/pull/2055) (andygrove)\n- Chore: implement string_space as ScalarUDFImpl [#2041](https://github.com/apache/datafusion-comet/pull/2041) (kazantsev-maksim)\n- docs : Change notes for `IntegralDivide` [#2054](https://github.com/apache/datafusion-comet/pull/2054) (coderfender)\n- Chore: refactor Comparison out of QueryPlanSerde [#2028](https://github.com/apache/datafusion-comet/pull/2028) (CuteChuanChuan)\n- chore: Use Datafusion's Sha2 and remove Comet's implementation. [#2063](https://github.com/apache/datafusion-comet/pull/2063) (rishvin)\n- chore: Adding dependabot [#2076](https://github.com/apache/datafusion-comet/pull/2076) (comphead)\n- chore: Fix clippy issues for Rust 1.89.0 [#2082](https://github.com/apache/datafusion-comet/pull/2082) (andygrove)\n- chore: Refactor string expression serde, part 1 [#2068](https://github.com/apache/datafusion-comet/pull/2068) (andygrove)\n- chore: Use `chr` function from datafusion-spark [#2080](https://github.com/apache/datafusion-comet/pull/2080) (andygrove)\n- minor: CometBuffer code cleanup [#2090](https://github.com/apache/datafusion-comet/pull/2090) (andygrove)\n- chore: Refactor string expression serde, part 2 [#2097](https://github.com/apache/datafusion-comet/pull/2097) (andygrove)\n- chore: create copy of fs-hdfs [#2062](https://github.com/apache/datafusion-comet/pull/2062) (parthchandra)\n- Chore: refactor datetime related expressions out of QueryPlanSerde [#2085](https://github.com/apache/datafusion-comet/pull/2085) (CuteChuanChuan)\n- chore(deps): bump actions/checkout from 3 to 4 [#2104](https://github.com/apache/datafusion-comet/pull/2104) (dependabot[bot])\n- chore(deps): bump libc from 0.2.174 to 0.2.175 in /native [#2107](https://github.com/apache/datafusion-comet/pull/2107) (dependabot[bot])\n- chore(deps): bump assertables from 9.8.1 to 9.8.2 in /native [#2108](https://github.com/apache/datafusion-comet/pull/2108) (dependabot[bot])\n- chore: Update dependabot label [#2110](https://github.com/apache/datafusion-comet/pull/2110) (mbutrovich)\n- chore: Move `stringDecode()` to `CommonStringExprs` trait [#2111](https://github.com/apache/datafusion-comet/pull/2111) (peter-toth)\n- chore(deps): bump uuid from 0.8.2 to 1.17.0 in /native [#2106](https://github.com/apache/datafusion-comet/pull/2106) (dependabot[bot])\n- chore(deps): bump actions/download-artifact from 4 to 5 [#2109](https://github.com/apache/datafusion-comet/pull/2109) (dependabot[bot])\n- chore(deps): bump tokio from 1.47.0 to 1.47.1 in /native [#2112](https://github.com/apache/datafusion-comet/pull/2112) (dependabot[bot])\n- chore(deps): bump actions/setup-java from 3 to 4 [#2105](https://github.com/apache/datafusion-comet/pull/2105) (dependabot[bot])\n- chore(deps): bump the proto group in /native with 2 updates [#2113](https://github.com/apache/datafusion-comet/pull/2113) (dependabot[bot])\n- chore: Add type parameter to `CometExpressionSerde` [#2114](https://github.com/apache/datafusion-comet/pull/2114) (peter-toth)\n- chore(deps): bump cc from 1.2.30 to 1.2.32 in /native [#2123](https://github.com/apache/datafusion-comet/pull/2123) (dependabot[bot])\n- chore(deps): bump bindgen from 0.64.0 to 0.69.5 in /native [#2124](https://github.com/apache/datafusion-comet/pull/2124) (dependabot[bot])\n- chore(deps): bump aws-credential-types from 1.2.4 to 1.2.5 in /native [#2125](https://github.com/apache/datafusion-comet/pull/2125) (dependabot[bot])\n- chore(deps): bump actions/checkout from 4 to 5 [#2126](https://github.com/apache/datafusion-comet/pull/2126) (dependabot[bot])\n- chore: fix `QueryPlanSerde` merge error [#2127](https://github.com/apache/datafusion-comet/pull/2127) (comphead)\n- chore(deps): bump slab from 0.4.10 to 0.4.11 in /native [#2128](https://github.com/apache/datafusion-comet/pull/2128) (dependabot[bot])\n- fix : implement_try_eval_mode_arithmetic [#2073](https://github.com/apache/datafusion-comet/pull/2073) (coderfender)\n- chore: Simplify approach to avoiding memory corruption due to buffer reuse [#2156](https://github.com/apache/datafusion-comet/pull/2156) (andygrove)\n- chore: upgrade to DataFusion 49.0.1 [#2077](https://github.com/apache/datafusion-comet/pull/2077) (mbutrovich)\n- chore: CometExecRule code cleanup [#2159](https://github.com/apache/datafusion-comet/pull/2159) (andygrove)\n- chore: Update `CometTestBase` to stop setting the scan implementation to `native_comet` [#2176](https://github.com/apache/datafusion-comet/pull/2176) (andygrove)\n- trivial: remove unnecessary clone() [#2066](https://github.com/apache/datafusion-comet/pull/2066) (isimluk)\n- chore: Pass Spark configs to native `createPlan` [#2180](https://github.com/apache/datafusion-comet/pull/2180) (andygrove)\n- (feat) add support for ArrayMin scalar function [#1944](https://github.com/apache/datafusion-comet/pull/1944) (dharanad)\n- chore: Upgrade to 49.0.2 [#2223](https://github.com/apache/datafusion-comet/pull/2223) (comphead)\n- chore(deps): bump bindgen from 0.69.5 to 0.72.0 in /native [#2222](https://github.com/apache/datafusion-comet/pull/2222) (dependabot[bot])\n- chore: move Round serde into object [#2237](https://github.com/apache/datafusion-comet/pull/2237) (andygrove)\n- chore: Improve expression fallback reporting [#2240](https://github.com/apache/datafusion-comet/pull/2240) (andygrove)\n- chore: Update stability suite to use `auto` scan instead of `native_comet` [#2178](https://github.com/apache/datafusion-comet/pull/2178) (andygrove)\n- chore: Improve documentation for `CometBatchIterator` and fix a potential issue [#2168](https://github.com/apache/datafusion-comet/pull/2168) (andygrove)\n- chore: Fix `array_intersect` test [#2246](https://github.com/apache/datafusion-comet/pull/2246) (comphead)\n- chore(deps): bump actions/checkout from 4 to 5 [#2229](https://github.com/apache/datafusion-comet/pull/2229) (dependabot[bot])\n- chore(deps): bump actions/setup-java from 4 to 5 [#2225](https://github.com/apache/datafusion-comet/pull/2225) (dependabot[bot])\n- chore: Introduce `strict-warning` profile for Scala [#2254](https://github.com/apache/datafusion-comet/pull/2254) (comphead)\n- chore: fix struct to string test for `native_iceberg_compat` [#2253](https://github.com/apache/datafusion-comet/pull/2253) (comphead)\n- chore: Add type parameter to CometAggregateExpressionSerde [#2249](https://github.com/apache/datafusion-comet/pull/2249) (andygrove)\n- Feat: Impl array flatten func [#2039](https://github.com/apache/datafusion-comet/pull/2039) (kazantsev-maksim)\n- Chore: Refactor serde for math expressions [#2259](https://github.com/apache/datafusion-comet/pull/2259) (kazantsev-maksim)\n- chore: Refactor serde for more array and struct expressions [#2257](https://github.com/apache/datafusion-comet/pull/2257) (andygrove)\n- chore: Refactor remaining predicate expression serde [#2265](https://github.com/apache/datafusion-comet/pull/2265) (andygrove)\n- chore(deps): bump procfs from 0.17.0 to 0.18.0 in /native [#2278](https://github.com/apache/datafusion-comet/pull/2278) (dependabot[bot])\n- chore(deps): bump cc from 1.2.34 to 1.2.35 in /native [#2277](https://github.com/apache/datafusion-comet/pull/2277) (dependabot[bot])\n- chore(deps): bump bindgen from 0.72.0 to 0.72.1 in /native [#2274](https://github.com/apache/datafusion-comet/pull/2274) (dependabot[bot])\n- chore(deps): bump aws-credential-types from 1.2.5 to 1.2.6 in /native [#2275](https://github.com/apache/datafusion-comet/pull/2275) (dependabot[bot])\n- minor: Remove useless ENABLE_COMET_SHUFFLE env [#2280](https://github.com/apache/datafusion-comet/pull/2280) (wForget)\n- chore: Refactor serde for conditional expressions [#2266](https://github.com/apache/datafusion-comet/pull/2266) (andygrove)\n- chore(deps): bump mimalloc from 0.1.47 to 0.1.48 in /native [#2276](https://github.com/apache/datafusion-comet/pull/2276) (dependabot[bot])\n- chore: docker publish and docs build only for apache repo [#2289](https://github.com/apache/datafusion-comet/pull/2289) (wForget)\n- minor: Reduce misleading fallback warnings [#2283](https://github.com/apache/datafusion-comet/pull/2283) (andygrove)\n- chore: Refactor `Cast` serde to avoid code duplication [#2242](https://github.com/apache/datafusion-comet/pull/2242) (andygrove)\n- chore: Refactor `hex`/`unhex` SerDe to avoid code duplication [#2287](https://github.com/apache/datafusion-comet/pull/2287) (hsiang-c)\n- minor: Improve exception message for unimplemented CometVector methods [#2291](https://github.com/apache/datafusion-comet/pull/2291) (andygrove)\n- chore: Align sort constraints w/ `arrow-rs` [#2279](https://github.com/apache/datafusion-comet/pull/2279) (hsiang-c)\n- chore: Collect fallback reasons for spark sql tests [#2313](https://github.com/apache/datafusion-comet/pull/2313) (wForget)\n- chore: Refactor serde for named expressions `alias` and `attributeReference` [#2290](https://github.com/apache/datafusion-comet/pull/2290) (andygrove)\n- chore(deps): bump log4rs from 1.3.0 to 1.4.0 in /native [#2334](https://github.com/apache/datafusion-comet/pull/2334) (dependabot[bot])\n- chore(deps): bump twox-hash from 2.1.1 to 2.1.2 in /native [#2335](https://github.com/apache/datafusion-comet/pull/2335) (dependabot[bot])\n- chore(deps): bump actions/setup-python from 5 to 6 [#2331](https://github.com/apache/datafusion-comet/pull/2331) (dependabot[bot])\n- chore(deps): bump actions/download-artifact from 4 to 5 [#2332](https://github.com/apache/datafusion-comet/pull/2332) (dependabot[bot])\n- chore(deps): bump cc from 1.2.35 to 1.2.36 in /native [#2337](https://github.com/apache/datafusion-comet/pull/2337) (dependabot[bot])\n- chore(deps): bump log from 0.4.27 to 0.4.28 in /native [#2333](https://github.com/apache/datafusion-comet/pull/2333) (dependabot[bot])\n- build: Specify SPARK_LOCAL_HOSTNAME to fix CI failures [#2353](https://github.com/apache/datafusion-comet/pull/2353) (andygrove)\n- chore: [branch-0.10] Bump version to 0.10.0 [#2356](https://github.com/apache/datafusion-comet/pull/2356) (andygrove)\n\n## Credits\n\nThank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.\n\n```\n 75\tAndy Grove\n 27\tdependabot[bot]\n 11\tOleks V\n 9\tZhen Wang\n 7\thsiang-c\n 5\tArtem Kupchinskiy\n 5\tB Vadlamani\n 5\tKazantsev Maksim\n 5\tMatt Butrovich\n 5\tParth Chandra\n 4\tRishab Joshi\n 3\tPeter Toth\n 3\tTal Glanzman\n 2\tDharan Aditya\n 2\tHuaxin Gao\n 2\tKAZUYUKI TANIMURA\n 2\tLeung Ming\n 2\tRaz Luvaton\n 2\tYu-Chuan Hung\n 1\tAsura7969\n 1\tEmily Matheys\n 1\tK.I. (Dennis) Jung\n 1\tKristin Cowalcijk\n 1\tPeter Nguyen\n 1\tcodetyri0n\n 1\tŠimon Lukašík\n```\n\nThank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.\n" }, { "path": "dev/changelog/0.11.0.md", "content": "\n\n# DataFusion Comet 0.11.0 Changelog\n\nThis release consists of 131 commits from 15 contributors. See credits at the end of this changelog for more information.\n\n**Fixed bugs:**\n\n- fix: temporarily ignore test for hdfs file systems [#2359](https://github.com/apache/datafusion-comet/pull/2359) (parthchandra)\n- fix: Check reused broadcast plan in non-AQE and make setNumPartitions thread safe [#2398](https://github.com/apache/datafusion-comet/pull/2398) (wForget)\n- fix: correct `missingInput` for `CometHashAggregateExec` [#2409](https://github.com/apache/datafusion-comet/pull/2409) (comphead)\n- fix:clippy errros rust 1.9.0 update [#2419](https://github.com/apache/datafusion-comet/pull/2419) (coderfender)\n- fix: Avoid spark plan execution cache preventing CometBatchRDD numPartitions change [#2420](https://github.com/apache/datafusion-comet/pull/2420) (wForget)\n- fix: regressions in `CometToPrettyStringSuite` [#2384](https://github.com/apache/datafusion-comet/pull/2384) (hsiang-c)\n- fix: Byte array Literals failed on cast [#2432](https://github.com/apache/datafusion-comet/pull/2432) (comphead)\n- fix: Do not push down subquery filters on native_datafusion scan [#2438](https://github.com/apache/datafusion-comet/pull/2438) (wForget)\n- fix: Improve error handling when resolving S3 bucket region [#2440](https://github.com/apache/datafusion-comet/pull/2440) (andygrove)\n- fix: [iceberg] additional parquet independent api for iceberg integration [#2442](https://github.com/apache/datafusion-comet/pull/2442) (parthchandra)\n- fix: Specify reqwest crate features [#2446](https://github.com/apache/datafusion-comet/pull/2446) (andygrove)\n- fix: distributed RangePartitioning bounds calculation with native shuffle [#2258](https://github.com/apache/datafusion-comet/pull/2258) (mbutrovich)\n- fix: fix regression in tpcbench.py [#2512](https://github.com/apache/datafusion-comet/pull/2512) (andygrove)\n- fix: [iceberg] Close reader instance in ReadConf [#2510](https://github.com/apache/datafusion-comet/pull/2510) (hsiang-c)\n- fix: Enable plan stability tests for `auto` scan [#2516](https://github.com/apache/datafusion-comet/pull/2516) (andygrove)\n- fix: Capture unexpected output when retrieving JVM 17 args in Makefile [#2566](https://github.com/apache/datafusion-comet/pull/2566) (zuston)\n\n**Performance related:**\n\n- perf: New Configuration from shared conf to avoid high costs [#2402](https://github.com/apache/datafusion-comet/pull/2402) (wForget)\n- perf: Use DataFusion's `count_udaf` instead of `SUM(IF(expr IS NOT NULL, 1, 0))` [#2407](https://github.com/apache/datafusion-comet/pull/2407) (andygrove)\n- perf: Improve BroadcastExchangeExec conversion [#2417](https://github.com/apache/datafusion-comet/pull/2417) (wForget)\n\n**Implemented enhancements:**\n\n- feat: Add dynamic `enabled` and `allowIncompat` configs for all supported expressions [#2329](https://github.com/apache/datafusion-comet/pull/2329) (andygrove)\n- feat: feature specific tests [#2372](https://github.com/apache/datafusion-comet/pull/2372) (parthchandra)\n- feat: Support more date part expressions [#2316](https://github.com/apache/datafusion-comet/pull/2316) (wForget)\n- feat: rpad support column for second arg instead of just literal [#2099](https://github.com/apache/datafusion-comet/pull/2099) (coderfender)\n- feat: Support comet native log level conf [#2379](https://github.com/apache/datafusion-comet/pull/2379) (wForget)\n- feat: Enable WeekDay function [#2411](https://github.com/apache/datafusion-comet/pull/2411) (wForget)\n- feat: Add nested Array literal support [#2181](https://github.com/apache/datafusion-comet/pull/2181) (comphead)\n- feat:add_additional_char_support_rpad [#2436](https://github.com/apache/datafusion-comet/pull/2436) (coderfender)\n- feat: do not fallback to Spark for `COUNT(distinct)` [#2429](https://github.com/apache/datafusion-comet/pull/2429) (comphead)\n- feat: implement_ansi_eval_mode_arithmetic [#2136](https://github.com/apache/datafusion-comet/pull/2136) (coderfender)\n- feat: Add plan conversion statistics to extended explain info [#2412](https://github.com/apache/datafusion-comet/pull/2412) (andygrove)\n- feat: implement_comet_native_lpad_expr [#2102](https://github.com/apache/datafusion-comet/pull/2102) (coderfender)\n- feat: Add `backtrace` feature to simplify enabling native backtraces in `CometNativeException` [#2515](https://github.com/apache/datafusion-comet/pull/2515) (andygrove)\n- feat: Support reverse function with ArrayType input [#2481](https://github.com/apache/datafusion-comet/pull/2481) (cfmcgrady)\n- feat: Change default off-heap memory pool from `greedy_unified` to `fair_unified` [#2526](https://github.com/apache/datafusion-comet/pull/2526) (andygrove)\n- feat: Make DiskManager `max_temp_directory_size` configurable [#2479](https://github.com/apache/datafusion-comet/pull/2479) (manuzhang)\n- feat: Parquet Modular Encryption with Spark KMS for native readers [#2447](https://github.com/apache/datafusion-comet/pull/2447) (mbutrovich)\n- feat: Add support for Spark-compatible cast from integral to decimal [#2472](https://github.com/apache/datafusion-comet/pull/2472) (coderfender)\n- feat:Support ANSI mode integral divide [#2421](https://github.com/apache/datafusion-comet/pull/2421) (coderfender)\n- feat: Add config to enable running Comet in onheap mode [#2554](https://github.com/apache/datafusion-comet/pull/2554) (andygrove)\n- feat:support ansi mode rounding function [#2542](https://github.com/apache/datafusion-comet/pull/2542) (coderfender)\n- feat:support ansi mode remainder function [#2556](https://github.com/apache/datafusion-comet/pull/2556) (coderfender)\n- feat: Implement array-to-string cast support [#2425](https://github.com/apache/datafusion-comet/pull/2425) (cfmcgrady)\n- feat: Various improvements to memory pool configuration, logging, and documentation [#2538](https://github.com/apache/datafusion-comet/pull/2538) (andygrove)\n- feat: Enable complex types for columnar shuffle [#2573](https://github.com/apache/datafusion-comet/pull/2573) (mbutrovich)\n- feat: support_decimal_types_bool_cast_native_impl [#2490](https://github.com/apache/datafusion-comet/pull/2490) (coderfender)\n- feat: Use buf write to reduce system call on index write [#2579](https://github.com/apache/datafusion-comet/pull/2579) (zuston)\n\n**Documentation updates:**\n\n- doc: Document usage IcebergCometBatchReader.java [#2347](https://github.com/apache/datafusion-comet/pull/2347) (comphead)\n- docs: Add changelog for 0.10.0 release [#2361](https://github.com/apache/datafusion-comet/pull/2361) (andygrove)\n- docs: Fix error in docs [#2373](https://github.com/apache/datafusion-comet/pull/2373) (andygrove)\n- docs: Fix more comet versions in docs [#2374](https://github.com/apache/datafusion-comet/pull/2374) (andygrove)\n- docs: Publish 0.10.0 user guide [#2394](https://github.com/apache/datafusion-comet/pull/2394) (andygrove)\n- doc: macos benches doc clarifications [#2418](https://github.com/apache/datafusion-comet/pull/2418) (comphead)\n- docs: update configs.md after #2422 [#2428](https://github.com/apache/datafusion-comet/pull/2428) (mbutrovich)\n- docs: update docs and tuning guide related to native shuffle [#2487](https://github.com/apache/datafusion-comet/pull/2487) (mbutrovich)\n- docs: Improve EC2 benchmarking guide [#2474](https://github.com/apache/datafusion-comet/pull/2474) (andygrove)\n- docs: docs_update_ansi_support [#2496](https://github.com/apache/datafusion-comet/pull/2496) (coderfender)\n- docs:support lpad expression documentation update [#2517](https://github.com/apache/datafusion-comet/pull/2517) (coderfender)\n- docs: doc changes to support ANSI mode integral divide [#2570](https://github.com/apache/datafusion-comet/pull/2570) (coderfender)\n- docs: Split configuration guide into different sections (scan, exec, shuffle, etc) [#2568](https://github.com/apache/datafusion-comet/pull/2568) (andygrove)\n- docs: doc update to support ANSI mode remainder function [#2576](https://github.com/apache/datafusion-comet/pull/2576) (coderfender)\n- docs: Documentation updates [#2581](https://github.com/apache/datafusion-comet/pull/2581) (andygrove)\n\n**Other:**\n\n- chore(deps): bump uuid from 1.18.0 to 1.18.1 in /native [#2336](https://github.com/apache/datafusion-comet/pull/2336) (dependabot[bot])\n- build: Check that all Scala test suites run in PR builds [#2304](https://github.com/apache/datafusion-comet/pull/2304) (andygrove)\n- chore: Start 0.11.0 development [#2365](https://github.com/apache/datafusion-comet/pull/2365) (andygrove)\n- chore: Split expression serde hash map into separate categories [#2322](https://github.com/apache/datafusion-comet/pull/2322) (andygrove)\n- chore: exclude Iceberg diffs from rat checks [#2376](https://github.com/apache/datafusion-comet/pull/2376) (hsiang-c)\n- chore: Refactor UnaryMinus serde [#2378](https://github.com/apache/datafusion-comet/pull/2378) (andygrove)\n- chore: Revert \"chore: [1941-Part1]: Introduce `map_sort` scalar function (#2… [#2381](https://github.com/apache/datafusion-comet/pull/2381) (comphead)\n- chore: Refactor Literal serde [#2377](https://github.com/apache/datafusion-comet/pull/2377) (andygrove)\n- chore: Output `BaseAggregateExec` accurate unsupported names [#2383](https://github.com/apache/datafusion-comet/pull/2383) (comphead)\n- chore: Improve Initcap test and docs [#2387](https://github.com/apache/datafusion-comet/pull/2387) (andygrove)\n- build: fix build of 'hdfs-opendal' feature for MacOS [#2392](https://github.com/apache/datafusion-comet/pull/2392) (parthchandra)\n- chore(deps): bump cc from 1.2.36 to 1.2.37 in /native [#2399](https://github.com/apache/datafusion-comet/pull/2399) (dependabot[bot])\n- chore: [iceberg] support Iceberg 1.9.1 [#2386](https://github.com/apache/datafusion-comet/pull/2386) (hsiang-c)\n- minor: Add deprecation notice to `datafusion-comet-spark-expr` crate [#2405](https://github.com/apache/datafusion-comet/pull/2405) (andygrove)\n- minor: Update benchmarking scripts to specify scan implementation [#2403](https://github.com/apache/datafusion-comet/pull/2403) (andygrove)\n- refactor: Scala hygiene - remove `scala.collection.JavaConverters` [#2393](https://github.com/apache/datafusion-comet/pull/2393) (hsiang-c)\n- chore: Improve test coverage for `count` aggregates [#2406](https://github.com/apache/datafusion-comet/pull/2406) (andygrove)\n- chore: upgrade to DataFusion 50.0.0, Arrow 56.1.0, Parquet 56.0.0 among others [#2286](https://github.com/apache/datafusion-comet/pull/2286) (mbutrovich)\n- chore: Support Spark 4.0.1 instead of 4.0.0 [#2414](https://github.com/apache/datafusion-comet/pull/2414) (andygrove)\n- chore: Respect native features env for cargo commands [#2296](https://github.com/apache/datafusion-comet/pull/2296) (wForget)\n- minor: Update TPC-DS microbenchmarks to remove \"scan only\" and \"exec only\" runs [#2396](https://github.com/apache/datafusion-comet/pull/2396) (andygrove)\n- minor: Add RDDScan to default value of sparkToColumnar.supportedOperatorList [#2422](https://github.com/apache/datafusion-comet/pull/2422) (wForget)\n- chore: new TPC-DS golden plans [#2426](https://github.com/apache/datafusion-comet/pull/2426) (mbutrovich)\n- chore: fix `pr_build*.yml` [#2434](https://github.com/apache/datafusion-comet/pull/2434) (comphead)\n- chore: Remove unused class [#2437](https://github.com/apache/datafusion-comet/pull/2437) (wForget)\n- chore(deps): bump cc from 1.2.37 to 1.2.38 in /native [#2439](https://github.com/apache/datafusion-comet/pull/2439) (dependabot[bot])\n- chore: add validate_workflows.yml [#2441](https://github.com/apache/datafusion-comet/pull/2441) (comphead)\n- test: potential native broadcast failure in scenarios with ReusedExhange [#2167](https://github.com/apache/datafusion-comet/pull/2167) (akupchinskiy)\n- chore: Improvements of fallback info [#2450](https://github.com/apache/datafusion-comet/pull/2450) (wForget)\n- chore: Upgrade Apache Release Audit Tool (RAT) to 0.16.1 [#2451](https://github.com/apache/datafusion-comet/pull/2451) (andygrove)\n- minor: Remove reference to SortExec deadlock issue that is now resolved [#2464](https://github.com/apache/datafusion-comet/pull/2464) (andygrove)\n- chore: Use checked operations when growing or shrinking unified memory pool [#2455](https://github.com/apache/datafusion-comet/pull/2455) (andygrove)\n- minor: Improve the log message of `CometTestBase#checkCometOperators` [#2458](https://github.com/apache/datafusion-comet/pull/2458) (cfmcgrady)\n- minor: Skip calculating per-task memory limit when in off-heap mode [#2462](https://github.com/apache/datafusion-comet/pull/2462) (andygrove)\n- Chore: Used DataFusion impl of bit_get function [#2466](https://github.com/apache/datafusion-comet/pull/2466) (kazantsev-maksim)\n- chore(deps): bump regex from 1.11.2 to 1.11.3 in /native [#2483](https://github.com/apache/datafusion-comet/pull/2483) (dependabot[bot])\n- chore: update TPS-DS plans after #2429 [#2486](https://github.com/apache/datafusion-comet/pull/2486) (mbutrovich)\n- chore(deps): bump thiserror from 2.0.16 to 2.0.17 in /native [#2485](https://github.com/apache/datafusion-comet/pull/2485) (dependabot[bot])\n- chore(deps): bump cc from 1.2.38 to 1.2.39 in /native [#2484](https://github.com/apache/datafusion-comet/pull/2484) (dependabot[bot])\n- chore: Support running specific benchmark query [#2491](https://github.com/apache/datafusion-comet/pull/2491) (comphead)\n- chore: Make CometColumnarToRowExec extends CometPlan [#2460](https://github.com/apache/datafusion-comet/pull/2460) (wForget)\n- chore: Update artifacts to 0.10.0 [#2500](https://github.com/apache/datafusion-comet/pull/2500) (comphead)\n- build: Stop caching libcomet in CI [#2498](https://github.com/apache/datafusion-comet/pull/2498) (andygrove)\n- chore: Upgrade Maven plugins [#2494](https://github.com/apache/datafusion-comet/pull/2494) (andygrove)\n- Chore: Used DataFusion impl of date_add and date_sub functions [#2473](https://github.com/apache/datafusion-comet/pull/2473) (kazantsev-maksim)\n- minor: include taskAttemptId in log messages [#2467](https://github.com/apache/datafusion-comet/pull/2467) (andygrove)\n- chore: Improve test assertions in plan stability suite [#2505](https://github.com/apache/datafusion-comet/pull/2505) (andygrove)\n- build: Add Spark 4.0 to release build script [#2514](https://github.com/apache/datafusion-comet/pull/2514) (parthchandra)\n- chore: Enable plan stability tests for `native_iceberg_compat` [#2519](https://github.com/apache/datafusion-comet/pull/2519) (andygrove)\n- chore(deps): bump parking_lot from 0.12.4 to 0.12.5 in /native [#2530](https://github.com/apache/datafusion-comet/pull/2530) (dependabot[bot])\n- chore(deps): bump cc from 1.2.39 to 1.2.40 in /native [#2529](https://github.com/apache/datafusion-comet/pull/2529) (dependabot[bot])\n- chore: Refactor serde for `ArrayCompact` and `ArrayFilter` [#2536](https://github.com/apache/datafusion-comet/pull/2536) (andygrove)\n- Chore: Fix Scala code warnings - common module [#2527](https://github.com/apache/datafusion-comet/pull/2527) (andy-hf-kwok)\n- chore: Refactor serde for `CheckOverflow` [#2537](https://github.com/apache/datafusion-comet/pull/2537) (andygrove)\n- build: Run scala tests against release build of native code [#2541](https://github.com/apache/datafusion-comet/pull/2541) (andygrove)\n- chore: Pass Comet configs to native `createPlan` [#2543](https://github.com/apache/datafusion-comet/pull/2543) (andygrove)\n- chore: Refactor serde for Length [#2547](https://github.com/apache/datafusion-comet/pull/2547) (andygrove)\n- chore: Include spark shim sources for spotless plugin and reformat [#2557](https://github.com/apache/datafusion-comet/pull/2557) (wForget)\n- chore(deps): bump opendal from 0.54.0 to 0.54.1 in /native [#2559](https://github.com/apache/datafusion-comet/pull/2559) (dependabot[bot])\n- chore: Finish moving Cast serde out of QueryPlanSerde [#2550](https://github.com/apache/datafusion-comet/pull/2550) (andygrove)\n- chore: Use cargo-nextest in CI [#2546](https://github.com/apache/datafusion-comet/pull/2546) (andygrove)\n- chore: Delete unused code [#2565](https://github.com/apache/datafusion-comet/pull/2565) (zuston)\n- chore: Improve plan comet transformation log [#2564](https://github.com/apache/datafusion-comet/pull/2564) (wForget)\n- chore(deps): bump cc from 1.2.40 to 1.2.41 in /native [#2560](https://github.com/apache/datafusion-comet/pull/2560) (dependabot[bot])\n- chore(deps): bump aws-credential-types from 1.2.6 to 1.2.7 in /native [#2563](https://github.com/apache/datafusion-comet/pull/2563) (dependabot[bot])\n- chore: Refactor serde for RegExpReplace [#2548](https://github.com/apache/datafusion-comet/pull/2548) (andygrove)\n- chore: use polymorphic map builders in shuffle. [#2571](https://github.com/apache/datafusion-comet/pull/2571) (ashdnazg)\n- chore: Move ToPrettyString serde into shim layer [#2549](https://github.com/apache/datafusion-comet/pull/2549) (andygrove)\n- chore(deps): bump DataFusion dependencies to 50.2.0, refresh Cargo.lock [#2575](https://github.com/apache/datafusion-comet/pull/2575) (mbutrovich)\n\n## Credits\n\nThank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.\n\n```\n 47\tAndy Grove\n 15\tZhen Wang\n 14\tB Vadlamani\n 12\tOleks V\n 11\tdependabot[bot]\n 10\tMatt Butrovich\n 5\tParth Chandra\n 5\thsiang-c\n 3\tFu Chen\n 3\tJunfan Zhang\n 2\tKazantsev Maksim\n 1\tArtem Kupchinskiy\n 1\tEshed Schacham\n 1\tManu Zhang\n 1\tandy-hf-kwok\n```\n\nThank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.\n" }, { "path": "dev/changelog/0.12.0.md", "content": "\n\n# DataFusion Comet 0.12.0 Changelog\n\nThis release consists of 105 commits from 13 contributors. See credits at the end of this changelog for more information.\n\n**Fixed bugs:**\n\n- fix: Fix `None.get` in `stringDecode` when `bin` child cannot be converted [#2606](https://github.com/apache/datafusion-comet/pull/2606) (cfmcgrady)\n- fix: Update FuzzDataGenerator to produce dictionary-encoded string arrays & fix bugs that this exposes [#2635](https://github.com/apache/datafusion-comet/pull/2635) (andygrove)\n- fix: Fallback to Spark for lpad/rpad for unsupported arguments & fix negative length handling [#2630](https://github.com/apache/datafusion-comet/pull/2630) (andygrove)\n- fix: Mark SortOrder with floating-point as incompatible [#2650](https://github.com/apache/datafusion-comet/pull/2650) (andygrove)\n- fix: Fall back to Spark for `trunc` / `date_trunc` functions when format string is unsupported, or is not a literal value [#2634](https://github.com/apache/datafusion-comet/pull/2634) (andygrove)\n- fix: [native_datafusion] only pass single partition of PartitionedFiles into DataSourceExec [#2675](https://github.com/apache/datafusion-comet/pull/2675) (mbutrovich)\n- fix: Fix subcommands options in fuzz-testing [#2684](https://github.com/apache/datafusion-comet/pull/2684) (manuzhang)\n- fix: Do not replace SMJ with HJ for `LeftSemi` [#2687](https://github.com/apache/datafusion-comet/pull/2687) (comphead)\n- fix: Apply spotless on Iceberg 1.8.1 diff [iceberg] [#2700](https://github.com/apache/datafusion-comet/pull/2700) (hsiang-c)\n- fix: Fix generate-user-guide-reference-docs failure when mvn command is not executed at root [#2691](https://github.com/apache/datafusion-comet/pull/2691) (manuzhang)\n- fix: Fix missing SortOrder fallback reason in range partitioning [#2716](https://github.com/apache/datafusion-comet/pull/2716) (andygrove)\n- fix: CometLiteral class cast exception with arrays [#2718](https://github.com/apache/datafusion-comet/pull/2718) (andygrove)\n- fix: NormalizeNaNAndZero::children() returns child's child [#2732](https://github.com/apache/datafusion-comet/pull/2732) (mbutrovich)\n- fix: checkSparkMaybeThrows should compare Spark and Comet results in success case [#2728](https://github.com/apache/datafusion-comet/pull/2728) (andygrove)\n- fix: Mark `WindowsExec` as incompatible [#2748](https://github.com/apache/datafusion-comet/pull/2748) (andygrove)\n- fix: Add strict floating point mode and fallback to Spark for min/max/sort on floating point inputs when enabled [#2747](https://github.com/apache/datafusion-comet/pull/2747) (andygrove)\n- fix: Implement producedAttributes for CometWindowExec [#2789](https://github.com/apache/datafusion-comet/pull/2789) (rahulbabarwal89)\n- fix: Pass all Comet configs to native plan [#2801](https://github.com/apache/datafusion-comet/pull/2801) (andygrove)\n\n**Implemented enhancements:**\n\n- feat: Add option to write benchmark results to file [#2640](https://github.com/apache/datafusion-comet/pull/2640) (andygrove)\n- feat: Implement metrics for iceberg compat [#2615](https://github.com/apache/datafusion-comet/pull/2615) (EmilyMatt)\n- feat: Define function signatures in CometFuzz [#2614](https://github.com/apache/datafusion-comet/pull/2614) (andygrove)\n- feat: cherry-pick UUID conversion logic from #2528 [#2648](https://github.com/apache/datafusion-comet/pull/2648) (mbutrovich)\n- feat: support `concat` for strings [#2604](https://github.com/apache/datafusion-comet/pull/2604) (comphead)\n- feat: Add support for `abs` [#2689](https://github.com/apache/datafusion-comet/pull/2689) (andygrove)\n- feat: Support variadic function in CometFuzz [#2682](https://github.com/apache/datafusion-comet/pull/2682) (manuzhang)\n- feat: CometExecRule refactor: Unify CometNativeExec creation with Serde in CometOperatorSerde trait [#2768](https://github.com/apache/datafusion-comet/pull/2768) (andygrove)\n- feat: support cot [#2755](https://github.com/apache/datafusion-comet/pull/2755) (psvri)\n- feat: Add bash script to build and run fuzz testing [#2686](https://github.com/apache/datafusion-comet/pull/2686) (manuzhang)\n- feat: Add getSupportLevel to CometAggregateExpressionSerde trait [#2777](https://github.com/apache/datafusion-comet/pull/2777) (andygrove)\n- feat: Add CI check to ensure generated docs are in sync with code [#2779](https://github.com/apache/datafusion-comet/pull/2779) (andygrove)\n- feat: Add prettier enforcement [#2783](https://github.com/apache/datafusion-comet/pull/2783) (andygrove)\n- feat: hyperbolic trig functions [#2784](https://github.com/apache/datafusion-comet/pull/2784) (psvri)\n- feat: [iceberg] Native scan by serializing FileScanTasks to iceberg-rust [#2528](https://github.com/apache/datafusion-comet/pull/2528) (mbutrovich)\n\n**Documentation updates:**\n\n- docs: Add changelog for 0.11.0 release [#2585](https://github.com/apache/datafusion-comet/pull/2585) (mbutrovich)\n- docs: Improve documentation layout [#2587](https://github.com/apache/datafusion-comet/pull/2587) (andygrove)\n- docs: Publish 0.11.0 user guide [#2589](https://github.com/apache/datafusion-comet/pull/2589) (andygrove)\n- docs: Put Comet logo in top nav bar, respect light/dark mode [#2591](https://github.com/apache/datafusion-comet/pull/2591) (andygrove)\n- docs: Improve main landing page [#2593](https://github.com/apache/datafusion-comet/pull/2593) (andygrove)\n- docs: Improve site navigation [#2597](https://github.com/apache/datafusion-comet/pull/2597) (andygrove)\n- docs: Update benchmark results [#2596](https://github.com/apache/datafusion-comet/pull/2596) (andygrove)\n- docs: Upgrade pydata-sphinx-theme to 0.16.1 [#2602](https://github.com/apache/datafusion-comet/pull/2602) (andygrove)\n- docs: Fix redirect [#2603](https://github.com/apache/datafusion-comet/pull/2603) (andygrove)\n- docs: Fix broken image link [#2613](https://github.com/apache/datafusion-comet/pull/2613) (andygrove)\n- docs: Add FFI docs to contributor guide [#2668](https://github.com/apache/datafusion-comet/pull/2668) (andygrove)\n- docs: Various documentation updates [#2674](https://github.com/apache/datafusion-comet/pull/2674) (andygrove)\n- docs: Add supported SortOrder expressions and fix a typo [#2694](https://github.com/apache/datafusion-comet/pull/2694) (andygrove)\n- docs: Minor docs update for running Spark SQL tests [#2712](https://github.com/apache/datafusion-comet/pull/2712) (andygrove)\n- docs: Update contributor guide for adding a new expression [#2704](https://github.com/apache/datafusion-comet/pull/2704) (andygrove)\n- docs: Documentation updates for `LocalTableScan` and `WindowExec` [#2742](https://github.com/apache/datafusion-comet/pull/2742) (andygrove)\n- docs: Typo fix [#2752](https://github.com/apache/datafusion-comet/pull/2752) (wForget)\n- docs: Categorize some configs as `testing` and add notes about known time zone issues [#2740](https://github.com/apache/datafusion-comet/pull/2740) (andygrove)\n- docs: Run prettier on all markdown files [#2782](https://github.com/apache/datafusion-comet/pull/2782) (andygrove)\n- docs: Ignore prettier formatting for generated tables [#2790](https://github.com/apache/datafusion-comet/pull/2790) (andygrove)\n- docs: Add new section to contributor guide, explaining how to add a new operator [#2758](https://github.com/apache/datafusion-comet/pull/2758) (andygrove)\n\n**Other:**\n\n- chore: Start 0.12.0 development [#2584](https://github.com/apache/datafusion-comet/pull/2584) (mbutrovich)\n- chore: Bump Spark from 3.5.6 to 3.5.7 [#2574](https://github.com/apache/datafusion-comet/pull/2574) (cfmcgrady)\n- chore(deps): bump parquet from 56.0.0 to 56.2.0 in /native [#2608](https://github.com/apache/datafusion-comet/pull/2608) (dependabot[bot])\n- chore(deps): bump tikv-jemallocator from 0.6.0 to 0.6.1 in /native [#2609](https://github.com/apache/datafusion-comet/pull/2609) (dependabot[bot])\n- chore(deps): bump tikv-jemalloc-ctl from 0.6.0 to 0.6.1 in /native [#2610](https://github.com/apache/datafusion-comet/pull/2610) (dependabot[bot])\n- tests: FuzzDataGenerator instead of Parquet-specific generator [#2616](https://github.com/apache/datafusion-comet/pull/2616) (mbutrovich)\n- chore: Simplify on-heap memory configuration [#2599](https://github.com/apache/datafusion-comet/pull/2599) (andygrove)\n- Feat: Add sha1 function impl [#2471](https://github.com/apache/datafusion-comet/pull/2471) (kazantsev-maksim)\n- chore: Refactor Parquet/DataFrame fuzz data generators [#2629](https://github.com/apache/datafusion-comet/pull/2629) (andygrove)\n- chore: Remove needless from_raw calls [#2638](https://github.com/apache/datafusion-comet/pull/2638) (EmilyMatt)\n- chore: support DataFusion 50.3.0 [#2605](https://github.com/apache/datafusion-comet/pull/2605) (comphead)\n- chore(deps): bump actions/upload-artifact from 4 to 5 [#2654](https://github.com/apache/datafusion-comet/pull/2654) (dependabot[bot])\n- chore(deps): bump cc from 1.2.42 to 1.2.43 in /native [#2653](https://github.com/apache/datafusion-comet/pull/2653) (dependabot[bot])\n- chore(deps): bump actions/download-artifact from 5 to 6 [#2652](https://github.com/apache/datafusion-comet/pull/2652) (dependabot[bot])\n- chore: extract comparison into separate tool [#2632](https://github.com/apache/datafusion-comet/pull/2632) (comphead)\n- chore: Various improvements to `checkSparkAnswer*` methods in `CometTestBase` [#2656](https://github.com/apache/datafusion-comet/pull/2656) (andygrove)\n- chore: Remove code for unpacking dictionaries prior to FilterExec [#2659](https://github.com/apache/datafusion-comet/pull/2659) (andygrove)\n- chore: display schema for datasets being compared [#2665](https://github.com/apache/datafusion-comet/pull/2665) (comphead)\n- chore: Remove `CopyExec` [#2663](https://github.com/apache/datafusion-comet/pull/2663) (andygrove)\n- chore: Add extended explain plans to stability suite [#2669](https://github.com/apache/datafusion-comet/pull/2669) (andygrove)\n- chore(deps): bump aws-config from 1.8.8 to 1.8.10 in /native [#2677](https://github.com/apache/datafusion-comet/pull/2677) (dependabot[bot])\n- chore(deps): bump cc from 1.2.43 to 1.2.44 in /native [#2678](https://github.com/apache/datafusion-comet/pull/2678) (dependabot[bot])\n- chore: `tpcbench` output `explain` just once and formatted [#2679](https://github.com/apache/datafusion-comet/pull/2679) (comphead)\n- chore: Add tolerance for `ComparisonTool` [#2699](https://github.com/apache/datafusion-comet/pull/2699) (comphead)\n- chore: Expand test coverage for `CometWindowsExec` [#2711](https://github.com/apache/datafusion-comet/pull/2711) (comphead)\n- chore: generate Float/Double NaN [#2695](https://github.com/apache/datafusion-comet/pull/2695) (hsiang-c)\n- minor: Combine two CI workflows for Spark SQL tests [#2727](https://github.com/apache/datafusion-comet/pull/2727) (andygrove)\n- chore: Improve framework for specifying that configs can be set with env vars [#2722](https://github.com/apache/datafusion-comet/pull/2722) (andygrove)\n- chore: Rename `COMET_EXPLAIN_VERBOSE_ENABLED` to `COMET_EXTENDED_EXPLAIN_FORMAT` and change default [#2644](https://github.com/apache/datafusion-comet/pull/2644) (andygrove)\n- chore: Fallback to Spark for windows functions [#2726](https://github.com/apache/datafusion-comet/pull/2726) (comphead)\n- chore: Refactor operator serde - part 1 [#2738](https://github.com/apache/datafusion-comet/pull/2738) (andygrove)\n- Feat: Add CometLocalTableScanExec operator [#2735](https://github.com/apache/datafusion-comet/pull/2735) (kazantsev-maksim)\n- chore(deps): bump cc from 1.2.44 to 1.2.45 in /native [#2750](https://github.com/apache/datafusion-comet/pull/2750) (dependabot[bot])\n- chore(deps): bump aws-credential-types from 1.2.8 to 1.2.9 in /native [#2751](https://github.com/apache/datafusion-comet/pull/2751) (dependabot[bot])\n- chore: Operator serde refactor part 2 [#2741](https://github.com/apache/datafusion-comet/pull/2741) (andygrove)\n- chore: Fallback to Spark for `array_reverse` for `array` [#2759](https://github.com/apache/datafusion-comet/pull/2759) (comphead)\n- chore: [iceberg] test iceberg 1.10.0 [#2709](https://github.com/apache/datafusion-comet/pull/2709) (manuzhang)\n- chore: Add `docs/comet-*` to rat exclude list [#2762](https://github.com/apache/datafusion-comet/pull/2762) (manuzhang)\n- Chore: Refactor static invoke exprs [#2671](https://github.com/apache/datafusion-comet/pull/2671) (kazantsev-maksim)\n- minor: Small refactor for consistent serde for hash aggregate [#2764](https://github.com/apache/datafusion-comet/pull/2764) (andygrove)\n- minor: Move `operator2Proto` to `CometExecRule` [#2767](https://github.com/apache/datafusion-comet/pull/2767) (andygrove)\n- chore: various refactoring changes for iceberg [iceberg] [#2680](https://github.com/apache/datafusion-comet/pull/2680) (parthchandra)\n- chore: Refactor CometExecRule handling of sink operators [#2771](https://github.com/apache/datafusion-comet/pull/2771) (andygrove)\n- minor: Refactor to move window-specific code from `QueryPlanSerde` to `CometWindowExec` [#2780](https://github.com/apache/datafusion-comet/pull/2780) (andygrove)\n- chore: Remove many references to `COMET_EXPR_ALLOW_INCOMPATIBLE` [#2775](https://github.com/apache/datafusion-comet/pull/2775) (andygrove)\n- chore: Remove COMET_EXPR_ALLOW_INCOMPATIBLE config [#2786](https://github.com/apache/datafusion-comet/pull/2786) (andygrove)\n- chore: check `missingInput` for Comet plan nodes [#2795](https://github.com/apache/datafusion-comet/pull/2795) (comphead)\n- chore: Finish refactoring expression serde out of `QueryPlanSerde` [#2791](https://github.com/apache/datafusion-comet/pull/2791) (andygrove)\n- chore: Update docs to fix CI after #2784 [#2799](https://github.com/apache/datafusion-comet/pull/2799) (mbutrovich)\n- chore: Update q79 golden plan for Spark 4.0 after #2795 [#2800](https://github.com/apache/datafusion-comet/pull/2800) (mbutrovich)\n- Fix: Fix null handling in CometVector implementations [#2643](https://github.com/apache/datafusion-comet/pull/2643) (cfmcgrady)\n\n## Credits\n\nThank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.\n\n```\n 54\tAndy Grove\n 11\tOleks V\n 10\tdependabot[bot]\n 9\tMatt Butrovich\n 6\tManu Zhang\n 3\tFu Chen\n 3\tKazantsev Maksim\n 2\tEmily Matheys\n 2\tVrishabh\n 2\thsiang-c\n 1\tParth Chandra\n 1\tZhen Wang\n 1\trahulbabarwal89\n```\n\nThank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.\n" }, { "path": "dev/changelog/0.13.0.md", "content": "\n\n# DataFusion Comet 0.13.0 Changelog\n\nThis release consists of 169 commits from 15 contributors. See credits at the end of this changelog for more information.\n\n**Fixed bugs:**\n\n- fix: NativeScan count assert firing for no reason [#2850](https://github.com/apache/datafusion-comet/pull/2850) (EmilyMatt)\n- fix: Correct link to tracing guide in CometConf [#2866](https://github.com/apache/datafusion-comet/pull/2866) (manuzhang)\n- fix: Fall back to Spark for MakeDecimal with unsupported input type [#2815](https://github.com/apache/datafusion-comet/pull/2815) (andygrove)\n- fix: Normalize s3 paths for PME key retriever [#2874](https://github.com/apache/datafusion-comet/pull/2874) (mbutrovich)\n- fix: modify CometNativeScan to generate the file partitions without instantiating RDD [#2891](https://github.com/apache/datafusion-comet/pull/2891) (mbutrovich)\n- fix: Modulus on decimal data type mismatch [#2922](https://github.com/apache/datafusion-comet/pull/2922) (andygrove)\n- fix: [iceberg] Mark nativeIcebergScanMetadata @transient [#2930](https://github.com/apache/datafusion-comet/pull/2930) (mbutrovich)\n- fix: enable cast tests for Spark 4.0 [#2919](https://github.com/apache/datafusion-comet/pull/2919) (manuzhang)\n- fix: Remove fallback for maps containing complex types [#2943](https://github.com/apache/datafusion-comet/pull/2943) (andygrove)\n- fix: CometShuffleManager hang by deferring SparkEnv access [#3002](https://github.com/apache/datafusion-comet/pull/3002) (Shekharrajak)\n- fix: format decimal to string when casting to short [#2916](https://github.com/apache/datafusion-comet/pull/2916) (manuzhang)\n- fix: [iceberg] reduce granularity of metrics updates in IcebergFileStream [#3050](https://github.com/apache/datafusion-comet/pull/3050) (mbutrovich)\n- fix: native shuffle now reports spill metrics correctly [#3197](https://github.com/apache/datafusion-comet/pull/3197) (andygrove)\n- fix: Prevent native write when input is not Arrow format [#3227](https://github.com/apache/datafusion-comet/pull/3227) (andygrove)\n- fix: Add JDK to Docker image for release build [#3262](https://github.com/apache/datafusion-comet/pull/3262) (hsiang-c)\n\n**Performance related:**\n\n- perf: [iceberg] Deduplicate serialized metadata for Iceberg native scan [#2933](https://github.com/apache/datafusion-comet/pull/2933) (mbutrovich)\n- perf: Use await instead of block_on in native shuffle writer [#2937](https://github.com/apache/datafusion-comet/pull/2937) (mbutrovich)\n- perf: refactor executePlan to try to avoid constantly entering Tokio runtime [#2938](https://github.com/apache/datafusion-comet/pull/2938) (mbutrovich)\n- perf: Optimize lpad/rpad to remove unnecessary memory allocations per element [#2963](https://github.com/apache/datafusion-comet/pull/2963) (andygrove)\n- perf: Improve performance of normalize_nan [#2999](https://github.com/apache/datafusion-comet/pull/2999) (andygrove)\n- perf: Improve string expression microbenchmarks [#3012](https://github.com/apache/datafusion-comet/pull/3012) (andygrove)\n- perf: Improve date/time microbenchmarks to avoid redundant/duplicate benchmarks [#3020](https://github.com/apache/datafusion-comet/pull/3020) (andygrove)\n- perf: Improve aggregate expression microbenchmarks [#3021](https://github.com/apache/datafusion-comet/pull/3021) (andygrove)\n- perf: Improve conditional expression microbenchmarks [#3024](https://github.com/apache/datafusion-comet/pull/3024) (andygrove)\n- perf: Improve performance of date truncate [#2997](https://github.com/apache/datafusion-comet/pull/2997) (andygrove)\n- perf: Add microbenchmark for comparison expressions [#3026](https://github.com/apache/datafusion-comet/pull/3026) (andygrove)\n- perf: Implement more microbenchmarks for cast expressions [#3031](https://github.com/apache/datafusion-comet/pull/3031) (andygrove)\n- perf: Add microbenchmark for hash expressions [#3028](https://github.com/apache/datafusion-comet/pull/3028) (andygrove)\n- perf: Improve performance of CAST from string to int [#3017](https://github.com/apache/datafusion-comet/pull/3017) (coderfender)\n- perf: Improve criterion benchmarks for cast string to int [#3049](https://github.com/apache/datafusion-comet/pull/3049) (andygrove)\n- perf: Additional optimizations for cast from string to int [#3048](https://github.com/apache/datafusion-comet/pull/3048) (andygrove)\n- perf: set DataFusion session context's target_partitions to match Spark's spark.task.cpus [#3062](https://github.com/apache/datafusion-comet/pull/3062) (mbutrovich)\n- perf: don't busy-poll Tokio stream for plans without CometScan [#3063](https://github.com/apache/datafusion-comet/pull/3063) (mbutrovich)\n- perf: minor optimizations in `process_sorted_row_partition` [#3059](https://github.com/apache/datafusion-comet/pull/3059) (andygrove)\n- perf: optimize complex-type hash implementations [#3140](https://github.com/apache/datafusion-comet/pull/3140) (mbutrovich)\n- perf: [iceberg] Remove IcebergFileStream, use iceberg-rust's parallelization, bump iceberg-rust to latest, cache SchemaAdapter [#3051](https://github.com/apache/datafusion-comet/pull/3051) (mbutrovich)\n- perf: [iceberg] reduce nativeIcebergScanMetadata serialization points [#3243](https://github.com/apache/datafusion-comet/pull/3243) (mbutrovich)\n- perf: reduce GC pressure in protobuf serialization [#3242](https://github.com/apache/datafusion-comet/pull/3242) (andygrove)\n- perf: cache serialized query plans to avoid per-partition serialization [#3246](https://github.com/apache/datafusion-comet/pull/3246) (andygrove)\n- perf: [iceberg] Use protobuf instead of JSON to serialize Iceberg partition values [#3247](https://github.com/apache/datafusion-comet/pull/3247) (parthchandra)\n\n**Implemented enhancements:**\n\n- feat: Add experimental support for native Parquet writes [#2812](https://github.com/apache/datafusion-comet/pull/2812) (andygrove)\n- feat: Partially implement file commit protocol for native Parquet writes [#2828](https://github.com/apache/datafusion-comet/pull/2828) (andygrove)\n- feat: CometNativeWriteExec support with native scan as a child [#2839](https://github.com/apache/datafusion-comet/pull/2839) (mbutrovich)\n- feat: Add support for `explode` and `explode_outer` for array inputs [#2836](https://github.com/apache/datafusion-comet/pull/2836) (andygrove)\n- feat: Support ANSI mode SUM (Decimal types) [#2826](https://github.com/apache/datafusion-comet/pull/2826) (coderfender)\n- feat: Add expression registry to native planner [#2851](https://github.com/apache/datafusion-comet/pull/2851) (andygrove)\n- feat: Implement native operator registry [#2875](https://github.com/apache/datafusion-comet/pull/2875) (andygrove)\n- feat: Improve fallback reporting for `native_datafusion` scan [#2879](https://github.com/apache/datafusion-comet/pull/2879) (andygrove)\n- feat: Enable bucket pruning with native_datafusion scans [#2888](https://github.com/apache/datafusion-comet/pull/2888) (mbutrovich)\n- feat: support_ansi-mode_aggregated_benchmarking [#2901](https://github.com/apache/datafusion-comet/pull/2901) (coderfender)\n- feat: [iceberg] REST catalog support for CometNativeIcebergScan [#2895](https://github.com/apache/datafusion-comet/pull/2895) (mbutrovich)\n- feat: [iceberg] Support session token in Iceberg Native scan [#2913](https://github.com/apache/datafusion-comet/pull/2913) (hsiang-c)\n- feat: Make shuffle writer buffer size configurable [#2899](https://github.com/apache/datafusion-comet/pull/2899) (andygrove)\n- feat: Add partial support for `from_json` [#2934](https://github.com/apache/datafusion-comet/pull/2934) (andygrove)\n- feat: Create benchmarks comet cast [#2932](https://github.com/apache/datafusion-comet/pull/2932) (coderfender)\n- feat: Support string decimal cast [#2925](https://github.com/apache/datafusion-comet/pull/2925) (coderfender)\n- feat: Remove unnecessary transition for native writes [#2960](https://github.com/apache/datafusion-comet/pull/2960) (comphead)\n- feat: Initial implementation of size for array inputs [#2862](https://github.com/apache/datafusion-comet/pull/2862) (andygrove)\n- feat: Support ANSI mode sum expr (int inputs) [#2600](https://github.com/apache/datafusion-comet/pull/2600) (coderfender)\n- feat: Support casting string float types [#2835](https://github.com/apache/datafusion-comet/pull/2835) (coderfender)\n- feat: Support ANSI mode avg expr (int inputs) [#2817](https://github.com/apache/datafusion-comet/pull/2817) (coderfender)\n- feat: Add support for remote Parquet HDFS writer with openDAL [#2929](https://github.com/apache/datafusion-comet/pull/2929) (comphead)\n- feat: Expand `murmur3` hash support to complex types [#3077](https://github.com/apache/datafusion-comet/pull/3077) (andygrove)\n- feat: Comet Writer should respect object store settings [#3042](https://github.com/apache/datafusion-comet/pull/3042) (comphead)\n- feat: add support for unix_date expression [#3141](https://github.com/apache/datafusion-comet/pull/3141) (andygrove)\n- feat: add partial support for date_format expression [#3201](https://github.com/apache/datafusion-comet/pull/3201) (andygrove)\n- feat: add complex type support to native Parquet writer [#3214](https://github.com/apache/datafusion-comet/pull/3214) (andygrove)\n- feat: implement framework to support multiple pyspark benchmarks [#3080](https://github.com/apache/datafusion-comet/pull/3080) (andygrove)\n- feat: add support for datediff expression [#3145](https://github.com/apache/datafusion-comet/pull/3145) (andygrove)\n- feat: Add support for `unix_timestamp` function [#2936](https://github.com/apache/datafusion-comet/pull/2936) (andygrove)\n- feat: add support for last_day expression [#3143](https://github.com/apache/datafusion-comet/pull/3143) (andygrove)\n- feat: Support left expression [#3206](https://github.com/apache/datafusion-comet/pull/3206) (Shekharrajak)\n- feat: Add support for round-robin partitioning in native shuffle [#3076](https://github.com/apache/datafusion-comet/pull/3076) (andygrove)\n- feat: Native columnar to row conversion (Phase 1) [#3221](https://github.com/apache/datafusion-comet/pull/3221) (andygrove)\n\n**Documentation updates:**\n\n- docs: add documentation for fully-native Iceberg scans [#2868](https://github.com/apache/datafusion-comet/pull/2868) (mbutrovich)\n- docs: Add documentation to contributor guide explaining native + JVM shuffle implementation [#3055](https://github.com/apache/datafusion-comet/pull/3055) (andygrove)\n- docs: add guidance on disabling constant folding for literal tests [#3200](https://github.com/apache/datafusion-comet/pull/3200) (andygrove)\n- docs: Add common pitfalls and improve PR checklist in development guide [#3231](https://github.com/apache/datafusion-comet/pull/3231) (andygrove)\n- docs: various documentation updates in preparation for next release [#3254](https://github.com/apache/datafusion-comet/pull/3254) (andygrove)\n- docs: Stop generating dynamic docs content in build [#3212](https://github.com/apache/datafusion-comet/pull/3212) (andygrove)\n- docs: document datetime rebasing and V2 API limitations for DataFusion-based scans [#3259](https://github.com/apache/datafusion-comet/pull/3259) (andygrove)\n- docs: Mark native_comet scan as deprecated [#3274](https://github.com/apache/datafusion-comet/pull/3274) (andygrove)\n\n**Other:**\n\n- chore: Add 0.12.0 changelog [#2811](https://github.com/apache/datafusion-comet/pull/2811) (andygrove)\n- chore: Prepare for 0.13.0 development [#2809](https://github.com/apache/datafusion-comet/pull/2809) (andygrove)\n- minor: Add microbenchmark for integer sum with grouping [#2805](https://github.com/apache/datafusion-comet/pull/2805) (andygrove)\n- test: extract conditional expression tests (`if`, `case_when` and `coalesce`) [#2807](https://github.com/apache/datafusion-comet/pull/2807) (rluvaton)\n- build: Disable caching for macOS PR builds [#2816](https://github.com/apache/datafusion-comet/pull/2816) (andygrove)\n- chore(deps): bump actions/checkout from 5 to 6 [#2818](https://github.com/apache/datafusion-comet/pull/2818) (dependabot[bot])\n- chore(deps): bump object_store_opendal from 0.54.1 to 0.55.0 in /native [#2819](https://github.com/apache/datafusion-comet/pull/2819) (dependabot[bot])\n- chore(deps): bump cc from 1.2.46 to 1.2.47 in /native [#2822](https://github.com/apache/datafusion-comet/pull/2822) (dependabot[bot])\n- chore(deps): bump opendal from 0.54.1 to 0.55.0 in /native [#2821](https://github.com/apache/datafusion-comet/pull/2821) (dependabot[bot])\n- chore: update `Iceberg` install docs [#2824](https://github.com/apache/datafusion-comet/pull/2824) (comphead)\n- chore(deps): bump cc from 1.2.47 to 1.2.48 in /native [#2833](https://github.com/apache/datafusion-comet/pull/2833) (dependabot[bot])\n- chore(deps): bump the proto group in /native with 2 updates [#2832](https://github.com/apache/datafusion-comet/pull/2832) (dependabot[bot])\n- minor: Clean up shuffle transformation code in `CometExecRule` [#2840](https://github.com/apache/datafusion-comet/pull/2840) (andygrove)\n- chore: fix broken link to Apache DataFusion Comet Overview in README [#2846](https://github.com/apache/datafusion-comet/pull/2846) (onestn)\n- chore: Refactor some of the scan and sink handling in `CometExecRule` to reduce duplicate code [#2844](https://github.com/apache/datafusion-comet/pull/2844) (andygrove)\n- deps: bump lz4_flex, downgrade prost from yanked version [#2847](https://github.com/apache/datafusion-comet/pull/2847) (mbutrovich)\n- minor: Move shuffle logic from `CometExecRule` to `CometShuffleExchangeExec` serde implementation [#2853](https://github.com/apache/datafusion-comet/pull/2853) (andygrove)\n- chore: remove coverage file auto generator [#2854](https://github.com/apache/datafusion-comet/pull/2854) (comphead)\n- chore(deps): bump cc from 1.2.48 to 1.2.49 in /native [#2858](https://github.com/apache/datafusion-comet/pull/2858) (dependabot[bot])\n- chore: Refactor `CometExecRule` handling of `BroadcastHashJoin` and fix fallback reporting [#2856](https://github.com/apache/datafusion-comet/pull/2856) (andygrove)\n- chore: update actions/checkout from v4 to v6 in setup-iceberg and set… [#2857](https://github.com/apache/datafusion-comet/pull/2857) (bjornjorgensen)\n- minor: Small refactor in `CometExecRule` to remove confusing code and fix more fallback reporting [#2860](https://github.com/apache/datafusion-comet/pull/2860) (andygrove)\n- chore: Add unit tests for `CometExecRule` [#2863](https://github.com/apache/datafusion-comet/pull/2863) (andygrove)\n- chore: Add unit tests for `CometScanRule` [#2867](https://github.com/apache/datafusion-comet/pull/2867) (andygrove)\n- minor: Pedantic refactoring to move some methods from `CometSparkSessionExtensions` to `CometScanRule` and `CometExecRule` [#2873](https://github.com/apache/datafusion-comet/pull/2873) (andygrove)\n- deps: [iceberg] upgrade DataFusion to 51, Arrow to 57, Iceberg to latest, MSRV to 1.88 [#2729](https://github.com/apache/datafusion-comet/pull/2729) (mbutrovich)\n- chore: Enable plan stability suite for `native_datafusion` scans [#2877](https://github.com/apache/datafusion-comet/pull/2877) (andygrove)\n- chore: `ScanExec::new` no longer fetches data [#2881](https://github.com/apache/datafusion-comet/pull/2881) (andygrove)\n- Chore: refactor bit_not [#2896](https://github.com/apache/datafusion-comet/pull/2896) (kazantsev-maksim)\n- chore(deps): bump actions/cache from 4 to 5 [#2909](https://github.com/apache/datafusion-comet/pull/2909) (dependabot[bot])\n- chore(deps): bump actions/upload-artifact from 5 to 6 [#2910](https://github.com/apache/datafusion-comet/pull/2910) (dependabot[bot])\n- chore: Refactor string benchmarks (~10x reduction in LOC) [#2907](https://github.com/apache/datafusion-comet/pull/2907) (andygrove)\n- chore(deps): bump actions/download-artifact from 6 to 7 [#2908](https://github.com/apache/datafusion-comet/pull/2908) (dependabot[bot])\n- chore: use datafusion impl of hex function [#2915](https://github.com/apache/datafusion-comet/pull/2915) (kazantsev-maksim)\n- chore: Use fixed seed in RNG in tests [#2917](https://github.com/apache/datafusion-comet/pull/2917) (andygrove)\n- chore: Remove `row_step` from `process_sorted_row_partition` [#2920](https://github.com/apache/datafusion-comet/pull/2920) (andygrove)\n- chore: Move string function handling to new expression registry [#2931](https://github.com/apache/datafusion-comet/pull/2931) (andygrove)\n- chore: Reduce syscalls in metrics update logic [#2940](https://github.com/apache/datafusion-comet/pull/2940) (andygrove)\n- chore: Add shuffle benchmark for deeply nested schemas [#2902](https://github.com/apache/datafusion-comet/pull/2902) (andygrove)\n- chore: Reduce timer overhead in native shuffle writer [#2941](https://github.com/apache/datafusion-comet/pull/2941) (andygrove)\n- chore: Remove low-level ffi/jvm timers from native `ScanExec` [#2939](https://github.com/apache/datafusion-comet/pull/2939) (andygrove)\n- build: Skip problematic Spark SQL test for Spark 4.0.x [#2947](https://github.com/apache/datafusion-comet/pull/2947) (andygrove)\n- build: Reinstate macOS CI builds of Comet with Spark 4.0 [#2950](https://github.com/apache/datafusion-comet/pull/2950) (manuzhang)\n- chore(deps): bump reqwest from 0.12.25 to 0.12.26 in /native [#2952](https://github.com/apache/datafusion-comet/pull/2952) (dependabot[bot])\n- chore(deps): bump cc from 1.2.49 to 1.2.50 in /native [#2954](https://github.com/apache/datafusion-comet/pull/2954) (dependabot[bot])\n- chore(deps): bump assertables from 9.8.2 to 9.8.3 in /native [#2953](https://github.com/apache/datafusion-comet/pull/2953) (dependabot[bot])\n- minor: Refactor expression microbenchmarks to remove duplicate code [#2956](https://github.com/apache/datafusion-comet/pull/2956) (andygrove)\n- build: fix missing import in `main` [#2962](https://github.com/apache/datafusion-comet/pull/2962) (andygrove)\n- build: Skip macOS Spark 4 fuzz test [#2966](https://github.com/apache/datafusion-comet/pull/2966) (andygrove)\n- Avoid duplicated writer nodes when AQE enabled [#2982](https://github.com/apache/datafusion-comet/pull/2982) (comphead)\n- build: Set thread thresholds envs for spark test on macOS [#2987](https://github.com/apache/datafusion-comet/pull/2987) (wForget)\n- chore: Add microbenchmark for casting string to temporal types [#2980](https://github.com/apache/datafusion-comet/pull/2980) (andygrove)\n- chore(deps): bump reqwest from 0.12.26 to 0.12.28 in /native [#3009](https://github.com/apache/datafusion-comet/pull/3009) (dependabot[bot])\n- chore(deps): bump tempfile from 3.23.0 to 3.24.0 in /native [#3006](https://github.com/apache/datafusion-comet/pull/3006) (dependabot[bot])\n- chore(deps): bump serde_json from 1.0.145 to 1.0.148 in /native [#3010](https://github.com/apache/datafusion-comet/pull/3010) (dependabot[bot])\n- chore: Add microbenchmark for casting string to numeric [#2979](https://github.com/apache/datafusion-comet/pull/2979) (andygrove)\n- chore: Skip some CI workflows for benchmark changes [#3030](https://github.com/apache/datafusion-comet/pull/3030) (andygrove)\n- chore: Skip more workflows on benchmark PRs [#3034](https://github.com/apache/datafusion-comet/pull/3034) (andygrove)\n- chore: Improve microbenchmark for string expressions [#2964](https://github.com/apache/datafusion-comet/pull/2964) (andygrove)\n- chore(deps): bump tokio from 1.48.0 to 1.49.0 in /native [#3039](https://github.com/apache/datafusion-comet/pull/3039) (dependabot[bot])\n- chore(deps): bump libc from 0.2.178 to 0.2.179 in /native [#3038](https://github.com/apache/datafusion-comet/pull/3038) (dependabot[bot])\n- chore(deps): bump actions/cache from 4 to 5 [#3037](https://github.com/apache/datafusion-comet/pull/3037) (dependabot[bot])\n- Chore: to_json unit/benchmark tests [#3011](https://github.com/apache/datafusion-comet/pull/3011) (kazantsev-maksim)\n- chore: Add checks to microbenchmarks for plan running natively in Comet [#3045](https://github.com/apache/datafusion-comet/pull/3045) (andygrove)\n- chore: Refactor `CometScanRule` to improve scan selection and fallback logic [#2978](https://github.com/apache/datafusion-comet/pull/2978) (andygrove)\n- chore: Respect to legacySizeOfNull option for size function [#3036](https://github.com/apache/datafusion-comet/pull/3036) (kazantsev-maksim)\n- chore: Add PySpark-based benchmarks, starting with ETL example [#3065](https://github.com/apache/datafusion-comet/pull/3065) (andygrove)\n- chore(deps): bump the proto group in /native with 2 updates [#3071](https://github.com/apache/datafusion-comet/pull/3071) (dependabot[bot])\n- chore: add MacOS file and event trace log to gitignore [#3070](https://github.com/apache/datafusion-comet/pull/3070) (manuzhang)\n- chore(deps): bump arrow from 57.1.0 to 57.2.0 in /native [#3073](https://github.com/apache/datafusion-comet/pull/3073) (dependabot[bot])\n- chore(deps): bump parquet from 57.1.0 to 57.2.0 in /native [#3074](https://github.com/apache/datafusion-comet/pull/3074) (dependabot[bot])\n- chore(deps): bump cc from 1.2.50 to 1.2.52 in /native [#3072](https://github.com/apache/datafusion-comet/pull/3072) (dependabot[bot])\n- chore: improve cast documentation to add support per eval mode [#3056](https://github.com/apache/datafusion-comet/pull/3056) (coderfender)\n- chore: Refactor JVM shuffle: Move `SpillSorter` to top level class and add tests [#3081](https://github.com/apache/datafusion-comet/pull/3081) (andygrove)\n- minor: Split CometShuffleExternalSorter into sync/async implementations [#3192](https://github.com/apache/datafusion-comet/pull/3192) (andygrove)\n- chore: Add pending PR shield [#3205](https://github.com/apache/datafusion-comet/pull/3205) (comphead)\n- chore: deprecate native_comet scan in favor of native_iceberg_compat [#2949](https://github.com/apache/datafusion-comet/pull/2949) (Shekharrajak)\n- chore: add script to regenerate golden files for plan stability tests [#3204](https://github.com/apache/datafusion-comet/pull/3204) (andygrove)\n- chore: fix clippy warnings for Rust 1.93 [#3239](https://github.com/apache/datafusion-comet/pull/3239) (andygrove)\n- build: build native library once and share across CI test jobs [#3249](https://github.com/apache/datafusion-comet/pull/3249) (andygrove)\n- Experimental: Native CSV files read [#3044](https://github.com/apache/datafusion-comet/pull/3044) (kazantsev-maksim)\n- build: add missing datafusion-datasource dependency [#3252](https://github.com/apache/datafusion-comet/pull/3252) (andygrove)\n- chore: Auto scan mode no longer falls back to `native_comet` [#3236](https://github.com/apache/datafusion-comet/pull/3236) (andygrove)\n- build: optimize CI cache usage and add fast lint gate [#3251](https://github.com/apache/datafusion-comet/pull/3251) (andygrove)\n- build: use `install` instead of `compile` in TPC CI jobs [#3263](https://github.com/apache/datafusion-comet/pull/3263) (andygrove)\n- build: remove dead code for 0.8/0.9 docs that broke CI [#3264](https://github.com/apache/datafusion-comet/pull/3264) (andygrove)\n- refactor: rename scan.allowIncompatible to scan.unsignedSmallIntSafetyCheck [#3238](https://github.com/apache/datafusion-comet/pull/3238) (andygrove)\n\n## Credits\n\nThank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.\n\n```\n 91\tAndy Grove\n 23\tdependabot[bot]\n 18\tMatt Butrovich\n 9\tB Vadlamani\n 7\tOleks V\n 5\tKazantsev Maksim\n 5\tManu Zhang\n 3\tShekhar Prasad Rajak\n 2\thsiang-c\n 1\tBjørn Jørgensen\n 1\tEmily Matheys\n 1\tParth Chandra\n 1\tRaz Luvaton\n 1\tWonseok Yang\n 1\tZhen Wang\n```\n\nThank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.\n" }, { "path": "dev/changelog/0.14.0.md", "content": "\n\n# DataFusion Comet 0.14.0 Changelog\n\nThis release consists of 189 commits from 21 contributors. See credits at the end of this changelog for more information.\n\n**Fixed bugs:**\n\n- fix: [iceberg] Fall back on dynamicpruning expressions for CometIcebergNativeScan [#3335](https://github.com/apache/datafusion-comet/pull/3335) (mbutrovich)\n- fix: [iceberg] Disable native c2r by default [#3348](https://github.com/apache/datafusion-comet/pull/3348) (andygrove)\n- fix: Fix `space()` with negative input [#3347](https://github.com/apache/datafusion-comet/pull/3347) (hsiang-c)\n- fix: respect scan impl config for v2 scan [#3357](https://github.com/apache/datafusion-comet/pull/3357) (andygrove)\n- fix: fix memory safety issue in native c2r [#3367](https://github.com/apache/datafusion-comet/pull/3367) (andygrove)\n- fix: preserve partitioning in CometNativeScanExec for bucketed scans [#3392](https://github.com/apache/datafusion-comet/pull/3392) (andygrove)\n- fix: unignore row index Spark SQL tests for native_datafusion [#3414](https://github.com/apache/datafusion-comet/pull/3414) (andygrove)\n- fix: fall back to Spark when Parquet field ID matching is enabled in native_datafusion [#3415](https://github.com/apache/datafusion-comet/pull/3415) (andygrove)\n- fix: Expose bucketing information from CometNativeScanExec [#3437](https://github.com/apache/datafusion-comet/pull/3437) (andygrove)\n- fix: support scalar processing for `space` function [#3408](https://github.com/apache/datafusion-comet/pull/3408) (kazantsev-maksim)\n- fix: Revert \"perf: Remove mutable buffers from scan partition/missing columns (#3411)\" [iceberg] [#3486](https://github.com/apache/datafusion-comet/pull/3486) (mbutrovich)\n- fix: unignore input_file_name Spark SQL tests for native_datafusion [#3458](https://github.com/apache/datafusion-comet/pull/3458) (andygrove)\n- fix: add scalar support for bit_count expression [#3361](https://github.com/apache/datafusion-comet/pull/3361) (hsiang-c)\n- fix: Support concat_ws with literal NULL separator [#3542](https://github.com/apache/datafusion-comet/pull/3542) (0lai0)\n- fix: handle type mismatches in native c2r conversion [#3583](https://github.com/apache/datafusion-comet/pull/3583) (andygrove)\n- fix: disable native C2R for legacy Iceberg scans [iceberg] [#3663](https://github.com/apache/datafusion-comet/pull/3663) (mbutrovich)\n- fix: resolve Miri UB in null struct field test, re-enable Miri on PRs [#3669](https://github.com/apache/datafusion-comet/pull/3669) (andygrove)\n- fix: Support on all-literal RLIKE expression [#3647](https://github.com/apache/datafusion-comet/pull/3647) (0lai0)\n- fix: Fix scan metrics test to run with both native_datafusion and native_iceberg_compat [#3690](https://github.com/apache/datafusion-comet/pull/3690) (andygrove)\n\n**Performance related:**\n\n- perf: refactor sum int with specialized implementations for each eval_mode [#3054](https://github.com/apache/datafusion-comet/pull/3054) (andygrove)\n- perf: Optimize contains expression with SIMD-based scalar pattern sea… [#2991](https://github.com/apache/datafusion-comet/pull/2991) (Shekharrajak)\n- perf: Add batch coalescing in BufBatchWriter to reduce IPC schema overhead [#3441](https://github.com/apache/datafusion-comet/pull/3441) (andygrove)\n- perf: Use `native_datafusion` scan in benchmark scripts (6% faster for TPC-H) [#3460](https://github.com/apache/datafusion-comet/pull/3460) (andygrove)\n- perf: Remove mutable buffers from scan partition/missing columns [#3411](https://github.com/apache/datafusion-comet/pull/3411) (andygrove)\n- perf: [iceberg] Single-pass FileScanTask validation [#3443](https://github.com/apache/datafusion-comet/pull/3443) (mbutrovich)\n- perf: Improve benchmarks for native row-to-columnar used by JVM shuffle [#3290](https://github.com/apache/datafusion-comet/pull/3290) (andygrove)\n- perf: executePlan uses a channel to park executor task thread instead of yield_now() [iceberg] [#3553](https://github.com/apache/datafusion-comet/pull/3553) (mbutrovich)\n- perf: Initialize tokio runtime worker threads from spark.executor.cores [#3555](https://github.com/apache/datafusion-comet/pull/3555) (andygrove)\n- perf: Add Comet config for native Iceberg reader's data file concurrency [iceberg] [#3584](https://github.com/apache/datafusion-comet/pull/3584) (mbutrovich)\n- perf: reuse CometConf.COMET_TRACING_ENABLED, Native, NativeUtil in NativeBatchDecoderIterator [#3627](https://github.com/apache/datafusion-comet/pull/3627) (mbutrovich)\n- perf: Improve performance of native row-to-columnar transition used by JVM shuffle [#3289](https://github.com/apache/datafusion-comet/pull/3289) (andygrove)\n- perf: use aligned pointer reads for SparkUnsafeRow field accessors [#3670](https://github.com/apache/datafusion-comet/pull/3670) (andygrove)\n- perf: Optimize some decimal expressions [#3619](https://github.com/apache/datafusion-comet/pull/3619) (andygrove)\n\n**Implemented enhancements:**\n\n- feat: Native columnar to row conversion (Phase 2) [#3266](https://github.com/apache/datafusion-comet/pull/3266) (andygrove)\n- feat: Enable native columnar-to-row by default [#3299](https://github.com/apache/datafusion-comet/pull/3299) (andygrove)\n- feat: add support for `width_bucket` expression [#3273](https://github.com/apache/datafusion-comet/pull/3273) (davidlghellin)\n- feat: Drop `native_comet` as a valid option for `COMET_NATIVE_SCAN_IMPL` config [#3358](https://github.com/apache/datafusion-comet/pull/3358) (andygrove)\n- feat: Support date to timestamp cast [#3383](https://github.com/apache/datafusion-comet/pull/3383) (coderfender)\n- feat: CometExecRDD supports per-partition plan data, reduce Iceberg native scan serialization, add DPP [iceberg] [#3349](https://github.com/apache/datafusion-comet/pull/3349) (mbutrovich)\n- feat: Support right expression [#3207](https://github.com/apache/datafusion-comet/pull/3207) (Shekharrajak)\n- feat: support map_contains_key expression [#3369](https://github.com/apache/datafusion-comet/pull/3369) (peterxcli)\n- feat: add support for make_date expression [#3147](https://github.com/apache/datafusion-comet/pull/3147) (andygrove)\n- feat: add support for next_day expression [#3148](https://github.com/apache/datafusion-comet/pull/3148) (andygrove)\n- feat: implement cast from whole numbers to binary format and bool to decimal [#3083](https://github.com/apache/datafusion-comet/pull/3083) (coderfender)\n- feat: Support for StringSplit [#2772](https://github.com/apache/datafusion-comet/pull/2772) (Shekharrajak)\n- feat: CometNativeScan per-partition plan serde [#3511](https://github.com/apache/datafusion-comet/pull/3511) (mbutrovich)\n- feat: Remove mutable buffers from scan partition/missing columns [iceberg] [#3514](https://github.com/apache/datafusion-comet/pull/3514) (andygrove)\n- feat: pass spark.comet.datafusion.\\* configs through to DataFusion session [#3455](https://github.com/apache/datafusion-comet/pull/3455) (andygrove)\n- feat: pass vended credentials to Iceberg native scan [#3523](https://github.com/apache/datafusion-comet/pull/3523) (tokoko)\n- feat: Cast date to Numeric (No Op) [#3544](https://github.com/apache/datafusion-comet/pull/3544) (coderfender)\n- feat: add support `crc32` expression [#3498](https://github.com/apache/datafusion-comet/pull/3498) (rafafrdz)\n- feat: Support int to timestamp casts [#3541](https://github.com/apache/datafusion-comet/pull/3541) (coderfender)\n- feat(benchmarks): add async-profiler support to TPC benchmark scripts [#3613](https://github.com/apache/datafusion-comet/pull/3613) (andygrove)\n- feat: Cast numeric (non int) to timestamp [#3559](https://github.com/apache/datafusion-comet/pull/3559) (coderfender)\n- feat: [ANSI] Ansi sql error messages [#3580](https://github.com/apache/datafusion-comet/pull/3580) (parthchandra)\n- feat: enable debug assertions in CI profile, fix unaligned memory access bug [#3652](https://github.com/apache/datafusion-comet/pull/3652) (andygrove)\n- feat: Enable native c2r by default, add debug asserts [#3649](https://github.com/apache/datafusion-comet/pull/3649) (andygrove)\n- feat: support Spark luhn_check expression [#3573](https://github.com/apache/datafusion-comet/pull/3573) (n0r0shi)\n\n**Documentation updates:**\n\n- docs: Add changelog for 0.13.0 [#3260](https://github.com/apache/datafusion-comet/pull/3260) (andygrove)\n- docs: fix bug in placement of prettier-ignore-end in generated docs [#3287](https://github.com/apache/datafusion-comet/pull/3287) (andygrove)\n- docs: Add contributor guide page for SQL file tests [#3333](https://github.com/apache/datafusion-comet/pull/3333) (andygrove)\n- docs: fix inaccurate claim about mutable buffers in parquet scan docs [#3378](https://github.com/apache/datafusion-comet/pull/3378) (andygrove)\n- docs: Improve documentation on maven usage for running tests [#3370](https://github.com/apache/datafusion-comet/pull/3370) (andygrove)\n- docs: move release process docs to contributor guide [#3492](https://github.com/apache/datafusion-comet/pull/3492) (andygrove)\n- docs: improve release process documentation [#3508](https://github.com/apache/datafusion-comet/pull/3508) (andygrove)\n- docs: update roadmap [#3543](https://github.com/apache/datafusion-comet/pull/3543) (mbutrovich)\n- docs: Update Parquet scan documentation [#3433](https://github.com/apache/datafusion-comet/pull/3433) (andygrove)\n- docs: recommend SQL file tests for new expressions [#3598](https://github.com/apache/datafusion-comet/pull/3598) (andygrove)\n- docs: add SAFETY comments to all unsafe blocks in shuffle spark_unsafe module [#3603](https://github.com/apache/datafusion-comet/pull/3603) (andygrove)\n- docs: Fix link to overview page [#3625](https://github.com/apache/datafusion-comet/pull/3625) (manuzhang)\n- doc: Document sql query error propagation [#3651](https://github.com/apache/datafusion-comet/pull/3651) (parthchandra)\n- docs: update Iceberg docs in advance of 0.14.0 [#3691](https://github.com/apache/datafusion-comet/pull/3691) (mbutrovich)\n\n**Other:**\n\n- chore(deps): bump actions/download-artifact from 4 to 7 [#3281](https://github.com/apache/datafusion-comet/pull/3281) (dependabot[bot])\n- chore(deps): bump cc from 1.2.53 to 1.2.54 in /native [#3284](https://github.com/apache/datafusion-comet/pull/3284) (dependabot[bot])\n- build: Fix docs workflow dependency resolution failure [#3275](https://github.com/apache/datafusion-comet/pull/3275) (andygrove)\n- chore(deps): bump actions/upload-artifact from 4 to 6 [#3280](https://github.com/apache/datafusion-comet/pull/3280) (dependabot[bot])\n- chore(deps): bump actions/cache from 4 to 5 [#3279](https://github.com/apache/datafusion-comet/pull/3279) (dependabot[bot])\n- chore(deps): bump uuid from 1.19.0 to 1.20.0 in /native [#3282](https://github.com/apache/datafusion-comet/pull/3282) (dependabot[bot])\n- build: reduce overhead of fuzz testing [#3257](https://github.com/apache/datafusion-comet/pull/3257) (andygrove)\n- chore: Start 0.14.0 development [#3288](https://github.com/apache/datafusion-comet/pull/3288) (andygrove)\n- chore: Add Comet released artifacts and links to maven [#3291](https://github.com/apache/datafusion-comet/pull/3291) (comphead)\n- chore: Add take/untake workflow for issue self-assignment [#3270](https://github.com/apache/datafusion-comet/pull/3270) (andygrove)\n- ci: Consolidate Spark SQL test jobs to reduce CI time [#3271](https://github.com/apache/datafusion-comet/pull/3271) (andygrove)\n- chore(deps): bump org.assertj:assertj-core from 3.23.1 to 3.27.7 [#3293](https://github.com/apache/datafusion-comet/pull/3293) (dependabot[bot])\n- chore: Add microbenchmark for IcebergScan operator serde roundtrip [#3296](https://github.com/apache/datafusion-comet/pull/3296) (andygrove)\n- chore: Remove IgnoreCometNativeScan from ParquetEncryptionSuite in 3.5.7 diff [#3304](https://github.com/apache/datafusion-comet/pull/3304) (andygrove)\n- chore: Enable native c2r in plan stability suite [#3302](https://github.com/apache/datafusion-comet/pull/3302) (andygrove)\n- chore: Add support for Spark 3.5.8 [#3323](https://github.com/apache/datafusion-comet/pull/3323) (manuzhang)\n- chore: Invert usingDataSourceExec test helper to usingLegacyNativeCometScan [#3310](https://github.com/apache/datafusion-comet/pull/3310) (andygrove)\n- tests: Add SQL test files covering edge cases for (almost) every Comet-supported expression [#3328](https://github.com/apache/datafusion-comet/pull/3328) (andygrove)\n- chore: Adapt caching from #3251 to [iceberg] workflows [#3353](https://github.com/apache/datafusion-comet/pull/3353) (mbutrovich)\n- bug: Fix string decimal type throw right exception [#3248](https://github.com/apache/datafusion-comet/pull/3248) (coderfender)\n- chore: Migrate `concat` tests to sql based testing framework [#3352](https://github.com/apache/datafusion-comet/pull/3352) (andygrove)\n- chore(deps): bump actions/setup-java from 4 to 5 [#3363](https://github.com/apache/datafusion-comet/pull/3363) (dependabot[bot])\n- chore: Annotate classes/methods/fields that are used by Apache Iceberg [#3237](https://github.com/apache/datafusion-comet/pull/3237) (andygrove)\n- Feat: map_from_entries [#2905](https://github.com/apache/datafusion-comet/pull/2905) (kazantsev-maksim)\n- chore: Move spark unsafe classes into spark_unsafe [#3373](https://github.com/apache/datafusion-comet/pull/3373) (EmilyMatt)\n- chore: Extract some tied down logic [#3374](https://github.com/apache/datafusion-comet/pull/3374) (EmilyMatt)\n- Fix: array contains null handling [#3372](https://github.com/apache/datafusion-comet/pull/3372) (Shekharrajak)\n- chore: stop uploading code coverage results [#3381](https://github.com/apache/datafusion-comet/pull/3381) (andygrove)\n- chore: update target-cpus in published binaries to x86-64-v3 and neoverse-n1 [#3368](https://github.com/apache/datafusion-comet/pull/3368) (mbutrovich)\n- chore: show line of error sql [#3390](https://github.com/apache/datafusion-comet/pull/3390) (peterxcli)\n- chore: Move writer-related logic to \"writers\" module [#3385](https://github.com/apache/datafusion-comet/pull/3385) (EmilyMatt)\n- chore(deps): bump bytes from 1.11.0 to 1.11.1 in /native [#3380](https://github.com/apache/datafusion-comet/pull/3380) (dependabot[bot])\n- chore: Clean up and split shuffle module [#3395](https://github.com/apache/datafusion-comet/pull/3395) (EmilyMatt)\n- chore: Make PR workflows match target-cpu flags in published jars [#3402](https://github.com/apache/datafusion-comet/pull/3402) (mbutrovich)\n- chore(deps): bump time from 0.3.45 to 0.3.47 in /native [#3412](https://github.com/apache/datafusion-comet/pull/3412) (dependabot[bot])\n- chore: Run Spark SQL tests with `native_datafusion` in CI [#3393](https://github.com/apache/datafusion-comet/pull/3393) (andygrove)\n- test: Add ANSI mode SQL test files for expressions that throw on invalid input [#3377](https://github.com/apache/datafusion-comet/pull/3377) (andygrove)\n- refactor: Split read benchmarks and add addParquetScanCases helper [#3407](https://github.com/apache/datafusion-comet/pull/3407) (andygrove)\n- chore: 4.5x reduction in number of golden files [#3399](https://github.com/apache/datafusion-comet/pull/3399) (andygrove)\n- Feat: to_csv [#3004](https://github.com/apache/datafusion-comet/pull/3004) (kazantsev-maksim)\n- minor: map_from_entries sql tests [#3394](https://github.com/apache/datafusion-comet/pull/3394) (kazantsev-maksim)\n- chore: add confirmation before tarball is released [#3439](https://github.com/apache/datafusion-comet/pull/3439) (milenkovicm)\n- chore(deps): bump cc from 1.2.54 to 1.2.55 in /native [#3451](https://github.com/apache/datafusion-comet/pull/3451) (dependabot[bot])\n- chore: Add Iceberg TPC-H benchmarking scripts [#3294](https://github.com/apache/datafusion-comet/pull/3294) (andygrove)\n- chore: Remove dead code paths for deprecated native_comet scan [#3396](https://github.com/apache/datafusion-comet/pull/3396) (andygrove)\n- chore(deps): bump arrow from 57.2.0 to 57.3.0 in /native [#3449](https://github.com/apache/datafusion-comet/pull/3449) (dependabot[bot])\n- chore(deps): bump aws-config from 1.8.12 to 1.8.13 in /native [#3450](https://github.com/apache/datafusion-comet/pull/3450) (dependabot[bot])\n- chore(deps): bump regex from 1.12.2 to 1.12.3 in /native [#3453](https://github.com/apache/datafusion-comet/pull/3453) (dependabot[bot])\n- chore(deps): bump rand from 0.9.2 to 0.10.0 in /native [#3465](https://github.com/apache/datafusion-comet/pull/3465) (manuzhang)\n- test: Add additional contains expression tests [#3462](https://github.com/apache/datafusion-comet/pull/3462) (andygrove)\n- chore: Adjust native artifact caching key in CI [#3476](https://github.com/apache/datafusion-comet/pull/3476) (mbutrovich)\n- chore: Add Comet writer nested types test assertion [#3480](https://github.com/apache/datafusion-comet/pull/3480) (comphead)\n- test: Add SQL file tests for left and right expressions [#3463](https://github.com/apache/datafusion-comet/pull/3463) (andygrove)\n- chore: Add GitHub workflow to close stale PRs [#3488](https://github.com/apache/datafusion-comet/pull/3488) (andygrove)\n- chore: Make `push` CI to be triggered for `main` branch only [#3474](https://github.com/apache/datafusion-comet/pull/3474) (comphead)\n- ci: disable Miri safety checks until compatibility is restored [#3504](https://github.com/apache/datafusion-comet/pull/3504) (andygrove)\n- chore: Add memory reservation debug logging [#3489](https://github.com/apache/datafusion-comet/pull/3489) (andygrove)\n- chore: enable GitHub button for updating PR branches with latest from main [#3505](https://github.com/apache/datafusion-comet/pull/3505) (andygrove)\n- chore: remove some dead cast code [#3513](https://github.com/apache/datafusion-comet/pull/3513) (andygrove)\n- chore(deps): bump aws-credential-types from 1.2.11 to 1.2.12 in /native [#3525](https://github.com/apache/datafusion-comet/pull/3525) (dependabot[bot])\n- chore(deps): bump libc from 0.2.180 to 0.2.182 in /native [#3527](https://github.com/apache/datafusion-comet/pull/3527) (dependabot[bot])\n- chore(deps): bump cc from 1.2.55 to 1.2.56 in /native [#3528](https://github.com/apache/datafusion-comet/pull/3528) (dependabot[bot])\n- chore(deps): bump tempfile from 3.24.0 to 3.25.0 in /native [#3529](https://github.com/apache/datafusion-comet/pull/3529) (dependabot[bot])\n- ci: Bump up `actions/upload-artifact` from v4 to v6 [#3533](https://github.com/apache/datafusion-comet/pull/3533) (manuzhang)\n- chore(deps): bump aws-config from 1.8.13 to 1.8.14 in /native [#3526](https://github.com/apache/datafusion-comet/pull/3526) (dependabot[bot])\n- chore: refactor array_repeat [#3516](https://github.com/apache/datafusion-comet/pull/3516) (kazantsev-maksim)\n- chore: Add envvars to override writer configs and cometConf minor clean up [#3540](https://github.com/apache/datafusion-comet/pull/3540) (comphead)\n- chore: Cast module refactor boolean module [#3491](https://github.com/apache/datafusion-comet/pull/3491) (coderfender)\n- chore: Consolidate TPC benchmark scripts [#3538](https://github.com/apache/datafusion-comet/pull/3538) (andygrove)\n- chore(deps): bump parquet from 57.2.0 to 57.3.0 in /native [#3568](https://github.com/apache/datafusion-comet/pull/3568) (dependabot[bot])\n- chore(deps): bump uuid from 1.20.0 to 1.21.0 in /native [#3567](https://github.com/apache/datafusion-comet/pull/3567) (dependabot[bot])\n- chore: Add TPC-\\* queries to repo [#3562](https://github.com/apache/datafusion-comet/pull/3562) (andygrove)\n- chore(deps): bump assertables from 9.8.4 to 9.8.6 in /native [#3570](https://github.com/apache/datafusion-comet/pull/3570) (dependabot[bot])\n- chore(deps): bump actions/stale from 10.1.1 to 10.2.0 [#3565](https://github.com/apache/datafusion-comet/pull/3565) (dependabot[bot])\n- chore(deps): bump aws-credential-types from 1.2.12 to 1.2.13 in /native [#3566](https://github.com/apache/datafusion-comet/pull/3566) (dependabot[bot])\n- chore: makes dependabot to group deps into single PR [#3578](https://github.com/apache/datafusion-comet/pull/3578) (comphead)\n- chore: Cast module refactor : String [#3577](https://github.com/apache/datafusion-comet/pull/3577) (coderfender)\n- chore(deps): bump the all-other-cargo-deps group in /native with 3 updates [#3581](https://github.com/apache/datafusion-comet/pull/3581) (dependabot[bot])\n- chore: Add Docker Compose support for TPC benchmarks [#3576](https://github.com/apache/datafusion-comet/pull/3576) (andygrove)\n- build: Runs-on for `PR Build (Linux)` [#3579](https://github.com/apache/datafusion-comet/pull/3579) (blaginin)\n- chore: Add consistency checks and result hashing to TPC benchmarks [#3582](https://github.com/apache/datafusion-comet/pull/3582) (andygrove)\n- chore: Remove all remaining uses of legacy BatchReader from Comet [iceberg] [#3468](https://github.com/apache/datafusion-comet/pull/3468) (andygrove)\n- build: Skip CI workflows for changes in benchmarks directory [#3599](https://github.com/apache/datafusion-comet/pull/3599) (andygrove)\n- build: fix runs-on tags for consistency [#3601](https://github.com/apache/datafusion-comet/pull/3601) (andygrove)\n- chore: Add Java Flight Recorder profiling to TPC benchmarks [#3597](https://github.com/apache/datafusion-comet/pull/3597) (andygrove)\n- deps: DataFusion 52.0.0 migration (SchemaAdapter changes, etc.) [iceberg] [#3536](https://github.com/apache/datafusion-comet/pull/3536) (comphead)\n- chore(deps): bump actions/download-artifact from 7 to 8 [#3609](https://github.com/apache/datafusion-comet/pull/3609) (dependabot[bot])\n- chore(deps): bump actions/upload-artifact from 6 to 7 [#3610](https://github.com/apache/datafusion-comet/pull/3610) (dependabot[bot])\n- chore: bump iceberg-rust dependency to latest [iceberg] [#3606](https://github.com/apache/datafusion-comet/pull/3606) (mbutrovich)\n- CI: Add CodeQL workflow for GitHub Actions security scanning [#3617](https://github.com/apache/datafusion-comet/pull/3617) (kevinjqliu)\n- CI: update codeql with pinned action versions [#3621](https://github.com/apache/datafusion-comet/pull/3621) (kevinjqliu)\n- chore: replace legacy datetime rebase tests with current scan coverage [iceberg] [#3605](https://github.com/apache/datafusion-comet/pull/3605) (andygrove)\n- build: More runners [#3626](https://github.com/apache/datafusion-comet/pull/3626) (blaginin)\n- deps: bump DataFusion to 52.2 [iceberg] [#3622](https://github.com/apache/datafusion-comet/pull/3622) (mbutrovich)\n- chore: use datafusion impl of `space` function [#3612](https://github.com/apache/datafusion-comet/pull/3612) (kazantsev-maksim)\n- chore: use datafusion impl of `bit_count` function [#3616](https://github.com/apache/datafusion-comet/pull/3616) (kazantsev-maksim)\n- chore: refactor cast module numeric data types [#3623](https://github.com/apache/datafusion-comet/pull/3623) (coderfender)\n- chore: Refactor cast module temporal types [#3624](https://github.com/apache/datafusion-comet/pull/3624) (coderfender)\n- chore: Fix clippy complaints [#3634](https://github.com/apache/datafusion-comet/pull/3634) (comphead)\n- chore(deps): bump docker/build-push-action from 6 to 7 [#3639](https://github.com/apache/datafusion-comet/pull/3639) (dependabot[bot])\n- chore(deps): bump github/codeql-action from 4.32.5 to 4.32.6 [#3637](https://github.com/apache/datafusion-comet/pull/3637) (dependabot[bot])\n- chore(deps): bump docker/setup-buildx-action from 3 to 4 [#3636](https://github.com/apache/datafusion-comet/pull/3636) (dependabot[bot])\n- chore(deps): bump docker/login-action from 3 to 4 [#3638](https://github.com/apache/datafusion-comet/pull/3638) (dependabot[bot])\n- deps: update to latest iceberg-rust to pick up get_byte_ranges [iceberg] [#3635](https://github.com/apache/datafusion-comet/pull/3635) (mbutrovich)\n- chore: Array literals tests enable [#3633](https://github.com/apache/datafusion-comet/pull/3633) (comphead)\n- chore: Add debug assertions before unsafe code blocks [#3655](https://github.com/apache/datafusion-comet/pull/3655) (andygrove)\n- chore: fix license header - ansi docs [#3662](https://github.com/apache/datafusion-comet/pull/3662) (coderfender)\n- chore(deps): bump quinn-proto from 0.11.13 to 0.11.14 in /native [#3660](https://github.com/apache/datafusion-comet/pull/3660) (dependabot[bot])\n- ci: add dedicated RAT license check workflow for all PRs [#3664](https://github.com/apache/datafusion-comet/pull/3664) (andygrove)\n- chore: Remove deprecated SCAN_NATIVE_COMET constant and related test code [#3671](https://github.com/apache/datafusion-comet/pull/3671) (andygrove)\n- chore: Upgrade to DF 52.3.0 [#3672](https://github.com/apache/datafusion-comet/pull/3672) (andygrove)\n- deps: update to iceberg-rust 0.9.0 rc1 [iceberg] [#3657](https://github.com/apache/datafusion-comet/pull/3657) (mbutrovich)\n- chore: Mark expressions with known correctness issues as incompatible [#3675](https://github.com/apache/datafusion-comet/pull/3675) (andygrove)\n- chore(deps): bump actions/setup-java from 4 to 5 [#3683](https://github.com/apache/datafusion-comet/pull/3683) (dependabot[bot])\n- chore(deps): bump runs-on/action from 2.0.3 to 2.1.0 [#3684](https://github.com/apache/datafusion-comet/pull/3684) (dependabot[bot])\n- chore(deps): bump actions/checkout from 4 to 6 [#3685](https://github.com/apache/datafusion-comet/pull/3685) (dependabot[bot])\n- ci: remove Java Iceberg integration tests from CI [iceberg] [#3673](https://github.com/apache/datafusion-comet/pull/3673) (andygrove)\n\n## Credits\n\nThank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.\n\n```\n 81\tAndy Grove\n 34\tdependabot[bot]\n 19\tMatt Butrovich\n 9\tB Vadlamani\n 8\tOleks V\n 7\tKazantsev Maksim\n 4\tEmily Matheys\n 4\tManu Zhang\n 4\tShekhar Prasad Rajak\n 2\tBhargava Vadlamani\n 2\tChenChen Lai\n 2\tDmitrii Blaginin\n 2\tKevin Liu\n 2\tParth Chandra\n 2\tPeter Lee\n 2\thsiang-c\n 1\tDavid López\n 1\tMarko Milenković\n 1\tRafael Fernández\n 1\tTornike Gurgenidze\n 1\tn0r0shi\n```\n\nThank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.\n" }, { "path": "dev/changelog/0.14.1.md", "content": "\n\n# DataFusion Comet 0.14.1 Changelog\n\nThis release consists of 5 commits from 1 contributors. See credits at the end of this changelog for more information.\n\n**Fixed bugs:**\n\n- fix: [branch-0.14] backport #3802 - cache object stores and bucket regions to reduce DNS query volume [#3935](https://github.com/apache/datafusion-comet/pull/3935) (andygrove)\n- fix: [branch-0.14] backport #3924 - share unified memory pools across native execution contexts [#3938](https://github.com/apache/datafusion-comet/pull/3938) (andygrove)\n- fix: [branch-0.14] backport #3879 - skip Comet columnar shuffle for stages with DPP scans [#3934](https://github.com/apache/datafusion-comet/pull/3934) (andygrove)\n- fix: [branch-0.14] backport #3914 - use min instead of max when capping write buffer size to Int range [#3936](https://github.com/apache/datafusion-comet/pull/3936) (andygrove)\n- fix: [branch-0.14] backport #3865 - handle ambiguous and non-existent local times [#3937](https://github.com/apache/datafusion-comet/pull/3937) (andygrove)\n\n## Credits\n\nThank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.\n\n```\n 5\tAndy Grove\n```\n\nThank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.\n" }, { "path": "dev/changelog/0.2.0.md", "content": "\n\n# DataFusion Comet 0.2.0 Changelog\n\nThis release consists of 87 commits from 14 contributors. See credits at the end of this changelog for more information.\n\n**Fixed bugs:**\n\n- fix: dictionary decimal vector optimization [#705](https://github.com/apache/datafusion-comet/pull/705) (kazuyukitanimura)\n- fix: Unsupported window expression should fall back to Spark [#710](https://github.com/apache/datafusion-comet/pull/710) (viirya)\n- fix: ReusedExchangeExec can be child operator of CometBroadcastExchangeExec [#713](https://github.com/apache/datafusion-comet/pull/713) (viirya)\n- fix: Fallback to Spark for window expression with range frame [#719](https://github.com/apache/datafusion-comet/pull/719) (viirya)\n- fix: Remove `skip.surefire.tests` mvn property [#739](https://github.com/apache/datafusion-comet/pull/739) (wForget)\n- fix: subquery execution under CometTakeOrderedAndProjectExec should not fail [#748](https://github.com/apache/datafusion-comet/pull/748) (viirya)\n- fix: skip negative scale checks for creating decimals [#723](https://github.com/apache/datafusion-comet/pull/723) (kazuyukitanimura)\n- fix: Fallback to Spark for unsupported partitioning [#759](https://github.com/apache/datafusion-comet/pull/759) (viirya)\n- fix: Unsupported types for SinglePartition should fallback to Spark [#765](https://github.com/apache/datafusion-comet/pull/765) (viirya)\n- fix: unwrap dictionaries in CreateNamedStruct [#754](https://github.com/apache/datafusion-comet/pull/754) (andygrove)\n- fix: Fallback to Spark for unsupported input besides ordering [#768](https://github.com/apache/datafusion-comet/pull/768) (viirya)\n- fix: Native window operator should be CometUnaryExec [#774](https://github.com/apache/datafusion-comet/pull/774) (viirya)\n- fix: Fallback to Spark when shuffling on struct with duplicate field name [#776](https://github.com/apache/datafusion-comet/pull/776) (viirya)\n- fix: withInfo was overwriting information in some cases [#780](https://github.com/apache/datafusion-comet/pull/780) (andygrove)\n- fix: Improve support for nested structs [#800](https://github.com/apache/datafusion-comet/pull/800) (eejbyfeldt)\n- fix: Sort on single struct should fallback to Spark [#811](https://github.com/apache/datafusion-comet/pull/811) (viirya)\n- fix: Check sort order of SortExec instead of child output [#821](https://github.com/apache/datafusion-comet/pull/821) (viirya)\n- fix: Fix panic in `avg` aggregate and disable `stddev` by default [#819](https://github.com/apache/datafusion-comet/pull/819) (andygrove)\n- fix: Supported nested types in HashJoin [#735](https://github.com/apache/datafusion-comet/pull/735) (eejbyfeldt)\n\n**Performance related:**\n\n- perf: Improve performance of CASE .. WHEN expressions [#703](https://github.com/apache/datafusion-comet/pull/703) (andygrove)\n- perf: Optimize IfExpr by delegating to CaseExpr [#681](https://github.com/apache/datafusion-comet/pull/681) (andygrove)\n- fix: optimize isNullAt [#732](https://github.com/apache/datafusion-comet/pull/732) (kazuyukitanimura)\n- perf: decimal decode improvements [#727](https://github.com/apache/datafusion-comet/pull/727) (parthchandra)\n- fix: Remove castting on decimals with a small precision to decimal256 [#741](https://github.com/apache/datafusion-comet/pull/741) (kazuyukitanimura)\n- fix: optimize some bit functions [#718](https://github.com/apache/datafusion-comet/pull/718) (kazuyukitanimura)\n- fix: Optimize getDecimal for small precision [#758](https://github.com/apache/datafusion-comet/pull/758) (kazuyukitanimura)\n- perf: add metrics to CopyExec and ScanExec [#778](https://github.com/apache/datafusion-comet/pull/778) (andygrove)\n- fix: Optimize decimal creation macros [#764](https://github.com/apache/datafusion-comet/pull/764) (kazuyukitanimura)\n- perf: Improve count aggregate performance [#784](https://github.com/apache/datafusion-comet/pull/784) (andygrove)\n- fix: Optimize read_side_padding [#772](https://github.com/apache/datafusion-comet/pull/772) (kazuyukitanimura)\n- perf: Remove some redundant copying of batches [#816](https://github.com/apache/datafusion-comet/pull/816) (andygrove)\n- perf: Remove redundant copying of batches after FilterExec [#835](https://github.com/apache/datafusion-comet/pull/835) (andygrove)\n- fix: Optimize CheckOverflow [#852](https://github.com/apache/datafusion-comet/pull/852) (kazuyukitanimura)\n- perf: Add benchmarks for Spark Scan + Comet Exec [#863](https://github.com/apache/datafusion-comet/pull/863) (andygrove)\n\n**Implemented enhancements:**\n\n- feat: Add support for time-zone, 3 & 5 digit years: Cast from string to timestamp. [#704](https://github.com/apache/datafusion-comet/pull/704) (akhilss99)\n- feat: Support count AggregateUDF for window function [#736](https://github.com/apache/datafusion-comet/pull/736) (huaxingao)\n- feat: Implement basic version of RLIKE [#734](https://github.com/apache/datafusion-comet/pull/734) (andygrove)\n- feat: show executed native plan with metrics when in debug mode [#746](https://github.com/apache/datafusion-comet/pull/746) (andygrove)\n- feat: Add GetStructField expression [#731](https://github.com/apache/datafusion-comet/pull/731) (Kimahriman)\n- feat: Add config to enable native upper and lower string conversion [#767](https://github.com/apache/datafusion-comet/pull/767) (andygrove)\n- feat: Improve native explain [#795](https://github.com/apache/datafusion-comet/pull/795) (andygrove)\n- feat: Add support for null literal with struct type [#797](https://github.com/apache/datafusion-comet/pull/797) (eejbyfeldt)\n- feat: Optimze CreateNamedStruct preserve dictionaries [#789](https://github.com/apache/datafusion-comet/pull/789) (eejbyfeldt)\n- feat: `CreateArray` support [#793](https://github.com/apache/datafusion-comet/pull/793) (Kimahriman)\n- feat: Add native thread configs [#828](https://github.com/apache/datafusion-comet/pull/828) (viirya)\n- feat: Add specific configs for converting Spark Parquet and JSON data to Arrow [#832](https://github.com/apache/datafusion-comet/pull/832) (andygrove)\n- feat: Support sum in window function [#802](https://github.com/apache/datafusion-comet/pull/802) (huaxingao)\n- feat: Simplify configs for enabling/disabling operators [#855](https://github.com/apache/datafusion-comet/pull/855) (andygrove)\n- feat: Enable `clippy::clone_on_ref_ptr` on `proto` and `spark_exprs` crates [#859](https://github.com/apache/datafusion-comet/pull/859) (comphead)\n- feat: Enable `clippy::clone_on_ref_ptr` on `core` crate [#860](https://github.com/apache/datafusion-comet/pull/860) (comphead)\n- feat: Use CometPlugin as main entrypoint [#853](https://github.com/apache/datafusion-comet/pull/853) (andygrove)\n\n**Documentation updates:**\n\n- doc: Update outdated spark.comet.columnar.shuffle.enabled configuration doc [#738](https://github.com/apache/datafusion-comet/pull/738) (wForget)\n- docs: Add explicit configs for enabling operators [#801](https://github.com/apache/datafusion-comet/pull/801) (andygrove)\n- doc: Document CometPlugin to start Comet in cluster mode [#836](https://github.com/apache/datafusion-comet/pull/836) (comphead)\n\n**Other:**\n\n- chore: Make rust clippy happy [#701](https://github.com/apache/datafusion-comet/pull/701) (Xuanwo)\n- chore: Update version to 0.2.0 and add 0.1.0 changelog [#696](https://github.com/apache/datafusion-comet/pull/696) (andygrove)\n- chore: Use rust-toolchain.toml for better toolchain support [#699](https://github.com/apache/datafusion-comet/pull/699) (Xuanwo)\n- chore(native): Make sure all targets in workspace been covered by clippy [#702](https://github.com/apache/datafusion-comet/pull/702) (Xuanwo)\n- Apache DataFusion Comet Logo [#697](https://github.com/apache/datafusion-comet/pull/697) (aocsa)\n- chore: Add logo to rat exclude list [#709](https://github.com/apache/datafusion-comet/pull/709) (andygrove)\n- chore: Use new logo in README and website [#724](https://github.com/apache/datafusion-comet/pull/724) (andygrove)\n- build: Add Comet logo files into exclude list [#726](https://github.com/apache/datafusion-comet/pull/726) (viirya)\n- chore: Remove TPC-DS benchmark results [#728](https://github.com/apache/datafusion-comet/pull/728) (andygrove)\n- chore: make Cast's logic reusable for other projects [#716](https://github.com/apache/datafusion-comet/pull/716) (Blizzara)\n- chore: move scalar_funcs into spark-expr [#712](https://github.com/apache/datafusion-comet/pull/712) (Blizzara)\n- chore: Bump DataFusion to rev 35c2e7e [#740](https://github.com/apache/datafusion-comet/pull/740) (andygrove)\n- chore: add more aggregate functions to benchmark test [#706](https://github.com/apache/datafusion-comet/pull/706) (huaxingao)\n- chore: Add criterion benchmark for decimal_div [#743](https://github.com/apache/datafusion-comet/pull/743) (andygrove)\n- build: Re-enable TPCDS q72 for broadcast and hash join configs [#781](https://github.com/apache/datafusion-comet/pull/781) (viirya)\n- chore: bump DataFusion to rev f4e519f [#783](https://github.com/apache/datafusion-comet/pull/783) (huaxingao)\n- chore: Upgrade to DataFusion rev bddb641 and disable \"skip partial aggregates\" feature [#788](https://github.com/apache/datafusion-comet/pull/788) (andygrove)\n- chore: Remove legacy code for adding a cast to a coalesce [#790](https://github.com/apache/datafusion-comet/pull/790) (andygrove)\n- chore: Use DataFusion 41.0.0-rc1 [#794](https://github.com/apache/datafusion-comet/pull/794) (andygrove)\n- chore: rename `CometRowToColumnar` and fix duplication bug [#785](https://github.com/apache/datafusion-comet/pull/785) (Kimahriman)\n- chore: Enable shuffle in micro benchmarks [#806](https://github.com/apache/datafusion-comet/pull/806) (andygrove)\n- Minor: ScanExec code cleanup and additional documentation [#804](https://github.com/apache/datafusion-comet/pull/804) (andygrove)\n- chore: Make it possible to run 'make benchmark-%' using jvm 17+ [#823](https://github.com/apache/datafusion-comet/pull/823) (eejbyfeldt)\n- chore: Add more unsupported cases to supportedSortType [#825](https://github.com/apache/datafusion-comet/pull/825) (viirya)\n- chore: Enable Comet shuffle with AQE coalesce partitions [#834](https://github.com/apache/datafusion-comet/pull/834) (viirya)\n- chore: Add GitHub workflow to publish Docker image [#847](https://github.com/apache/datafusion-comet/pull/847) (andygrove)\n- chore: Revert \"fix: change the not exists base image apache/spark:3.4.3 to 3.4.2\" [#854](https://github.com/apache/datafusion-comet/pull/854) (haoxins)\n- chore: fix docker-publish attempt 1 [#851](https://github.com/apache/datafusion-comet/pull/851) (andygrove)\n- minor: stop warning that AQEShuffleRead cannot run natively [#842](https://github.com/apache/datafusion-comet/pull/842) (andygrove)\n- chore: Improve ObjectHashAggregate fallback error message [#849](https://github.com/apache/datafusion-comet/pull/849) (andygrove)\n- chore: Fix docker image publishing (specify ghcr.io in tag) [#856](https://github.com/apache/datafusion-comet/pull/856) (andygrove)\n- chore: Use Git tag as Comet version when publishing Docker images [#857](https://github.com/apache/datafusion-comet/pull/857) (andygrove)\n\n## Credits\n\nThank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.\n\n```\n 36\tAndy Grove\n 16\tLiang-Chi Hsieh\n 9\tKAZUYUKI TANIMURA\n 5\tEmil Ejbyfeldt\n 4\tHuaxin Gao\n 3\tAdam Binford\n 3\tOleks V\n 3\tXuanwo\n 2\tArttu\n 2\tZhen Wang\n 1\tAkhil S S\n 1\tAlexander Ocsa\n 1\tParth Chandra\n 1\tXin Hao\n```\n\nThank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.\n" }, { "path": "dev/changelog/0.3.0.md", "content": "\n\n# DataFusion Comet 0.3.0 Changelog\n\nThis release consists of 57 commits from 12 contributors. See credits at the end of this changelog for more information.\n\n**Fixed bugs:**\n\n- fix: Support type coercion for ScalarUDFs [#865](https://github.com/apache/datafusion-comet/pull/865) (Kimahriman)\n- fix: CometTakeOrderedAndProjectExec native scan node should use child operator's output [#896](https://github.com/apache/datafusion-comet/pull/896) (viirya)\n- fix: Fix various memory leaks problems [#890](https://github.com/apache/datafusion-comet/pull/890) (Kontinuation)\n- fix: Add output to Comet operators equal and hashCode [#902](https://github.com/apache/datafusion-comet/pull/902) (viirya)\n- fix: Fallback to Spark when cannot resolve AttributeReference [#926](https://github.com/apache/datafusion-comet/pull/926) (viirya)\n- fix: Fix memory bloat caused by holding too many unclosed `ArrowReaderIterator`s [#929](https://github.com/apache/datafusion-comet/pull/929) (Kontinuation)\n- fix: Normalize NaN and zeros for floating number comparison [#953](https://github.com/apache/datafusion-comet/pull/953) (viirya)\n- fix: window function range offset should be long instead of int [#733](https://github.com/apache/datafusion-comet/pull/733) (huaxingao)\n- fix: CometScanExec on Spark 3.5.2 [#915](https://github.com/apache/datafusion-comet/pull/915) (Kimahriman)\n- fix: div and rem by negative zero [#960](https://github.com/apache/datafusion-comet/pull/960) (kazuyukitanimura)\n\n**Performance related:**\n\n- perf: Optimize CometSparkToColumnar for columnar input [#892](https://github.com/apache/datafusion-comet/pull/892) (mbutrovich)\n- perf: Fall back to Spark if query uses DPP with v1 data sources [#897](https://github.com/apache/datafusion-comet/pull/897) (andygrove)\n- perf: Report accurate total time for scans [#916](https://github.com/apache/datafusion-comet/pull/916) (andygrove)\n- perf: Add metric for time spent casting in native scan [#919](https://github.com/apache/datafusion-comet/pull/919) (andygrove)\n- perf: Add criterion benchmark for aggregate expressions [#948](https://github.com/apache/datafusion-comet/pull/948) (andygrove)\n- perf: Add metric for time spent in CometSparkToColumnarExec [#931](https://github.com/apache/datafusion-comet/pull/931) (mbutrovich)\n- perf: Optimize decimal precision check in decimal aggregates (sum and avg) [#952](https://github.com/apache/datafusion-comet/pull/952) (andygrove)\n\n**Implemented enhancements:**\n\n- feat: Add config option to enable converting CSV to columnar [#871](https://github.com/apache/datafusion-comet/pull/871) (andygrove)\n- feat: Implement basic version of string to float/double/decimal [#870](https://github.com/apache/datafusion-comet/pull/870) (andygrove)\n- feat: Implement to_json for subset of types [#805](https://github.com/apache/datafusion-comet/pull/805) (andygrove)\n- feat: Add ShuffleQueryStageExec to direct child node for CometBroadcastExchangeExec [#880](https://github.com/apache/datafusion-comet/pull/880) (viirya)\n- feat: Support sort merge join with a join condition [#553](https://github.com/apache/datafusion-comet/pull/553) (viirya)\n- feat: Array element extraction [#899](https://github.com/apache/datafusion-comet/pull/899) (Kimahriman)\n- feat: date_add and date_sub functions [#910](https://github.com/apache/datafusion-comet/pull/910) (mbutrovich)\n- feat: implement scripts for binary release build [#932](https://github.com/apache/datafusion-comet/pull/932) (parthchandra)\n- feat: Publish artifacts to maven [#946](https://github.com/apache/datafusion-comet/pull/946) (parthchandra)\n\n**Documentation updates:**\n\n- doc: Documenting Helm chart for Comet Kube execution [#874](https://github.com/apache/datafusion-comet/pull/874) (comphead)\n- doc: Update native code path in development [#921](https://github.com/apache/datafusion-comet/pull/921) (viirya)\n- docs: Add more detailed architecture documentation [#922](https://github.com/apache/datafusion-comet/pull/922) (andygrove)\n\n**Other:**\n\n- chore: Update installation.md [#869](https://github.com/apache/datafusion-comet/pull/869) (haoxins)\n- chore: Update versions to 0.3.0 / 0.3.0-SNAPSHOT [#868](https://github.com/apache/datafusion-comet/pull/868) (andygrove)\n- chore: Add documentation on running benchmarks with Microk8s [#848](https://github.com/apache/datafusion-comet/pull/848) (andygrove)\n- chore: Improve CometExchange metrics [#873](https://github.com/apache/datafusion-comet/pull/873) (viirya)\n- chore: Add spilling metrics of SortMergeJoin [#878](https://github.com/apache/datafusion-comet/pull/878) (viirya)\n- chore: change shuffle mode default from jvm to auto [#877](https://github.com/apache/datafusion-comet/pull/877) (andygrove)\n- chore: Enable shuffle by default [#881](https://github.com/apache/datafusion-comet/pull/881) (andygrove)\n- chore: print Comet native version to logs after Comet is initialized [#900](https://github.com/apache/datafusion-comet/pull/900) (SemyonSinchenko)\n- chore: Revise batch pull approach to more follow C Data interface semantics [#893](https://github.com/apache/datafusion-comet/pull/893) (viirya)\n- chore: Close dictionary provider when iterator is closed [#904](https://github.com/apache/datafusion-comet/pull/904) (andygrove)\n- chore: Remove unused function [#906](https://github.com/apache/datafusion-comet/pull/906) (viirya)\n- chore: Upgrade to latest DataFusion revision [#909](https://github.com/apache/datafusion-comet/pull/909) (andygrove)\n- build: fix build [#917](https://github.com/apache/datafusion-comet/pull/917) (andygrove)\n- chore: Revise array import to more follow C Data Interface semantics [#905](https://github.com/apache/datafusion-comet/pull/905) (viirya)\n- chore: Address reviews [#920](https://github.com/apache/datafusion-comet/pull/920) (viirya)\n- chore: Enable Comet shuffle for Spark core-1 test [#924](https://github.com/apache/datafusion-comet/pull/924) (viirya)\n- build: Add maven-compiler-plugin for java cross-build [#911](https://github.com/apache/datafusion-comet/pull/911) (viirya)\n- build: Disable upload-test-reports for macos-13 runner [#933](https://github.com/apache/datafusion-comet/pull/933) (viirya)\n- minor: cast timestamp test #468 [#923](https://github.com/apache/datafusion-comet/pull/923) (himadripal)\n- build: Set Java version arg for scala-maven-plugin [#934](https://github.com/apache/datafusion-comet/pull/934) (viirya)\n- chore: Remove redundant RowToColumnar from CometShuffleExchangeExec for columnar shuffle [#944](https://github.com/apache/datafusion-comet/pull/944) (viirya)\n- minor: rename CometMetricNode `add` to `set` and update documentation [#940](https://github.com/apache/datafusion-comet/pull/940) (andygrove)\n- chore: Add config for enabling SMJ with join condition [#937](https://github.com/apache/datafusion-comet/pull/937) (andygrove)\n- chore: Change maven group ID to `org.apache.datafusion` [#941](https://github.com/apache/datafusion-comet/pull/941) (andygrove)\n- chore: Upgrade to DataFusion 42.0.0 [#945](https://github.com/apache/datafusion-comet/pull/945) (andygrove)\n- build: Fix regression in jar packaging [#950](https://github.com/apache/datafusion-comet/pull/950) (andygrove)\n- chore: Show reason for falling back to Spark when SMJ with join condition is not enabled [#956](https://github.com/apache/datafusion-comet/pull/956) (andygrove)\n- chore: clarify tarball installation [#959](https://github.com/apache/datafusion-comet/pull/959) (comphead)\n\n## Credits\n\nThank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.\n\n```\n 22\tAndy Grove\n 18\tLiang-Chi Hsieh\n 3\tAdam Binford\n 3\tMatt Butrovich\n 2\tKristin Cowalcijk\n 2\tOleks V\n 2\tParth Chandra\n 1\tHimadri Pal\n 1\tHuaxin Gao\n 1\tKAZUYUKI TANIMURA\n 1\tSemyon\n 1\tXin Hao\n```\n\nThank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.\n" }, { "path": "dev/changelog/0.4.0.md", "content": "\n\n# DataFusion Comet 0.4.0 Changelog\n\nThis release consists of 51 commits from 10 contributors. See credits at the end of this changelog for more information.\n\n**Fixed bugs:**\n\n- fix: Use the number of rows from underlying arrays instead of logical row count from RecordBatch [#972](https://github.com/apache/datafusion-comet/pull/972) (viirya)\n- fix: The spilled_bytes metric of CometSortExec should be size instead of time [#984](https://github.com/apache/datafusion-comet/pull/984) (Kontinuation)\n- fix: Properly handle Java exceptions without error messages; fix loading of comet native library from java.library.path [#982](https://github.com/apache/datafusion-comet/pull/982) (Kontinuation)\n- fix: Fallback to Spark if scan has meta columns [#997](https://github.com/apache/datafusion-comet/pull/997) (viirya)\n- fix: Fallback to Spark if named_struct contains duplicate field names [#1016](https://github.com/apache/datafusion-comet/pull/1016) (viirya)\n- fix: Make comet-git-info.properties optional [#1027](https://github.com/apache/datafusion-comet/pull/1027) (andygrove)\n- fix: TopK operator should return correct results on dictionary column with nulls [#1033](https://github.com/apache/datafusion-comet/pull/1033) (viirya)\n- fix: need default value for getSizeAsMb(EXECUTOR_MEMORY.key) [#1046](https://github.com/apache/datafusion-comet/pull/1046) (neyama)\n\n**Performance related:**\n\n- perf: Remove one redundant CopyExec for SMJ [#962](https://github.com/apache/datafusion-comet/pull/962) (andygrove)\n- perf: Add experimental feature to replace SortMergeJoin with ShuffledHashJoin [#1007](https://github.com/apache/datafusion-comet/pull/1007) (andygrove)\n- perf: Cache jstrings during metrics collection [#1029](https://github.com/apache/datafusion-comet/pull/1029) (mbutrovich)\n\n**Implemented enhancements:**\n\n- feat: Support `GetArrayStructFields` expression [#993](https://github.com/apache/datafusion-comet/pull/993) (Kimahriman)\n- feat: Implement bloom_filter_agg [#987](https://github.com/apache/datafusion-comet/pull/987) (mbutrovich)\n- feat: Support more types with BloomFilterAgg [#1039](https://github.com/apache/datafusion-comet/pull/1039) (mbutrovich)\n- feat: Implement CAST from struct to string [#1066](https://github.com/apache/datafusion-comet/pull/1066) (andygrove)\n- feat: Use official DataFusion 43 release [#1070](https://github.com/apache/datafusion-comet/pull/1070) (andygrove)\n- feat: Implement CAST between struct types [#1074](https://github.com/apache/datafusion-comet/pull/1074) (andygrove)\n- feat: support array_append [#1072](https://github.com/apache/datafusion-comet/pull/1072) (NoeB)\n- feat: Require offHeap memory to be enabled (always use unified memory) [#1062](https://github.com/apache/datafusion-comet/pull/1062) (andygrove)\n\n**Documentation updates:**\n\n- doc: add documentation interlinks [#975](https://github.com/apache/datafusion-comet/pull/975) (comphead)\n- docs: Add IntelliJ documentation for generated source code [#985](https://github.com/apache/datafusion-comet/pull/985) (mbutrovich)\n- docs: Update tuning guide [#995](https://github.com/apache/datafusion-comet/pull/995) (andygrove)\n- docs: Various documentation improvements [#1005](https://github.com/apache/datafusion-comet/pull/1005) (andygrove)\n- docs: clarify that Maven central only has jars for Linux [#1009](https://github.com/apache/datafusion-comet/pull/1009) (andygrove)\n- doc: fix K8s links and doc [#1058](https://github.com/apache/datafusion-comet/pull/1058) (comphead)\n- docs: Update benchmarking.md [#1085](https://github.com/apache/datafusion-comet/pull/1085) (rluvaton-flarion)\n\n**Other:**\n\n- chore: Generate changelog for 0.3.0 release [#964](https://github.com/apache/datafusion-comet/pull/964) (andygrove)\n- chore: fix publish-to-maven script [#966](https://github.com/apache/datafusion-comet/pull/966) (andygrove)\n- chore: Update benchmarks results based on 0.3.0-rc1 [#969](https://github.com/apache/datafusion-comet/pull/969) (andygrove)\n- chore: update rem expression guide [#976](https://github.com/apache/datafusion-comet/pull/976) (kazuyukitanimura)\n- chore: Enable additional CreateArray tests [#928](https://github.com/apache/datafusion-comet/pull/928) (Kimahriman)\n- chore: fix compatibility guide [#978](https://github.com/apache/datafusion-comet/pull/978) (kazuyukitanimura)\n- chore: Update for 0.3.0 release, prepare for 0.4.0 development [#970](https://github.com/apache/datafusion-comet/pull/970) (andygrove)\n- chore: Don't transform the HashAggregate to CometHashAggregate if Comet shuffle is disabled [#991](https://github.com/apache/datafusion-comet/pull/991) (viirya)\n- chore: Make parquet reader options Comet options instead of Hadoop options [#968](https://github.com/apache/datafusion-comet/pull/968) (parthchandra)\n- chore: remove legacy comet-spark-shell [#1013](https://github.com/apache/datafusion-comet/pull/1013) (andygrove)\n- chore: Reserve memory for native shuffle writer per partition [#988](https://github.com/apache/datafusion-comet/pull/988) (viirya)\n- chore: Bump arrow-rs to 53.1.0 and datafusion [#1001](https://github.com/apache/datafusion-comet/pull/1001) (kazuyukitanimura)\n- chore: Revert \"chore: Reserve memory for native shuffle writer per partition (#988)\" [#1020](https://github.com/apache/datafusion-comet/pull/1020) (viirya)\n- minor: Remove hard-coded version number from Dockerfile [#1025](https://github.com/apache/datafusion-comet/pull/1025) (andygrove)\n- chore: Reserve memory for native shuffle writer per partition [#1022](https://github.com/apache/datafusion-comet/pull/1022) (viirya)\n- chore: Improve error handling when native lib fails to load [#1000](https://github.com/apache/datafusion-comet/pull/1000) (andygrove)\n- chore: Use twox-hash 2.0 xxhash64 oneshot api instead of custom implementation [#1041](https://github.com/apache/datafusion-comet/pull/1041) (NoeB)\n- chore: Refactor Arrow Array and Schema allocation in ColumnReader and MetadataColumnReader [#1047](https://github.com/apache/datafusion-comet/pull/1047) (viirya)\n- minor: Refactor binary expr serde to reduce code duplication [#1053](https://github.com/apache/datafusion-comet/pull/1053) (andygrove)\n- chore: Upgrade to DataFusion 43.0.0-rc1 [#1057](https://github.com/apache/datafusion-comet/pull/1057) (andygrove)\n- chore: Refactor UnaryExpr and MathExpr in protobuf [#1056](https://github.com/apache/datafusion-comet/pull/1056) (andygrove)\n- minor: use defaults instead of hard-coding values [#1060](https://github.com/apache/datafusion-comet/pull/1060) (andygrove)\n- minor: refactor UnaryExpr handling to make code more concise [#1065](https://github.com/apache/datafusion-comet/pull/1065) (andygrove)\n- chore: Refactor binary and math expression serde code [#1069](https://github.com/apache/datafusion-comet/pull/1069) (andygrove)\n- chore: Simplify CometShuffleMemoryAllocator to use Spark unified memory allocator [#1063](https://github.com/apache/datafusion-comet/pull/1063) (viirya)\n- test: Restore one test in CometExecSuite by adding COMET_SHUFFLE_MODE config [#1087](https://github.com/apache/datafusion-comet/pull/1087) (viirya)\n\n## Credits\n\nThank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.\n\n```\n 19\tAndy Grove\n 13\tMatt Butrovich\n 8\tLiang-Chi Hsieh\n 3\tKAZUYUKI TANIMURA\n 2\tAdam Binford\n 2\tKristin Cowalcijk\n 1\tNoeB\n 1\tOleks V\n 1\tParth Chandra\n 1\tneyama\n```\n\nThank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.\n" }, { "path": "dev/changelog/0.5.0.md", "content": "\n\n# DataFusion Comet 0.5.0 Changelog\n\nThis release consists of 69 commits from 15 contributors. See credits at the end of this changelog for more information.\n\n**Fixed bugs:**\n\n- fix: Unsigned type related bugs [#1095](https://github.com/apache/datafusion-comet/pull/1095) (kazuyukitanimura)\n- fix: Use RDD partition index [#1112](https://github.com/apache/datafusion-comet/pull/1112) (viirya)\n- fix: Various metrics bug fixes and improvements [#1111](https://github.com/apache/datafusion-comet/pull/1111) (andygrove)\n- fix: Don't create CometScanExec for subclasses of ParquetFileFormat [#1129](https://github.com/apache/datafusion-comet/pull/1129) (Kimahriman)\n- fix: Fix metrics regressions [#1132](https://github.com/apache/datafusion-comet/pull/1132) (andygrove)\n- fix: Enable scenarios accidentally commented out in CometExecBenchmark [#1151](https://github.com/apache/datafusion-comet/pull/1151) (mbutrovich)\n- fix: Spark 4.0-preview1 SPARK-47120 [#1156](https://github.com/apache/datafusion-comet/pull/1156) (kazuyukitanimura)\n- fix: Document enabling comet explain plan usage in Spark (4.0) [#1176](https://github.com/apache/datafusion-comet/pull/1176) (parthchandra)\n- fix: stddev_pop should not directly return 0.0 when count is 1.0 [#1184](https://github.com/apache/datafusion-comet/pull/1184) (viirya)\n- fix: fix missing explanation for then branch in case when [#1200](https://github.com/apache/datafusion-comet/pull/1200) (rluvaton)\n- fix: Fall back to Spark for unsupported partition or sort expressions in window aggregates [#1253](https://github.com/apache/datafusion-comet/pull/1253) (andygrove)\n- fix: Fall back to Spark for distinct aggregates [#1262](https://github.com/apache/datafusion-comet/pull/1262) (andygrove)\n- fix: disable initCap by default [#1276](https://github.com/apache/datafusion-comet/pull/1276) (kazuyukitanimura)\n\n**Performance related:**\n\n- perf: Stop passing Java config map into native createPlan [#1101](https://github.com/apache/datafusion-comet/pull/1101) (andygrove)\n- feat: Make native shuffle compression configurable and respect `spark.shuffle.compress` [#1185](https://github.com/apache/datafusion-comet/pull/1185) (andygrove)\n- perf: Improve query planning to more reliably fall back to columnar shuffle when native shuffle is not supported [#1209](https://github.com/apache/datafusion-comet/pull/1209) (andygrove)\n- feat: Move shuffle block decompression and decoding to native code and add LZ4 & Snappy support [#1192](https://github.com/apache/datafusion-comet/pull/1192) (andygrove)\n- feat: Implement custom RecordBatch serde for shuffle for improved performance [#1190](https://github.com/apache/datafusion-comet/pull/1190) (andygrove)\n\n**Implemented enhancements:**\n\n- feat: support array_insert [#1073](https://github.com/apache/datafusion-comet/pull/1073) (SemyonSinchenko)\n- feat: enable decimal to decimal cast of different precision and scale [#1086](https://github.com/apache/datafusion-comet/pull/1086) (himadripal)\n- feat: Improve ScanExec native metrics [#1133](https://github.com/apache/datafusion-comet/pull/1133) (andygrove)\n- feat: Add Spark-compatible implementation of SchemaAdapterFactory [#1169](https://github.com/apache/datafusion-comet/pull/1169) (andygrove)\n- feat: Improve shuffle metrics (second attempt) [#1175](https://github.com/apache/datafusion-comet/pull/1175) (andygrove)\n- feat: Add a `spark.comet.exec.memoryPool` configuration for experimenting with various datafusion memory pool setups. [#1021](https://github.com/apache/datafusion-comet/pull/1021) (Kontinuation)\n- feat: Reenable tests for filtered SMJ anti join [#1211](https://github.com/apache/datafusion-comet/pull/1211) (comphead)\n- feat: add support for array_remove expression [#1179](https://github.com/apache/datafusion-comet/pull/1179) (jatin510)\n\n**Documentation updates:**\n\n- docs: Update documentation for 0.4.0 release [#1096](https://github.com/apache/datafusion-comet/pull/1096) (andygrove)\n- docs: Fix readme typo FGPA -> FPGA [#1117](https://github.com/apache/datafusion-comet/pull/1117) (gstvg)\n- docs: Add more technical detail and new diagram to Comet plugin overview [#1119](https://github.com/apache/datafusion-comet/pull/1119) (andygrove)\n- docs: Add some documentation explaining how shuffle works [#1148](https://github.com/apache/datafusion-comet/pull/1148) (andygrove)\n- docs: Update TPC-H benchmark results [#1257](https://github.com/apache/datafusion-comet/pull/1257) (andygrove)\n\n**Other:**\n\n- chore: Add changelog for 0.4.0 [#1089](https://github.com/apache/datafusion-comet/pull/1089) (andygrove)\n- chore: Prepare for 0.5.0 development [#1090](https://github.com/apache/datafusion-comet/pull/1090) (andygrove)\n- build: Skip installation of spark-integration and fuzz testing modules [#1091](https://github.com/apache/datafusion-comet/pull/1091) (parthchandra)\n- minor: Add hint for finding the GPG key to use when publishing to maven [#1093](https://github.com/apache/datafusion-comet/pull/1093) (andygrove)\n- chore: Include first ScanExec batch in metrics [#1105](https://github.com/apache/datafusion-comet/pull/1105) (andygrove)\n- chore: Improve CometScan metrics [#1100](https://github.com/apache/datafusion-comet/pull/1100) (andygrove)\n- chore: Add custom metric for native shuffle fetching batches from JVM [#1108](https://github.com/apache/datafusion-comet/pull/1108) (andygrove)\n- chore: Remove unused StringView struct [#1143](https://github.com/apache/datafusion-comet/pull/1143) (andygrove)\n- test: enable more Spark 4.0 tests [#1145](https://github.com/apache/datafusion-comet/pull/1145) (kazuyukitanimura)\n- chore: Refactor cast to use SparkCastOptions param [#1146](https://github.com/apache/datafusion-comet/pull/1146) (andygrove)\n- chore: Move more expressions from core crate to spark-expr crate [#1152](https://github.com/apache/datafusion-comet/pull/1152) (andygrove)\n- chore: Remove dead code [#1155](https://github.com/apache/datafusion-comet/pull/1155) (andygrove)\n- chore: Move string kernels and expressions to spark-expr crate [#1164](https://github.com/apache/datafusion-comet/pull/1164) (andygrove)\n- chore: Move remaining expressions to spark-expr crate + some minor refactoring [#1165](https://github.com/apache/datafusion-comet/pull/1165) (andygrove)\n- chore: Add ignored tests for reading complex types from Parquet [#1167](https://github.com/apache/datafusion-comet/pull/1167) (andygrove)\n- test: enabling Spark tests with offHeap requirement [#1177](https://github.com/apache/datafusion-comet/pull/1177) (kazuyukitanimura)\n- minor: move shuffle classes from common to spark [#1193](https://github.com/apache/datafusion-comet/pull/1193) (andygrove)\n- minor: refactor to move decodeBatches to broadcast exchange code as private function [#1195](https://github.com/apache/datafusion-comet/pull/1195) (andygrove)\n- minor: refactor prepare_output so that it does not require an ExecutionContext [#1194](https://github.com/apache/datafusion-comet/pull/1194) (andygrove)\n- minor: remove unused source files [#1202](https://github.com/apache/datafusion-comet/pull/1202) (andygrove)\n- chore: Upgrade to DataFusion 44.0.0-rc2 [#1154](https://github.com/apache/datafusion-comet/pull/1154) (andygrove)\n- chore: Add safety check to CometBuffer [#1050](https://github.com/apache/datafusion-comet/pull/1050) (viirya)\n- chore: Remove unreachable code [#1213](https://github.com/apache/datafusion-comet/pull/1213) (andygrove)\n- test: Enable Comet by default except some tests in SparkSessionExtensionSuite [#1201](https://github.com/apache/datafusion-comet/pull/1201) (kazuyukitanimura)\n- chore: extract `struct` expressions to folders based on spark grouping [#1216](https://github.com/apache/datafusion-comet/pull/1216) (rluvaton)\n- chore: extract static invoke expressions to folders based on spark grouping [#1217](https://github.com/apache/datafusion-comet/pull/1217) (rluvaton)\n- chore: Follow-on PR to fully enable onheap memory usage [#1210](https://github.com/apache/datafusion-comet/pull/1210) (andygrove)\n- chore: extract agg_funcs expressions to folders based on spark grouping [#1224](https://github.com/apache/datafusion-comet/pull/1224) (rluvaton)\n- chore: extract datetime_funcs expressions to folders based on spark grouping [#1222](https://github.com/apache/datafusion-comet/pull/1222) (rluvaton)\n- chore: Upgrade to DataFusion 44.0.0 from 44.0.0 RC2 [#1232](https://github.com/apache/datafusion-comet/pull/1232) (rluvaton)\n- chore: extract strings file to `strings_func` like in spark grouping [#1215](https://github.com/apache/datafusion-comet/pull/1215) (rluvaton)\n- chore: extract predicate_functions expressions to folders based on spark grouping [#1218](https://github.com/apache/datafusion-comet/pull/1218) (rluvaton)\n- build(deps): bump protobuf version to 3.21.12 [#1234](https://github.com/apache/datafusion-comet/pull/1234) (wForget)\n- chore: extract json_funcs expressions to folders based on spark grouping [#1220](https://github.com/apache/datafusion-comet/pull/1220) (rluvaton)\n- test: Enable shuffle by default in Spark tests [#1240](https://github.com/apache/datafusion-comet/pull/1240) (kazuyukitanimura)\n- chore: extract hash_funcs expressions to folders based on spark grouping [#1221](https://github.com/apache/datafusion-comet/pull/1221) (rluvaton)\n- build: Fix test failure caused by merging conflicting PRs [#1259](https://github.com/apache/datafusion-comet/pull/1259) (andygrove)\n\n## Credits\n\nThank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.\n\n```\n 37\tAndy Grove\n 10\tRaz Luvaton\n 7\tKAZUYUKI TANIMURA\n 3\tLiang-Chi Hsieh\n 2\tParth Chandra\n 1\tAdam Binford\n 1\tDharan Aditya\n 1\tHimadri Pal\n 1\tJagdish Parihar\n 1\tKristin Cowalcijk\n 1\tMatt Butrovich\n 1\tOleks V\n 1\tSem\n 1\tZhen Wang\n 1\tgstvg\n```\n\nThank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.\n" }, { "path": "dev/changelog/0.6.0.md", "content": "\n\n# DataFusion Comet 0.6.0 Changelog\n\n**Fixed bugs:**\n\n- fix: cast timestamp to decimal is unsupported [#1281](https://github.com/apache/datafusion-comet/pull/1281) (wForget)\n- fix: partially fix consistency issue of hash functions with decimal input [#1295](https://github.com/apache/datafusion-comet/pull/1295) (wForget)\n- fix: Improve testing for array_remove and fallback to Spark for unsupported types [#1308](https://github.com/apache/datafusion-comet/pull/1308) (andygrove)\n- fix: address post merge comet-parquet-exec review comments [#1327](https://github.com/apache/datafusion-comet/pull/1327) (parthchandra)\n- fix: memory pool error type [#1346](https://github.com/apache/datafusion-comet/pull/1346) (kazuyukitanimura)\n- fix: Fall back to Spark when hashing decimals with precision > 18 [#1325](https://github.com/apache/datafusion-comet/pull/1325) (andygrove)\n- fix: expressions doc for ArrayRemove [#1356](https://github.com/apache/datafusion-comet/pull/1356) (kazuyukitanimura)\n- fix: pass scale to DF round in spark_round [#1341](https://github.com/apache/datafusion-comet/pull/1341) (cht42)\n- fix: Mark cast from float/double to decimal as incompatible [#1372](https://github.com/apache/datafusion-comet/pull/1372) (andygrove)\n- fix: Passthrough condition in StaticInvoke case block [#1392](https://github.com/apache/datafusion-comet/pull/1392) (EmilyMatt)\n- fix: disable checking for uint_8 and uint_16 if complex type readers are enabled [#1376](https://github.com/apache/datafusion-comet/pull/1376) (parthchandra)\n\n**Performance related:**\n\n- perf: improve performance of update metrics [#1329](https://github.com/apache/datafusion-comet/pull/1329) (wForget)\n- perf: Use DataFusion FilterExec for experimental native scans [#1395](https://github.com/apache/datafusion-comet/pull/1395) (mbutrovich)\n\n**Implemented enhancements:**\n\n- feat: Add HasRowIdMapping interface [#1288](https://github.com/apache/datafusion-comet/pull/1288) (viirya)\n- feat: Upgrade to DataFusion 45 [#1364](https://github.com/apache/datafusion-comet/pull/1364) (andygrove)\n- feat: Add fair unified memory pool [#1369](https://github.com/apache/datafusion-comet/pull/1369) (kazuyukitanimura)\n- feat: Add unbounded memory pool [#1386](https://github.com/apache/datafusion-comet/pull/1386) (kazuyukitanimura)\n- feat: make random seed configurable in fuzz-testing [#1401](https://github.com/apache/datafusion-comet/pull/1401) (wForget)\n- feat: override executor overhead memory only when comet unified memory manager is disabled [#1379](https://github.com/apache/datafusion-comet/pull/1379) (wForget)\n\n**Documentation updates:**\n\n- docs: Fix links and provide complete benchmarking scripts [#1284](https://github.com/apache/datafusion-comet/pull/1284) (andygrove)\n- doc: update memory tuning guide [#1394](https://github.com/apache/datafusion-comet/pull/1394) (kazuyukitanimura)\n\n**Other:**\n\n- chore: Start 0.6.0 development [#1286](https://github.com/apache/datafusion-comet/pull/1286) (andygrove)\n- minor: update compatibility [#1303](https://github.com/apache/datafusion-comet/pull/1303) (kazuyukitanimura)\n- chore: extract conversion_funcs, conditional_funcs, bitwise_funcs and array_funcs expressions to folders based on spark grouping [#1223](https://github.com/apache/datafusion-comet/pull/1223) (rluvaton)\n- chore: extract math_funcs expressions to folders based on spark grouping [#1219](https://github.com/apache/datafusion-comet/pull/1219) (rluvaton)\n- chore: merge comet-parquet-exec branch into main [#1318](https://github.com/apache/datafusion-comet/pull/1318) (andygrove)\n- Feat: Support array_intersect function [#1271](https://github.com/apache/datafusion-comet/pull/1271) (erenavsarogullari)\n- build(deps): bump pprof from 0.13.0 to 0.14.0 in /native [#1319](https://github.com/apache/datafusion-comet/pull/1319) (dependabot[bot])\n- chore: Fix merge conflicts from merging comet-parquet-exec into main [#1320](https://github.com/apache/datafusion-comet/pull/1320) (andygrove)\n- chore: Revert accidental re-introduction of off-heap memory requirement [#1326](https://github.com/apache/datafusion-comet/pull/1326) (andygrove)\n- chore: Fix merge conflicts from merging comet-parquet-exec into main [#1323](https://github.com/apache/datafusion-comet/pull/1323) (mbutrovich)\n- Feat: Support array_join function [#1290](https://github.com/apache/datafusion-comet/pull/1290) (erenavsarogullari)\n- Fix missing slash in spark script [#1334](https://github.com/apache/datafusion-comet/pull/1334) (xleoken)\n- chore: Refactor QueryPlanSerde to allow logic to be moved to individual classes per expression [#1331](https://github.com/apache/datafusion-comet/pull/1331) (andygrove)\n- build: re-enable upload-test-reports for macos-13 runner [#1335](https://github.com/apache/datafusion-comet/pull/1335) (viirya)\n- chore: Upgrade to Arrow 53.4.0 [#1338](https://github.com/apache/datafusion-comet/pull/1338) (andygrove)\n- Feat: Support arrays_overlap function [#1312](https://github.com/apache/datafusion-comet/pull/1312) (erenavsarogullari)\n- chore: Move all array\\_\\* serde to new framework, use correct INCOMPAT config [#1349](https://github.com/apache/datafusion-comet/pull/1349) (andygrove)\n- chore: Prepare for DataFusion 45 (bump to DataFusion rev 5592834 + Arrow 54.0.0) [#1332](https://github.com/apache/datafusion-comet/pull/1332) (andygrove)\n- minor: commit compatibility doc [#1358](https://github.com/apache/datafusion-comet/pull/1358) (kazuyukitanimura)\n- minor: update fuzz dependency [#1357](https://github.com/apache/datafusion-comet/pull/1357) (kazuyukitanimura)\n- chore: Remove redundant processing from exprToProtoInternal [#1351](https://github.com/apache/datafusion-comet/pull/1351) (andygrove)\n- chore: Adding an optional `hdfs` crate [#1377](https://github.com/apache/datafusion-comet/pull/1377) (comphead)\n- chore: Refactor aggregate expression serde [#1380](https://github.com/apache/datafusion-comet/pull/1380) (andygrove)\n" }, { "path": "dev/changelog/0.7.0.md", "content": "\n\n# DataFusion Comet 0.7.0 Changelog\n\nThis release consists of 46 commits from 11 contributors. See credits at the end of this changelog for more information.\n\n**Fixed bugs:**\n\n- fix: Change default value of COMET_SCAN_ALLOW_INCOMPATIBLE and add documentation [#1398](https://github.com/apache/datafusion-comet/pull/1398) (andygrove)\n- fix: Reduce cast.rs and utils.rs logic from parquet_support.rs for experimental native scans [#1387](https://github.com/apache/datafusion-comet/pull/1387) (mbutrovich)\n- fix: Remove more cast.rs logic from parquet_support.rs for experimental native scans [#1413](https://github.com/apache/datafusion-comet/pull/1413) (mbutrovich)\n- fix: fix various unit test failures in native_datafusion and native_iceberg_compat readers [#1415](https://github.com/apache/datafusion-comet/pull/1415) (parthchandra)\n- fix: metrics tests for native_datafusion experimental native scan [#1445](https://github.com/apache/datafusion-comet/pull/1445) (mbutrovich)\n- fix: Reduce number of shuffle spill files, fix spilled_bytes metric, add some unit tests [#1440](https://github.com/apache/datafusion-comet/pull/1440) (andygrove)\n- fix: Executor memory overhead overriding [#1462](https://github.com/apache/datafusion-comet/pull/1462) (LukMRVC)\n- fix: Stop copying rust-toolchain to docker file [#1475](https://github.com/apache/datafusion-comet/pull/1475) (andygrove)\n- fix: PartitionBuffers should not have their own MemoryConsumer [#1496](https://github.com/apache/datafusion-comet/pull/1496) (EmilyMatt)\n- fix: enable full decimal to decimal support [#1385](https://github.com/apache/datafusion-comet/pull/1385) (himadripal)\n- fix: use common implementation of handling object store and hdfs urls for native_datafusion and native_iceberg_compat [#1494](https://github.com/apache/datafusion-comet/pull/1494) (parthchandra)\n- fix: Simplify CometShuffleMemoryAllocator logic, rename classes, remove config [#1485](https://github.com/apache/datafusion-comet/pull/1485) (mbutrovich)\n- fix: check overflow for decimal integral division [#1512](https://github.com/apache/datafusion-comet/pull/1512) (wForget)\n\n**Performance related:**\n\n- perf: Update RewriteJoin logic to choose optimal build side [#1424](https://github.com/apache/datafusion-comet/pull/1424) (andygrove)\n- perf: Reduce native shuffle memory overhead by 50% [#1452](https://github.com/apache/datafusion-comet/pull/1452) (andygrove)\n\n**Implemented enhancements:**\n\n- feat: CometNativeScan metrics from ParquetFileMetrics and FileStreamMetrics [#1172](https://github.com/apache/datafusion-comet/pull/1172) (mbutrovich)\n- feat: add experimental remote HDFS support for native DataFusion reader [#1359](https://github.com/apache/datafusion-comet/pull/1359) (comphead)\n- feat: add Win-amd64 profile [#1410](https://github.com/apache/datafusion-comet/pull/1410) (wForget)\n- feat: Support IntegralDivide function [#1428](https://github.com/apache/datafusion-comet/pull/1428) (wForget)\n- feat: Add div operator for fuzz testing and update expression doc [#1464](https://github.com/apache/datafusion-comet/pull/1464) (wForget)\n- feat: Upgrade to DataFusion 46.0.0-rc2 [#1423](https://github.com/apache/datafusion-comet/pull/1423) (andygrove)\n- feat: Add support for rpad [#1470](https://github.com/apache/datafusion-comet/pull/1470) (andygrove)\n- feat: Use official DataFusion 46.0.0 release [#1484](https://github.com/apache/datafusion-comet/pull/1484) (andygrove)\n\n**Documentation updates:**\n\n- docs: Add changelog for 0.6.0 release [#1402](https://github.com/apache/datafusion-comet/pull/1402) (andygrove)\n- docs: Improve documentation for running stability plan tests [#1469](https://github.com/apache/datafusion-comet/pull/1469) (andygrove)\n\n**Other:**\n\n- test: Add experimental native scans to CometReadBenchmark [#1150](https://github.com/apache/datafusion-comet/pull/1150) (mbutrovich)\n- chore: Prepare for 0.7.0 development [#1404](https://github.com/apache/datafusion-comet/pull/1404) (andygrove)\n- chore: Update released version in documentation [#1418](https://github.com/apache/datafusion-comet/pull/1418) (andygrove)\n- chore: Update protobuf to 3.25.5 [#1434](https://github.com/apache/datafusion-comet/pull/1434) (kazuyukitanimura)\n- chore: Update guava to 33.2.1-jre [#1435](https://github.com/apache/datafusion-comet/pull/1435) (kazuyukitanimura)\n- test: Register Spark-compatible expressions with a DataFusion context [#1432](https://github.com/apache/datafusion-comet/pull/1432) (viczsaurav)\n- chore: fixes for kube build [#1421](https://github.com/apache/datafusion-comet/pull/1421) (comphead)\n- build: pin machete to version 0.7.0 [#1444](https://github.com/apache/datafusion-comet/pull/1444) (andygrove)\n- chore: Re-organize shuffle writer code [#1439](https://github.com/apache/datafusion-comet/pull/1439) (andygrove)\n- chore: faster maven mirror [#1447](https://github.com/apache/datafusion-comet/pull/1447) (comphead)\n- build: Use stable channel in rust-toolchain [#1465](https://github.com/apache/datafusion-comet/pull/1465) (andygrove)\n- Feat: support array_compact function [#1321](https://github.com/apache/datafusion-comet/pull/1321) (kazantsev-maksim)\n- chore: Upgrade to Spark 3.5.4 [#1471](https://github.com/apache/datafusion-comet/pull/1471) (andygrove)\n- chore: Enable CI checks for `native_datafusion` scan [#1479](https://github.com/apache/datafusion-comet/pull/1479) (andygrove)\n- chore: Add `native_iceberg_compat` CI checks [#1487](https://github.com/apache/datafusion-comet/pull/1487) (andygrove)\n- chore: Stop disabling readside padding in TPC stability suite [#1491](https://github.com/apache/datafusion-comet/pull/1491) (andygrove)\n- chore: Remove num partitions from repartitioner [#1498](https://github.com/apache/datafusion-comet/pull/1498) (EmilyMatt)\n- test: fix Spark 3.5 tests [#1482](https://github.com/apache/datafusion-comet/pull/1482) (kazuyukitanimura)\n- minor: Remove hard-coded config default [#1503](https://github.com/apache/datafusion-comet/pull/1503) (andygrove)\n- chore: Use Datafusion's existing empty stream [#1517](https://github.com/apache/datafusion-comet/pull/1517) (EmilyMatt)\n\n## Credits\n\nThank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.\n\n```\n 20\tAndy Grove\n 6\tMatt Butrovich\n 4\tZhen Wang\n 3\tEmily Matheys\n 3\tKAZUYUKI TANIMURA\n 3\tOleks V\n 2\tHimadri Pal\n 2\tParth Chandra\n 1\tKazantsev Maksim\n 1\tLukas Moravec\n 1\tSaurav Verma\n```\n\nThank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.\n" }, { "path": "dev/changelog/0.8.0.md", "content": "\n\n# DataFusion Comet 0.8.0 Changelog\n\nThis release consists of 81 commits from 11 contributors. See credits at the end of this changelog for more information.\n\n**Fixed bugs:**\n\n- fix: remove code duplication in native_datafusion and native_iceberg_compat implementations [#1443](https://github.com/apache/datafusion-comet/pull/1443) (parthchandra)\n- fix: Refactor CometScanRule and fix bugs [#1483](https://github.com/apache/datafusion-comet/pull/1483) (andygrove)\n- fix: check if handle has been initialized before closing [#1554](https://github.com/apache/datafusion-comet/pull/1554) (wForget)\n- fix: Taking slicing into account when writing BooleanBuffers as fast-encoding format [#1522](https://github.com/apache/datafusion-comet/pull/1522) (Kontinuation)\n- fix: isCometEnabled name conflict [#1569](https://github.com/apache/datafusion-comet/pull/1569) (kazuyukitanimura)\n- fix: make register_object_store use same session_env as file scan [#1555](https://github.com/apache/datafusion-comet/pull/1555) (wForget)\n- fix: adjust CometNativeScan's doCanonicalize and hashCode for AQE, use DataSourceScanExec trait [#1578](https://github.com/apache/datafusion-comet/pull/1578) (mbutrovich)\n- fix: corrected the logic of eliminating CometSparkToColumnarExec [#1597](https://github.com/apache/datafusion-comet/pull/1597) (wForget)\n- fix: avoid panic caused by close null handle of parquet reader [#1604](https://github.com/apache/datafusion-comet/pull/1604) (wForget)\n- fix: Make AQE capable of converting Comet shuffled joins to Comet broadcast hash joins [#1605](https://github.com/apache/datafusion-comet/pull/1605) (Kontinuation)\n- fix: Making shuffle files generated in native shuffle mode reclaimable [#1568](https://github.com/apache/datafusion-comet/pull/1568) (Kontinuation)\n- fix: Support per-task shuffle write rows and shuffle write time metrics [#1617](https://github.com/apache/datafusion-comet/pull/1617) (Kontinuation)\n- fix: Modify Spark SQL core 2 tests for `native_datafusion` reader, change 3.5.5 diff hash length to 11 [#1641](https://github.com/apache/datafusion-comet/pull/1641) (mbutrovich)\n- fix: fix spark/sql test failures in native_iceberg_compat [#1593](https://github.com/apache/datafusion-comet/pull/1593) (parthchandra)\n- fix: handle missing field correctly in native_iceberg_compat [#1656](https://github.com/apache/datafusion-comet/pull/1656) (parthchandra)\n- fix: better int96 support for experimental native scans [#1652](https://github.com/apache/datafusion-comet/pull/1652) (mbutrovich)\n- fix: respect `ignoreNulls` flag in `first_value` and `last_value` [#1626](https://github.com/apache/datafusion-comet/pull/1626) (andygrove)\n- fix: update row groups count in internal metrics accumulator [#1658](https://github.com/apache/datafusion-comet/pull/1658) (parthchandra)\n- fix: Shuffle should maintain insertion order [#1660](https://github.com/apache/datafusion-comet/pull/1660) (EmilyMatt)\n\n**Performance related:**\n\n- perf: Use a global tokio runtime [#1614](https://github.com/apache/datafusion-comet/pull/1614) (andygrove)\n- perf: Respect Spark's PARQUET_FILTER_PUSHDOWN_ENABLED config [#1619](https://github.com/apache/datafusion-comet/pull/1619) (andygrove)\n- perf: Experimental fix to avoid join strategy regression [#1674](https://github.com/apache/datafusion-comet/pull/1674) (andygrove)\n\n**Implemented enhancements:**\n\n- feat: add read array support [#1456](https://github.com/apache/datafusion-comet/pull/1456) (comphead)\n- feat: introduce hadoop mini cluster to test native scan on hdfs [#1556](https://github.com/apache/datafusion-comet/pull/1556) (wForget)\n- feat: make parquet native scan schema case insensitive [#1575](https://github.com/apache/datafusion-comet/pull/1575) (wForget)\n- feat: enable iceberg compat tests, more tests for complex types [#1550](https://github.com/apache/datafusion-comet/pull/1550) (comphead)\n- feat: pushdown filter for native_iceberg_compat [#1566](https://github.com/apache/datafusion-comet/pull/1566) (wForget)\n- feat: Fix struct of arrays schema issue [#1592](https://github.com/apache/datafusion-comet/pull/1592) (comphead)\n- feat: adding more struct/arrays tests [#1594](https://github.com/apache/datafusion-comet/pull/1594) (comphead)\n- feat: respect `batchSize/workerThreads/blockingThreads` configurations for native_iceberg_compat scan [#1587](https://github.com/apache/datafusion-comet/pull/1587) (wForget)\n- feat: add MAP type support for first level [#1603](https://github.com/apache/datafusion-comet/pull/1603) (comphead)\n- feat: Add more tests for nested types combinations for `native_datafusion` [#1632](https://github.com/apache/datafusion-comet/pull/1632) (comphead)\n- feat: Override MapBuilder values field with expected schema [#1643](https://github.com/apache/datafusion-comet/pull/1643) (comphead)\n- feat: track unified memory pool [#1651](https://github.com/apache/datafusion-comet/pull/1651) (wForget)\n- feat: Add support for complex types in native shuffle [#1655](https://github.com/apache/datafusion-comet/pull/1655) (andygrove)\n\n**Documentation updates:**\n\n- docs: Update configuration guide to show optional configs [#1524](https://github.com/apache/datafusion-comet/pull/1524) (andygrove)\n- docs: Add changelog for 0.7.0 release [#1527](https://github.com/apache/datafusion-comet/pull/1527) (andygrove)\n- docs: Use a shallow clone for Spark SQL test instructions [#1547](https://github.com/apache/datafusion-comet/pull/1547) (mbutrovich)\n- docs: Update benchmark results for 0.7.0 release [#1548](https://github.com/apache/datafusion-comet/pull/1548) (andygrove)\n- doc: Renew `kubernetes.md` [#1549](https://github.com/apache/datafusion-comet/pull/1549) (comphead)\n- docs: various improvements to tuning guide [#1525](https://github.com/apache/datafusion-comet/pull/1525) (andygrove)\n- docs: Update supported Spark versions [#1580](https://github.com/apache/datafusion-comet/pull/1580) (andygrove)\n- docs: change OSX/OS X to macOS [#1584](https://github.com/apache/datafusion-comet/pull/1584) (mbutrovich)\n- docs: docs for benchmarking in aws ec2 [#1601](https://github.com/apache/datafusion-comet/pull/1601) (andygrove)\n- docs: Update compatibility docs for new native scans [#1657](https://github.com/apache/datafusion-comet/pull/1657) (andygrove)\n- doc: Document local HDFS setup [#1673](https://github.com/apache/datafusion-comet/pull/1673) (comphead)\n\n**Other:**\n\n- chore: fix issue in release process [#1528](https://github.com/apache/datafusion-comet/pull/1528) (andygrove)\n- chore: Remove all subdependencies [#1514](https://github.com/apache/datafusion-comet/pull/1514) (EmilyMatt)\n- chore: Drop support for Spark 3.3 (EOL) [#1529](https://github.com/apache/datafusion-comet/pull/1529) (andygrove)\n- chore: Prepare for 0.8.0 development [#1530](https://github.com/apache/datafusion-comet/pull/1530) (andygrove)\n- chore: Re-enable GitHub discussions [#1535](https://github.com/apache/datafusion-comet/pull/1535) (andygrove)\n- chore: [FOLLOWUP] Drop support for Spark 3.3 (EOL) [#1534](https://github.com/apache/datafusion-comet/pull/1534) (kazuyukitanimura)\n- build: Use unique name for surefire artifacts [#1544](https://github.com/apache/datafusion-comet/pull/1544) (andygrove)\n- chore: Update links for released version [#1540](https://github.com/apache/datafusion-comet/pull/1540) (andygrove)\n- chore: Enable Comet explicitly in `CometTPCDSQueryTestSuite` [#1559](https://github.com/apache/datafusion-comet/pull/1559) (andygrove)\n- chore: Fix some inconsistencies in memory pool configuration [#1561](https://github.com/apache/datafusion-comet/pull/1561) (andygrove)\n- upgraded spark 3.5.4 to 3.5.5 [#1565](https://github.com/apache/datafusion-comet/pull/1565) (YanivKunda)\n- minor: fix typo [#1570](https://github.com/apache/datafusion-comet/pull/1570) (wForget)\n- Chore: simplify array related functions impl [#1490](https://github.com/apache/datafusion-comet/pull/1490) (kazantsev-maksim)\n- added fallback using reflection for backward-compatibility [#1573](https://github.com/apache/datafusion-comet/pull/1573) (YanivKunda)\n- chore: Override node name for CometSparkToColumnar [#1577](https://github.com/apache/datafusion-comet/pull/1577) (l0kr)\n- chore: Reimplement ShuffleWriterExec using interleave_record_batch [#1511](https://github.com/apache/datafusion-comet/pull/1511) (Kontinuation)\n- chore: Run Comet tests for more Spark versions [#1582](https://github.com/apache/datafusion-comet/pull/1582) (andygrove)\n- Feat: support array_except function [#1343](https://github.com/apache/datafusion-comet/pull/1343) (kazantsev-maksim)\n- minor: Fix clippy warnings [#1606](https://github.com/apache/datafusion-comet/pull/1606) (Kontinuation)\n- chore: Remove some unwraps in hashing code [#1600](https://github.com/apache/datafusion-comet/pull/1600) (andygrove)\n- chore: Remove redundant shims for getFailOnError [#1608](https://github.com/apache/datafusion-comet/pull/1608) (andygrove)\n- chore: Making comet native operators write spill files to spark local dir [#1581](https://github.com/apache/datafusion-comet/pull/1581) (Kontinuation)\n- chore: Refactor QueryPlanSerde to use idiomatic Scala and reduce verbosity [#1609](https://github.com/apache/datafusion-comet/pull/1609) (andygrove)\n- chore: Create simple fuzz test as part of test suite [#1610](https://github.com/apache/datafusion-comet/pull/1610) (andygrove)\n- chore: Document `testSingleLineQuery` test method [#1628](https://github.com/apache/datafusion-comet/pull/1628) (comphead)\n- chore: Parquet fuzz testing [#1623](https://github.com/apache/datafusion-comet/pull/1623) (andygrove)\n- chore: Change default Spark version to 3.5 [#1620](https://github.com/apache/datafusion-comet/pull/1620) (andygrove)\n- chore: Add manually-triggered CI jobs for testing Spark SQL with native scans [#1624](https://github.com/apache/datafusion-comet/pull/1624) (andygrove)\n- chore: refactor v2 scan conversion [#1621](https://github.com/apache/datafusion-comet/pull/1621) (andygrove)\n- chore: clean up `planner.rs` [#1650](https://github.com/apache/datafusion-comet/pull/1650) (comphead)\n- chore: correct name of pipelines for native_datafusion ci workflow [#1653](https://github.com/apache/datafusion-comet/pull/1653) (parthchandra)\n- chore: Upgrade to datafusion 47.0.0-rc1 and arrow-rs 55.0.0 [#1563](https://github.com/apache/datafusion-comet/pull/1563) (andygrove)\n- chore: Upgrade to datafusion 47.0.0 [#1663](https://github.com/apache/datafusion-comet/pull/1663) (YanivKunda)\n- chore: Enable CometFuzzTestSuite int96 test for experimental native scans (without complex types) [#1664](https://github.com/apache/datafusion-comet/pull/1664) (mbutrovich)\n- chore: Refactor Memory Pools [#1662](https://github.com/apache/datafusion-comet/pull/1662) (EmilyMatt)\n\n## Credits\n\nThank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.\n\n```\n 31\tAndy Grove\n 11\tOleks V\n 10\tZhen Wang\n 7\tKristin Cowalcijk\n 6\tMatt Butrovich\n 5\tParth Chandra\n 3\tEmily Matheys\n 3\tYaniv Kunda\n 2\tKAZUYUKI TANIMURA\n 2\tKazantsev Maksim\n 1\tŁukasz\n```\n\nThank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.\n" }, { "path": "dev/changelog/0.9.0.md", "content": "\n\n# DataFusion Comet 0.9.0 Changelog\n\nThis release consists of 139 commits from 24 contributors. See credits at the end of this changelog for more information.\n\n**Fixed bugs:**\n\n- fix: typo for `instr` in fuzz testing [#1686](https://github.com/apache/datafusion-comet/pull/1686) (mbutrovich)\n- fix: Bucketed scan fallback for native_datafusion Parquet scan [#1720](https://github.com/apache/datafusion-comet/pull/1720) (mbutrovich)\n- fix: Skip row index Spark SQL tests for native_datafusion Parquet scan [#1724](https://github.com/apache/datafusion-comet/pull/1724) (mbutrovich)\n- fix: Check acquired memory when CometMemoryPool grows [#1732](https://github.com/apache/datafusion-comet/pull/1732) (wForget)\n- fix: Fix data race in memory profiling [#1727](https://github.com/apache/datafusion-comet/pull/1727) (andygrove)\n- fix: Enable some DPP Spark SQL tests [#1734](https://github.com/apache/datafusion-comet/pull/1734) (andygrove)\n- fix: support literal null list and map [#1742](https://github.com/apache/datafusion-comet/pull/1742) (kazuyukitanimura)\n- fix: get_struct field is incorrect when struct in array [#1687](https://github.com/apache/datafusion-comet/pull/1687) (comphead)\n- fix: cast map types correctly in schema adapter [#1771](https://github.com/apache/datafusion-comet/pull/1771) (parthchandra)\n- fix: correct schema type checking in native_iceberg_compat [#1755](https://github.com/apache/datafusion-comet/pull/1755) (parthchandra)\n- fix: default values for native_datafusion scan [#1756](https://github.com/apache/datafusion-comet/pull/1756) (mbutrovich)\n- fix: [native_scans] Support `CASE_SENSITIVE` when reading Parquet [#1782](https://github.com/apache/datafusion-comet/pull/1782) (andygrove)\n- fix: cargo install tpchgen-cli in benchmark doc [#1797](https://github.com/apache/datafusion-comet/pull/1797) (zhuqi-lucas)\n- fix: support `map_keys` [#1788](https://github.com/apache/datafusion-comet/pull/1788) (comphead)\n- fix: fall back on nested types for default values [#1799](https://github.com/apache/datafusion-comet/pull/1799) (mbutrovich)\n- fix: Re-enable Spark 4 tests on Linux [#1806](https://github.com/apache/datafusion-comet/pull/1806) (andygrove)\n- fix: fallback to Spark scan if encryption is enabled (native_datafusion/native_iceberg_compat) [#1785](https://github.com/apache/datafusion-comet/pull/1785) (parthchandra)\n- fix: native_iceberg_compat: move checking parquet types above fetching batch [#1809](https://github.com/apache/datafusion-comet/pull/1809) (mbutrovich)\n- fix: translate missing or corrupt file exceptions, fall back if asked to ignore [#1765](https://github.com/apache/datafusion-comet/pull/1765) (mbutrovich)\n- fix: Fix Spark SQL AQE exchange reuse test failures [#1811](https://github.com/apache/datafusion-comet/pull/1811) (coderfender)\n- fix: Enable more Spark SQL tests [#1834](https://github.com/apache/datafusion-comet/pull/1834) (andygrove)\n- fix: support `map_values` [#1835](https://github.com/apache/datafusion-comet/pull/1835) (comphead)\n- fix: Handle case where num_cols == 0 in native execution [#1840](https://github.com/apache/datafusion-comet/pull/1840) (andygrove)\n- fix: Fix shuffle writing rows containing null struct fields [#1845](https://github.com/apache/datafusion-comet/pull/1845) (Kontinuation)\n- fix: Fall back to Spark for `RANGE BETWEEN` window expressions [#1848](https://github.com/apache/datafusion-comet/pull/1848) (andygrove)\n- fix: Remove COMET_SHUFFLE_FALLBACK_TO_COLUMNAR hack [#1865](https://github.com/apache/datafusion-comet/pull/1865) (andygrove)\n- fix: support read Struct by user schema [#1860](https://github.com/apache/datafusion-comet/pull/1860) (comphead)\n- fix: map parquet field_id correctly (native_iceberg_compat) [#1815](https://github.com/apache/datafusion-comet/pull/1815) (parthchandra)\n- fix: cast_struct_to_struct aligns to Spark behavior [#1879](https://github.com/apache/datafusion-comet/pull/1879) (mbutrovich)\n- fix: correctly handle schemas with nested array of struct (native_iceberg_compat) [#1883](https://github.com/apache/datafusion-comet/pull/1883) (parthchandra)\n- fix: set RangePartitioning for native shuffle default to false [#1907](https://github.com/apache/datafusion-comet/pull/1907) (mbutrovich)\n- fix: conflict between #1905 and #1892. [#1919](https://github.com/apache/datafusion-comet/pull/1919) (mbutrovich)\n- fix: Add overflow check to evaluate of sum decimal accumulator [#1922](https://github.com/apache/datafusion-comet/pull/1922) (leung-ming)\n- fix: Fix overflow handling when casting float to decimal [#1914](https://github.com/apache/datafusion-comet/pull/1914) (leung-ming)\n- fix: Ignore a test case fails on Miri [#1951](https://github.com/apache/datafusion-comet/pull/1951) (leung-ming)\n\n**Performance related:**\n\n- perf: Add memory profiling [#1702](https://github.com/apache/datafusion-comet/pull/1702) (andygrove)\n- perf: Add performance tracing capability [#1706](https://github.com/apache/datafusion-comet/pull/1706) (andygrove)\n- perf: Add `COMET_RESPECT_PARQUET_FILTER_PUSHDOWN` config [#1936](https://github.com/apache/datafusion-comet/pull/1936) (andygrove)\n\n**Implemented enhancements:**\n\n- feat: add jemalloc as optional custom allocator [#1679](https://github.com/apache/datafusion-comet/pull/1679) (mbutrovich)\n- feat: support `array_repeat` [#1680](https://github.com/apache/datafusion-comet/pull/1680) (comphead)\n- feat: More warning info for users [#1667](https://github.com/apache/datafusion-comet/pull/1667) (hsiang-c)\n- feat: decode() expression when using 'utf-8' encoding [#1697](https://github.com/apache/datafusion-comet/pull/1697) (mbutrovich)\n- feat: regexp_replace() expression with no starting offset [#1700](https://github.com/apache/datafusion-comet/pull/1700) (mbutrovich)\n- feat: Improve performance tracing feature [#1730](https://github.com/apache/datafusion-comet/pull/1730) (andygrove)\n- feat: Set/cancel with job tag and make max broadcast table size configurable [#1693](https://github.com/apache/datafusion-comet/pull/1693) (wForget)\n- feat: Add support for `expm1` expression from `datafusion-spark` crate [#1711](https://github.com/apache/datafusion-comet/pull/1711) (andygrove)\n- feat: Add config option for showing all Comet plan transformations [#1780](https://github.com/apache/datafusion-comet/pull/1780) (andygrove)\n- feat: Support Type widening: byte → short/int/long, short → int/long [#1770](https://github.com/apache/datafusion-comet/pull/1770) (huaxingao)\n- feat: Translate Hadoop S3A configurations to object_store configurations [#1817](https://github.com/apache/datafusion-comet/pull/1817) (Kontinuation)\n- feat: Upgrade to official DataFusion 48.0.0 release [#1877](https://github.com/apache/datafusion-comet/pull/1877) (andygrove)\n- feat: Add experimental auto mode for `COMET_PARQUET_SCAN_IMPL` [#1747](https://github.com/apache/datafusion-comet/pull/1747) (andygrove)\n- feat: support RangePartitioning with native shuffle [#1862](https://github.com/apache/datafusion-comet/pull/1862) (mbutrovich)\n- feat: Add support for signum expression [#1889](https://github.com/apache/datafusion-comet/pull/1889) (andygrove)\n- feat: Add support to lookup map by key [#1898](https://github.com/apache/datafusion-comet/pull/1898) (comphead)\n- feat: support array_max [#1892](https://github.com/apache/datafusion-comet/pull/1892) (drexler-sky)\n- feat: pass ignore_nulls flag to first and last [#1866](https://github.com/apache/datafusion-comet/pull/1866) (rluvaton)\n- feat: Implement ToPrettyString [#1921](https://github.com/apache/datafusion-comet/pull/1921) (andygrove)\n- feat: Support hadoop s3a config in native_iceberg_compat [#1925](https://github.com/apache/datafusion-comet/pull/1925) (parthchandra)\n- feat: rand expression support [#1199](https://github.com/apache/datafusion-comet/pull/1199) (akupchinskiy)\n- feat: supports array_distinct [#1923](https://github.com/apache/datafusion-comet/pull/1923) (drexler-sky)\n- feat: `auto` scan mode should check for supported file location [#1930](https://github.com/apache/datafusion-comet/pull/1930) (andygrove)\n- feat: Encapsulate Parquet objects [#1920](https://github.com/apache/datafusion-comet/pull/1920) (huaxingao)\n- feat: Change default value of `COMET_NATIVE_SCAN_IMPL` to `auto` [#1933](https://github.com/apache/datafusion-comet/pull/1933) (andygrove)\n- feat: Supports array_union [#1945](https://github.com/apache/datafusion-comet/pull/1945) (drexler-sky)\n\n**Documentation updates:**\n\n- docs: Add changelog for 0.8.0 [#1675](https://github.com/apache/datafusion-comet/pull/1675) (andygrove)\n- docs: Add instructions on running TPC-H on macOS [#1647](https://github.com/apache/datafusion-comet/pull/1647) (andygrove)\n- docs: Add documentation for accelerating Iceberg Parquet scans with Comet [#1683](https://github.com/apache/datafusion-comet/pull/1683) (andygrove)\n- docs: Add note on setting `core.abbrev` when generating diffs [#1735](https://github.com/apache/datafusion-comet/pull/1735) (andygrove)\n- docs: Remove outdated param in macos bench guide [#1748](https://github.com/apache/datafusion-comet/pull/1748) (ding-young)\n- docs: Add instructions for running individual Spark SQL tests from sbt [#1752](https://github.com/apache/datafusion-comet/pull/1752) (coderfender)\n- docs: Add documentation for native_datafusion Parquet scanner's S3 support [#1832](https://github.com/apache/datafusion-comet/pull/1832) (Kontinuation)\n- docs: Add docs stating that Comet does not support reading decimals encoded in Parquet BINARY format [#1895](https://github.com/apache/datafusion-comet/pull/1895) (andygrove)\n\n**Other:**\n\n- chore: Start 0.9.0 development [#1676](https://github.com/apache/datafusion-comet/pull/1676) (andygrove)\n- chore: Update viable crates [#1677](https://github.com/apache/datafusion-comet/pull/1677) (EmilyMatt)\n- chore: match Maven plugin versions with Spark 3.5 [#1668](https://github.com/apache/datafusion-comet/pull/1668) (hsiang-c)\n- chore: Remove fallback reason \"because the children were not native\" [#1672](https://github.com/apache/datafusion-comet/pull/1672) (andygrove)\n- chore: Rename `scalarExprToProto` to `scalarFunctionExprToProto` [#1688](https://github.com/apache/datafusion-comet/pull/1688) (comphead)\n- chore: fix build errors [#1690](https://github.com/apache/datafusion-comet/pull/1690) (comphead)\n- chore: Make Aggregate transformation more compact [#1670](https://github.com/apache/datafusion-comet/pull/1670) (EmilyMatt)\n- chore: update dev/release/rat_exclude_files.txt [#1689](https://github.com/apache/datafusion-comet/pull/1689) (hsiang-c)\n- chore: Move Comet rules into their own files [#1695](https://github.com/apache/datafusion-comet/pull/1695) (andygrove)\n- chore: Remove fast encoding option [#1703](https://github.com/apache/datafusion-comet/pull/1703) (andygrove)\n- chore: fix CI job name [#1712](https://github.com/apache/datafusion-comet/pull/1712) (hsiang-c)\n- minor: Warn if memory pool is dropped with bytes still reserved [#1721](https://github.com/apache/datafusion-comet/pull/1721) (andygrove)\n- chore: Correct memory acquired size in unified memory pool [#1738](https://github.com/apache/datafusion-comet/pull/1738) (zuston)\n- chore: allow large errors for Clippy [#1743](https://github.com/apache/datafusion-comet/pull/1743) (comphead)\n- chore: Refactor DataTypeSupport [#1741](https://github.com/apache/datafusion-comet/pull/1741) (andygrove)\n- chore: More refactoring of type checking logic [#1744](https://github.com/apache/datafusion-comet/pull/1744) (andygrove)\n- chore: Enable more complex type tests [#1753](https://github.com/apache/datafusion-comet/pull/1753) (andygrove)\n- chore: Add `scanImpl` attribute to `CometScanExec` [#1746](https://github.com/apache/datafusion-comet/pull/1746) (andygrove)\n- chore: Prepare for DataFusion 48.0.0 [#1710](https://github.com/apache/datafusion-comet/pull/1710) (andygrove)\n- Docs: Setup Comet on IntelliJ [#1760](https://github.com/apache/datafusion-comet/pull/1760) (coderfender)\n- chore: Reenable nested types for CometFuzzTestSuite with int96 [#1761](https://github.com/apache/datafusion-comet/pull/1761) (mbutrovich)\n- chore: Enable partial Spark SQL tests for `native_iceberg_compat` scan [#1762](https://github.com/apache/datafusion-comet/pull/1762) (andygrove)\n- chore: [native_iceberg_compat / native_datafusion] Ignore Spark SQL Parquet encryption tests [#1763](https://github.com/apache/datafusion-comet/pull/1763) (andygrove)\n- build: Ignore array_repeat test to fix CI issues [#1774](https://github.com/apache/datafusion-comet/pull/1774) (andygrove)\n- chore: Upload crash logs if Java tests fail [#1779](https://github.com/apache/datafusion-comet/pull/1779) (andygrove)\n- chore: Drop support for Java 8 [#1777](https://github.com/apache/datafusion-comet/pull/1777) (andygrove)\n- chore: Bump arrow to 18.3.0 [#1773](https://github.com/apache/datafusion-comet/pull/1773) (Kontinuation)\n- build: Stop running Comet's Spark 4 tests on Linux for PR builds [#1802](https://github.com/apache/datafusion-comet/pull/1802) (andygrove)\n- Chore: Moved strings expressions to separate file [#1792](https://github.com/apache/datafusion-comet/pull/1792) (kazantsev-maksim)\n- chore: Speed up \"PR Builds\" CI workflows [#1807](https://github.com/apache/datafusion-comet/pull/1807) (andygrove)\n- chore: [native scans] Ignore Spark SQL test for string predicate pushdown [#1768](https://github.com/apache/datafusion-comet/pull/1768) (andygrove)\n- chore: Bump DataFusion to git rev 2c2f225 [#1814](https://github.com/apache/datafusion-comet/pull/1814) (andygrove)\n- Feat: support bit_count function [#1602](https://github.com/apache/datafusion-comet/pull/1602) (kazantsev-maksim)\n- Chore: implement bit_not as ScalarUDFImpl [#1825](https://github.com/apache/datafusion-comet/pull/1825) (kazantsev-maksim)\n- build: Specify -Dsbt.log.noformat=true in sbt CI runs [#1822](https://github.com/apache/datafusion-comet/pull/1822) (andygrove)\n- chore: Use unique artifact names in Java test run [#1818](https://github.com/apache/datafusion-comet/pull/1818) (andygrove)\n- minor: Refactor PhysicalPlanner::default() to avoid duplicate code [#1821](https://github.com/apache/datafusion-comet/pull/1821) (andygrove)\n- Chore: implement bit_count as ScalarUDFImpl [#1826](https://github.com/apache/datafusion-comet/pull/1826) (kazantsev-maksim)\n- chore: IgnoreCometNativeScan on a few more Spark SQL tests [#1837](https://github.com/apache/datafusion-comet/pull/1837) (mbutrovich)\n- chore: Enable tests in RemoveRedundantProjectsSuite.scala related to issue #242 [#1838](https://github.com/apache/datafusion-comet/pull/1838) (rishvin)\n- minor: Replace many instances of `checkSparkAnswer` with `checkSparkAnswerAndOperator` [#1851](https://github.com/apache/datafusion-comet/pull/1851) (andygrove)\n- chore: Update documentation and ignore Spark SQL tests for known issue with count distinct on NaN in aggregate [#1847](https://github.com/apache/datafusion-comet/pull/1847) (andygrove)\n- chore: Ignore Spark SQL WholeStageCodegenSuite tests [#1859](https://github.com/apache/datafusion-comet/pull/1859) (andygrove)\n- chore: Upgrade to DataFusion 48.0.0-rc3 [#1863](https://github.com/apache/datafusion-comet/pull/1863) (andygrove)\n- upgraded spark 3.5.5 to 3.5.6 [#1861](https://github.com/apache/datafusion-comet/pull/1861) (YanivKunda)\n- build: Disable some rounding tests when miri is enabled [#1873](https://github.com/apache/datafusion-comet/pull/1873) (andygrove)\n- chore: Enable Spark SQL tests for `native_iceberg_compat` [#1876](https://github.com/apache/datafusion-comet/pull/1876) (andygrove)\n- chore: Enable more Spark SQL tests [#1869](https://github.com/apache/datafusion-comet/pull/1869) (andygrove)\n- chore: refactor planner read schema tests [#1886](https://github.com/apache/datafusion-comet/pull/1886) (comphead)\n- chore: Implement date_trunc as ScalarUDFImpl [#1880](https://github.com/apache/datafusion-comet/pull/1880) (leung-ming)\n- Chore: implement datetime funcs as ScalarUDFImpl [#1874](https://github.com/apache/datafusion-comet/pull/1874) (trompa)\n- minor: Improve testing of math scalar functions [#1896](https://github.com/apache/datafusion-comet/pull/1896) (andygrove)\n- minor: Avoid rewriting join to unsupported join [#1888](https://github.com/apache/datafusion-comet/pull/1888) (andygrove)\n- chore: Enable `native_iceberg_compat` Spark SQL tests (for real, this time) [#1910](https://github.com/apache/datafusion-comet/pull/1910) (andygrove)\n- chore: rename makeParquetFileAllTypes to makeParquetFileAllPrimitiveTypes [#1905](https://github.com/apache/datafusion-comet/pull/1905) (parthchandra)\n- chore: add a test case to read from an arbitrarily complex type schema [#1911](https://github.com/apache/datafusion-comet/pull/1911) (parthchandra)\n- test: Trigger Spark 3.4.3 SQL tests for iceberg-compat [#1912](https://github.com/apache/datafusion-comet/pull/1912) (kazuyukitanimura)\n- build: Fix conflict between #1910 and #1912 [#1924](https://github.com/apache/datafusion-comet/pull/1924) (andygrove)\n- minor: fix kube/Dockerfile build failed [#1918](https://github.com/apache/datafusion-comet/pull/1918) (zhangxffff)\n- chore: Improve reporting of fallback reasons for CollectLimit [#1694](https://github.com/apache/datafusion-comet/pull/1694) (andygrove)\n- chore: move udf registration to better place [#1899](https://github.com/apache/datafusion-comet/pull/1899) (rluvaton)\n- chore: Comet + Iceberg (1.8.1) CI [#1715](https://github.com/apache/datafusion-comet/pull/1715) (hsiang-c)\n- chore: Introduce `exprHandlers` map in QueryPlanSerde [#1903](https://github.com/apache/datafusion-comet/pull/1903) (andygrove)\n- chore: Enable Spark SQL tests for auto scan mode [#1885](https://github.com/apache/datafusion-comet/pull/1885) (andygrove)\n- Feat: support bit_get function [#1713](https://github.com/apache/datafusion-comet/pull/1713) (kazantsev-maksim)\n- chore: Clippy fixes for Rust 1.88 [#1939](https://github.com/apache/datafusion-comet/pull/1939) (andygrove)\n- Minor: Add unit tests for `ceil`/`floor` functions [#1728](https://github.com/apache/datafusion-comet/pull/1728) (tlm365)\n\n## Credits\n\nThank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.\n\n```\n 62\tAndy Grove\n 16\tMatt Butrovich\n 10\tOleks V\n 8\tParth Chandra\n 5\tKazantsev Maksim\n 5\thsiang-c\n 4\tKristin Cowalcijk\n 4\tLeung Ming\n 3\tB Vadlamani\n 3\tdrexler-sky\n 2\tEmily Matheys\n 2\tHuaxin Gao\n 2\tKAZUYUKI TANIMURA\n 2\tRaz Luvaton\n 2\tZhen Wang\n 1\tArtem Kupchinskiy\n 1\tJunfan Zhang\n 1\tQi Zhu\n 1\tRishab Joshi\n 1\tTai Le Manh\n 1\tYaniv Kunda\n 1\tZhang Xiaofeng\n 1\tding-young\n 1\ttrompa\n```\n\nThank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.\n" }, { "path": "dev/changelog/0.9.1.md", "content": "\n\n# DataFusion Comet 0.9.1 Changelog\n\nThis release consists of 2 commits from 1 contributors. See credits at the end of this changelog for more information.\n\n**Fixed bugs:**\n\n- fix: [branch-0.9] Backport FFI fix [#2164](https://github.com/apache/datafusion-comet/pull/2164) (andygrove)\n- fix: [branch-0.9] Avoid double free in CometUnifiedShuffleMemoryAllocator [#2201](https://github.com/apache/datafusion-comet/pull/2201) (andygrove)\n\n## Credits\n\nThank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.\n\n```\n 2\tAndy Grove\n```\n\nThank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.\n" }, { "path": "dev/checkstyle-suppressions.xml", "content": "\n\n\n\n\n\n\n\n\n" }, { "path": "dev/ci/check-suites.py", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nimport sys\nfrom pathlib import Path\n\ndef file_to_class_name(path: Path) -> str | None:\n parts = path.parts\n if \"org\" not in parts or \"apache\" not in parts:\n return None\n org_index = parts.index(\"org\")\n package_parts = parts[org_index:]\n class_name = \".\".join(package_parts)\n class_name = class_name.replace(\".scala\", \"\")\n return class_name\n\nif __name__ == \"__main__\":\n\n # ignore traits, abstract classes, and intentionally skipped test suites\n ignore_list = [\n \"org.apache.comet.parquet.ParquetReadSuite\", # abstract\n \"org.apache.comet.parquet.ParquetReadFromS3Suite\", # manual test suite\n \"org.apache.comet.IcebergReadFromS3Suite\", # manual test suite\n \"org.apache.spark.sql.comet.CometPlanStabilitySuite\", # abstract\n \"org.apache.spark.sql.comet.ParquetDatetimeRebaseSuite\", # abstract\n \"org.apache.comet.exec.CometColumnarShuffleSuite\" # abstract\n ]\n\n for workflow_filename in [\".github/workflows/pr_build_linux.yml\", \".github/workflows/pr_build_macos.yml\"]:\n workflow = open(workflow_filename, encoding=\"utf-8\").read()\n\n root = Path(\".\")\n for path in root.rglob(\"*Suite.scala\"):\n class_name = file_to_class_name(path)\n if class_name:\n if \"Shim\" in class_name:\n continue\n if class_name in ignore_list:\n continue\n if class_name not in workflow:\n print(f\"Suite not found in workflow {workflow_filename}: {class_name}\")\n sys.exit(-1)\n print(f\"Found {class_name} in {workflow_filename}\")\n" }, { "path": "dev/ci/check-working-tree-clean.sh", "content": "#!/bin/bash\n#\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\nset -euo pipefail # Exit on errors, undefined vars, pipe failures\n\n# Check if we're in a git repository\nif ! git rev-parse --git-dir > /dev/null 2>&1; then\n echo \"Error: Not in a git repository\"\n # exit 1\nfi\n\n# Fail if there are any local changes (staged, unstaged, or untracked)\nif [ -n \"$(git status --porcelain)\" ]; then\n echo \"Working tree is not clean:\"\n git status --short\n echo \"Full diff:\"\n git diff\n echo \"\"\n echo \"Please commit, stash, or clean these changes before proceeding.\"\n exit 1\nelse\n echo \"Working tree is clean\"\nfi\n\n" }, { "path": "dev/copyright/java-header.txt", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n" }, { "path": "dev/diffs/3.4.3.diff", "content": "diff --git a/pom.xml b/pom.xml\nindex d3544881af1..377683b10c5 100644\n--- a/pom.xml\n+++ b/pom.xml\n@@ -148,6 +148,8 @@\n 0.10.0\n 2.5.1\n 2.0.8\n+ 3.4\n+ 0.15.0-SNAPSHOT\n \n 2.5.1\n 2.0.8\n+ 3.5\n+ 0.15.0-SNAPSHOT\n \n \n org.apache.datasketches\ndiff --git a/sql/core/pom.xml b/sql/core/pom.xml\nindex dcf6223a98b..0458a5bb640 100644\n--- a/sql/core/pom.xml\n+++ b/sql/core/pom.xml\n@@ -90,6 +90,10 @@\n org.apache.spark\n spark-tags_${scala.binary.version}\n \n+ \n+ org.apache.datafusion\n+ comet-spark-spark${spark.version.short}_${scala.binary.version}\n+ \n \n \\n\"\"\")\n\n print(f\"# DataFusion Comet {version} Changelog\\n\")\n\n # get the number of commits\n commit_count = subprocess.check_output(f\"git log --pretty=oneline {tag1}..{tag2} | wc -l\", shell=True, text=True).strip()\n\n # get number of contributors\n contributor_count = subprocess.check_output(f\"git shortlog -sn {tag1}..{tag2} | wc -l\", shell=True, text=True).strip()\n\n print(f\"This release consists of {commit_count} commits from {contributor_count} contributors. \"\n f\"See credits at the end of this changelog for more information.\\n\")\n\n print_pulls(repo_name, \"Breaking changes\", breaking)\n print_pulls(repo_name, \"Fixed bugs\", bugs)\n print_pulls(repo_name, \"Performance related\", performance)\n print_pulls(repo_name, \"Implemented enhancements\", enhancements)\n print_pulls(repo_name, \"Documentation updates\", docs)\n print_pulls(repo_name, \"Other\", other)\n\n # show code contributions\n credits = subprocess.check_output(f\"git shortlog -sn {tag1}..{tag2}\", shell=True, text=True).rstrip()\n\n print(\"## Credits\\n\")\n print(\"Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) \"\n \"per contributor.\\n\")\n print(\"```\")\n print(credits)\n print(\"```\\n\")\n\n print(\"Thank you also to everyone who contributed in other ways such as filing issues, reviewing \"\n \"PRs, and providing feedback on this release.\\n\")\n\ndef resolve_ref(ref):\n \"\"\"Resolve a git ref (e.g. HEAD, branch name) to a full commit SHA.\"\"\"\n try:\n return subprocess.check_output(\n [\"git\", \"rev-parse\", ref], text=True\n ).strip()\n except subprocess.CalledProcessError:\n # If it can't be resolved locally, return as-is (e.g. a tag name\n # that the GitHub API can resolve)\n return ref\n\n\ndef cli(args=None):\n \"\"\"Process command line arguments.\"\"\"\n if not args:\n args = sys.argv[1:]\n\n parser = argparse.ArgumentParser()\n parser.add_argument(\"tag1\", help=\"The previous commit or tag (e.g. 0.1.0)\")\n parser.add_argument(\"tag2\", help=\"The current commit or tag (e.g. HEAD)\")\n parser.add_argument(\"version\", help=\"The version number to include in the changelog\")\n args = parser.parse_args()\n\n # Resolve refs to SHAs so the GitHub API compares the same commits\n # as the local git log. Without this, refs like HEAD get resolved by\n # the GitHub API to the default branch instead of the current branch.\n tag1 = resolve_ref(args.tag1)\n tag2 = resolve_ref(args.tag2)\n\n token = os.getenv(\"GITHUB_TOKEN\")\n project = \"apache/datafusion-comet\"\n\n g = Github(token)\n repo = g.get_repo(project)\n generate_changelog(repo, project, tag1, tag2, args.version)\n\nif __name__ == \"__main__\":\n cli()" }, { "path": "dev/release/publish-to-maven.sh", "content": "#!/bin/bash\n\n#\n# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n\n###\n# This file is based on the script release-build.sh in Spark\n###\n\nfunction usage {\n local NAME=$(basename $0)\n cat << EOF\nusage: $NAME options\n\nPublish signed artifacts to Maven.\n\nOptions\n -u ASF_USERNAME - Username of ASF committer account\n -r LOCAL_REPO - path to temporary local maven repo (created and written to by 'build-release-comet.sh')\n\nThe following will be prompted for -\n ASF_PASSWORD - Password of ASF committer account\n GPG_KEY - GPG key used to sign release artifacts\n GPG_PASSPHRASE - Passphrase for GPG key\nEOF\n exit 1\n}\n\nSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" >/dev/null && pwd)\"\nCOMET_HOME_DIR=$SCRIPT_DIR/../..\n\nASF_USERNAME=\"\"\nASF_PASSWORD=\"\"\n\nNEXUS_ROOT=https://repository.apache.org/service/local/staging\nNEXUS_PROFILE=789e15c00fd47\n\nwhile getopts \"u:r:h\" opt; do\n case $opt in\n u) ASF_USERNAME=\"$OPTARG\" ;;\n r) LOCAL_REPO=\"$OPTARG\" ;;\n h) usage ;;\n \\?) error \"Invalid option. Run with -h for help.\" ;;\n esac\ndone\n\nif [ \"$LOCAL_REPO\" == \"\" ]\nthen\n \"Please provide the local Maven repository (from 'build-release-comet.sh')\"\n usage\nfi\n\nif [ \"$ASF_USERNAME\" == \"\" ]\nthen\n read -p \"ASF Username : \" ASF_USERNAME && echo \"\"\nfi\n# Read some secret information\nread -s -p \"ASF Password : \" ASF_PASSWORD && echo \"\"\nread -s -p \"GPG Key (Optional): \" GPG_KEY && echo \"\"\nread -s -p \"GPG Passphrase : \" GPG_PASSPHRASE && echo \"\"\n\nif [ \"$ASF_USERNAME\" == \"\" ] || [ \"$ASF_PASSWORD\" == \"\" ] || [ \"$GPG_PASSPHRASE\" = \"\" ]\nthen\n echo \"Missing credentials\"\n exit 1\nfi\n\n# Default GPG command to use\nGPG=\"gpg --pinentry-mode loopback\"\nif [ \"$GPG_KEY\" != \"\" ]\nthen\n GPG=\"$GPG -u $GPG_KEY\"\nfi\n\n# sha1sum on linux, shasum on macos\nSHA1SUM=$( which sha1sum || which shasum)\n\nGIT_HASH=$(git rev-parse --short HEAD)\n\n# REF: https://support.sonatype.com/hc/en-us/articles/213465818-How-can-I-programmatically-upload-an-artifact-into-Nexus-Repo-2\n# REF: https://support.sonatype.com/hc/en-us/articles/213465868-Uploading-to-a-Nexus-Repository-2-Staging-Repository-via-REST-API\necho \"Creating Nexus staging repository\"\n\nREPO_REQUEST=\"Apache Datafusion Comet $COMET_VERSION (commit $GIT_HASH)\"\nREPO_REQUEST_RESPONSE=$(curl -X POST -d \"$REPO_REQUEST\" -u $ASF_USERNAME:$ASF_PASSWORD \\\n -H \"Content-Type:application/xml\" \\\n $NEXUS_ROOT/profiles/$NEXUS_PROFILE/start)\nif [ $? -ne 0 ]\nthen\n echo \"Error creating staged repository\"\n echo \"$REPO_REQUEST_RESPONSE\"\n exit 1\nfi\n\nSTAGED_REPO_ID=$(echo $REPO_REQUEST_RESPONSE | xmllint --xpath \"//stagedRepositoryId/text()\" -)\necho \"Created Nexus staging repository: $STAGED_REPO_ID\"\n\nif [ \"$STAGED_REPO_ID\" == \"\" ]\nthen\n echo \"Error creating staged repository\"\n echo \"$REPO_REQUEST_RESPONSE\"\n exit 1\nfi\n\necho \"Deploying artifacts from $LOCAL_REPO\"\n\npushd $LOCAL_REPO/org/apache/datafusion\n\n# Remove any extra files generated during install\nfind . -type f |grep -v \\.jar |grep -v \\.pom | xargs rm\n\necho \"Creating hash and signature files\"\nfor file in $(find . -type f)\ndo\necho $GPG_PASSPHRASE | $GPG --passphrase-fd 0 --output $file.asc \\\n --detach-sig --armour $file;\nif [ $(command -v md5) ]; then\n # Available on macOS; -q to keep only hash\n md5 -q $file > $file.md5\nelse\n # Available on Linux; cut to keep only hash\n md5sum $file | cut -f1 -d' ' > $file.md5\nfi\n$SHA1SUM $file | cut -f1 -d' ' > $file.sha1\ndone\n\nNEXUS_UPLOAD=$NEXUS_ROOT/deployByRepositoryId/$STAGED_REPO_ID\necho \"Uploading files to $NEXUS_UPLOAD\"\nfor file in $(find . -type f)\ndo\n # strip leading ./\n FILE_SHORT=$(echo $file | sed -e \"s/\\.\\///\")\n DEST_URL=\"$NEXUS_UPLOAD/org/apache/datafusion/$FILE_SHORT\"\n echo \" Uploading $FILE_SHORT\"\n curl -u $ASF_USERNAME:$ASF_PASSWORD --upload-file $FILE_SHORT $DEST_URL\n if [ $? -ne 0 ]\n then\n echo \" - Failed\"\n exit 2\n fi\ndone\n\necho \"Closing nexus staging repository\"\nREPO_REQUEST=\"$STAGED_REPO_IDApache Datafusion Comet $COMET_VERSION (commit $GIT_HASH)\"\nREPO_REQUEST_RESPONSE=$(curl -X POST -d \"$REPO_REQUEST\" -u $ASF_USERNAME:$ASF_PASSWORD \\\n -H \"Content-Type:application/xml\" -v \\\n $NEXUS_ROOT/profiles/$NEXUS_PROFILE/finish)\necho \"Closed Nexus staging repository: $STAGED_REPO_ID\"\n\npopd" }, { "path": "dev/release/rat_exclude_files.txt", "content": "*.gitignore\n*.dockerignore\n.github/pull_request_template.md\n.gitmodules\nnative/Cargo.lock\nnative/testdata/backtrace.txt\nnative/testdata/stacktrace.txt\nnative/core/testdata/backtrace.txt\nnative/core/testdata/stacktrace.txt\ndev/copyright/scala-header.txt\ndev/diffs/iceberg/*.diff\ndev/release/requirements.txt\ndev/release/rat_exclude_files.txt\ndocs/comet-*\ndocs/source/_static/images/*.svg\ndocs/source/_static/images/comet-dataflow.excalidraw\ndocs/source/contributor-guide/benchmark-results/**/*.json\ndocs/logos/*.png\ndocs/logos/*.svg\ndocs/source/contributor-guide/*.svg\nrust-toolchain\nspark/src/test/resources/tpcds-extended/q*.sql\nspark/src/test/resources/tpcds-query-results/*.out\nspark/src/test/resources/tpcds-micro-benchmarks/*.sql\nspark/src/test/resources/tpcds-plan-stability/approved-plans*/**/explain.txt\nspark/src/test/resources/tpcds-plan-stability/approved-plans*/**/simplified.txt\nspark/src/test/resources/tpcds-plan-stability/approved-plans*/**/extended.txt\nspark/src/test/resources/tpch-query-results/*.out\nspark/src/test/resources/tpch-extended/q*.sql\nspark/src/test/resources/test-data/*.csv\nspark/src/test/resources/test-data/*.ndjson\nspark/inspections/CometTPC*results.txt\nbenchmarks/tpc/queries/**/*.sql\n" }, { "path": "dev/release/release-tarball.sh", "content": "#!/bin/bash\n#\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n# Adapted from https://github.com/apache/arrow-rs/tree/master/dev/release/release-tarball.sh\n\n# This script copies a tarball from the \"dev\" area of the\n# dist.apache.datafusion repository to the \"release\" area\n#\n# This script should only be run after the release has been approved\n# by the Apache DataFusion PMC committee.\n#\n# See release/README.md for full release instructions\n#\n# Based in part on post-01-upload.sh from apache/arrow\n\n\nset -e\nset -u\n\nif [ \"$#\" -ne 2 ]; then\n echo \"Usage: $0 \"\n echo \"ex. $0 4.1.0 2\"\n exit\nfi\n\nversion=$1\nrc=$2\n\nread -r -p \"Proceed to release tarball for ${version}-rc${rc}? [y/N]: \" answer\nanswer=${answer:-no}\nif [ \"${answer}\" != \"y\" ]; then\n echo \"Cancelled tarball release!\"\n exit 1\nfi\n\ntmp_dir=tmp-apache-datafusion-comet-dist\n\necho \"Recreate temporary directory: ${tmp_dir}\"\nrm -rf ${tmp_dir}\nmkdir -p ${tmp_dir}\n\necho \"Clone dev dist repository\"\nsvn \\\n co \\\n https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-comet-${version}-rc${rc} \\\n ${tmp_dir}/dev\n\necho \"Clone release dist repository\"\nsvn co https://dist.apache.org/repos/dist/release/datafusion ${tmp_dir}/release\n\necho \"Copy ${version}-rc${rc} to release working copy\"\nrelease_version=datafusion-comet-${version}\nmkdir -p ${tmp_dir}/release/${release_version}\ncp -r ${tmp_dir}/dev/* ${tmp_dir}/release/${release_version}/\nsvn add ${tmp_dir}/release/${release_version}\n\necho \"Commit release\"\nsvn ci -m \"Apache DataFusion Comet ${version}\" ${tmp_dir}/release\n\necho \"Clean up\"\nrm -rf ${tmp_dir}\n\necho \"Success! The release is available here:\"\necho \" https://dist.apache.org/repos/dist/release/datafusion/${release_version}\"\n" }, { "path": "dev/release/requirements.txt", "content": "PyGitHub" }, { "path": "dev/release/run-rat.sh", "content": "#!/bin/bash\n#\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\nRAT_VERSION=0.16.1\n\n# download apache rat\nif [ ! -f apache-rat-${RAT_VERSION}.jar ]; then\n curl -s https://repo1.maven.org/maven2/org/apache/rat/apache-rat/${RAT_VERSION}/apache-rat-${RAT_VERSION}.jar > apache-rat-${RAT_VERSION}.jar\nfi\n\nRAT=\"java -jar apache-rat-${RAT_VERSION}.jar -x \"\n\nRELEASE_DIR=$(cd \"$(dirname \"$BASH_SOURCE\")\"; pwd)\n\n# generate the rat report\n$RAT $1 > rat.txt\npython3 $RELEASE_DIR/check-rat-report.py $RELEASE_DIR/rat_exclude_files.txt rat.txt > filtered_rat.txt\nUNAPPROVED=`cat filtered_rat.txt | grep \"NOT APPROVED\" | wc -l`\n\nif [ \"0\" -eq \"${UNAPPROVED}\" ]; then\n echo \"No unapproved licenses\"\nelse\n echo \"${UNAPPROVED} unapproved licences. Check rat report: rat.txt\"\n cat filtered_rat.txt\n exit 1\nfi\n" }, { "path": "dev/release/verify-release-candidate.sh", "content": "#!/bin/bash\n#\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\ncase $# in\n 2) VERSION=\"$1\"\n RC_NUMBER=\"$2\"\n ;;\n *) echo \"Usage: $0 X.Y.Z RC_NUMBER\"\n exit 1\n ;;\nesac\n\nset -e\nset -x\nset -o pipefail\n\nSOURCE_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]:-$0}\")\" && pwd)\"\nCOMET_DIR=\"$(dirname $(dirname ${SOURCE_DIR}))\"\nCOMET_DIST_URL='https://dist.apache.org/repos/dist/dev/datafusion'\n\ndownload_dist_file() {\n curl \\\n --silent \\\n --show-error \\\n --fail \\\n --location \\\n --remote-name $COMET_DIST_URL/$1\n}\n\ndownload_rc_file() {\n download_dist_file apache-datafusion-comet-${VERSION}-rc${RC_NUMBER}/$1\n}\n\nimport_gpg_keys() {\n download_dist_file KEYS\n gpg --import KEYS\n}\n\nif type shasum >/dev/null 2>&1; then\n sha256_verify=\"shasum -a 256 -c\"\n sha512_verify=\"shasum -a 512 -c\"\nelse\n sha256_verify=\"sha256sum -c\"\n sha512_verify=\"sha512sum -c\"\nfi\n\nfetch_archive() {\n local dist_name=$1\n download_rc_file ${dist_name}.tar.gz\n download_rc_file ${dist_name}.tar.gz.asc\n download_rc_file ${dist_name}.tar.gz.sha256\n download_rc_file ${dist_name}.tar.gz.sha512\n verify_dir_artifact_signatures\n}\n\nverify_dir_artifact_signatures() {\n # verify the signature and the checksums of each artifact\n find . -name '*.asc' | while read sigfile; do\n artifact=${sigfile/.asc/}\n gpg --verify $sigfile $artifact || exit 1\n\n # go into the directory because the checksum files contain only the\n # basename of the artifact\n pushd $(dirname $artifact)\n base_artifact=$(basename $artifact)\n ${sha256_verify} $base_artifact.sha256 || exit 1\n ${sha512_verify} $base_artifact.sha512 || exit 1\n popd\n done\n}\n\nsetup_tempdir() {\n cleanup() {\n if [ \"${TEST_SUCCESS}\" = \"yes\" ]; then\n rm -fr \"${COMET_TMPDIR}\"\n else\n echo \"Failed to verify release candidate. See ${COMET_TMPDIR} for details.\"\n fi\n }\n\n if [ -z \"${COMET_TMPDIR}\" ]; then\n # clean up automatically if COMET_TMPDIR is not defined\n COMET_TMPDIR=$(mktemp -d -t \"$1.XXXXX\")\n trap cleanup EXIT\n else\n # don't clean up automatically\n mkdir -p \"${COMET_TMPDIR}\"\n fi\n}\n\ntest_source_distribution() {\n set -e\n pushd native\n RUSTFLAGS=\"-Ctarget-cpu=native\" cargo build --release\n popd\n # test with the latest supported version of Spark\n ./mvnw verify -Prelease -DskipTests -P\"spark-3.4\" -Dmaven.gitcommitid.skip=true\n}\n\nTEST_SUCCESS=no\n\nsetup_tempdir \"datafusion-comet-${VERSION}\"\necho \"Working in sandbox ${COMET_TMPDIR}\"\ncd ${COMET_TMPDIR}\n\ndist_name=\"apache-datafusion-comet-${VERSION}\"\nimport_gpg_keys\nfetch_archive ${dist_name}\ntar xf ${dist_name}.tar.gz\npushd ${dist_name}\n test_source_distribution\npopd\n\nTEST_SUCCESS=yes\necho 'Release candidate looks good!'\nexit 0\n" }, { "path": "dev/release/verifying-release-candidates.md", "content": "\n\n# Verifying DataFusion Comet Release Candidates\n\nThe `dev/release/verify-release-candidate.sh` script in this repository can assist in the verification\nprocess. It checks the hashes and runs the build. It does not run the test suite because this takes a long time\nfor this project and the test suites already run in CI before we create the release candidate, so running them\nagain is somewhat redundant.\n\n```shell\n./dev/release/verify-release-candidate.sh 0.1.0 1\n```\n\nThe following command can be used to build a release for testing.\n\n```shell\nmake release-nogit\n```\n\nWe hope that users will verify the release beyond running this script by testing the release candidate with their\nexisting Spark jobs and report any functional issues or performance regressions.\n\nThe email announcing the vote should contain a link to pre-built jar files in a Maven staging repository.\n\nAnother way of verifying the release is to follow the\n[Comet Benchmarking Guide](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html) and compare\nperformance with the previous release.\n" }, { "path": "dev/scalastyle-config.xml", "content": "\n\n\n Scalastyle standard configuration\n\n \n \n \n\n \n\n \n\n \n\n \n\n \n\n \n \n \n \n true\n \n \n\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n\n \n\n \n\n \n\n \n\n \n \n \n \n \n \n\n \n\n \n\n \n\n \n\n \n \n ARROW, EQUALS, ELSE, TRY, CATCH, FINALLY, LARROW, RARROW\n \n \n\n \n \n ARROW, EQUALS, COMMA, COLON, IF, ELSE, DO, WHILE, FOR, MATCH, TRY, CATCH, FINALLY, LARROW, RARROW\n \n \n\n \n \n\n \n ^println$\n \n \n\n \n @VisibleForTesting\n \n \n\n \n Runtime\\.getRuntime\\.addShutdownHook\n \n \n\n \n mutable\\.SynchronizedBuffer\n \n \n\n \n Class\\.forName\n \n \n\n \n Await\\.result\n \n \n\n \n Await\\.ready\n \n \n\n \n (\\.toUpperCase|\\.toLowerCase)(?!(\\(|\\(Locale.ROOT\\)))\n \n \n\n \n throw new \\w+Error\\(\n \n \n\n \n org\\.apache\\.commons\\.lang\\.\n Use Commons Lang 3 classes (package org.apache.commons.lang3.*) instead\n of Commons Lang 2 (package org.apache.commons.lang.*)\n \n\n \n FileSystem.get\\([a-zA-Z_$][a-zA-Z_$0-9]*\\)\n \n \n\n \n \n java,scala,org,apache,3rdParty,comet\n javax?\\..*\n scala\\..*\n org\\.(?!apache).*\n org\\.apache\\.(?!comet).*\n (?!(javax?\\.|scala\\.|org\\.apache\\.comet\\.)).*\n org\\.apache\\.comet\\..*\n \n \n\n \n \n COMMA\n \n \n\n \n \n \\)\\{\n \n \n\n \n (?m)^(\\s*)/[*][*].*$(\\r|)\\n^\\1 [*]\n Use Javadoc style indentation for multiline comments\n \n\n \n case[^\\n>]*=>\\s*\\{\n Omit braces in case clauses.\n \n\n \n new (java\\.lang\\.)?(Byte|Integer|Long|Short)\\(\n Use static factory 'valueOf' or 'parseXXX' instead of the deprecated constructors.\n \n\n \n \n\n \n\n \n \n \n \n\n \n \n Please use Apache Log4j 2 instead.\n \n\n \n \n \n\n \n \n \n \n \n \n \n\n \n\n \n \n \n\n \n \n \n\n \n \n\n \n \n\n \n \n\n \n \n\n \n \n\n \n \n 800>\n \n\n \n \n 30\n \n\n \n \n 10\n \n\n \n \n 50\n \n\n \n \n \n \n \n\n \n \n -1,0,1,2,3\n \n\n \n Objects.toStringHelper\n Avoid using Object.toStringHelper. Use ToStringBuilder instead.\n \n\n" }, { "path": "docs/.gitignore", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nbuild\ntemp\nvenv/\n.python-version\ncomet-*\n" }, { "path": "docs/Makefile", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n#\n# Minimal makefile for Sphinx documentation\n#\n\n# You can set these variables from the command line, and also\n# from the environment for the first two.\nSPHINXOPTS ?=\nSPHINXBUILD ?= sphinx-build\nSOURCEDIR = source\nBUILDDIR = build\n\n# Put it first so that \"make\" without argument is like \"make help\".\nhelp:\n\t@$(SPHINXBUILD) -M help \"$(SOURCEDIR)\" \"$(BUILDDIR)\" $(SPHINXOPTS) $(O)\n\n.PHONY: help Makefile\n\n# Catch-all target: route all unknown targets to Sphinx using the new\n# \"make mode\" option. $(O) is meant as a shortcut for $(SPHINXOPTS).\n%: Makefile\n\t@$(SPHINXBUILD) -M $@ \"$(SOURCEDIR)\" \"$(BUILDDIR)\" $(SPHINXOPTS) $(O)\n" }, { "path": "docs/README.md", "content": "\n\n# Apache DataFusion Comet Documentation\n\nThis folder contains the source content for the Apache DataFusion Comet documentation site. This content is published\nto when any changes are merged into the main branch.\n\n## Dependencies\n\nIt's recommended to install build dependencies and build the documentation\ninside a Python virtualenv.\n\n- Python\n- `pip install -r requirements.txt`\n\n## Build & Preview\n\nRun the provided script to build the HTML pages.\n\n```bash\n./build.sh\n```\n\nThe HTML will be generated into a `build` directory.\n\nPreview the site on Linux by running this command.\n\n```bash\nfirefox build/html/index.html\n```\n\n## Making Changes\n\nTo make changes to the docs, simply make a Pull Request with your\nproposed changes as normal. When the PR is merged the docs will be\nautomatically updated.\n\n## Release Process\n\nThis documentation is hosted at \n\nWhen the PR is merged to the `main` branch of the `datafusion-comet`\nrepository, a [GitHub workflow](https://github.com/apache/datafusion-comet/blob/main/.github/workflows/docs.yaml) which:\n\n1. Builds the html content\n2. Pushes the html content to the [`asf-site`](https://github.com/apache/datafusion-comet/tree/asf-site) branch in this repository.\n\nThe Apache Software Foundation provides ,\nwhich serves content based on the configuration in\n[.asf.yaml](https://github.com/apache/datafusion-comet/blob/main/.asf.yaml),\nwhich specifies the target as .\n" }, { "path": "docs/build.sh", "content": "#!/bin/bash\n#\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\nset -e\nrm -rf build 2> /dev/null\nrm -rf temp 2> /dev/null\nmkdir temp\ncp -rf source/* temp/\n\n# Add user guide from published releases\nrm -rf comet-0.11\nrm -rf comet-0.12\nrm -rf comet-0.13\npython3 generate-versions.py\n\n# Generate dynamic content (configs, compatibility matrices) for latest docs\n# This runs GenerateDocs against the temp copy, not source files\necho \"Generating dynamic documentation content...\"\ncd ..\n./mvnw -q package -Pgenerate-docs -DskipTests -Dmaven.test.skip=true \\\n -Dexec.arguments=\"$(pwd)/docs/temp/user-guide/latest/\"\ncd docs\n\nmake SOURCEDIR=`pwd`/temp html\n" }, { "path": "docs/generate-versions.py", "content": "#!/usr/bin/python\n##############################################################################\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n##############################################################################\n\n# This script clones release branches such as branch-0.9 and then copies the user guide markdown\n# for inclusion in the published documentation so that we publish user guides for released versions\n# of Comet\n\nimport os\nimport xml.etree.ElementTree as ET\nfrom pathlib import Path\n\ndef get_major_minor_version(version: str):\n parts = version.split('.')\n return f\"{parts[0]}.{parts[1]}\"\n\ndef get_version_from_pom():\n pom_path = Path(__file__).parent.parent / \"pom.xml\"\n tree = ET.parse(pom_path)\n root = tree.getroot()\n \n namespace = {'maven': 'http://maven.apache.org/POM/4.0.0'}\n version_element = root.find('maven:version', namespace)\n \n if version_element is not None:\n return version_element.text\n else:\n raise ValueError(\"Could not find version in pom.xml\")\n\ndef replace_in_files(root: str, filename_pattern: str, search: str, replace: str):\n root_path = Path(root)\n for file in root_path.rglob(filename_pattern):\n text = file.read_text(encoding=\"utf-8\")\n updated = text.replace(search, replace)\n if text != updated:\n file.write_text(updated, encoding=\"utf-8\")\n print(f\"Replaced {search} with {replace} in {file}\")\n\ndef insert_warning_after_asf_header(root: str, warning: str):\n root_path = Path(root)\n for file in root_path.rglob(\"*.md\"):\n lines = file.read_text(encoding=\"utf-8\").splitlines(keepends=True)\n new_lines = []\n inserted = False\n for line in lines:\n new_lines.append(line)\n if not inserted and \"-->\" in line:\n new_lines.append(warning + \"\\n\")\n inserted = True\n file.write_text(\"\".join(new_lines), encoding=\"utf-8\")\n\n\ndef get_user_guide_dir(major_minor: str):\n if major_minor == \"0.8\" or major_minor == \"0.9\":\n return \"docs/source/user-guide\"\n else:\n return \"docs/source/user-guide/latest\"\n\ndef publish_released_version(version: str):\n major_minor = get_major_minor_version(version)\n dir = get_user_guide_dir(major_minor)\n os.system(f\"git clone --depth 1 https://github.com/apache/datafusion-comet.git -b branch-{major_minor} comet-{major_minor}\")\n os.system(f\"mkdir temp/user-guide/{major_minor}\")\n os.system(f\"cp -rf comet-{major_minor}/{dir}/* temp/user-guide/{major_minor}\")\n # Replace $COMET_VERSION with actual version\n for file_pattern in [\"*.md\", \"*.rst\"]:\n replace_in_files(f\"temp/user-guide/{major_minor}\", file_pattern, \"$COMET_VERSION\", version)\n\ndef generate_docs(snapshot_version: str, latest_released_version: str, previous_versions: list[str]):\n\n # Replace $COMET_VERSION with actual version for snapshot version\n for file_pattern in [\"*.md\", \"*.rst\"]:\n replace_in_files(f\"temp/user-guide/latest\", file_pattern, \"$COMET_VERSION\", snapshot_version)\n\n # Add user guide content for latest released versions\n publish_released_version(latest_released_version)\n\n # Add user guide content for older released versions\n for version in previous_versions:\n publish_released_version(version)\n # add warning that this is out-of-date documentation\n warning = f\"\"\"```{{warning}}\nThis is **out-of-date** documentation. The latest Comet release is version {latest_released_version}.\n```\"\"\"\n major_minor = get_major_minor_version(version)\n insert_warning_after_asf_header(f\"temp/user-guide/{major_minor}\", warning)\n\nif __name__ == \"__main__\":\n print(\"Generating versioned user guide docs...\")\n snapshot_version = get_version_from_pom()\n latest_released_version = \"0.14.0\"\n previous_versions = [\"0.11.0\", \"0.12.0\", \"0.13.0\"]\n generate_docs(snapshot_version, latest_released_version, previous_versions)" }, { "path": "docs/make.bat", "content": "@rem Licensed to the Apache Software Foundation (ASF) under one\r\n@rem or more contributor license agreements. See the NOTICE file\r\n@rem distributed with this work for additional information\r\n@rem regarding copyright ownership. The ASF licenses this file\r\n@rem to you under the Apache License, Version 2.0 (the\r\n@rem \"License\"); you may not use this file except in compliance\r\n@rem with the License. You may obtain a copy of the License at\r\n@rem\r\n@rem http://www.apache.org/licenses/LICENSE-2.0\r\n@rem\r\n@rem Unless required by applicable law or agreed to in writing,\r\n@rem software distributed under the License is distributed on an\r\n@rem \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\r\n@rem KIND, either express or implied. See the License for the\r\n@rem specific language governing permissions and limitations\r\n@rem under the License.\r\n\r\n@ECHO OFF\r\n\r\npushd %~dp0\r\n\r\nREM Command file for Sphinx documentation\r\n\r\nif \"%SPHINXBUILD%\" == \"\" (\r\n\tset SPHINXBUILD=sphinx-build\r\n)\r\nset SOURCEDIR=source\r\nset BUILDDIR=build\r\n\r\nif \"%1\" == \"\" goto help\r\n\r\n%SPHINXBUILD% >NUL 2>NUL\r\nif errorlevel 9009 (\r\n\techo.\r\n\techo.The 'sphinx-build' command was not found. Make sure you have Sphinx\r\n\techo.installed, then set the SPHINXBUILD environment variable to point\r\n\techo.to the full path of the 'sphinx-build' executable. Alternatively you\r\n\techo.may add the Sphinx directory to PATH.\r\n\techo.\r\n\techo.If you don't have Sphinx installed, grab it from\r\n\techo.http://sphinx-doc.org/\r\n\texit /b 1\r\n)\r\n\r\n%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%\r\ngoto end\r\n\r\n:help\r\n%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%\r\n\r\n:end\r\npopd\r\n" }, { "path": "docs/requirements.txt", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nsetuptools\nsphinx>=7.0,<8.0\nsphinx-reredirects\npydata-sphinx-theme>=0.16.1,<0.17.0\nmyst-parser>=2.0,<4.0\nmaturin\njinja2\n" }, { "path": "docs/source/_static/theme_overrides.css", "content": "/**\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n\n/* Customizing with theme CSS variables */\n\n:root {\n --pst-color-active-navigation: 215, 70, 51;\n --pst-color-link-hover: 215, 70, 51;\n --pst-color-headerlink: 215, 70, 51;\n /* Use normal text color (like h3, ..) instead of primary color */\n --pst-color-h1: var(--color-text-base);\n --pst-color-h2: var(--color-text-base);\n /* Use softer blue from bootstrap's default info color */\n --pst-color-info: 23, 162, 184;\n --pst-content-max-width: 100%; /* center column */\n --pst-font-size-base: 0.9rem;\n}\n\n/* Scale down the logo in the top navbar */\n.bd-header .navbar-brand img.logo__image {\n height: 40px;\n width: auto;\n margin-right: 0.25rem;\n vertical-align: middle;\n}\n\n/* --- remove the right (secondary) sidebar entirely --- */\n.bd-sidebar-secondary { display: none !important; }\n\n/* Some versions still reserve the grid column for it — collapse it */\n.bd-main {\n /* left sidebar + content only */\n grid-template-columns: 20rem minmax(0, 1fr) !important;\n}\n\n/* --- make the left (primary) sidebar small --- */\n.bd-sidebar-primary {\n width: 20rem !important;\n flex: 0 0 20rem !important;\n}\n\n/* Optional: make its text a bit tighter */\n.bd-sidebar-primary .bd-sidebar {\n font-size: 0.9rem;\n}\n\n/* --- let the center content use all remaining width --- */\n.bd-content, .bd-article-container {\n max-width: none !important; /* remove internal cap */\n}\n\n\ncode {\n color: rgb(215, 70, 51);\n}\n\n.footer {\n text-align: center;\n}\n\n/* Ensure the logo is properly displayed */\n\n.navbar-brand {\n height: auto;\n width: auto;\n padding: 0 2em;\n}\n\n/* This is the bootstrap CSS style for \"table-striped\". Since the theme does\nnot yet provide an easy way to configure this globally, it easier to simply\ninclude this snippet here than updating each table in all rst files to\nadd \":class: table-striped\" */\n\n.table tbody tr:nth-of-type(odd) {\n background-color: rgba(0, 0, 0, 0.05);\n}\n\n\n/* Limit the max height of the sidebar navigation section. Because in our\ncustomized template, there is more content above the navigation, i.e.\nlarger logo: if we don't decrease the max-height, it will overlap with\nthe footer.\nDetails: 8rem for search box etc*/\n\n@media (min-width:720px) {\n @supports (position:-webkit-sticky) or (position:sticky) {\n .bd-links {\n max-height: calc(100vh - 8rem)\n }\n }\n}\n\n\n/* Fix table text wrapping in RTD theme,\n * see https://rackerlabs.github.io/docs-rackspace/tools/rtd-tables.html\n */\n\n@media screen {\n table.docutils td {\n /* !important prevents the common CSS stylesheets from overriding\n this as on RTD they are loaded after this stylesheet */\n white-space: normal !important;\n }\n}\n" }, { "path": "docs/source/_templates/docs-sidebar.html", "content": "\n\n

\n\n" }, { "path": "docs/source/_templates/layout.html", "content": "\n\n{% extends \"pydata_sphinx_theme/layout.html\" %}\n\n\n{% block footer %}\n\n
\n
\n {% for footer_item in theme_footer_items %}\n
\n {% include footer_item %}\n
\n {% endfor %}\n
\n

Apache DataFusion, Apache DataFusion Comet, Apache, the Apache feather logo, and the Apache DataFusion project logo

\n

are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.

\n
\n
\n
\n\n{% endblock %}\n" }, { "path": "docs/source/about/gluten_comparison.md", "content": "\n\n# Comparison of Comet and Gluten\n\nThis document provides a comparison of the Comet and Gluten projects to help guide users who are looking to choose\nbetween them. This document is likely biased because the Comet community maintains it.\n\nWe recommend trying out both Comet and Gluten to see which is the best fit for your needs.\n\nThis document is based on Comet 0.9.0 and Gluten 1.4.0.\n\n## Architecture\n\nComet and Gluten have very similar architectures. Both are Spark plugins that translate Spark physical plans to\na serialized representation and pass the serialized plan to native code for execution.\n\nGluten serializes the plans using the Substrait format and has an extensible architecture that supports execution\nagainst multiple engines. Velox and Clickhouse are currently supported, but Velox is more widely used.\n\nComet serializes the plans in a proprietary Protocol Buffer format. Execution is delegated to Apache DataFusion. Comet\ndoes not plan to support multiple engines, but rather focus on a tight integration between Spark and DataFusion.\n\n## Underlying Execution Engine: DataFusion vs Velox\n\nOne of the main differences between Comet and Gluten is the choice of native execution engine.\n\nGluten uses Velox, which is an open-source C++ vectorized query engine created by Meta.\n\nComet uses Apache DataFusion, which is an open-source vectorized query engine implemented in Rust and is governed by the\nApache Software Foundation.\n\nVelox and DataFusion are both mature query engines that are growing in popularity.\n\nFrom the point of view of the usage of these query engines in Gluten and Comet, the most significant difference is\nthe choice of implementation language (Rust vs C++) and this may be the main factor that users should consider when\nchoosing a solution. For users wishing to implement UDFs in Rust, Comet would likely be a better choice. For users\nwishing to implement UDFs in C++, Gluten would likely be a better choice.\n\nIf users are just interested in speeding up their existing Spark jobs and do not need to implement UDFs in native\ncode, then we suggest benchmarking with both solutions and choosing the fastest one for your use case.\n\n![github-stars-datafusion-velox.png](/_static/images/github-stars-datafusion-velox.png)\n\n## Compatibility\n\nComet relies on the full Spark SQL test suite (consisting of more than 24,000 tests) as well its own unit and\nintegration tests to ensure compatibility with Spark. Features that are known to have compatibility differences with\nSpark are disabled by default, but users can opt in. See the [Comet Compatibility Guide] for more information.\n\n[Comet Compatibility Guide]: /user-guide/latest/compatibility.md\n\nGluten also aims to provide compatibility with Spark, and includes a subset of the Spark SQL tests in its own test\nsuite. See the [Gluten Compatibility Guide] for more information.\n\n[Gluten Compatibility Guide]: https://apache.github.io/incubator-gluten-site/archives/v1.3.0/velox-backend/limitations/\n\n## Performance\n\nWhen running a benchmark derived from TPC-H on a single node against local Parquet files, we see that both Comet\nand Gluten provide an impressive speedup when compared to Spark. Comet provides a 2.4x speedup compares to a 2.8x speedup\nwith Gluten.\n\nGluten is currently faster than Comet for this particular benchmark, but we expect to close that gap over time.\n\nAlthough TPC-H is a good benchmark for operators such as joins and aggregates, it doesn't necessarily represent\nreal-world queries, especially for ETL use cases. For example, there are no complex types involved and no string\nmanipulation, regular expressions, or other advanced expressions. We recommend running your own benchmarks based\non your existing Spark jobs.\n\n![tpch_allqueries_comet_gluten.png](/_static/images/tpch_allqueries_comet_gluten.png)\n\nThe scripts that were used to generate these results can be found [here](https://github.com/apache/datafusion-comet/tree/main/benchmarks/tpc).\n\n## Ease of Development & Contributing\n\nSetting up a local development environment with Comet is generally easier than with Gluten due to Rust's package\nmanagement capabilities vs the complexities around installing C++ dependencies.\n\n## Summary\n\nComet and Gluten are both good solutions for accelerating Spark jobs. We recommend trying both to see which is the\nbest fit for your needs.\n" }, { "path": "docs/source/about/index.md", "content": "\n\n# Comet Overview\n\nApache DataFusion Comet is a high-performance accelerator for Apache Spark, built on top of the powerful\n[Apache DataFusion] query engine. Comet is designed to significantly enhance the\nperformance of Apache Spark workloads while leveraging commodity hardware and seamlessly integrating with the\nSpark ecosystem without requiring any code changes.\n\n[Apache DataFusion]: https://datafusion.apache.org\n\nThe following diagram provides an overview of Comet's architecture.\n\n![Comet Overview](/_static/images/comet-overview.png)\n\n## Architecture\n\nThe following diagram shows how Comet integrates with Apache Spark.\n\n![Comet System Diagram](/_static/images/comet-system-diagram.png)\n\n## Feature Parity with Apache Spark\n\nThe project strives to keep feature parity with Apache Spark, that is,\nusers should expect the same behavior (w.r.t features, configurations,\nquery results, etc) with Comet turned on or turned off in their Spark\njobs. In addition, Comet extension should automatically detect unsupported\nfeatures and fallback to Spark engine.\n\n## Comparison with other open-source Spark accelerators\n\nThere are two other major open-source Spark accelerators:\n\n- [Apache Gluten (incubating)](https://github.com/apache/incubator-gluten)\n- [NVIDIA Spark RAPIDS](https://github.com/NVIDIA/spark-rapids)\n\nWe have a detailed guide [comparing Apache DataFusion Comet with Apache Gluten].\n\nSpark RAPIDS is a solution that provides hardware acceleration on NVIDIA GPUs. Comet does not require specialized\nhardware.\n\n[comparing Apache DataFusion Comet with Apache Gluten]: gluten_comparison.md\n\n## Getting Started\n\nRefer to the [Comet Installation Guide] to get started.\n\n[Comet Installation Guide]: /user-guide/latest/installation.md\n\n```{toctree}\n:maxdepth: 1\n:caption: About\n:hidden:\n\nComparison with Gluten \n```\n" }, { "path": "docs/source/asf/index.md", "content": "\n\n# ASF Links\n\n```{toctree}\n:maxdepth: 1\n:caption: ASF Links\n\nApache Software Foundation \nLicense \nDonate \nThanks \nSecurity \nCode of conduct \n```\n" }, { "path": "docs/source/conf.py", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n# Configuration file for the Sphinx documentation builder.\n#\n# This file only contains a selection of the most common options. For a full\n# list see the documentation:\n# https://www.sphinx-doc.org/en/master/usage/configuration.html\n\n# -- Path setup --------------------------------------------------------------\n\n# If extensions (or modules to document with autodoc) are in another directory,\n# add these directories to sys.path here. If the directory is relative to the\n# documentation root, use os.path.abspath to make it absolute, like shown here.\n#\n# import os\n# import sys\n# sys.path.insert(0, os.path.abspath('.'))\n\n# -- Project information -----------------------------------------------------\n\nproject = 'Apache DataFusion Comet'\ncopyright = '2023-2024, Apache Software Foundation'\nauthor = 'Apache Software Foundation'\n\n\n# -- General configuration ---------------------------------------------------\n\n# Add any Sphinx extension module names here, as strings. They can be\n# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom\n# ones.\nextensions = [\n 'sphinx.ext.autodoc',\n 'sphinx.ext.autosummary',\n 'sphinx.ext.doctest',\n 'sphinx.ext.ifconfig',\n 'sphinx.ext.mathjax',\n 'sphinx.ext.viewcode',\n 'sphinx.ext.napoleon',\n 'myst_parser',\n 'sphinx_reredirects',\n]\n\nsource_suffix = {\n '.rst': 'restructuredtext',\n '.md': 'markdown',\n}\n\n# Add any paths that contain templates here, relative to this directory.\ntemplates_path = ['_templates']\n\n# List of patterns, relative to source directory, that match files and\n# directories to ignore when looking for source files.\n# This pattern also affects html_static_path and html_extra_path.\nexclude_patterns = []\n\n# Show members for classes in .. autosummary\nautodoc_default_options = {\n \"members\": None,\n \"undoc-members\": None,\n \"show-inheritance\": None,\n \"inherited-members\": None,\n}\n\nautosummary_generate = True\n\n# -- Options for HTML output -------------------------------------------------\n\n# The theme to use for HTML and HTML Help pages. See the documentation for\n# a list of builtin themes.\n#\nhtml_theme = 'pydata_sphinx_theme'\n\nhtml_logo = \"_static/images/DataFusionComet-Logo-Light.png\"\n\nhtml_theme_options = {\n \"logo\": {\n \"image_light\": \"_static/images/DataFusionComet-Logo-Light.png\",\n \"image_dark\": \"_static/images/DataFusionComet-Logo-Dark.png\",\n },\n\n \"use_edit_page_button\": False,\n \"secondary_sidebar_items\": [],\n \"collapse_navigation\": True,\n \"navbar_start\": [\"navbar-logo\"],\n \"navbar_center\": [\"navbar-nav\"],\n \"navbar_end\": [\"navbar-icon-links\", \"theme-switcher\"],\n \"icon_links\": [\n {\n \"name\": \"GitHub\",\n \"url\": \"https://github.com/apache/datafusion-comet\",\n \"icon\": \"fa-brands fa-github\",\n },\n ],\n\n}\n\nhtml_context = {\n \"default_mode\": \"light\",\n \"github_user\": \"apache\",\n \"github_repo\": \"datafusion-comet\",\n \"github_version\": \"main\",\n \"doc_path\": \"docs/source\",\n}\n\n# Add any paths that contain custom static files (such as style sheets) here,\n# relative to this directory. They are copied after the builtin static files,\n# so a file named \"default.css\" will overwrite the builtin \"default.css\".\nhtml_static_path = ['_static']\n\nhtml_css_files = [\n \"theme_overrides.css\"\n]\n\nhtml_js_files = [\n (\"https://buttons.github.io/buttons.js\", {'async': 'true', 'defer': 'true'}),\n]\n\nhtml_sidebars = {\n \"**\": [\"docs-sidebar.html\"],\n}\n\n# tell myst_parser to auto-generate anchor links for headers h1, h2, h3\nmyst_heading_anchors = 3\n\n# enable nice rendering of checkboxes for the task lists\nmyst_enable_extensions = [\"colon_fence\", \"deflist\", \"tasklist\"]\n\nredirects = {\n \"overview.html\": \"about/index.html\",\n \"gluten_comparison.html\": \"about/gluten_comparison.html\",\n \"user-guide/overview.html\": \"../about/overview.html\",\n \"user-guide/gluten_comparison.html\": \"../about/gluten_comparison.html\",\n \"user-guide/compatibility.html\": \"latest/compatibility.html\",\n \"user-guide/configs.html\": \"latest/configs.html\",\n \"user-guide/datasource.html\": \"latest/datasource.html\",\n \"user-guide/datatypes.html\": \"latest/datatypes.html\",\n \"user-guide/expressions.html\": \"latest/expressions.html\",\n \"user-guide/iceberg.html\": \"latest/iceberg.html\",\n \"user-guide/installation.html\": \"latest/installation.html\",\n \"user-guide/kubernetes.html\": \"latest/kubernetes.html\",\n \"user-guide/metrics.html\": \"latest/metrics.html\",\n \"user-guide/operators.html\": \"latest/operators.html\",\n \"user-guide/source.html\": \"latest/source.html\",\n \"user-guide/tuning.html\": \"latest/tuning.html\",\n}" }, { "path": "docs/source/contributor-guide/adding_a_new_expression.md", "content": "\n\n# Adding a New Expression\n\nThere are a number of Spark expression that are not supported by DataFusion Comet yet, and implementing them is a good way to contribute to the project.\n\nBefore you start, have a look through [these slides](https://docs.google.com/presentation/d/1H0fF2MOkkBK8fPBlnqK6LejUeLcVD917JhVWfp3mb8A/edit#slide=id.p) as they provide a conceptual overview. And a video of a presentation on those slides is available [here](https://drive.google.com/file/d/1POU4lFAZfYwZR8zV1X2eoLiAmc1GDtSP/view?usp=sharing).\n\n## Finding an Expression to Add\n\nYou may have a specific expression in mind that you'd like to add, but if not, you can review the [expression coverage document](https://github.com/apache/datafusion-comet/blob/main/docs/spark_expressions_support.md) to see which expressions are not yet supported.\n\n## Implementing the Expression\n\nOnce you have the expression you'd like to add, you should take inventory of the following:\n\n1. What is the Spark expression's behavior across different Spark versions? These make good test cases and will inform you of any compatibility issues, such as an API change that will have to be addressed.\n2. Check if the expression is already implemented in DataFusion and if it is compatible with the Spark expression.\n 1. If it is, you can potentially reuse the existing implementation though you'll need to add tests to verify compatibility.\n 2. If it's not, consider an initial version in DataFusion for expressions that are common across different engines. For expressions that are specific to Spark, consider an initial version in DataFusion Comet.\n3. Test cases for the expression. As mentioned, you can refer to Spark's test cases for a good idea of what to test.\n\nOnce you know what you want to add, you'll need to update the query planner to recognize the new expression in Scala and potentially add a new expression implementation in the Rust package.\n\n### Adding the Expression in Scala\n\nDataFusion Comet uses a framework based on the `CometExpressionSerde` trait for converting Spark expressions to protobuf. Instead of a large match statement, each expression type has its own serialization handler. For aggregate expressions, use the `CometAggregateExpressionSerde` trait instead.\n\n#### Creating a CometExpressionSerde Implementation\n\nFirst, create an object that extends `CometExpressionSerde[T]` where `T` is the Spark expression type. This is typically added to one of the serde files in `spark/src/main/scala/org/apache/comet/serde/` (e.g., `math.scala`, `strings.scala`, `arrays.scala`, etc.).\n\nFor example, the `unhex` function looks like this:\n\n```scala\nobject CometUnhex extends CometExpressionSerde[Unhex] {\n override def convert(\n expr: Unhex,\n inputs: Seq[Attribute],\n binding: Boolean): Option[ExprOuterClass.Expr] = {\n val childExpr = exprToProtoInternal(expr.child, inputs, binding)\n val failOnErrorExpr = exprToProtoInternal(Literal(expr.failOnError), inputs, binding)\n\n val optExpr =\n scalarFunctionExprToProtoWithReturnType(\n \"unhex\",\n expr.dataType,\n false,\n childExpr,\n failOnErrorExpr)\n optExprWithInfo(optExpr, expr, expr.child)\n }\n}\n```\n\nThe `CometExpressionSerde` trait provides three methods you can override:\n\n- `convert(expr: T, inputs: Seq[Attribute], binding: Boolean): Option[Expr]` - **Required**. Converts the Spark expression to protobuf. Return `None` if the expression cannot be converted.\n- `getSupportLevel(expr: T): SupportLevel` - Optional. Returns the level of support for the expression. See \"Using getSupportLevel\" section below for details.\n- `getExprConfigName(expr: T): String` - Optional. Returns a short name for configuration keys. Defaults to the Spark class name.\n\nFor simple scalar functions that map directly to a DataFusion function, you can use the built-in `CometScalarFunction` implementation:\n\n```scala\nclassOf[Cos] -> CometScalarFunction(\"cos\")\n```\n\n#### Registering the Expression Handler\n\nOnce you've created your `CometExpressionSerde` implementation, register it in `QueryPlanSerde.scala` by adding it to the appropriate expression map (e.g., `mathExpressions`, `stringExpressions`, `predicateExpressions`, etc.):\n\n```scala\nprivate val mathExpressions: Map[Class[_ <: Expression], CometExpressionSerde[_]] = Map(\n // ... other expressions ...\n classOf[Unhex] -> CometUnhex,\n classOf[Hex] -> CometHex)\n```\n\nThe `exprToProtoInternal` method will automatically use this mapping to find and invoke your handler when it encounters the corresponding Spark expression type.\n\nA few things to note:\n\n- The `convert` method is recursively called on child expressions using `exprToProtoInternal`, so you'll need to make sure that the child expressions are also converted to protobuf.\n- `scalarFunctionExprToProtoWithReturnType` is for scalar functions that need to return type information. Your expression may use a different method depending on the type of expression.\n- Use helper methods like `createBinaryExpr` and `createUnaryExpr` from `QueryPlanSerde` for common expression patterns.\n\n#### Using getSupportLevel\n\nThe `getSupportLevel` method allows you to control whether an expression should be executed by Comet based on various conditions such as data types, parameter values, or other expression-specific constraints. This is particularly useful when:\n\n1. Your expression only supports specific data types\n2. Your expression has known incompatibilities with Spark's behavior\n3. Your expression has edge cases that aren't yet supported\n\nThe method returns one of three `SupportLevel` values:\n\n- **`Compatible(notes: Option[String] = None)`** - Comet supports this expression with full compatibility with Spark, or may have known differences in specific edge cases that are unlikely to be an issue for most users. This is the default if you don't override `getSupportLevel`.\n- **`Incompatible(notes: Option[String] = None)`** - Comet supports this expression but results can be different from Spark. The expression will only be used if `spark.comet.expr.allowIncompatible=true` or the expression-specific config `spark.comet.expr..allowIncompatible=true` is set.\n- **`Unsupported(notes: Option[String] = None)`** - Comet does not support this expression under the current conditions. The expression will not be used and Spark will fall back to its native execution.\n\nAll three support levels accept an optional `notes` parameter to provide additional context about the support level.\n\n##### Examples\n\n**Example 1: Restricting to specific data types**\n\nThe `Abs` expression only supports numeric types:\n\n```scala\nobject CometAbs extends CometExpressionSerde[Abs] {\n override def getSupportLevel(expr: Abs): SupportLevel = {\n expr.child.dataType match {\n case _: NumericType =>\n Compatible()\n case _ =>\n // Spark supports NumericType, DayTimeIntervalType, and YearMonthIntervalType\n Unsupported(Some(\"Only integral, floating-point, and decimal types are supported\"))\n }\n }\n\n override def convert(\n expr: Abs,\n inputs: Seq[Attribute],\n binding: Boolean): Option[ExprOuterClass.Expr] = {\n // ... conversion logic ...\n }\n}\n```\n\n**Example 2: Validating parameter values**\n\nThe `TruncDate` expression only supports specific format strings:\n\n```scala\nobject CometTruncDate extends CometExpressionSerde[TruncDate] {\n val supportedFormats: Seq[String] =\n Seq(\"year\", \"yyyy\", \"yy\", \"quarter\", \"mon\", \"month\", \"mm\", \"week\")\n\n override def getSupportLevel(expr: TruncDate): SupportLevel = {\n expr.format match {\n case Literal(fmt: UTF8String, _) =>\n if (supportedFormats.contains(fmt.toString.toLowerCase(Locale.ROOT))) {\n Compatible()\n } else {\n Unsupported(Some(s\"Format $fmt is not supported\"))\n }\n case _ =>\n Incompatible(\n Some(\"Invalid format strings will throw an exception instead of returning NULL\"))\n }\n }\n\n override def convert(\n expr: TruncDate,\n inputs: Seq[Attribute],\n binding: Boolean): Option[ExprOuterClass.Expr] = {\n // ... conversion logic ...\n }\n}\n```\n\n**Example 3: Marking known incompatibilities**\n\nThe `ArrayAppend` expression has known behavioral differences from Spark:\n\n```scala\nobject CometArrayAppend extends CometExpressionSerde[ArrayAppend] {\n override def getSupportLevel(expr: ArrayAppend): SupportLevel = Incompatible(None)\n\n override def convert(\n expr: ArrayAppend,\n inputs: Seq[Attribute],\n binding: Boolean): Option[ExprOuterClass.Expr] = {\n // ... conversion logic ...\n }\n}\n```\n\nThis expression will only be used when users explicitly enable incompatible expressions via configuration.\n\n##### How getSupportLevel Affects Execution\n\nWhen the query planner encounters an expression:\n\n1. It first checks if the expression is explicitly disabled via `spark.comet.expr..enabled=false`\n2. It then calls `getSupportLevel` on the expression handler\n3. Based on the result:\n - `Compatible()`: Expression proceeds to conversion\n - `Incompatible()`: Expression is skipped unless `spark.comet.expr.allowIncompatible=true` or expression-specific allow config is set\n - `Unsupported()`: Expression is skipped and a fallback to Spark occurs\n\nAny notes provided will be logged to help with debugging and understanding why an expression was not used.\n\n#### Adding Spark-side Tests for the New Expression\n\nIt is important to verify that the new expression is correctly recognized by the native execution engine and matches the expected Spark behavior. The preferred way to add test coverage is to write a SQL test file using the SQL file test framework. This approach is simpler than writing Scala test code and makes it easy to cover many input combinations and edge cases.\n\n##### Writing a SQL test file\n\nCreate a `.sql` file under the appropriate subdirectory in `spark/src/test/resources/sql-tests/expressions/` (e.g., `string/`, `math/`, `array/`). The file should create a table with test data, then run queries that exercise the expression. Here is an example for the `unhex` expression:\n\n```sql\nstatement\nCREATE TABLE test_unhex(col string) USING parquet\n\nstatement\nINSERT INTO test_unhex VALUES\n ('537061726B2053514C'),\n ('737472696E67'),\n ('\\0'),\n (''),\n ('###'),\n ('G123'),\n ('hello'),\n ('A1B'),\n ('0A1B'),\n (NULL)\n\n-- column argument\nquery\nSELECT unhex(col) FROM test_unhex\n\n-- literal arguments\nquery\nSELECT unhex('48656C6C6F'), unhex(''), unhex(NULL)\n```\n\nEach `query` block automatically runs the SQL through both Spark and Comet and compares results, and also verifies that Comet executes the expression natively (not falling back to Spark).\n\nRun the test with:\n\n```shell\n./mvnw test -Dsuites=\"org.apache.comet.CometSqlFileTestSuite unhex\" -Dtest=none\n```\n\nFor full documentation on the test file format — including directives like `ConfigMatrix`, query modes like `spark_answer_only` and `tolerance`, handling known bugs with `ignore(...)`, and tips for writing thorough tests — see the [SQL File Tests](sql-file-tests.md) guide.\n\n##### Tips\n\n- **Cover both column references and literals.** Comet often uses different code paths for each. The SQL file test suite automatically disables constant folding, so all-literal queries are evaluated natively.\n- **Include edge cases** such as `NULL`, empty strings, boundary values, `NaN`, and multibyte UTF-8 characters.\n- **Keep one file per expression** to make failures easy to locate.\n\n##### Scala tests (alternative)\n\nFor cases that require programmatic setup or custom assertions beyond what SQL files support, you can also add Scala test cases in `CometExpressionSuite` using the `checkSparkAnswerAndOperator` method:\n\n```scala\ntest(\"unhex\") {\n val table = \"unhex_table\"\n withTable(table) {\n sql(s\"create table $table(col string) using parquet\")\n\n sql(s\"\"\"INSERT INTO $table VALUES\n |('537061726B2053514C'),\n |('737472696E67'),\n |('\\\\0'),\n |(''),\n |('###'),\n |('G123'),\n |('hello'),\n |('A1B'),\n |('0A1B')\"\"\".stripMargin)\n\n checkSparkAnswerAndOperator(s\"SELECT unhex(col) FROM $table\")\n }\n}\n```\n\nWhen writing Scala tests with literal values (e.g., `SELECT my_func('literal')`), Spark's constant folding optimizer may evaluate the expression at planning time, bypassing Comet. To prevent this, disable constant folding:\n\n```scala\ntest(\"my_func with literals\") {\n withSQLConf(SQLConf.OPTIMIZER_EXCLUDED_RULES.key ->\n \"org.apache.spark.sql.catalyst.optimizer.ConstantFolding\") {\n checkSparkAnswerAndOperator(\"SELECT my_func('literal_value')\")\n }\n}\n```\n\n### Adding the Expression To the Protobuf Definition\n\nOnce you have the expression implemented in Scala, you might need to update the protobuf definition to include the new expression. You may not need to do this if the expression is already covered by the existing protobuf definition (e.g. you're adding a new scalar function that uses the `ScalarFunc` message).\n\nYou can find the protobuf definition in `native/proto/src/proto/expr.proto`, and in particular the `Expr` or potentially the `AggExpr` messages. If you were to add a new expression called `Add2`, you would add a new case to the `Expr` message like so:\n\n```proto\nmessage Expr {\n oneof expr_struct {\n ...\n Add2 add2 = 100; // Choose the next available number\n }\n}\n```\n\nThen you would define the `Add2` message like so:\n\n```proto\nmessage Add2 {\n Expr left = 1;\n Expr right = 2;\n}\n```\n\n### Adding the Expression in Rust\n\nWith the serialization complete, the next step is to implement the expression in Rust and ensure that the incoming plan can make use of it.\n\nHow this works is somewhat dependent on the type of expression you're adding. Expression implementations live in the `native/spark-expr/src/` directory, organized by category (e.g., `math_funcs/`, `string_funcs/`, `array_funcs/`).\n\n#### Generally Adding a New Expression\n\nIf you're adding a new expression that requires custom protobuf serialization, you may need to:\n\n1. Add a new message to the protobuf definition in `native/proto/src/proto/expr.proto`\n2. Add a native expression handler in `expression_registry.rs` to deserialize the new protobuf message type and\n create a native expression\n\nFor most expressions, you can skip this step if you're using the existing scalar function infrastructure.\n\n#### Adding a New Scalar Function Expression\n\nFor a new scalar function, you can reuse a lot of code by updating the `create_comet_physical_fun` method in `native/spark-expr/src/comet_scalar_funcs.rs`. Add a match case for your function name:\n\n```rust\nmatch fun_name {\n // ... other functions ...\n \"unhex\" => {\n let func = Arc::new(spark_unhex);\n make_comet_scalar_udf!(\"unhex\", func, without data_type)\n }\n // ... more functions ...\n}\n```\n\nThe `make_comet_scalar_udf!` macro has several variants depending on whether your function needs:\n\n- A data type parameter: `make_comet_scalar_udf!(\"ceil\", spark_ceil, data_type)`\n- No data type parameter: `make_comet_scalar_udf!(\"unhex\", func, without data_type)`\n- An eval mode: `make_comet_scalar_udf!(\"decimal_div\", spark_decimal_div, data_type, eval_mode)`\n- A fail_on_error flag: `make_comet_scalar_udf!(\"spark_modulo\", func, without data_type, fail_on_error)`\n\n#### Implementing the Function\n\nThen implement your function in an appropriate module under `native/spark-expr/src/`. The function signature will look like:\n\n```rust\npub fn spark_unhex(args: &[ColumnarValue]) -> Result {\n // Do the work here\n}\n```\n\nIf your function uses the data type or eval mode, the signature will include those as additional parameters:\n\n```rust\npub fn spark_ceil(\n args: &[ColumnarValue],\n data_type: &DataType\n) -> Result {\n // Implementation\n}\n```\n\n### API Differences Between Spark Versions\n\nIf the expression you're adding has different behavior across different Spark versions, you'll need to account for that in your implementation. There are two tools at your disposal to help with this:\n\n1. Shims that exist in `spark/src/main/spark-$SPARK_VERSION/org/apache/comet/shims/CometExprShim.scala` for each Spark version. These shims are used to provide compatibility between different Spark versions.\n2. Variables that correspond to the Spark version, such as `isSpark33Plus`, which can be used to conditionally execute code based on the Spark version.\n\n## Shimming to Support Different Spark Versions\n\nIf the expression you're adding has different behavior across different Spark versions, you can use the shim system located in `spark/src/main/spark-$SPARK_VERSION/org/apache/comet/shims/CometExprShim.scala` for each Spark version.\n\nThe `CometExprShim` trait provides several mechanisms for handling version differences:\n\n1. **Version-specific methods** - Override methods in the trait to provide version-specific behavior\n2. **Version-specific expression handling** - Use `versionSpecificExprToProtoInternal` to handle expressions that only exist in certain Spark versions\n\nFor example, the `StringDecode` expression only exists in certain Spark versions. The shim handles this:\n\n```scala\ntrait CometExprShim {\n def versionSpecificExprToProtoInternal(\n expr: Expression,\n inputs: Seq[Attribute],\n binding: Boolean): Option[Expr] = {\n expr match {\n case s: StringDecode =>\n stringDecode(expr, s.charset, s.bin, inputs, binding)\n case _ => None\n }\n }\n}\n```\n\nThe `QueryPlanSerde.exprToProtoInternal` method calls `versionSpecificExprToProtoInternal` first, allowing shims to intercept and handle version-specific expressions before falling back to the standard expression maps.\n\nYour `CometExpressionSerde` implementation can also access shim methods by mixing in the `CometExprShim` trait, though in most cases you can directly access the expression properties if they're available across all supported Spark versions.\n\n## Resources\n\n- [Variance PR](https://github.com/apache/datafusion-comet/pull/297)\n - Aggregation function\n- [Unhex PR](https://github.com/apache/datafusion-comet/pull/342)\n - Basic scalar function with shims for different Spark versions\n" }, { "path": "docs/source/contributor-guide/adding_a_new_operator.md", "content": "\n\n# Adding a New Operator\n\nThis guide explains how to add support for a new Spark physical operator in Apache DataFusion Comet.\n\n## Overview\n\n`CometExecRule` is responsible for replacing Spark operators with Comet operators. There are different approaches to\nimplementing Comet operators depending on where they execute and how they integrate with the native execution engine.\n\n### Types of Comet Operators\n\n`CometExecRule` maintains two distinct maps of operators:\n\n#### 1. Native Operators (`nativeExecs` map)\n\nThese operators run entirely in native Rust code and are the primary way to accelerate Spark workloads. Native\noperators are registered in the `nativeExecs` map in `CometExecRule.scala`.\n\nKey characteristics of native operators:\n\n- They are converted to their corresponding native protobuf representation\n- They execute as DataFusion operators in the native engine\n- The `CometOperatorSerde` implementation handles enable/disable checks, support validation, and protobuf serialization\n\nExamples: `ProjectExec`, `FilterExec`, `SortExec`, `HashAggregateExec`, `SortMergeJoinExec`, `ExpandExec`, `WindowExec`\n\n#### 2. Sink Operators (`sinks` map)\n\nSink operators serve as entry points (data sources) for native execution blocks. They are registered in the `sinks`\nmap in `CometExecRule.scala`.\n\nKey characteristics of sinks:\n\n- They become `ScanExec` operators in the native plan (see `operator2Proto` in `CometExecRule.scala`)\n- They can be leaf nodes that feed data into native execution blocks\n- They are wrapped with `CometScanWrapper` or `CometSinkPlaceHolder` during plan transformation\n- Examples include operators that bring data from various sources into native execution\n\nExamples: `UnionExec`, `CoalesceExec`, `CollectLimitExec`, `TakeOrderedAndProjectExec`\n\nSpecial sinks (not in the `sinks` map but also treated as sinks):\n\n- `CometScanExec` - File scans\n- `CometSparkToColumnarExec` - Conversion from Spark row format\n- `ShuffleExchangeExec` / `BroadcastExchangeExec` - Exchange operators\n\n#### 3. Comet JVM Operators\n\nThese operators run in the JVM but are part of the Comet execution path. For JVM operators, all checks happen\nin `CometExecRule` rather than using `CometOperatorSerde`, because they don't need protobuf serialization.\n\nExamples: `CometBroadcastExchangeExec`, `CometShuffleExchangeExec`\n\n### Choosing the Right Operator Type\n\nWhen adding a new operator, choose based on these criteria:\n\n**Use Native Operators when:**\n\n- The operator transforms data (e.g., project, filter, sort, aggregate, join)\n- The operator has a direct DataFusion equivalent or custom implementation\n- The operator consumes native child operators and produces native output\n- The operator is in the middle of an execution pipeline\n\n**Use Sink Operators when:**\n\n- The operator serves as a data source for native execution (becomes a `ScanExec`)\n- The operator brings data from non-native sources (e.g., `UnionExec` combining multiple inputs)\n- The operator is typically a leaf or near-leaf node in the execution tree\n- The operator needs special handling to interface with the native engine\n\n**Implementation Note for Sinks:**\n\nSink operators are handled specially in `CometExecRule.operator2Proto`. Instead of converting to their own operator\ntype, they are converted to `ScanExec` in the native plan. This allows them to serve as entry points for native\nexecution blocks. The original Spark operator is wrapped with `CometScanWrapper` or `CometSinkPlaceHolder` which\nmanages the boundary between JVM and native execution.\n\n## Implementing a Native Operator\n\nThis section focuses on adding a native operator, which is the most common and complex case.\n\n### Step 1: Define the Protobuf Message\n\nFirst, add the operator definition to `native/proto/src/proto/operator.proto`.\n\n#### Add to the Operator Message\n\nAdd your new operator to the `oneof op_struct` in the main `Operator` message:\n\n```proto\nmessage Operator {\n repeated Operator children = 1;\n uint32 plan_id = 2;\n\n oneof op_struct {\n Scan scan = 100;\n Projection projection = 101;\n Filter filter = 102;\n // ... existing operators ...\n YourNewOperator your_new_operator = 112; // Choose next available number\n }\n}\n```\n\n#### Define the Operator Message\n\nCreate a message for your operator with the necessary fields:\n\n```proto\nmessage YourNewOperator {\n // Fields specific to your operator\n repeated spark.spark_expression.Expr expressions = 1;\n // Add other configuration fields as needed\n}\n```\n\nFor reference, see existing operators like `Filter` (simple), `HashAggregate` (complex), or `Sort` (with ordering).\n\n### Step 2: Create a CometOperatorSerde Implementation\n\nCreate a new Scala file in `spark/src/main/scala/org/apache/comet/serde/operator/` (e.g., `CometYourOperator.scala`) that extends `CometOperatorSerde[T]` where `T` is the Spark operator type.\n\nThe `CometOperatorSerde` trait provides several key methods:\n\n- `enabledConfig: Option[ConfigEntry[Boolean]]` - Configuration to enable/disable this operator\n- `getSupportLevel(operator: T): SupportLevel` - Determines if the operator is supported\n- `convert(op: T, builder: Operator.Builder, childOp: Operator*): Option[Operator]` - Converts to protobuf\n- `createExec(nativeOp: Operator, op: T): CometNativeExec` - Creates the Comet execution operator wrapper\n\nThe validation workflow in `CometExecRule.isOperatorEnabled`:\n\n1. Checks if the operator is enabled via `enabledConfig`\n2. Calls `getSupportLevel()` to determine compatibility\n3. Handles Compatible/Incompatible/Unsupported cases with appropriate fallback messages\n\n#### Simple Example (Filter)\n\n```scala\nobject CometFilterExec extends CometOperatorSerde[FilterExec] {\n\n override def enabledConfig: Option[ConfigEntry[Boolean]] =\n Some(CometConf.COMET_EXEC_FILTER_ENABLED)\n\n override def convert(\n op: FilterExec,\n builder: Operator.Builder,\n childOp: OperatorOuterClass.Operator*): Option[OperatorOuterClass.Operator] = {\n val cond = exprToProto(op.condition, op.child.output)\n\n if (cond.isDefined && childOp.nonEmpty) {\n val filterBuilder = OperatorOuterClass.Filter\n .newBuilder()\n .setPredicate(cond.get)\n Some(builder.setFilter(filterBuilder).build())\n } else {\n withInfo(op, op.condition, op.child)\n None\n }\n }\n\n override def createExec(nativeOp: Operator, op: FilterExec): CometNativeExec = {\n CometFilterExec(nativeOp, op, op.output, op.condition, op.child, SerializedPlan(None))\n }\n}\n\ncase class CometFilterExec(\n override val nativeOp: Operator,\n override val originalPlan: SparkPlan,\n override val output: Seq[Attribute],\n condition: Expression,\n child: SparkPlan,\n override val serializedPlanOpt: SerializedPlan)\n extends CometUnaryExec {\n\n override def outputPartitioning: Partitioning = child.outputPartitioning\n\n override def outputOrdering: Seq[SortOrder] = child.outputOrdering\n\n override protected def withNewChildInternal(newChild: SparkPlan): SparkPlan =\n this.copy(child = newChild)\n}\n```\n\n#### More Complex Example (Project)\n\n```scala\nobject CometProjectExec extends CometOperatorSerde[ProjectExec] {\n\n override def enabledConfig: Option[ConfigEntry[Boolean]] =\n Some(CometConf.COMET_EXEC_PROJECT_ENABLED)\n\n override def convert(\n op: ProjectExec,\n builder: Operator.Builder,\n childOp: Operator*): Option[OperatorOuterClass.Operator] = {\n val exprs = op.projectList.map(exprToProto(_, op.child.output))\n\n if (exprs.forall(_.isDefined) && childOp.nonEmpty) {\n val projectBuilder = OperatorOuterClass.Projection\n .newBuilder()\n .addAllProjectList(exprs.map(_.get).asJava)\n Some(builder.setProjection(projectBuilder).build())\n } else {\n withInfo(op, op.projectList: _*)\n None\n }\n }\n\n override def createExec(nativeOp: Operator, op: ProjectExec): CometNativeExec = {\n CometProjectExec(nativeOp, op, op.output, op.projectList, op.child, SerializedPlan(None))\n }\n}\n\ncase class CometProjectExec(\n override val nativeOp: Operator,\n override val originalPlan: SparkPlan,\n override val output: Seq[Attribute],\n projectList: Seq[NamedExpression],\n child: SparkPlan,\n override val serializedPlanOpt: SerializedPlan)\n extends CometUnaryExec\n with PartitioningPreservingUnaryExecNode {\n\n override def producedAttributes: AttributeSet = outputSet\n\n override protected def withNewChildInternal(newChild: SparkPlan): SparkPlan =\n this.copy(child = newChild)\n}\n```\n\n#### Using getSupportLevel\n\nOverride `getSupportLevel` to control operator support based on specific conditions:\n\n```scala\noverride def getSupportLevel(operator: YourOperatorExec): SupportLevel = {\n // Check for unsupported features\n if (operator.hasUnsupportedFeature) {\n return Unsupported(Some(\"Feature X is not supported\"))\n }\n\n // Check for incompatible behavior\n if (operator.hasKnownDifferences) {\n return Incompatible(Some(\"Known differences in edge case Y\"))\n }\n\n Compatible()\n}\n```\n\nSupport levels:\n\n- **`Compatible()`** - Fully compatible with Spark (default)\n- **`Incompatible()`** - Supported but may differ; requires explicit opt-in\n- **`Unsupported()`** - Not supported under current conditions\n\nNote that Comet will treat an operator as incompatible if any of the child expressions are incompatible.\n\n### Step 3: Register the Operator\n\nAdd your operator to the appropriate map in `CometExecRule.scala`:\n\n#### For Native Operators\n\nAdd to the `nativeExecs` map (`CometExecRule.scala`):\n\n```scala\nval nativeExecs: Map[Class[_ <: SparkPlan], CometOperatorSerde[_]] =\n Map(\n classOf[ProjectExec] -> CometProjectExec,\n classOf[FilterExec] -> CometFilterExec,\n // ... existing operators ...\n classOf[YourOperatorExec] -> CometYourOperator,\n )\n```\n\n#### For Sink Operators\n\nIf your operator is a sink (becomes a `ScanExec` in the native plan), add to the `sinks` map (`CometExecRule.scala`):\n\n```scala\nval sinks: Map[Class[_ <: SparkPlan], CometOperatorSerde[_]] =\n Map(\n classOf[CoalesceExec] -> CometCoalesceExec,\n classOf[UnionExec] -> CometUnionExec,\n // ... existing operators ...\n classOf[YourSinkOperatorExec] -> CometYourSinkOperator,\n )\n```\n\nNote: The `allExecs` map automatically combines both `nativeExecs` and `sinks`, so you only need to add to one of the two maps.\n\n### Step 4: Add Configuration Entry\n\nAdd a configuration entry in `common/src/main/scala/org/apache/comet/CometConf.scala`:\n\n```scala\nval COMET_EXEC_YOUR_OPERATOR_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.exec.yourOperator.enabled\")\n .doc(\"Whether to enable your operator in Comet\")\n .booleanConf\n .createWithDefault(true)\n```\n\nRun `make` to update the user guide. The new configuration option will be added to `docs/source/user-guide/latest/configs.md`.\n\n### Step 5: Implement the Native Operator in Rust\n\n#### Update the Planner\n\nIn `native/core/src/execution/planner.rs`, add a match case in the operator deserialization logic to handle your new protobuf message:\n\n```rust\nuse datafusion_comet_proto::spark_operator::operator::OpStruct;\n\n// In the create_plan or similar method:\nmatch op.op_struct.as_ref() {\n Some(OpStruct::Scan(scan)) => {\n // ... existing cases ...\n }\n Some(OpStruct::YourNewOperator(your_op)) => {\n create_your_operator_exec(your_op, children, session_ctx)\n }\n // ... other cases ...\n}\n```\n\n#### Implement the Operator\n\nCreate the operator implementation, either in an existing file or a new file in `native/core/src/execution/operators/`:\n\n```rust\nuse datafusion::physical_plan::{ExecutionPlan, ...};\nuse datafusion_comet_proto::spark_operator::YourNewOperator;\n\npub fn create_your_operator_exec(\n op: &YourNewOperator,\n children: Vec>,\n session_ctx: &SessionContext,\n) -> Result, ExecutionError> {\n // Deserialize expressions and configuration\n // Create and return the execution plan\n\n // Option 1: Use existing DataFusion operator\n // Ok(Arc::new(SomeDataFusionExec::try_new(...)?))\n\n // Option 2: Implement custom operator (see ExpandExec for example)\n // Ok(Arc::new(YourCustomExec::new(...)))\n}\n```\n\nFor custom operators, you'll need to implement the `ExecutionPlan` trait. See `native/core/src/execution/operators/expand.rs` or `scan.rs` for examples.\n\n### Step 6: Add Tests\n\n#### Scala Integration Tests\n\nAdd tests in `spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala` or a related test suite:\n\n```scala\ntest(\"your operator\") {\n withTable(\"test_table\") {\n sql(\"CREATE TABLE test_table(col1 INT, col2 STRING) USING parquet\")\n sql(\"INSERT INTO test_table VALUES (1, 'a'), (2, 'b')\")\n\n // Test query that uses your operator\n checkSparkAnswerAndOperator(\n \"SELECT * FROM test_table WHERE col1 > 1\"\n )\n }\n}\n```\n\nThe `checkSparkAnswerAndOperator` helper verifies:\n\n1. Results match Spark's native execution\n2. Your operator is actually being used (not falling back)\n\n#### Rust Unit Tests\n\nAdd unit tests in your Rust implementation file:\n\n```rust\n#[cfg(test)]\nmod tests {\n use super::*;\n\n #[test]\n fn test_your_operator() {\n // Test operator creation and execution\n }\n}\n```\n\n### Step 7: Update Documentation\n\nAdd your operator to the supported operators list in `docs/source/user-guide/latest/compatibility.md` or similar documentation.\n\n## Implementing a Sink Operator\n\nSink operators are converted to `ScanExec` in the native plan and serve as entry points for native execution. The implementation is simpler than native operators because sink operators extend the `CometSink` base class which provides the conversion logic.\n\n### Step 1: Create a CometOperatorSerde Implementation\n\nCreate a new Scala file in `spark/src/main/scala/org/apache/spark/sql/comet/` (e.g., `CometYourSinkOperator.scala`):\n\n```scala\nimport org.apache.comet.serde.operator.CometSink\n\nobject CometYourSinkOperator extends CometSink[YourSinkExec] {\n\n override def enabledConfig: Option[ConfigEntry[Boolean]] =\n Some(CometConf.COMET_EXEC_YOUR_SINK_ENABLED)\n\n // Optional: Override if the data produced is FFI safe\n override def isFfiSafe: Boolean = false\n\n override def createExec(\n nativeOp: OperatorOuterClass.Operator,\n op: YourSinkExec): CometNativeExec = {\n CometSinkPlaceHolder(\n nativeOp,\n op,\n CometYourSinkExec(op, op.output, /* other parameters */, op.child))\n }\n\n // Optional: Override getSupportLevel if you need custom validation beyond data types\n override def getSupportLevel(operator: YourSinkExec): SupportLevel = {\n // CometSink base class already checks data types in convert()\n // Add any additional validation here\n Compatible()\n }\n}\n\n/**\n * Comet implementation of YourSinkExec that supports columnar processing\n */\ncase class CometYourSinkExec(\n override val originalPlan: SparkPlan,\n override val output: Seq[Attribute],\n /* other parameters */,\n child: SparkPlan)\n extends CometExec\n with UnaryExecNode {\n\n override protected def doExecuteColumnar(): RDD[ColumnarBatch] = {\n // Implement columnar execution logic\n val rdd = child.executeColumnar()\n // Apply your sink operator's logic\n rdd\n }\n\n override def outputPartitioning: Partitioning = {\n // Define output partitioning\n }\n\n override protected def withNewChildInternal(newChild: SparkPlan): SparkPlan =\n this.copy(child = newChild)\n}\n```\n\n**Key Points:**\n\n- Extend `CometSink[T]` which provides the `convert()` method that transforms the operator to `ScanExec`\n- The `CometSink.convert()` method (in `CometSink.scala`) automatically handles:\n - Data type validation\n - Conversion to `ScanExec` in the native plan\n - Setting FFI safety flags\n- You must implement `createExec()` to wrap the operator appropriately\n- You typically need to create a corresponding `CometYourSinkExec` class that implements columnar execution\n\n### Step 2: Register the Sink\n\nAdd your sink to the `sinks` map in `CometExecRule.scala`:\n\n```scala\nval sinks: Map[Class[_ <: SparkPlan], CometOperatorSerde[_]] =\n Map(\n classOf[CoalesceExec] -> CometCoalesceExec,\n classOf[UnionExec] -> CometUnionExec,\n classOf[YourSinkExec] -> CometYourSinkOperator,\n )\n```\n\n### Step 3: Add Configuration\n\nAdd a configuration entry in `CometConf.scala`:\n\n```scala\nval COMET_EXEC_YOUR_SINK_ENABLED: ConfigEntry[Boolean] =\n conf(\"spark.comet.exec.yourSink.enabled\")\n .doc(\"Whether to enable your sink operator in Comet\")\n .booleanConf\n .createWithDefault(true)\n```\n\n### Step 4: Add Tests\n\nTest that your sink operator correctly feeds data into native execution:\n\n```scala\ntest(\"your sink operator\") {\n withTable(\"test_table\") {\n sql(\"CREATE TABLE test_table(col1 INT, col2 STRING) USING parquet\")\n sql(\"INSERT INTO test_table VALUES (1, 'a'), (2, 'b')\")\n\n // Test query that uses your sink operator followed by native operators\n checkSparkAnswerAndOperator(\n \"SELECT col1 + 1 FROM (/* query that produces YourSinkExec */)\"\n )\n }\n}\n```\n\n**Important Notes for Sinks:**\n\n- Sinks extend the `CometSink` base class, which provides the `convert()` method implementation\n- The `CometSink.convert()` method automatically handles conversion to `ScanExec` in the native plan\n- You don't need to add protobuf definitions for sink operators - they use the standard `Scan` message\n- You don't need Rust implementation for sinks - they become standard `ScanExec` operators that read from the JVM\n- Sink implementations should provide a columnar-compatible execution class (e.g., `CometCoalesceExec`)\n- The `createExec()` method wraps the operator with `CometSinkPlaceHolder` to manage the JVM-to-native boundary\n- See `CometCoalesceExec.scala` or `CometUnionExec` in `spark/src/main/scala/org/apache/spark/sql/comet/` for reference implementations\n\n## Implementing a JVM Operator\n\nFor operators that run in the JVM:\n\n1. Create a new operator class extending appropriate Spark base classes in `spark/src/main/scala/org/apache/comet/`\n2. Add matching logic in `CometExecRule.scala` to transform the Spark operator\n3. No protobuf or Rust implementation needed\n\nExample pattern from `CometExecRule.scala`:\n\n```scala\ncase s: ShuffleExchangeExec if nativeShuffleSupported(s) =>\n CometShuffleExchangeExec(s, shuffleType = CometNativeShuffle)\n```\n\n## Common Patterns and Helpers\n\n### Expression Conversion\n\nUse `QueryPlanSerde.exprToProto` to convert Spark expressions to protobuf:\n\n```scala\nval protoExpr = exprToProto(sparkExpr, inputSchema)\n```\n\n### Handling Fallback\n\nUse `withInfo` to tag operators with fallback reasons:\n\n```scala\nif (!canConvert) {\n withInfo(op, \"Reason for fallback\", childNodes: _*)\n return None\n}\n```\n\n### Child Operator Validation\n\nAlways check that child operators were successfully converted:\n\n```scala\nif (childOp.isEmpty) {\n // Cannot convert if children failed\n return None\n}\n```\n\n## Debugging Tips\n\n1. **Enable verbose logging**: Set `spark.comet.explain.format=verbose` to see detailed plan transformations\n2. **Check fallback reasons**: Set `spark.comet.logFallbackReasons.enabled=true` to log why operators fall back to Spark\n3. **Verify protobuf**: Add debug prints in Rust to inspect deserialized operators\n4. **Use EXPLAIN**: Run `EXPLAIN EXTENDED` on queries to see the physical plan\n" }, { "path": "docs/source/contributor-guide/benchmark-results/blaze-0.5.0-tpcds.json", "content": "{\n \"engine\": \"datafusion-comet\",\n \"benchmark\": \"tpcds\",\n \"data_path\": \"/mnt/bigdata/tpcds/sf100\",\n \"query_path\": \"/mnt/bigdata/tpcds/queries\",\n \"spark_conf\": {\n \"spark.blaze.forceShuffledHashJoin\": \"true\",\n \"spark.driver.port\": \"35949\",\n \"spark.eventLog.enabled\": \"true\",\n \"spark.sql.analyzer.canonicalization.multiCommutativeOpMemoryOptThreshold\": \"2147483647\",\n \"spark.cores.max\": \"16\",\n \"spark.driver.extraClassPath\": \"/opt/blaze/blaze-engine-spark-3.5-release-5.0.0-SNAPSHOT.jar\",\n \"spark.app.submitTime\": \"1753976545258\",\n \"spark.executor.memory\": \"16g\",\n \"spark.shuffle.manager\": \"org.apache.spark.sql.execution.blaze.shuffle.BlazeShuffleManager\",\n \"spark.executor.instances\": \"2\",\n \"spark.app.startTime\": \"1753976545559\",\n \"spark.serializer.objectStreamReset\": \"100\",\n \"spark.driver.host\": \"10.0.0.118\",\n \"spark.submit.deployMode\": \"client\",\n \"spark.app.name\": \"blaze benchmark derived from tpcds\",\n \"spark.executor.cores\": \"8\",\n \"spark.driver.extraJavaOptions\": \"-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false\",\n \"spark.executor.memoryOverhead\": \"16g\",\n \"spark.sql.extensions\": \"org.apache.spark.sql.blaze.BlazeSparkSessionExtension\",\n \"spark.blaze.enable\": \"true\",\n \"spark.sql.adaptive.enabled\": \"true\",\n \"spark.executor.id\": \"driver\",\n \"spark.master\": \"spark://woody:7077\",\n \"spark.sql.adaptive.forceApply\": \"true\",\n \"spark.repl.local.jars\": \"file:///opt/blaze/blaze-engine-spark-3.5-release-5.0.0-SNAPSHOT.jar\",\n \"spark.memory.offHeap.enabled\": \"false\",\n \"spark.driver.memory\": \"8G\",\n \"spark.jars\": \"file:///opt/blaze/blaze-engine-spark-3.5-release-5.0.0-SNAPSHOT.jar\",\n \"spark.app.initial.jar.urls\": \"spark://10.0.0.118:35949/jars/blaze-engine-spark-3.5-release-5.0.0-SNAPSHOT.jar\",\n \"spark.app.id\": \"app-20250731094225-0000\",\n \"spark.rdd.compress\": \"True\",\n \"spark.executor.extraJavaOptions\": \"-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false\",\n \"spark.submit.pyFiles\": \"\",\n \"spark.sql.warehouse.dir\": \"file:/home/andy/git/apache/datafusion-comet/dev/benchmarks/spark-warehouse\",\n \"spark.executor.extraClassPath\": \"/opt/blaze/blaze-engine-spark-3.5-release-5.0.0-SNAPSHOT.jar\"\n },\n \"1\": [\n 4.29833984375\n ],\n \"2\": [\n 3.2578773498535156\n ],\n \"3\": [\n 1.8652851581573486\n ],\n \"4\": [\n 28.0850727558136\n ],\n \"5\": [\n 5.483048439025879\n ],\n \"6\": [\n 1.8989448547363281\n ],\n \"7\": [\n 2.7168705463409424\n ],\n \"8\": [\n 2.0041682720184326\n ],\n \"9\": [\n 6.036026239395142\n ],\n \"10\": [\n 2.158907175064087\n ],\n \"11\": [\n 12.088945388793945\n ],\n \"12\": [\n 1.6278820037841797\n ],\n \"13\": [\n 4.141969919204712\n ],\n \"14\": [\n 27.16408944129944\n ],\n \"15\": [\n 7.972369909286499\n ],\n \"16\": [\n 19.509708166122437\n ],\n \"17\": [\n 2.9771337509155273\n ],\n \"18\": [\n 3.184837818145752\n ],\n \"19\": [\n 3.512895107269287\n ],\n \"20\": [\n 2.5250091552734375\n ],\n \"21\": [\n 2.032010316848755\n ],\n \"22\": [\n 10.054096698760986\n ],\n \"23\": [\n 37.075517892837524\n ],\n \"24\": [\n 4.5132224559783936\n ],\n \"25\": [\n 3.260453939437866\n ],\n \"26\": [\n 1.6490797996520996\n ],\n \"27\": [\n 2.562084436416626\n ],\n \"28\": [\n 6.1220972537994385\n ],\n \"29\": [\n 3.355574607849121\n ],\n \"30\": [\n 1.3987014293670654\n ],\n \"31\": [\n 5.99055290222168\n ],\n \"32\": [\n 1.860522985458374\n ],\n \"33\": [\n 3.1227269172668457\n ],\n \"34\": [\n 1.4920480251312256\n ],\n \"35\": [\n 3.3718972206115723\n ],\n \"36\": [\n 3.137023687362671\n ],\n \"37\": [\n 2.0510008335113525\n ],\n \"38\": [\n 3.81714129447937\n ],\n \"39\": [\n 13.598978281021118\n ],\n \"40\": [\n 3.6551194190979004\n ],\n \"41\": [\n 0.20307397842407227\n ],\n \"42\": [\n 1.4140422344207764\n ],\n \"43\": [\n 1.4719886779785156\n ],\n \"44\": [\n 1.0411145687103271\n ],\n \"45\": [\n 2.1550612449645996\n ],\n \"46\": [\n 4.502093076705933\n ],\n \"47\": [\n 5.857606649398804\n ],\n \"48\": [\n 4.466501235961914\n ],\n \"49\": [\n 4.6020188331604\n ],\n \"50\": [\n 3.786076784133911\n ],\n \"51\": [\n 9.466694355010986\n ],\n \"52\": [\n 1.7045836448669434\n ],\n \"53\": [\n 1.6680505275726318\n ],\n \"54\": [\n 3.2161707878112793\n ],\n \"55\": [\n 1.3395226001739502\n ],\n \"56\": [\n 2.775632619857788\n ],\n \"57\": [\n 2.668205976486206\n ],\n \"58\": [\n 4.912959575653076\n ],\n \"59\": [\n 3.08463454246521\n ],\n \"60\": [\n 2.8756072521209717\n ],\n \"61\": [\n 4.997696161270142\n ],\n \"62\": [\n 1.0771088600158691\n ],\n \"63\": [\n 1.4476990699768066\n ],\n \"64\": [\n 8.340136051177979\n ],\n \"65\": [\n 9.196853160858154\n ],\n \"66\": [\n 3.8863117694854736\n ],\n \"67\": [\n 24.989885330200195\n ],\n \"68\": [\n 4.229811429977417\n ],\n \"69\": [\n 2.1849424839019775\n ],\n \"70\": [\n 3.462109088897705\n ],\n \"71\": [\n 2.9780192375183105\n ],\n \"72\": [\n 98.07282423973083\n ],\n \"73\": [\n 1.2939915657043457\n ],\n \"74\": [\n 17.994235277175903\n ],\n \"75\": [\n 9.07810640335083\n ],\n \"76\": [\n 1.987945795059204\n ],\n \"77\": [\n 3.2832038402557373\n ],\n \"78\": [\n 15.809038877487183\n ],\n \"79\": [\n 2.3230249881744385\n ],\n \"80\": [\n 9.651153087615967\n ],\n \"81\": [\n 1.4056549072265625\n ],\n \"82\": [\n 2.151021718978882\n ],\n \"83\": [\n 0.9110991954803467\n ],\n \"84\": [\n 0.9381260871887207\n ],\n \"85\": [\n 4.502171277999878\n ],\n \"86\": [\n 0.8276948928833008\n ],\n \"87\": [\n 3.5236589908599854\n ],\n \"88\": [\n 7.260309457778931\n ],\n \"89\": [\n 1.6737573146820068\n ],\n \"90\": [\n 0.6564974784851074\n ],\n \"91\": [\n 0.5676999092102051\n ],\n \"92\": [\n 0.9616613388061523\n ],\n \"93\": [\n 4.848900318145752\n ],\n \"94\": [\n 9.96233868598938\n ],\n \"95\": [\n 36.08780312538147\n ],\n \"96\": [\n 0.9135136604309082\n ],\n \"97\": [\n 4.4856956005096436\n ],\n \"98\": [\n 4.316124677658081\n ],\n \"99\": [\n 1.8331043720245361\n ]\n}" }, { "path": "docs/source/contributor-guide/benchmark-results/blaze-0.5.0-tpch.json", "content": "{\n \"engine\": \"datafusion-comet\",\n \"benchmark\": \"tpch\",\n \"data_path\": \"/mnt/bigdata/tpch/sf100/\",\n \"query_path\": \"/mnt/bigdata/tpch/queries/\",\n \"spark_conf\": {\n \"spark.app.name\": \"blaze benchmark derived from tpch\",\n \"spark.blaze.forceShuffledHashJoin\": \"true\",\n \"spark.eventLog.enabled\": \"true\",\n \"spark.app.id\": \"app-20250731095942-0000\",\n \"spark.sql.analyzer.canonicalization.multiCommutativeOpMemoryOptThreshold\": \"2147483647\",\n \"spark.driver.extraClassPath\": \"/opt/blaze/blaze-engine-spark-3.5-release-5.0.0-SNAPSHOT.jar\",\n \"spark.app.submitTime\": \"1753977581869\",\n \"spark.app.initial.jar.urls\": \"spark://10.0.0.118:34855/jars/blaze-engine-spark-3.5-release-5.0.0-SNAPSHOT.jar\",\n \"spark.executor.memory\": \"16g\",\n \"spark.shuffle.manager\": \"org.apache.spark.sql.execution.blaze.shuffle.BlazeShuffleManager\",\n \"spark.serializer.objectStreamReset\": \"100\",\n \"spark.driver.host\": \"10.0.0.118\",\n \"spark.submit.deployMode\": \"client\",\n \"spark.executor.cores\": \"8\",\n \"spark.driver.extraJavaOptions\": \"-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false\",\n \"spark.executor.memoryOverhead\": \"16g\",\n \"spark.sql.extensions\": \"org.apache.spark.sql.blaze.BlazeSparkSessionExtension\",\n \"spark.blaze.enable\": \"true\",\n \"spark.sql.adaptive.enabled\": \"true\",\n \"spark.executor.id\": \"driver\",\n \"spark.master\": \"spark://woody:7077\",\n \"spark.sql.adaptive.forceApply\": \"true\",\n \"spark.repl.local.jars\": \"file:///opt/blaze/blaze-engine-spark-3.5-release-5.0.0-SNAPSHOT.jar\",\n \"spark.memory.offHeap.enabled\": \"false\",\n \"spark.driver.memory\": \"8G\",\n \"spark.jars\": \"file:///opt/blaze/blaze-engine-spark-3.5-release-5.0.0-SNAPSHOT.jar\",\n \"spark.driver.port\": \"34855\",\n \"spark.rdd.compress\": \"True\",\n \"spark.executor.extraJavaOptions\": \"-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false\",\n \"spark.app.startTime\": \"1753977582179\",\n \"spark.executor.instances\": \"1\",\n \"spark.cores.max\": \"8\",\n \"spark.submit.pyFiles\": \"\",\n \"spark.sql.warehouse.dir\": \"file:/home/andy/git/apache/datafusion-comet/dev/benchmarks/spark-warehouse\",\n \"spark.executor.extraClassPath\": \"/opt/blaze/blaze-engine-spark-3.5-release-5.0.0-SNAPSHOT.jar\"\n },\n \"1\": [\n 9.673133850097656\n ],\n \"2\": [\n 5.968065500259399\n ],\n \"3\": [\n 11.941619157791138\n ],\n \"4\": [\n 11.886268138885498\n ],\n \"5\": [\n 23.566248416900635\n ],\n \"6\": [\n 2.833730697631836\n ],\n \"7\": [\n 13.405823469161987\n ],\n \"8\": [\n 23.069715976715088\n ],\n \"9\": [\n 39.501715898513794\n ],\n \"10\": [\n 11.25059962272644\n ],\n \"11\": [\n 4.793748140335083\n ],\n \"12\": [\n 6.490489959716797\n ],\n \"13\": [\n 6.752679824829102\n ],\n \"14\": [\n 3.9720609188079834\n ],\n \"15\": [\n 8.393324375152588\n ],\n \"16\": [\n 2.637575149536133\n ],\n \"17\": [\n 29.3641459941864\n ],\n \"18\": [\n 35.25342917442322\n ],\n \"19\": [\n 6.869648456573486\n ],\n \"20\": [\n 7.14982533454895\n ],\n \"21\": [\n 51.66389608383179\n ],\n \"22\": [\n 4.223745107650757\n ]\n}" }, { "path": "docs/source/contributor-guide/benchmark-results/gluten-1.4.0-tpcds.json", "content": "{\n \"engine\": \"datafusion-comet\",\n \"benchmark\": \"tpcds\",\n \"data_path\": \"/mnt/bigdata/tpcds/sf100\",\n \"query_path\": \"/mnt/bigdata/tpcds/queries\",\n \"spark_conf\": {\n \"spark.eventLog.enabled\": \"true\",\n \"spark.gluten.sql.session.timeZone.default\": \"UTC\",\n \"spark.plugins\": \"org.apache.gluten.GlutenPlugin\",\n \"spark.app.name\": \"gluten benchmark derived from tpcds\",\n \"spark.cores.max\": \"16\",\n \"spark.gluten.memoryOverhead.size.in.bytes\": \"5153751040\",\n \"spark.memory.offHeap.enabled\": \"true\",\n \"spark.repl.local.jars\": \"file:///opt/gluten/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar\",\n \"spark.executor.extraClassPath\": \"/opt/gluten/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar\",\n \"spark.executor.instances\": \"2\",\n \"spark.executor.memory\": \"16G\",\n \"spark.driver.extraClassPath\": \"/opt/gluten/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar\",\n \"spark.driver.port\": \"38403\",\n \"spark.driver.host\": \"10.0.0.118\",\n \"spark.serializer.objectStreamReset\": \"100\",\n \"spark.app.submitTime\": \"1753671747082\",\n \"spark.submit.deployMode\": \"client\",\n \"spark.app.id\": \"app-20250728030228-0000\",\n \"spark.executor.cores\": \"8\",\n \"spark.app.initial.jar.urls\": \"spark://10.0.0.118:38403/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar\",\n \"spark.jars\": \"file:///opt/gluten/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar\",\n \"spark.driver.extraJavaOptions\": \"-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false\",\n \"spark.gluten.memory.offHeap.size.in.bytes\": \"17179869184\",\n \"spark.shuffle.manager\": \"org.apache.spark.shuffle.sort.ColumnarShuffleManager\",\n \"spark.executor.id\": \"driver\",\n \"spark.master\": \"spark://woody:7077\",\n \"spark.executor.memoryOverhead\": \"4915\",\n \"spark.sql.extensions\": \"org.apache.gluten.extension.GlutenSessionExtensions\",\n \"spark.gluten.numTaskSlotsPerExecutor\": \"8\",\n \"spark.driver.memory\": \"8G\",\n \"spark.gluten.memory.task.offHeap.size.in.bytes\": \"2147483648\",\n \"spark.memory.offHeap.size\": \"17179869184\",\n \"spark.rdd.compress\": \"True\",\n \"spark.executor.extraJavaOptions\": \"-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false\",\n \"spark.app.startTime\": \"1753671747382\",\n \"spark.gluten.memory.conservative.task.offHeap.size.in.bytes\": \"1073741824\",\n \"spark.gluten.sql.columnar.forceShuffledHashJoin\": \"true\",\n \"spark.sql.adaptive.customCostEvaluatorClass\": \"org.apache.spark.sql.execution.adaptive.GlutenCostEvaluator\",\n \"spark.submit.pyFiles\": \"\",\n \"spark.gluten.memory.dynamic.offHeap.sizing.enabled\": \"false\",\n \"spark.sql.session.timeZone\": \"UTC\",\n \"spark.sql.warehouse.dir\": \"file:/home/andy/git/apache/datafusion-comet/dev/benchmarks/spark-warehouse\"\n },\n \"1\": [\n 3.0712015628814697\n ],\n \"2\": [\n 3.0666799545288086\n ],\n \"3\": [\n 1.0671255588531494\n ],\n \"4\": [\n 17.276039361953735\n ],\n \"5\": [\n 4.336325407028198\n ],\n \"6\": [\n 2.240464925765991\n ],\n \"7\": [\n 2.1070384979248047\n ],\n \"8\": [\n 2.2761263847351074\n ],\n \"9\": [\n 3.4592885971069336\n ],\n \"10\": [\n 2.2161333560943604\n ],\n \"11\": [\n 8.601247549057007\n ],\n \"12\": [\n 1.3066213130950928\n ],\n \"13\": [\n 2.623806953430176\n ],\n \"14\": [\n 23.665879487991333\n ],\n \"15\": [\n 2.3319854736328125\n ],\n \"16\": [\n 6.435860633850098\n ],\n \"17\": [\n 3.3381738662719727\n ],\n \"18\": [\n 2.4940454959869385\n ],\n \"19\": [\n 3.214183807373047\n ],\n \"20\": [\n 0.8629763126373291\n ],\n \"21\": [\n 1.1388742923736572\n ],\n \"22\": [\n 13.283171892166138\n ],\n \"23\": [\n 33.25598192214966\n ],\n \"24\": [\n 13.960798025131226\n ],\n \"25\": [\n 3.493340492248535\n ],\n \"26\": [\n 1.003934383392334\n ],\n \"27\": [\n 1.8391008377075195\n ],\n \"28\": [\n 4.88685941696167\n ],\n \"29\": [\n 3.245540142059326\n ],\n \"30\": [\n 1.6746954917907715\n ],\n \"31\": [\n 7.37675404548645\n ],\n \"32\": [\n 1.4505951404571533\n ],\n \"33\": [\n 2.6308183670043945\n ],\n \"34\": [\n 1.872574806213379\n ],\n \"35\": [\n 2.4917962551116943\n ],\n \"36\": [\n 2.046686887741089\n ],\n \"37\": [\n 0.798964262008667\n ],\n \"38\": [\n 3.3161089420318604\n ],\n \"39\": [\n 4.972580671310425\n ],\n \"40\": [\n 3.0317022800445557\n ],\n \"41\": [\n 0.292935848236084\n ],\n \"42\": [\n 0.9157257080078125\n ],\n \"43\": [\n 1.17030668258667\n ],\n \"44\": [\n 0.5907480716705322\n ],\n \"45\": [\n 1.6078379154205322\n ],\n \"46\": [\n 3.1738409996032715\n ],\n \"47\": [\n 3.6137521266937256\n ],\n \"48\": [\n 2.1740095615386963\n ],\n \"49\": [\n 4.5014050006866455\n ],\n \"50\": [\n 3.800581216812134\n ],\n \"51\": [\n 5.458707571029663\n ],\n \"52\": [\n 0.9787416458129883\n ],\n \"53\": [\n 1.2668261528015137\n ],\n \"54\": [\n 4.599064111709595\n ],\n \"55\": [\n 0.9526393413543701\n ],\n \"56\": [\n 2.744847059249878\n ],\n \"57\": [\n 2.3258957862854004\n ],\n \"58\": [\n 2.707801580429077\n ],\n \"59\": [\n 2.9688880443573\n ],\n \"60\": [\n 2.8286445140838623\n ],\n \"61\": [\n 7.713818073272705\n ],\n \"62\": [\n 1.3007659912109375\n ],\n \"63\": [\n 1.6228349208831787\n ],\n \"64\": [\n 10.753086805343628\n ],\n \"65\": [\n 5.098642826080322\n ],\n \"66\": [\n 1.919985055923462\n ],\n \"67\": [\n 31.3416805267334\n ],\n \"68\": [\n 2.760091781616211\n ],\n \"69\": [\n 2.1082839965820312\n ],\n \"70\": [\n 3.231919050216675\n ],\n \"71\": [\n 3.1762571334838867\n ],\n \"72\": [\n 811.9890258312225\n ],\n \"73\": [\n 1.1076979637145996\n ],\n \"74\": [\n 6.423391580581665\n ],\n \"75\": [\n 6.315303325653076\n ],\n \"76\": [\n 3.0667858123779297\n ],\n \"77\": [\n 2.4472310543060303\n ],\n \"78\": [\n 9.947726488113403\n ],\n \"79\": [\n 2.0369961261749268\n ],\n \"80\": [\n 7.902697801589966\n ],\n \"81\": [\n 1.4782285690307617\n ],\n \"82\": [\n 1.0187582969665527\n ],\n \"83\": [\n 1.3076961040496826\n ],\n \"84\": [\n 0.7356717586517334\n ],\n \"85\": [\n 2.1237404346466064\n ],\n \"86\": [\n 1.196605920791626\n ],\n \"87\": [\n 3.310823917388916\n ],\n \"88\": [\n 5.557470083236694\n ],\n \"89\": [\n 1.4879262447357178\n ],\n \"90\": [\n 1.3077824115753174\n ],\n \"91\": [\n 1.2596931457519531\n ],\n \"92\": [\n 1.1216387748718262\n ],\n \"93\": [\n 3.8458316326141357\n ],\n \"94\": [\n 3.957663059234619\n ],\n \"95\": [\n 18.828344583511353\n ],\n \"96\": [\n 0.8511285781860352\n ],\n \"97\": [\n 3.0931687355041504\n ],\n \"98\": [\n 2.2251434326171875\n ],\n \"99\": [\n 1.0991463661193848\n ]\n}" }, { "path": "docs/source/contributor-guide/benchmark-results/gluten-1.4.0-tpch.json", "content": "{\n \"engine\": \"datafusion-comet\",\n \"benchmark\": \"tpch\",\n \"data_path\": \"/mnt/bigdata/tpch/sf100/\",\n \"query_path\": \"/mnt/bigdata/tpch/queries/\",\n \"spark_conf\": {\n \"spark.eventLog.enabled\": \"true\",\n \"spark.gluten.sql.session.timeZone.default\": \"UTC\",\n \"spark.app.submitTime\": \"1752337239601\",\n \"spark.plugins\": \"org.apache.gluten.GlutenPlugin\",\n \"spark.gluten.memoryOverhead.size.in.bytes\": \"5153751040\",\n \"spark.memory.offHeap.enabled\": \"true\",\n \"spark.repl.local.jars\": \"file:///opt/gluten/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar\",\n \"spark.executor.extraClassPath\": \"/opt/gluten/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar\",\n \"spark.driver.port\": \"46545\",\n \"spark.executor.memory\": \"16G\",\n \"spark.driver.extraClassPath\": \"/opt/gluten/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar\",\n \"spark.serializer.objectStreamReset\": \"100\",\n \"spark.driver.host\": \"10.0.0.118\",\n \"spark.submit.deployMode\": \"client\",\n \"spark.executor.cores\": \"8\",\n \"spark.jars\": \"file:///opt/gluten/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar\",\n \"spark.driver.extraJavaOptions\": \"-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false\",\n \"spark.gluten.memory.offHeap.size.in.bytes\": \"17179869184\",\n \"spark.shuffle.manager\": \"org.apache.spark.shuffle.sort.ColumnarShuffleManager\",\n \"spark.app.id\": \"app-20250712162041-0000\",\n \"spark.executor.id\": \"driver\",\n \"spark.master\": \"spark://woody:7077\",\n \"spark.app.name\": \"gluten benchmark derived from tpch\",\n \"spark.executor.memoryOverhead\": \"4915\",\n \"spark.app.startTime\": \"1752337239956\",\n \"spark.sql.extensions\": \"org.apache.gluten.extension.GlutenSessionExtensions\",\n \"spark.gluten.numTaskSlotsPerExecutor\": \"8\",\n \"spark.driver.memory\": \"8G\",\n \"spark.gluten.memory.task.offHeap.size.in.bytes\": \"2147483648\",\n \"spark.memory.offHeap.size\": \"17179869184\",\n \"spark.app.initial.jar.urls\": \"spark://10.0.0.118:46545/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar\",\n \"spark.rdd.compress\": \"True\",\n \"spark.executor.extraJavaOptions\": \"-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false\",\n \"spark.gluten.memory.conservative.task.offHeap.size.in.bytes\": \"1073741824\",\n \"spark.gluten.sql.columnar.forceShuffledHashJoin\": \"true\",\n \"spark.sql.adaptive.customCostEvaluatorClass\": \"org.apache.spark.sql.execution.adaptive.GlutenCostEvaluator\",\n \"spark.submit.pyFiles\": \"\",\n \"spark.executor.instances\": \"1\",\n \"spark.cores.max\": \"8\",\n \"spark.sql.session.timeZone\": \"UTC\",\n \"spark.sql.warehouse.dir\": \"file:/home/andy/git/apache/datafusion-comet/dev/benchmarks/spark-warehouse\",\n \"spark.gluten.memory.dynamic.offHeap.sizing.enabled\": \"false\"\n },\n \"1\": [\n 7.653544187545776\n ],\n \"2\": [\n 4.837377309799194\n ],\n \"3\": [\n 7.874017000198364\n ],\n \"4\": [\n 7.201406717300415\n ],\n \"5\": [\n 15.82085108757019\n ],\n \"6\": [\n 1.7845516204833984\n ],\n \"7\": [\n 10.334960460662842\n ],\n \"8\": [\n 16.844615697860718\n ],\n \"9\": [\n 24.79072380065918\n ],\n \"10\": [\n 8.695719480514526\n ],\n \"11\": [\n 3.6076815128326416\n ],\n \"12\": [\n 4.583712339401245\n ],\n \"13\": [\n 7.016078233718872\n ],\n \"14\": [\n 3.003683567047119\n ],\n \"15\": [\n 7.852885007858276\n ],\n \"16\": [\n 3.1703848838806152\n ],\n \"17\": [\n 19.128700017929077\n ],\n \"18\": [\n 20.703757524490356\n ],\n \"19\": [\n 3.294731378555298\n ],\n \"20\": [\n 6.121671199798584\n ],\n \"21\": [\n 41.27772641181946\n ],\n \"22\": [\n 3.542634963989258\n ]\n}" }, { "path": "docs/source/contributor-guide/benchmark-results/spark-3.5.3-tpcds.json", "content": "{\n \"engine\": \"datafusion-comet\",\n \"benchmark\": \"tpcds\",\n \"data_path\": \"/mnt/bigdata/tpcds/sf100/\",\n \"query_path\": \"/mnt/bigdata/tpcds/queries/\",\n \"spark_conf\": {\n \"spark.eventLog.enabled\": \"true\",\n \"spark.driver.extraJavaOptions\": \"-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false\",\n \"spark.cores.max\": \"16\",\n \"spark.app.submitTime\": \"1760657346549\",\n \"spark.app.startTime\": \"1760657346845\",\n \"spark.hadoop.fs.s3a.aws.credentials.provider\": \"com.amazonaws.auth.DefaultAWSCredentialsProviderChain\",\n \"spark.driver.port\": \"41151\",\n \"spark.master\": \"spark://woody:7077\",\n \"spark.executor.id\": \"driver\",\n \"spark.memory.offHeap.enabled\": \"true\",\n \"spark.driver.memory\": \"8G\",\n \"spark.executor.memory\": \"16g\",\n \"spark.hadoop.fs.s3a.impl\": \"org.apache.hadoop.fs.s3a.S3AFileSystem\",\n \"spark.app.id\": \"app-20251016172907-0000\",\n \"spark.rdd.compress\": \"True\",\n \"spark.executor.instances\": \"2\",\n \"spark.executor.extraJavaOptions\": \"-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false\",\n \"spark.serializer.objectStreamReset\": \"100\",\n \"spark.submit.pyFiles\": \"\",\n \"spark.submit.deployMode\": \"client\",\n \"spark.sql.warehouse.dir\": \"file:/home/andy/git/apache/datafusion-comet/dev/benchmarks/spark-warehouse\",\n \"spark.app.name\": \"spark benchmark derived from tpcds\",\n \"spark.executor.cores\": \"8\",\n \"spark.driver.host\": \"woody.lan\",\n \"spark.memory.offHeap.size\": \"16g\"\n },\n \"1\": [\n 4.523299217224121\n ],\n \"2\": [\n 3.061577081680298\n ],\n \"3\": [\n 1.5242812633514404\n ],\n \"4\": [\n 50.726277589797974\n ],\n \"5\": [\n 12.996135711669922\n ],\n \"6\": [\n 1.9649882316589355\n ],\n \"7\": [\n 3.0779569149017334\n ],\n \"8\": [\n 2.091928720474243\n ],\n \"9\": [\n 5.696160078048706\n ],\n \"10\": [\n 2.6276841163635254\n ],\n \"11\": [\n 14.976734399795532\n ],\n \"12\": [\n 1.0809345245361328\n ],\n \"13\": [\n 3.9019551277160645\n ],\n \"14\": [\n 40.51576566696167\n ],\n \"15\": [\n 3.2053213119506836\n ],\n \"16\": [\n 12.394548416137695\n ],\n \"17\": [\n 4.5201404094696045\n ],\n \"18\": [\n 3.710134506225586\n ],\n \"19\": [\n 2.388089895248413\n ],\n \"20\": [\n 1.1710665225982666\n ],\n \"21\": [\n 1.8241875171661377\n ],\n \"22\": [\n 22.95406436920166\n ],\n \"23\": [\n 91.60468244552612\n ],\n \"24\": [\n 14.566667556762695\n ],\n \"25\": [\n 4.0827412605285645\n ],\n \"26\": [\n 1.9840991497039795\n ],\n \"27\": [\n 2.4599902629852295\n ],\n \"28\": [\n 8.550125122070312\n ],\n \"29\": [\n 4.786677122116089\n ],\n \"30\": [\n 1.974135398864746\n ],\n \"31\": [\n 5.949168920516968\n ],\n \"32\": [\n 1.8300220966339111\n ],\n \"33\": [\n 3.0961456298828125\n ],\n \"34\": [\n 1.5340814590454102\n ],\n \"35\": [\n 4.07404637336731\n ],\n \"36\": [\n 2.289029121398926\n ],\n \"37\": [\n 4.415144681930542\n ],\n \"38\": [\n 7.2151618003845215\n ],\n \"39\": [\n 8.983118057250977\n ],\n \"40\": [\n 8.090219020843506\n ],\n \"41\": [\n 0.2758638858795166\n ],\n \"42\": [\n 1.4861207008361816\n ],\n \"43\": [\n 1.4047019481658936\n ],\n \"44\": [\n 1.1311781406402588\n ],\n \"45\": [\n 1.7742793560028076\n ],\n \"46\": [\n 2.6378352642059326\n ],\n \"47\": [\n 5.033041954040527\n ],\n \"48\": [\n 7.473065137863159\n ],\n \"49\": [\n 7.34236478805542\n ],\n \"50\": [\n 10.566041946411133\n ],\n \"51\": [\n 13.90496039390564\n ],\n \"52\": [\n 1.2180531024932861\n ],\n \"53\": [\n 1.8595950603485107\n ],\n \"54\": [\n 2.967790365219116\n ],\n \"55\": [\n 1.1428017616271973\n ],\n \"56\": [\n 2.6576313972473145\n ],\n \"57\": [\n 3.1256778240203857\n ],\n \"58\": [\n 3.110593795776367\n ],\n \"59\": [\n 3.6817147731781006\n ],\n \"60\": [\n 2.342132568359375\n ],\n \"61\": [\n 3.5699379444122314\n ],\n \"62\": [\n 1.0830762386322021\n ],\n \"63\": [\n 1.5091686248779297\n ],\n \"64\": [\n 21.24241876602173\n ],\n \"65\": [\n 11.174801588058472\n ],\n \"66\": [\n 2.968890428543091\n ],\n \"67\": [\n 56.65302085876465\n ],\n \"68\": [\n 2.7817704677581787\n ],\n \"69\": [\n 2.238774061203003\n ],\n \"70\": [\n 2.8807759284973145\n ],\n \"71\": [\n 2.878760576248169\n ],\n \"72\": [\n 87.73793363571167\n ],\n \"73\": [\n 1.3357255458831787\n ],\n \"74\": [\n 17.069884300231934\n ],\n \"75\": [\n 12.418950080871582\n ],\n \"76\": [\n 2.7246549129486084\n ],\n \"77\": [\n 3.2758727073669434\n ],\n \"78\": [\n 31.570815801620483\n ],\n \"79\": [\n 2.1316466331481934\n ],\n \"80\": [\n 25.685710191726685\n ],\n \"81\": [\n 2.383108615875244\n ],\n \"82\": [\n 2.1166131496429443\n ],\n \"83\": [\n 1.825127124786377\n ],\n \"84\": [\n 1.4185044765472412\n ],\n \"85\": [\n 4.894887924194336\n ],\n \"86\": [\n 0.9917283058166504\n ],\n \"87\": [\n 7.2068891525268555\n ],\n \"88\": [\n 5.049949645996094\n ],\n \"89\": [\n 1.8566343784332275\n ],\n \"90\": [\n 0.8218870162963867\n ],\n \"91\": [\n 1.0735762119293213\n ],\n \"92\": [\n 1.3357479572296143\n ],\n \"93\": [\n 21.765851259231567\n ],\n \"94\": [\n 6.729715585708618\n ],\n \"95\": [\n 23.40083408355713\n ],\n \"96\": [\n 0.6852657794952393\n ],\n \"97\": [\n 11.261420965194702\n ],\n \"98\": [\n 2.003811836242676\n ],\n \"99\": [\n 1.5077838897705078\n ]\n}" }, { "path": "docs/source/contributor-guide/benchmark-results/spark-3.5.3-tpch.json", "content": "{\n \"engine\": \"datafusion-comet\",\n \"benchmark\": \"tpch\",\n \"data_path\": \"/mnt/bigdata/tpch/sf100/\",\n \"query_path\": \"/mnt/bigdata/tpch/queries/\",\n \"spark_conf\": {\n \"spark.app.submitTime\": \"1760654734485\",\n \"spark.eventLog.enabled\": \"true\",\n \"spark.driver.port\": \"45867\",\n \"spark.driver.extraJavaOptions\": \"-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false\",\n \"spark.executor.id\": \"driver\",\n \"spark.hadoop.fs.s3a.aws.credentials.provider\": \"com.amazonaws.auth.DefaultAWSCredentialsProviderChain\",\n \"spark.app.startTime\": \"1760654734783\",\n \"spark.master\": \"spark://woody:7077\",\n \"spark.memory.offHeap.enabled\": \"true\",\n \"spark.driver.memory\": \"8G\",\n \"spark.executor.memory\": \"16g\",\n \"spark.hadoop.fs.s3a.impl\": \"org.apache.hadoop.fs.s3a.S3AFileSystem\",\n \"spark.rdd.compress\": \"True\",\n \"spark.app.name\": \"spark benchmark derived from tpch\",\n \"spark.executor.extraJavaOptions\": \"-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false\",\n \"spark.serializer.objectStreamReset\": \"100\",\n \"spark.executor.instances\": \"1\",\n \"spark.cores.max\": \"8\",\n \"spark.app.id\": \"app-20251016164535-0000\",\n \"spark.submit.pyFiles\": \"\",\n \"spark.submit.deployMode\": \"client\",\n \"spark.sql.warehouse.dir\": \"file:/home/andy/git/apache/datafusion-comet/dev/benchmarks/spark-warehouse\",\n \"spark.executor.cores\": \"8\",\n \"spark.driver.host\": \"woody.lan\",\n \"spark.memory.offHeap.size\": \"16g\"\n },\n \"1\": [\n 82.14922451972961\n ],\n \"2\": [\n 13.39389944076538\n ],\n \"3\": [\n 26.29530620574951\n ],\n \"4\": [\n 22.093881368637085\n ],\n \"5\": [\n 52.291497230529785\n ],\n \"6\": [\n 3.5577659606933594\n ],\n \"7\": [\n 23.285813808441162\n ],\n \"8\": [\n 36.04274129867554\n ],\n \"9\": [\n 77.91567945480347\n ],\n \"10\": [\n 20.83555793762207\n ],\n \"11\": [\n 13.609780311584473\n ],\n \"12\": [\n 13.592298984527588\n ],\n \"13\": [\n 22.524098873138428\n ],\n \"14\": [\n 6.217005252838135\n ],\n \"15\": [\n 16.401182413101196\n ],\n \"16\": [\n 7.700634241104126\n ],\n \"17\": [\n 70.78637218475342\n ],\n \"18\": [\n 77.76704359054565\n ],\n \"19\": [\n 7.267014503479004\n ],\n \"20\": [\n 10.419305562973022\n ],\n \"21\": [\n 72.9061770439148\n ],\n \"22\": [\n 9.621031522750854\n ]\n}" }, { "path": "docs/source/contributor-guide/benchmark-results/tpc-ds.md", "content": "\n\n# Apache DataFusion Comet: Benchmarks Derived From TPC-DS\n\nThe following benchmarks were performed on a Linux workstation with PCIe 5, AMD 7950X CPU (16 cores), 128 GB RAM, and\ndata stored locally in Parquet format on NVMe storage. Performance characteristics will vary in different environments\nand we encourage you to run these benchmarks in your own environments.\n\nThe operating system is Ubuntu 22.04.5 LTS.\n\nThe tracking issue for improving TPC-DS performance is [#858](https://github.com/apache/datafusion-comet/issues/858).\n\n![](../../_static/images/benchmark-results/0.11.0/tpcds_allqueries.png)\n\nHere is a breakdown showing relative performance of Spark and Comet for each query.\n\n![](../../_static/images/benchmark-results/0.11.0/tpcds_queries_compare.png)\n\nThe following chart shows how much Comet currently accelerates each query from the benchmark in relative terms.\n\n![](../../_static/images/benchmark-results/0.11.0/tpcds_queries_speedup_rel.png)\n\nThe following chart shows how much Comet currently accelerates each query from the benchmark in absolute terms.\n\n![](../../_static/images/benchmark-results/0.11.0/tpcds_queries_speedup_abs.png)\n\nThe raw results of these benchmarks in JSON format is available here:\n\n- [Spark](spark-3.5.3-tpcds.json)\n- [Comet](comet-0.11.0-tpcds.json)\n\nThe scripts that were used to generate these results can be found [here](https://github.com/apache/datafusion-comet/tree/main/benchmarks/tpc).\n" }, { "path": "docs/source/contributor-guide/benchmark-results/tpc-h.md", "content": "\n\n# Apache DataFusion Comet: Benchmarks Derived From TPC-H\n\nThe following benchmarks were performed on a Linux workstation with PCIe 5, AMD 7950X CPU (16 cores), 128 GB RAM, and\ndata stored locally in Parquet format on NVMe storage. Performance characteristics will vary in different environments\nand we encourage you to run these benchmarks in your own environments.\n\nThe operating system is Ubuntu 22.04.5 LTS.\n\nThe tracking issue for improving TPC-H performance is [#391](https://github.com/apache/datafusion-comet/issues/391).\n\n![](../../_static/images/benchmark-results/0.11.0/tpch_allqueries.png)\n\nHere is a breakdown showing relative performance of Spark and Comet for each query.\n\n![](../../_static/images/benchmark-results/0.11.0/tpch_queries_compare.png)\n\nThe following chart shows how much Comet currently accelerates each query from the benchmark in relative terms.\n\n![](../../_static/images/benchmark-results/0.11.0/tpch_queries_speedup_rel.png)\n\nThe following chart shows how much Comet currently accelerates each query from the benchmark in absolute terms.\n\n![](../../_static/images/benchmark-results/0.11.0/tpch_queries_speedup_abs.png)\n\nThe raw results of these benchmarks in JSON format is available here:\n\n- [Spark](spark-3.5.3-tpch.json)\n- [Comet](comet-0.11.0-tpch.json)\n\nThe scripts that were used to generate these results can be found [here](https://github.com/apache/datafusion-comet/tree/main/benchmarks/tpc).\n" }, { "path": "docs/source/contributor-guide/benchmarking.md", "content": "\n\n# Comet Benchmarking Guide\n\nTo track progress on performance, we regularly run benchmarks derived from TPC-H and TPC-DS.\n\nThe benchmarking scripts are contained [here](https://github.com/apache/datafusion-comet/tree/main/benchmarks/tpc).\n\nData generation scripts are available in the [DataFusion Benchmarks](https://github.com/apache/datafusion-benchmarks) GitHub repository.\n\n## Current Benchmark Results\n\nThe published benchmarks are performed on a Linux workstation with PCIe 5, AMD 7950X CPU (16 cores), 128 GB RAM, and\ndata stored locally in Parquet format on NVMe storage. Performance characteristics will vary in different environments\nand we encourage you to run these benchmarks in your own environments.\n\nThe operating system used was Ubuntu 22.04.5 LTS.\n\n- [Benchmarks derived from TPC-H](benchmark-results/tpc-h)\n- [Benchmarks derived from TPC-DS](benchmark-results/tpc-ds)\n\n## Benchmarking Guides\n\nAvailable benchmarking guides:\n\n- [Benchmarking on macOS](benchmarking_macos.md)\n- [Benchmarking on AWS EC2](benchmarking_aws_ec2)\n- [TPC-DS Benchmarking with spark-sql-perf](benchmarking_spark_sql_perf.md)\n\nWe also have many micro benchmarks that can be run from an IDE located [here](https://github.com/apache/datafusion-comet/tree/main/spark/src/test/scala/org/apache/spark/sql/benchmark).\n" }, { "path": "docs/source/contributor-guide/benchmarking_aws_ec2.md", "content": "\n\n# Comet Benchmarking in EC2\n\nThis guide is for setting up benchmarks on AWS EC2 with a single node with Parquet files either located on an\nattached EBS volume, or stored in S3.\n\n## Create EC2 Instance\n\n- Create an EC2 instance with an EBS volume sized for approximately 2x the size of\n the dataset to be generated (200 GB for scale factor 100, 2 TB for scale factor 1000, and so on)\n- Optionally, create an S3 bucket to store the Parquet files\n\nRecommendation: Use `c7i.4xlarge` instance type with `provisioned iops 2` EBS volume (8000 IOPS).\n\n## Install Prerequisites\n\n```shell\nsudo yum update -y\nsudo yum install -y git protobuf-compiler java-17-amazon-corretto-headless java-17-amazon-corretto-devel\nsudo yum groupinstall -y \"Development Tools\"\nexport JAVA_HOME=/usr/lib/jvm/java-17-amazon-corretto\n```\n\n## Generate Benchmark Data\n\n```shell\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\nRUSTFLAGS='-C target-cpu=native' cargo install tpchgen-cli\ntpchgen-cli -s 100 --format parquet --parts 32 --output-dir data\n```\n\nRename the generated directories so that they have a `.parquet` suffix. For example, rename `customer` to\n`customer.parquet`.\n\nSet the `TPCH_DATA` environment variable. This will be referenced by the benchmarking scripts.\n\n```shell\nexport TPCH_DATA=/home/ec2-user/data\n```\n\n## Install Apache Spark\n\n```shell\nexport SPARK_VERSION=3.5.6\nwget https://archive.apache.org/dist/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop3.tgz\ntar xzf spark-$SPARK_VERSION-bin-hadoop3.tgz\nsudo mv spark-$SPARK_VERSION-bin-hadoop3 /opt\nexport SPARK_HOME=/opt/spark-$SPARK_VERSION-bin-hadoop3/\nmkdir /tmp/spark-events\n```\n\nSet `SPARK_MASTER` env var (IP address will need to be edited):\n\n```shell\nexport SPARK_MASTER=spark://172.31.34.87:7077\n```\n\nSet `SPARK_LOCAL_DIRS` to point to EBS volume\n\n```shell\nsudo mkdir /mnt/tmp\nsudo chmod 777 /mnt/tmp\nmv $SPARK_HOME/conf/spark-env.sh.template $SPARK_HOME/conf/spark-env.sh\n```\n\nAdd the following entry to `spark-env.sh`:\n\n```shell\nSPARK_LOCAL_DIRS=/mnt/tmp\n```\n\n## Git Clone DataFusion Repositories\n\n```shell\ngit clone https://github.com/apache/datafusion-benchmarks.git\ngit clone https://github.com/apache/datafusion-comet.git\n```\n\nBuild Comet\n\n```shell\ncd datafusion-comet\nmake release\n```\n\nSet `COMET_JAR` environment variable.\n\n```shell\nexport COMET_JAR=/home/ec2-user/datafusion-comet/spark/target/comet-spark-spark3.5_2.12-$COMET_VERSION.jar\n```\n\n## Run Benchmarks\n\nUse the scripts in `benchmarks/tpc` in the Comet repository.\n\n```shell\ncd benchmarks/tpc\nexport TPCH_QUERIES=/home/ec2-user/datafusion-benchmarks/tpch/queries/\n```\n\nRun Spark benchmark:\n\n```shell\npython3 run.py --engine spark --benchmark tpch\n```\n\nRun Comet benchmark:\n\n```shell\npython3 run.py --engine comet --benchmark tpch\n```\n\n## Running Benchmarks with S3\n\nCopy the Parquet data to an S3 bucket.\n\n```shell\naws s3 cp /home/ec2-user/data s3://your-bucket-name/--recursive\n```\n\nUpdate `TPCH_DATA` environment variable.\n\n```shell\nexport TPCH_DATA=s3a://your-bucket-name\n```\n\nInstall Hadoop jar files:\n\n```shell\nwget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar -P $SPARK_HOME/jars\nwget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.11.1026/aws-java-sdk-bundle-1.11.1026.jar -P $SPARK_HOME/jars\n```\n\nAdd credentials to `~/.aws/credentials`:\n\n```shell\n[default]\naws_access_key_id=your-access-key\naws_secret_access_key=your-secret-key\n```\n\nModify the scripts to add the following configurations.\n\n```shell\n--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \\\n--conf spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain \\\n```\n\nNow run the benchmarks:\n\n```shell\npython3 run.py --engine spark --benchmark tpch\npython3 run.py --engine comet --benchmark tpch\n```\n" }, { "path": "docs/source/contributor-guide/benchmarking_macos.md", "content": "\n\n# Comet Benchmarking on macOS\n\nThis guide is for setting up TPC-H benchmarks locally on macOS using the 100 GB dataset.\n\nNote that running this benchmark on macOS is not ideal because we cannot force Spark or Comet to use performance\ncores rather than efficiency cores, and background processes are sharing these cores. Also, power and thermal\nmanagement may throttle CPU cores.\n\n## Prerequisites\n\nJava and Rust must be installed locally.\n\n## Data Generation\n\n```shell\ncargo install tpchgen-cli\nmkdir benchmark_data\ncd benchmark_data\ntpchgen-cli -s 100 --format=parquet\nexport $BENCH_DATA=`pwd`\n```\n\nCreate a temp folder for spark events emitted during benchmarking\n\n```shell\nmkdir /tmp/spark-events\n```\n\n## Clone the DataFusion Benchmarks Repository\n\n```shell\ngit clone https://github.com/apache/datafusion-benchmarks.git\ncd\nexport DF_BENCH=`pwd`\n```\n\n## Install Spark\n\nInstall Apache Spark. This example refers to 3.5.4 version.\n\n```shell\nwget https://archive.apache.org/dist/spark/spark-3.5.4/spark-3.5.4-bin-hadoop3.tgz\ntar xzf spark-3.5.4-bin-hadoop3.tgz\nsudo mv spark-3.5.4-bin-hadoop3 /opt\nexport SPARK_HOME=/opt/spark-3.5.4-bin-hadoop3/\n```\n\nStart Spark in standalone mode:\n\n```shell\n$SPARK_HOME/sbin/start-master.sh\n```\n\nSet `SPARK_MASTER` env var (host name will need to be edited):\n\n```shell\nexport SPARK_MASTER=spark://Rustys-MacBook-Pro.local:7077\n```\n\n```shell\n$SPARK_HOME/sbin/start-worker.sh $SPARK_MASTER\n```\n\n### Start local Apache Spark cluster using `spark-class`\n\nFor Apache Spark distributions installed using `brew` tool, it may happen there is no `$SPARK_HOME/sbin` folder on your machine.\nIn order to start local Apache Spark cluster on `localhost:7077` port, run:\n\n```shell\n$SPARK_HOME/bin/spark-class org.apache.spark.deploy.master.Master --host 127.0.0.1 --port 7077 --webui-port 8080\n```\n\nOnce master has started, in separate console start the worker referring the spark master uri on `localhost:7077`\n\n```shell\n$SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker --cores 8 --memory 16G spark://localhost:7077\n```\n\n## Run Spark Benchmarks\n\nRun the following command (the `--data` parameter will need to be updated to point to your TPC-H data):\n\n```shell\n$SPARK_HOME/bin/spark-submit \\\n --master $SPARK_MASTER \\\n --conf spark.driver.memory=8G \\\n --conf spark.executor.instances=1 \\\n --conf spark.executor.cores=8 \\\n --conf spark.cores.max=8 \\\n --conf spark.executor.memory=16g \\\n --conf spark.memory.offHeap.enabled=true \\\n --conf spark.memory.offHeap.size=16g \\\n --conf spark.eventLog.enabled=true \\\n $DF_BENCH/runners/datafusion-comet/tpcbench.py \\\n --benchmark tpch \\\n --data $BENCH_DATA/tpch-data/ \\\n --queries $DF_BENCH/tpch/queries \\\n --output . \\\n --iterations 1\n```\n\n## Run Comet Benchmarks\n\nBuild Comet from source, with `mimalloc` enabled.\n\n```shell\nmake release COMET_FEATURES=mimalloc\n```\n\nSet `COMET_JAR` to point to the location of the Comet jar file. Example for Comet 0.8\n\n```shell\nexport COMET_JAR=`pwd`/spark/target/comet-spark-spark3.5_2.12-0.8.0-SNAPSHOT.jar\n```\n\nRun the following command (the `--data` parameter will need to be updated to point to your S3 bucket):\n\n```shell\n$SPARK_HOME/bin/spark-submit \\\n --master $SPARK_MASTER \\\n --conf spark.driver.memory=8G \\\n --conf spark.executor.instances=1 \\\n --conf spark.executor.cores=8 \\\n --conf spark.cores.max=8 \\\n --conf spark.executor.memory=16g \\\n --conf spark.memory.offHeap.enabled=true \\\n --conf spark.memory.offHeap.size=16g \\\n --conf spark.eventLog.enabled=true \\\n --jars $COMET_JAR \\\n --driver-class-path $COMET_JAR \\\n --conf spark.driver.extraClassPath=$COMET_JAR \\\n --conf spark.executor.extraClassPath=$COMET_JAR \\\n --conf spark.plugins=org.apache.spark.CometPlugin \\\n --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \\\n --conf spark.comet.enabled=true \\\n --conf spark.comet.exec.shuffle.enableFastEncoding=true \\\n --conf spark.comet.exec.shuffle.fallbackToColumnar=true \\\n --conf spark.comet.exec.replaceSortMergeJoin=true \\\n --conf spark.comet.expression.allowIncompatible=true \\\n $DF_BENCH/runners/datafusion-comet/tpcbench.py \\\n --benchmark tpch \\\n --data $BENCH_DATA/tpch-data/ \\\n --queries $DF_BENCH/tpch/queries \\\n --output . \\\n --iterations 1\n```\n" }, { "path": "docs/source/contributor-guide/benchmarking_spark_sql_perf.md", "content": "\n\n# TPC-DS Benchmarking with spark-sql-perf\n\nThis guide explains how to generate TPC-DS data and run TPC-DS benchmarks using the\n[KubedAI/spark-sql-perf](https://github.com/KubedAI/spark-sql-perf) framework (a fork of\n[databricks/spark-sql-perf](https://github.com/databricks/spark-sql-perf)).\n\nWe use the KubedAI fork because it adds two features not present in the upstream Databricks repository:\n\n- **Apache Iceberg support** — allows generating and registering TPC-DS tables in Iceberg format, which is\n needed for Comet Iceberg benchmarking.\n- **TPC-DS v4.0 queries** — adds the full set of TPC-DS v4.0 queries alongside the existing v1.4 and v2.4 sets.\n\nThe spark-sql-perf approach uses the TPC-DS `dsdgen` tool to generate data directly through Spark, which handles\npartitioning and writing to Parquet format automatically.\n\n## Prerequisites\n\n- Java 17 (for Spark 3.5+)\n- Apache Spark 3.5.x\n- SBT (Scala Build Tool)\n- C compiler toolchain (`gcc`, `make`, `flex`, `bison`, `byacc`)\n\n## Step 1: Build tpcds-kit\n\nThe `dsdgen` tool from [databricks/tpcds-kit](https://github.com/databricks/tpcds-kit) is required for data\ngeneration. This is a modified fork of the official TPC-DS toolkit that outputs to stdout, allowing Spark to\ningest the data directly.\n\n**Linux (Ubuntu/Debian):**\n\n```shell\nsudo apt-get install -y gcc make flex bison byacc git\ngit clone https://github.com/databricks/tpcds-kit.git\ncd tpcds-kit/tools\nmake OS=LINUX\n```\n\n**Linux (CentOS/RHEL/Amazon Linux):**\n\n```shell\nsudo yum install -y gcc make flex bison byacc git\ngit clone https://github.com/databricks/tpcds-kit.git\ncd tpcds-kit/tools\nmake OS=LINUX\n```\n\n**macOS:**\n\n```shell\nxcode-select --install\ngit clone https://github.com/databricks/tpcds-kit.git\ncd tpcds-kit/tools\nmake OS=MACOS\n```\n\nVerify the build succeeded:\n\n```shell\nls -l dsdgen\n```\n\n## Step 2: Build spark-sql-perf\n\n```shell\ngit clone https://github.com/KubedAI/spark-sql-perf.git\ncd spark-sql-perf\ngit checkout support-iceberg-tpcds-v4.0\nsbt package\n```\n\nThis produces a JAR file at `target/scala-2.12/spark-sql-perf_2.12-0.5.1-SNAPSHOT.jar` (the exact version may\nvary). Note the path to this JAR for later use.\n\n## Step 3: Install and Start Spark\n\nIf you do not already have Spark installed:\n\n```shell\nexport SPARK_VERSION=3.5.6\nwget https://archive.apache.org/dist/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop3.tgz\ntar xzf spark-$SPARK_VERSION-bin-hadoop3.tgz\nsudo mv spark-$SPARK_VERSION-bin-hadoop3 /opt\nexport SPARK_HOME=/opt/spark-$SPARK_VERSION-bin-hadoop3/\nexport JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64\nmkdir -p /tmp/spark-events\n```\n\nStart Spark in standalone mode:\n\n```shell\n$SPARK_HOME/sbin/start-master.sh\nexport SPARK_MASTER=spark://$(hostname):7077\n$SPARK_HOME/sbin/start-worker.sh $SPARK_MASTER\n```\n\n## Step 4: Generate TPC-DS Data\n\nLaunch `spark-shell` with the spark-sql-perf JAR loaded:\n\n```shell\n$SPARK_HOME/bin/spark-shell \\\n --master $SPARK_MASTER \\\n --jars /path/to/spark-sql-perf/target/scala-2.12/spark-sql-perf_2.12-0.5.1-SNAPSHOT.jar \\\n --conf spark.driver.memory=8G \\\n --conf spark.executor.instances=1 \\\n --conf spark.executor.cores=8 \\\n --conf spark.executor.memory=16g\n```\n\nIn the Spark shell, run the following to generate data. Adjust `scaleFactor` and paths as needed:\n\n```scala\nimport com.databricks.spark.sql.perf.tpcds.TPCDSTables\n\nval tpcdsKit = \"/path/to/tpcds-kit/tools\"\nval scaleFactor = \"100\" // 100 GB\nval dataDir = \"/path/to/tpcds-data\"\nval format = \"parquet\"\nval numPartitions = 32 // adjust based on cluster size\n\nval tables = new TPCDSTables(spark.sqlContext, dsdgenDir = tpcdsKit, scaleFactor = scaleFactor)\ntables.genData(\n location = dataDir,\n format = format,\n overwrite = true,\n partitionTables = true,\n clusterByPartitionColumns = true,\n filterOutNullPartitionValues = false,\n numPartitions = numPartitions\n)\n```\n\nData generation for SF100 typically takes 20-60 minutes depending on hardware. When complete, exit the shell:\n\n```scala\n:quit\n```\n\nVerify the data was generated:\n\n```shell\nls /path/to/tpcds-data/\n```\n\nYou should see directories for each TPC-DS table (`store_sales`, `catalog_sales`, `web_sales`, `customer`,\n`date_dim`, etc.).\n\nSet the `TPCDS_DATA` environment variable:\n\n```shell\nexport TPCDS_DATA=/path/to/tpcds-data\n```\n\n## Step 5: Run TPC-DS Benchmarks\n\n### Register Tables\n\nLaunch `spark-shell` with the spark-sql-perf JAR (same as Step 4) and register the generated data as tables:\n\n```scala\nimport com.databricks.spark.sql.perf.tpcds.{TPCDS, TPCDSTables}\n\nval scaleFactor = \"100\"\nval dataDir = \"/path/to/tpcds-data\"\nval format = \"parquet\"\nval databaseName = \"tpcds\"\n\n// Create database and register tables\nsql(s\"CREATE DATABASE IF NOT EXISTS $databaseName\")\nval tables = new TPCDSTables(spark.sqlContext, dsdgenDir = \"\", scaleFactor = scaleFactor)\ntables.createExternalTables(\n location = dataDir,\n format = format,\n databaseName = databaseName,\n overwrite = true,\n discoverPartitions = true\n)\n\nsql(s\"USE $databaseName\")\n```\n\n### Run Spark Baseline\n\n```scala\nval tpcds = new TPCDS(spark.sqlContext)\n\n// Choose a query set: tpcds1_4Queries, tpcds2_4Queries, or tpcds4_0Queries\nval queries = tpcds.tpcds2_4Queries\n\nval experiment = tpcds.runExperiment(\n executionsToRun = queries,\n iterations = 3,\n resultLocation = \"/path/to/results/spark\",\n tags = Map(\"engine\" -> \"spark\", \"scale_factor\" -> \"100\"),\n forkThread = true\n)\n\nexperiment.waitForFinish(86400)\n```\n\nResults are saved as JSON to the `resultLocation` path.\n\n### Run with Comet\n\nBuild Comet from source and launch `spark-shell` with both the Comet and spark-sql-perf JARs:\n\n```shell\nmake release\nexport COMET_JAR=$(pwd)/spark/target/comet-spark-spark3.5_2.12-*.jar\n\n$SPARK_HOME/bin/spark-shell \\\n --master $SPARK_MASTER \\\n --jars /path/to/spark-sql-perf/target/scala-2.12/spark-sql-perf_2.12-0.5.1-SNAPSHOT.jar,$COMET_JAR \\\n --conf spark.driver.memory=8G \\\n --conf spark.executor.instances=1 \\\n --conf spark.executor.cores=8 \\\n --conf spark.executor.memory=8g \\\n --conf spark.memory.offHeap.enabled=true \\\n --conf spark.memory.offHeap.size=8g \\\n --conf spark.executor.extraClassPath=$COMET_JAR \\\n --conf spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions \\\n --conf spark.comet.enabled=true \\\n --conf spark.comet.exec.enabled=true \\\n --conf spark.comet.exec.all.enabled=true \\\n --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \\\n --conf spark.comet.exec.shuffle.enabled=true \\\n --conf spark.comet.columnar.shuffle.enabled=true\n```\n\nThen register tables and run the benchmark the same way as the Spark baseline, changing the tags and result\nlocation:\n\n```scala\nval experiment = tpcds.runExperiment(\n executionsToRun = queries,\n iterations = 3,\n resultLocation = \"/path/to/results/comet\",\n tags = Map(\"engine\" -> \"comet\", \"scale_factor\" -> \"100\"),\n forkThread = true\n)\n\nexperiment.waitForFinish(86400)\n```\n\n### View Results\n\nResults are saved as JSON under the `resultLocation`. You can query them directly in Spark:\n\n```scala\nval results = spark.read.json(\"/path/to/results/spark\")\nresults.select(\"name\", \"parsingTime\", \"analysisTime\", \"optimizationTime\", \"planningTime\", \"executionTime\")\n .withColumn(\"totalTime\", (col(\"parsingTime\") + col(\"analysisTime\") +\n col(\"optimizationTime\") + col(\"planningTime\") + col(\"executionTime\")) / 1000.0)\n .orderBy(\"name\")\n .show(200, false)\n```\n\n## Alternative: Command-Line Data Generation\n\nYou can also generate TPC-DS data without the Spark shell using `spark-submit`:\n\n```shell\n$SPARK_HOME/bin/spark-submit \\\n --master $SPARK_MASTER \\\n --class com.databricks.spark.sql.perf.tpcds.GenTPCDSData \\\n --conf spark.driver.memory=8G \\\n --conf spark.executor.instances=1 \\\n --conf spark.executor.cores=8 \\\n --conf spark.executor.memory=16g \\\n /path/to/spark-sql-perf/target/scala-2.12/spark-sql-perf_2.12-0.5.1-SNAPSHOT.jar \\\n -d /path/to/tpcds-kit/tools \\\n -s 100 \\\n -l /path/to/tpcds-data \\\n -f parquet\n```\n\n## Troubleshooting\n\n### dsdgen not found\n\nEnsure `tpcds-kit/tools/dsdgen` exists and is executable. The `dsdgenDir` parameter in the Spark shell (or `-d`\nflag in the CLI) must point to the directory containing the `dsdgen` binary, not the binary itself.\n\n### Out of memory during data generation\n\nFor large scale factors (SF1000+), increase executor memory and the number of partitions:\n\n```shell\n--conf spark.executor.memory=32g\n```\n\nAnd in the Spark shell, use a higher `numPartitions` value (e.g., 200+).\n" }, { "path": "docs/source/contributor-guide/bug_triage.md", "content": "\n\n# Bug Triage Guide\n\nThis guide describes how we prioritize and triage bugs in the Comet project. The goal is to ensure\nthat the most impactful bugs — especially correctness issues that produce wrong results — are\nidentified and addressed before less critical issues.\n\n## Priority Labels\n\nEvery bug should have exactly one priority label. When filing or triaging a bug, apply the\nappropriate label from the table below.\n\n| Label | Color | Description | Examples |\n| ------------------- | ------ | ------------------------------------------------------------------------------------ | --------------------------------------------------------------------- |\n| `priority:critical` | Red | Data corruption, silent wrong results, security vulnerabilities | Wrong aggregation results, FFI data corruption, incorrect cast output |\n| `priority:high` | Orange | Crashes, panics, segfaults, major functional breakage affecting production workloads | Native engine panic, JVM segfault, NPE on supported code path |\n| `priority:medium` | Yellow | Functional bugs, performance regressions, broken features that have workarounds | Missing expression support, writer feature gaps, excessive spilling |\n| `priority:low` | Green | Minor issues, test-only failures, tooling, CI flakes, cosmetic issues | Flaky CI test, build script edge case, documentation generator bug |\n\n### How to Choose a Priority\n\nUse this decision tree:\n\n1. **Can this bug cause silent wrong results?** If yes → `priority:critical`. These are the most\n dangerous bugs because users may not notice the incorrect output.\n2. **Does this bug crash the JVM or native engine?** If yes → `priority:high`. Crashes are\n disruptive but at least visible to the user.\n3. **Does this bug break a feature or cause significant performance degradation?** If yes →\n `priority:medium`. The user can work around it (e.g., falling back to Spark) but it impacts\n the value of Comet.\n4. **Everything else** → `priority:low`. Test failures, CI issues, tooling, and cosmetic problems.\n\n### Escalation Triggers\n\nA bug should be escalated to a higher priority if:\n\n- A `priority:high` crash is discovered to also produce wrong results silently in some cases →\n escalate to `priority:critical`\n- A `priority:medium` bug is reported by multiple users or affects a common workload → consider\n escalating to `priority:high`\n- A `priority:low` CI flake is blocking PR merges consistently → escalate to `priority:medium`\n\n## Area Labels\n\nArea labels indicate which subsystem is affected. A bug may have multiple area labels. These\nhelp contributors find bugs in their area of expertise.\n\n| Label | Description |\n| ------------------ | ----------------------------------------- |\n| `area:writer` | Native writer (Parquet and other formats) |\n| `area:shuffle` | Shuffle (JVM and native) |\n| `area:aggregation` | Hash aggregates, aggregate expressions |\n| `area:scan` | Data source scan (Parquet, CSV, Iceberg) |\n| `area:expressions` | Expression evaluation |\n| `area:ffi` | Arrow FFI / JNI boundary |\n| `area:ci` | CI/CD, GitHub Actions, build tooling |\n\nThe following pre-existing labels also serve as area indicators: `native_datafusion`,\n`native_iceberg_compat`, `spark 4`, `spark sql tests`.\n\n## Triage Process\n\nEvery new issue is automatically labeled with `requires-triage` when it is opened. This makes it\neasy to find issues that have not yet been triaged by filtering on that label. Once an issue has\nbeen triaged, remove the `requires-triage` label and apply the appropriate priority and area labels.\n\n### For New Issues\n\nWhen a new bug is filed:\n\n1. **Reproduce or verify** the issue if possible. If the report lacks reproduction steps, ask\n the reporter for more details.\n2. **Assess correctness impact first.** Ask: \"Could this produce wrong results silently?\" This\n is more important than whether it crashes.\n3. **Apply a priority label** using the decision tree above.\n4. **Apply area labels** to indicate the affected subsystem(s).\n5. **Apply `good first issue`** if the fix is likely straightforward and well-scoped.\n6. **Remove the `requires-triage` label** to indicate triage is complete.\n\n### For Existing Bugs\n\nPeriodically review open bugs to ensure priorities are still accurate:\n\n- Has a `priority:medium` bug been open for a long time with user reports? Consider escalating.\n- Has a `priority:high` bug been fixed by a related change? Close it.\n- Are there clusters of related bugs that should be tracked under an EPIC?\n\n### Prioritization Principles\n\n1. **Correctness over crashes.** A bug that silently returns wrong results is worse than one that\n crashes, because crashes are at least visible.\n2. **User-reported over test-only.** A bug hit by a real user on a real workload takes priority\n over one found only in test suites.\n3. **Core path over experimental.** Bugs in the default scan mode (`native_comet`) or widely-used\n expressions take priority over bugs in experimental features like `native_datafusion` or\n `native_iceberg_compat`.\n4. **Production safety over feature completeness.** Fixing a data corruption bug is more important\n than adding support for a new expression.\n\n## Common Bug Categories\n\n### Correctness Bugs (`priority:critical`)\n\nThese are bugs where Comet produces different results than Spark without any error or warning.\nExamples include:\n\n- Incorrect cast behavior (e.g., negative zero to string)\n- Aggregate functions ignoring configuration (e.g., `ignoreNulls`)\n- Data corruption in FFI boundary (e.g., boolean arrays with non-zero offset)\n- Type mismatches between partial and final aggregation stages\n\nWhen fixing correctness bugs, always add a regression test that verifies the output matches Spark.\n\n### Crash Bugs (`priority:high`)\n\nThese are bugs where the native engine panics, segfaults, or throws an unhandled exception.\nCommon patterns include:\n\n- **All-scalar inputs:** Some expressions assume at least one columnar input and panic when all\n inputs are literals (e.g., when `ConstantFolding` is disabled)\n- **Type mismatches:** Downcasting to the wrong Arrow array type\n- **Memory safety:** FFI boundary issues, unaligned arrays, GlobalRef lifecycle\n\n### Aggregate Planning Bugs\n\nSeveral bugs relate to how Comet plans hash aggregates across stage boundaries. The key issue is\nthat Spark's AQE may materialize a Comet partial aggregate but then run the final aggregate in\nSpark (or vice versa), and the intermediate formats may not be compatible. See the\n[EPIC #2892](https://github.com/apache/datafusion-comet/issues/2892) for the full picture.\n\n### Native Writer Bugs\n\nThe native Parquet writer has a cluster of known test failures tracked as individual issues\n(#3417–#3430). These are lower priority since the native writer is still maturing, but they\nshould be addressed before the writer is promoted to production-ready status.\n\n## How to Help with Triage\n\nTriage is a valuable contribution that doesn't require writing code. You can help by:\n\n- Reviewing new issues and suggesting a priority label\n- Reproducing reported bugs and adding details\n- Identifying duplicate issues\n- Linking related issues together\n- Testing whether old bugs have been fixed by recent changes\n" }, { "path": "docs/source/contributor-guide/contributing.md", "content": "\n\n# Contributing to Apache DataFusion Comet\n\nWe welcome contributions to Comet in many areas, and encourage new contributors to get involved.\n\nHere are some areas where you can help:\n\n- Testing Comet with existing Spark jobs and reporting issues for any bugs or performance issues\n- Contributing code to support Spark expressions, operators, and data types that are not currently supported\n- Reviewing pull requests and helping to test new features for correctness and performance\n- Improving documentation\n\n## Finding issues to work on\n\nWe maintain a list of good first issues in GitHub [here](https://github.com/apache/datafusion-comet/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22). We also have a [roadmap](roadmap.md).\n\nTo assign yourself an issue, comment `take` on the issue. To unassign yourself, comment `untake`.\n\n## Reporting issues\n\nWe use [GitHub issues](https://github.com/apache/datafusion-comet/issues) for bug reports and feature requests.\n\n## Asking for Help\n\nThe Comet project uses the same Slack and Discord channels as the main Apache DataFusion project. See details at\n[Apache DataFusion Communications]. There are dedicated Comet channels in both Slack and Discord.\n\n## Regular public meetings\n\nThe Comet contributors hold regular video calls where new and current contributors are welcome to ask questions and\ncoordinate on issues that they are working on.\n\nSee the [Apache DataFusion Comet community meeting] Google document for more information.\n\n[Apache DataFusion Communications]: https://datafusion.apache.org/contributor-guide/communication.html\n[Apache DataFusion Comet community meeting]: https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8/edit?usp=sharing\n" }, { "path": "docs/source/contributor-guide/debugging.md", "content": "\n\n# Comet Debugging Guide\n\nThis HOWTO describes how to debug JVM code and Native code concurrently. The guide assumes you have:\n\n1. IntelliJ as the Java IDE\n2. CLion as the Native IDE. For Rust code, the CLion Rust language plugin is required. Note that the\n IntelliJ Rust plugin is not sufficient.\n3. CLion/LLDB as the native debugger. CLion ships with a bundled LLDB and the Rust community has\n its own packaging of LLDB (`lldb-rust`). Both provide a better display of Rust symbols than plain\n LLDB or the LLDB that is bundled with XCode. We will use the LLDB packaged with CLion for this guide.\n4. We will use a Comet _unit_ test as the canonical use case.\n\n_Caveat: The steps here have only been tested with JDK 11_ on Mac (M1)\n\n## Debugging for Advanced Developers\n\nAdd a `.lldbinit` to comet/native. This is not strictly necessary but will be useful if you want to\nuse advanced `lldb` debugging. For example, we can ignore some exceptions and signals that are not relevant\nto our debugging and would otherwise cause the debugger to stop.\n\n```\nsettings set platform.plugin.darwin.ignored-exceptions EXC_BAD_ACCESS|EXC_BAD_INSTRUCTION\nprocess handle -n true -p true -s false SIGBUS SIGSEGV SIGILL\n```\n\n### In IntelliJ\n\n1. Set a breakpoint in `NativeBase.load()`, at a point _after_ the Comet library has been loaded.\n\n1. Add a Debug Configuration for the unit test\n\n1. In the Debug Configuration for that unit test add `-Xint` as a JVM parameter. This option is\n undocumented _magic_. Without this, the LLDB debugger hits a EXC_BAD_ACCESS (or EXC_BAD_INSTRUCTION) from\n which one cannot recover.\n\n1. Add a println to the unit test to print the PID of the JVM process. (jps can also be used but this is less error prone if you have multiple jvm processes running)\n\n ```scala\n println(\"Waiting for Debugger: PID - \", ManagementFactory.getRuntimeMXBean().getName())\n ```\n\n This will print something like : `PID@your_machine_name`.\n\n For JDK9 and newer\n\n ```scala\n println(\"Waiting for Debugger: PID - \", ProcessHandle.current.pid)\n ```\n\n ==> Note the PID\n\n1. Debug-run the test in IntelliJ and wait for the breakpoint to be hit\n\n### In CLion\n\n1. After the breakpoint is hit in IntelliJ, in Clion (or LLDB from terminal or editor) -\n 1. Attach to the jvm process (make sure the PID matches). In CLion, this is `Run -> Attach to process`\n\n 1. Put your breakpoint in the native code\n\n1. Go back to IntelliJ and resume the process.\n\n1. Most debugging in CLion is similar to IntelliJ. For advanced LLDB based debugging the LLDB command line can be accessed from the LLDB tab in the Debugger view. Refer to the [LLDB manual](https://lldb.llvm.org/use/tutorial.html) for LLDB commands.\n\n### After your debugging is done\n\n1. In CLion, detach from the process if not already detached\n\n2. In IntelliJ, the debugger might have lost track of the process. If so, the debugger tab\n will show the process as running (even if the test/job is shown as completed).\n\n3. Close the debugger tab, and if the IDS asks whether it should terminate the process,\n click Yes.\n\n4. In terminal, use jps to identify the process with the process id you were debugging. If\n it shows up as running, kill -9 [pid]. If that doesn't remove the process, don't bother,\n the process will be left behind as a zombie and will consume no (significant) resources.\n Eventually it will be cleaned up when you reboot possibly after a software update.\n\n### Additional Info\n\nOpenJDK mailing list on debugging the JDK on MacOS\n\n\nDetecting the debugger\n).\n\n## Verbose debug\n\n### Enabling Native Backtraces\n\nBy default, Comet does not show native backtraces when exceptions happen in native code:\n\n```scala\nscala> spark.sql(\"my_failing_query\").show(false)\n\n24/03/05 17:00:07 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)/ 1]\norg.apache.comet.CometNativeException: Internal error: MIN/MAX is not expected to receive scalars of incompatible types (Date32(\"NULL\"), Int32(15901)).\nThis was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker\n at org.apache.comet.Native.executePlan(Native Method)\n at org.apache.comet.CometExecIterator.executeNative(CometExecIterator.scala:65)\n at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:111)\n at org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:126)\n\n```\n\nComet can be built with DataFusion's [backtrace] feature enabled, which will include native back traces in `CometNativeException`.\n\n[backtrace]: https://arrow.apache.org/datafusion/user-guide/example-usage.html#enable-backtraces\n\nTo build Comet with this feature enabled:\n\n```shell\nmake release COMET_FEATURES=backtrace\n```\n\nSet `RUST_BACKTRACE=1` for the Spark worker/executor process, or for `spark-submit` if running in local mode.\n\n```console\nRUST_BACKTRACE=1 $SPARK_HOME/spark-shell --jars spark/target/comet-spark-spark3.5_2.12-$COMET_VERSION.jar --conf spark.plugins=org.apache.spark.CometPlugin --conf spark.comet.enabled=true --conf spark.comet.exec.enabled=true\n```\n\nGet the expanded exception details\n\n```scala\nscala> spark.sql(\"my_failing_query\").show(false)\n24/03/05 17:00:49 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)\norg.apache.comet.CometNativeException: Internal error: MIN/MAX is not expected to receive scalars of incompatible types (Date32(\"NULL\"), Int32(15901))\n\nbacktrace:\n 0: std::backtrace::Backtrace::create\n 1: datafusion::physical_expr::aggregate::min_max::min\n 2: ::update_batch\n 3: as futures_core::stream::Stream>::poll_next\n 4: comet::execution::jni_api::Java_org_apache_comet_Native_executePlan::{{closure}}\n 5: _Java_org_apache_comet_Native_executePlan\n (reduced)\n\nThis was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker\n at org.apache.comet.Native.executePlan(Native Method)\nat org.apache.comet.CometExecIterator.executeNative(CometExecIterator.scala:65)\nat org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:111)\nat org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:126)\n(reduced)\n\n```\n\nNote:\n\n- The backtrace coverage in DataFusion is still improving. So there is a chance the error still not covered, if so feel free to file a [ticket](https://github.com/apache/arrow-datafusion/issues)\n- The backtrace evaluation comes with performance cost and intended mostly for debugging purposes\n\n### Native log configuration\n\nBy default, Comet emits native-side logs at the `INFO` level to `stderr`.\n\nYou can use the `COMET_LOG_LEVEL` environment variable to specify the log level. Supported values are: `OFF`, `ERROR`, `WARN`, `INFO`, `DEBUG`, `TRACE`.\n\nFor example, to configure native logs at the `DEBUG` level on spark executor:\n\n```\nspark.executorEnv.COMET_LOG_LEVEL=DEBUG\n```\n\nThis produces output like the following:\n\n```\n25/09/15 20:17:42 INFO core/src/lib.rs: Comet native library version 0.11.0 initialized\n25/09/15 20:17:44 DEBUG /xxx/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/datafusion-execution-49.0.2/src/disk_manager.rs: Created local dirs [TempDir { path: \"/private/var/folders/4p/9gtjq1s10fd6frkv9kzy0y740000gn/T/blockmgr-ba524f95-a792-4d79-b49c-276ba324941e/datafusion-qrpApx\" }] as DataFusion working directory\n25/09/15 20:17:44 DEBUG /xxx/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/datafusion-functions-nested-49.0.2/src/lib.rs: Overwrite existing UDF: array_to_string\n25/09/15 20:17:44 DEBUG /xxx/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/datafusion-functions-nested-49.0.2/src/lib.rs: Overwrite existing UDF: string_to_array\n25/09/15 20:17:44 DEBUG /xxx/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/datafusion-functions-nested-49.0.2/src/lib.rs: Overwrite existing UDF: range\n...\n```\n\nAdditionally, you can place a `log4rs.yaml` configuration file inside the Comet configuration directory specified by the `COMET_CONF_DIR` environment variable to enable more advanced logging configurations. This file uses the [log4rs YAML configuration format](https://docs.rs/log4rs/latest/log4rs/#configuration-via-a-yaml-file).\nFor example, see: [log4rs.yaml](https://github.com/apache/datafusion-comet/blob/main/conf/log4rs.yaml).\n\n### Debugging Memory Reservations\n\nSet `spark.comet.debug.memory=true` to log all calls that grow or shrink memory reservations.\n\nExample log output:\n\n```\n[Task 486] MemoryPool[ExternalSorter[6]].try_grow(256232960) returning Ok\n[Task 486] MemoryPool[ExternalSorter[6]].try_grow(256375168) returning Ok\n[Task 486] MemoryPool[ExternalSorter[6]].try_grow(256899456) returning Ok\n[Task 486] MemoryPool[ExternalSorter[6]].try_grow(257296128) returning Ok\n[Task 486] MemoryPool[ExternalSorter[6]].try_grow(257820416) returning Err\n[Task 486] MemoryPool[ExternalSorterMerge[6]].shrink(10485760)\n[Task 486] MemoryPool[ExternalSorter[6]].shrink(150464)\n[Task 486] MemoryPool[ExternalSorter[6]].shrink(146688)\n[Task 486] MemoryPool[ExternalSorter[6]].shrink(137856)\n[Task 486] MemoryPool[ExternalSorter[6]].shrink(141952)\n[Task 486] MemoryPool[ExternalSorterMerge[6]].try_grow(0) returning Ok\n[Task 486] MemoryPool[ExternalSorterMerge[6]].try_grow(0) returning Ok\n[Task 486] MemoryPool[ExternalSorter[6]].shrink(524288)\n[Task 486] MemoryPool[ExternalSorterMerge[6]].try_grow(0) returning Ok\n[Task 486] MemoryPool[ExternalSorterMerge[6]].try_grow(68928) returning Ok\n```\n\nWhen backtraces are enabled (see earlier section) then backtraces will be included for failed allocations.\n" }, { "path": "docs/source/contributor-guide/development.md", "content": "\n\n# Comet Development Guide\n\n## Project Layout\n\n```\n├── common <- common Java/Scala code\n├── conf <- configuration files\n├── native <- native code, in Rust\n├── spark <- Spark integration\n```\n\n## Threading Architecture\n\nComet's native execution runs on a shared tokio multi-threaded runtime. Understanding this\narchitecture is important because it affects how you write native operators and JVM callbacks.\n\n### How execution works\n\nSpark calls into native code via JNI from an **executor task thread**. There are two execution\npaths depending on whether the plan reads data from the JVM:\n\n**Async I/O path (no JVM data sources, e.g. Iceberg scans):** The DataFusion stream is spawned\nonto a tokio worker thread and batches are delivered to the executor thread via an `mpsc` channel.\nThe executor thread parks in `blocking_recv()` until the next batch is ready. This avoids\nbusy-polling on I/O-bound workloads.\n\n**JVM data source path (ScanExec present):** The executor thread calls `block_on()` and polls the\nDataFusion stream directly, interleaving `pull_input_batches()` calls on `Poll::Pending` to feed\ndata from the JVM into ScanExec operators.\n\nIn both cases, DataFusion operators execute on **tokio worker threads**, not on the Spark executor\ntask thread. All Spark tasks on an executor share one tokio runtime.\n\n### Rules for native code\n\n**Do not use `thread_local!` or assume thread identity.** Tokio may run your operator's `poll`\nmethod on any worker thread, and may move it between threads across polls. Any state must live\nin the operator struct or be shared via `Arc`.\n\n**JNI calls work from any thread, but have overhead.** `JVMClasses::get_env()` calls\n`AttachCurrentThread`, which acquires JVM internal locks. The `AttachGuard` detaches the thread\nwhen dropped. Repeated attach/detach cycles on tokio workers add overhead, so avoid calling\ninto the JVM on hot paths during stream execution.\n\n**Do not call `TaskContext.get()` from JVM callbacks during execution.** Spark's `TaskContext` is\na `ThreadLocal` on the executor task thread. JVM methods invoked from tokio worker threads will\nsee `null`. If you need task metadata, capture it at construction time (in `createPlan` or\noperator setup) and store it in the operator. See `CometTaskMemoryManager` for an example — it\ncaptures `TaskContext.get().taskMemoryManager()` in its constructor and uses the stored reference\nthereafter.\n\n**Memory pool operations call into the JVM.** `CometUnifiedMemoryPool` and `CometFairMemoryPool`\ncall `acquireMemory()` / `releaseMemory()` via JNI whenever DataFusion operators grow or shrink\nmemory reservations. This happens on whatever thread the operator is executing on. These calls\nare thread-safe (they use stored `GlobalRef`s, not thread-locals), but they do trigger\n`AttachCurrentThread`.\n\n**Scalar subqueries call into the JVM.** `Subquery::evaluate()` calls static methods on\n`CometScalarSubquery` via JNI. These use a static `HashMap`, not thread-locals, so they are\nsafe from any thread.\n\n**Parquet encryption calls into the JVM.** `CometKeyRetriever::retrieve_key()` calls the JVM\nto unwrap decryption keys during Parquet reads. It uses a stored `GlobalRef` and a cached\n`JMethodID`, so it is safe from any thread.\n\n### The tokio runtime\n\nThe runtime is created once per executor JVM in a `Lazy` static:\n\n- **Worker threads:** `num_cpus` by default, configurable via `COMET_WORKER_THREADS`\n- **Max blocking threads:** 512 by default, configurable via `COMET_MAX_BLOCKING_THREADS`\n- All async I/O (S3, HTTP, Parquet reads) runs on worker threads as non-blocking futures\n\n### Summary of what is safe and what is not\n\n| Pattern | Safe? | Notes |\n| ----------------------------------------- | ------ | ---------------------------------------- |\n| `Arc` shared across operators | Yes | Standard Rust thread safety |\n| `JVMClasses::get_env()` from tokio worker | Yes | Attaches thread to JVM automatically |\n| `thread_local!` in operator code | **No** | Tokio moves tasks between threads |\n| `TaskContext.get()` in JVM callback | **No** | Returns `null` on non-executor threads |\n| Storing `JNIEnv` in an operator | **No** | `JNIEnv` is thread-specific |\n| Capturing state at plan creation time | Yes | Runs on executor thread, store in struct |\n\n## Global singletons\n\nComet code runs in both the driver and executor JVM processes, and different parts of the\ncodebase run in each. Global singletons have **process lifetime** — they are created once and\nnever dropped until the JVM exits. Since multiple Spark jobs, queries, and tasks share the same\nprocess, this makes it difficult to reason about what state a singleton holds and whether it is\nstill valid.\n\n### How to recognize them\n\n**Rust:** `static` variables using `OnceLock`, `LazyLock`, `OnceCell`, `Lazy`, or `lazy_static!`:\n\n```rust\nstatic TOKIO_RUNTIME: OnceLock = OnceLock::new();\nstatic TASK_SHARED_MEMORY_POOLS: Lazy>> = Lazy::new(..);\n```\n\n**Java:** `static` fields, especially mutable collections:\n\n```java\nprivate static final HashMap> subqueryMap = new HashMap<>();\n```\n\n**Scala:** `object` declarations (companion objects are JVM singletons) holding mutable state:\n\n```scala\nobject MyCache {\n private val cache = new ConcurrentHashMap[String, Value]()\n}\n```\n\n### Why they are dangerous\n\n- **Credential staleness.** A singleton caching an authenticated client will hold stale\n credentials after token rotation, causing silent failures mid-job.\n- **Unbounded growth.** A cache keyed by file path or configuration grows with every query\n but never shrinks. Over hours of process uptime this becomes a memory leak.\n- **Cross-job contamination.** Different Spark jobs on the same process may use different\n configurations. A singleton initialized by the first job silently serves wrong state to\n subsequent jobs.\n- **Testing difficulty.** Global state persists across test cases, making tests\n order-dependent.\n\n### When a singleton is acceptable\n\nSome state genuinely has process lifetime:\n\n| Singleton | Why it is safe |\n| --------------------------------------------- | --------------------------------------------------- |\n| `TOKIO_RUNTIME` | One runtime per executor, no configuration variance |\n| `JAVA_VM` / `JVM_CLASSES` | One JVM per process, set once at JNI load |\n| `OperatorRegistry` / `ExpressionRegistry` | Immutable after initialization |\n| Compiled `Regex` patterns (`LazyLock`) | Stateless and immutable |\n\n### When to avoid a singleton\n\nIf any of these apply, do **not** use a global singleton:\n\n- The state depends on configuration that can vary between jobs or queries\n- The state holds credentials or authenticated connections that will not expire or invalidate appropriately\n- The state grows proportionally to the number of queries or files processed\n- The state needs cleanup or refresh during process lifetime\n\nInstead, scope state to the plan or task by adding the cache as a field in an existing session or context object.\n\nIf a singleton is truly needed, add a comment explaining why `static` is the right lifetime,\nwhether the cache is bounded, and how credential refresh is handled (if applicable).\n\n## Development Setup\n\n1. Make sure `JAVA_HOME` is set and point to JDK using [support matrix](../user-guide/latest/installation.md)\n2. Install Rust toolchain. The easiest way is to use\n [rustup](https://rustup.rs).\n\n## Build & Test\n\nA few common commands are specified in project's `Makefile`:\n\n- `make`: compile the entire project, but don't run tests\n- `make test-rust`: compile the project and run tests in Rust side\n- `make test-jvm`: compile the project and run tests in Java side\n- `make test`: compile the project and run tests in both Rust and Java\n side.\n- `make release`: compile the project and creates a release build. This\n is useful when you want to test Comet local installation in another project\n such as Spark.\n- `make clean`: clean up the workspace\n\n## Common Build and Test Pitfalls\n\n### Native Code Must Be Built First\n\nThe native Rust code must be compiled before running JVM tests. If you skip this step, tests will\nfail because they cannot find the native library. Always run `make core` (or `cd native && cargo build`)\nbefore running Maven tests.\n\n```sh\n# Correct order\nmake core # Build native code first\n./mvnw test -Dsuites=\"...\" # Then run JVM tests\n```\n\n### Debug vs Release Mode\n\nThere is no need to use release mode (`make release`) during normal development. Debug builds\nare faster to compile and provide better error messages. Only use release mode when:\n\n- Running benchmarks\n- Testing performance\n- Creating a release build for deployment\n\nFor regular development and testing, use `make` or `make core` which build in debug mode.\n\n### Running Rust Tests\n\nWhen running Rust tests directly with `cargo test`, the JVM library (`libjvm.so`) must be on\nyour library path. Set the `LD_LIBRARY_PATH` environment variable to include your JDK's `lib/server`\ndirectory:\n\n```sh\n# Find your libjvm.so location (example for typical JDK installation)\nexport LD_LIBRARY_PATH=$JAVA_HOME/lib/server:$LD_LIBRARY_PATH\n\n# Now you can run Rust tests\ncd native && cargo test\n```\n\nAlternatively, use `make test-rust` which handles the JVM compilation dependency automatically.\n\n### Avoid Using `-pl` to Select Modules\n\nWhen running Maven tests, avoid using `-pl spark` to select only the spark module. This can cause\nMaven to pick up the `common` module from your local Maven repository instead of using the current\ncodebase, leading to inconsistent test results:\n\n```sh\n# Avoid this - may use stale common module from local repo\n./mvnw test -pl spark -Dsuites=\"...\"\n\n# Do this instead - builds and tests with current code\n./mvnw test -Dsuites=\"...\"\n```\n\n### Use `wildcardSuites` for Running Tests\n\nWhen running specific test suites, use `wildcardSuites` instead of `suites` for more flexible\nmatching. The `wildcardSuites` parameter allows partial matching of suite names:\n\n```sh\n# Run all suites containing \"CometCast\"\n./mvnw test -DwildcardSuites=\"CometCast\"\n\n# Run specific suite with filter\n./mvnw test -Dsuites=\"org.apache.comet.CometCastSuite valid\"\n```\n\n## Development Environment\n\nComet is a multi-language project with native code written in Rust and JVM code written in Java and Scala.\nFor Rust code, the CLion IDE is recommended. For JVM code, IntelliJ IDEA is recommended.\n\nBefore opening the project in an IDE, make sure to run `make` first to generate the necessary files for the IDEs. Currently, it's mostly about\ngenerating protobuf message classes for the JVM side. It's only required to run `make` once after cloning the repo.\n\n### IntelliJ IDEA\n\nFirst make sure to install the Scala plugin in IntelliJ IDEA.\nAfter that, you can open the project in IntelliJ IDEA. The IDE should automatically detect the project structure and import as a Maven project.\n\nComet uses generated source files that are too large for IntelliJ's default size limit for code inspections. To avoid IDE errors\n(missing definitions, etc.) caused by IntelliJ skipping these generated files, modify\n[IntelliJ's Platform Properties](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties)\nby going to `Help -> Edit Custom Properties...`. For example, adding `idea.max.intellisense.filesize=16384` increases the file\nsize limit to 16 MB.\n\n### CLion\n\nFirst make sure to install the Rust plugin in CLion or you can use the dedicated Rust IDE: RustRover.\nAfter that you can open the project in CLion. The IDE should automatically detect the project structure and import as a Cargo project.\n\n### SQL file tests (recommended for expressions)\n\nFor testing expressions and operators, prefer using SQL file tests over writing Scala test\ncode. SQL file tests are plain `.sql` files that are automatically discovered and executed --\nno Scala code to write, and no recompilation needed when tests change. This makes it easy to\niterate quickly and to get good coverage of edge cases and argument combinations.\n\nSee the [SQL File Tests](sql-file-tests) guide for the full documentation on how to write\nand run these tests.\n\n### Running Tests in IDEA\n\nLike other Maven projects, you can run tests in IntelliJ IDEA by right-clicking on the test class or test method and selecting \"Run\" or \"Debug\".\nHowever if the tests is related to the native side. Please make sure to run `make core` or `cd native && cargo build` before running the tests in IDEA.\n\n### Running Tests from command line\n\nIt is possible to specify which ScalaTest suites you want to run from the CLI using the `suites`\nargument, for example if you only want to execute the test cases that contains _valid_\nin their name in `org.apache.comet.CometCastSuite` you can use\n\n```sh\n./mvnw test -Dtest=none -Dsuites=\"org.apache.comet.CometCastSuite valid\"\n```\n\nOther options for selecting specific suites are described in the [ScalaTest Maven Plugin documentation](https://www.scalatest.org/user_guide/using_the_scalatest_maven_plugin)\n\n## Plan Stability Testing\n\nComet has a plan stability testing framework that can be used to test the stability of the query plans generated by Comet.\nThe plan stability testing framework is located in the `spark` module.\n\n### Using the Helper Script\n\nThe easiest way to regenerate golden files is to use the provided script:\n\n```sh\n# Regenerate golden files for all Spark versions\n./dev/regenerate-golden-files.sh\n\n# Regenerate only for a specific Spark version\n./dev/regenerate-golden-files.sh --spark-version 3.5\n```\n\nThe script verifies that JDK 17+ is configured (required for Spark 4.0), installs Comet for each\nSpark version, and runs the plan stability tests with `SPARK_GENERATE_GOLDEN_FILES=1`.\n\n### Manual Instructions\n\nAlternatively, you can run the tests manually using the following commands.\n\nNote that the output files get written to `$SPARK_HOME`.\n\nThe tests can be run with:\n\n```sh\nexport SPARK_HOME=`pwd`\n./mvnw -Dsuites=\"org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite\" -Pspark-3.4 -nsu test\n./mvnw -Dsuites=\"org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite\" -Pspark-3.5 -nsu test\n./mvnw -Dsuites=\"org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite\" -Pspark-4.0 -nsu test\n```\n\nand\n\n```sh\nexport SPARK_HOME=`pwd`\n./mvnw -Dsuites=\"org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite\" -Pspark-3.4 -nsu test\n./mvnw -Dsuites=\"org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite\" -Pspark-3.5 -nsu test\n./mvnw -Dsuites=\"org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite\" -Pspark-4.0 -nsu test\n```\n\nIf your pull request changes the query plans generated by Comet, you should regenerate the golden files.\nTo regenerate the golden files, you can run the following commands.\n\n```sh\nexport SPARK_HOME=`pwd`\nSPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -Dsuites=\"org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite\" -Pspark-3.4 -nsu test\nSPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -Dsuites=\"org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite\" -Pspark-3.5 -nsu test\nSPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -Dsuites=\"org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite\" -Pspark-4.0 -nsu test\n```\n\nand\n\n```sh\nexport SPARK_HOME=`pwd`\nSPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -Dsuites=\"org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite\" -Pspark-3.4 -nsu test\nSPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -Dsuites=\"org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite\" -Pspark-3.5 -nsu test\nSPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -Dsuites=\"org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite\" -Pspark-4.0 -nsu test\n```\n\n## Benchmark\n\nThere's a `make` command to run micro benchmarks in the repo. For\ninstance:\n\n```\nmake benchmark-org.apache.spark.sql.benchmark.CometReadBenchmark\n```\n\nTo run TPC-H or TPC-DS micro benchmarks, please follow the instructions\nin the respective source code, e.g., `CometTPCHQueryBenchmark`.\n\n## Debugging\n\nComet is a multi-language project with native code written in Rust and JVM code written in Java and Scala.\nIt is possible to debug both native and JVM code concurrently as described in the [DEBUGGING guide](debugging)\n\n## Submitting a Pull Request\n\nBefore submitting a pull request, follow this checklist to ensure your changes are ready:\n\n### 1. Format Your Code\n\nComet uses `cargo fmt`, [Scalafix](https://github.com/scalacenter/scalafix) and [Spotless](https://github.com/diffplug/spotless/tree/main/plugin-maven) to\nautomatically format the code. Run the following command to format all code:\n\n```sh\nmake format\n```\n\n### 2. Build and Verify\n\nAfter formatting, run a full build to ensure everything compiles correctly and generated\ndocumentation is up to date:\n\n```sh\nmake\n```\n\nThis builds both native and JVM code. Fix any compilation errors before proceeding.\n\n### 3. Run Clippy (Recommended)\n\nIt's strongly recommended to run Clippy locally to catch potential issues before the CI/CD pipeline does. You can run the same Clippy checks used in CI/CD with:\n\n```bash\ncd native\ncargo clippy --color=never --all-targets --workspace -- -D warnings\n```\n\nMake sure to resolve any Clippy warnings before submitting your pull request, as the CI/CD pipeline will fail if warnings are present.\n\n### 4. Run Tests\n\nRun the relevant tests for your changes:\n\n```sh\n# Run all tests\nmake test\n\n# Or run only Rust tests\nmake test-rust\n\n# Or run only JVM tests (native must be built first)\nmake test-jvm\n```\n\n### 5. Register New Test Suites in CI\n\nComet's CI does not automatically discover test suites. Instead, test suites are explicitly listed\nin the GitHub Actions workflow files so they can be grouped by category and run as separate parallel\njobs. This reduces overall CI time.\n\nIf you add a new Scala test suite, you must add it to the `suite` matrix in **both** workflow files:\n\n- `.github/workflows/pr_build_linux.yml`\n- `.github/workflows/pr_build_macos.yml`\n\nEach file contains a `suite` matrix with named groups such as `fuzz`, `shuffle`, `parquet`, `csv`,\n`exec`, `expressions`, and `sql`. Add your new suite's fully qualified class name to the\nappropriate group. For example, if you add a new expression test suite, add it to the\n`expressions` group:\n\n```yaml\n- name: \"expressions\"\n value: |\n org.apache.comet.CometExpressionSuite\n # ... existing suites ...\n org.apache.comet.YourNewExpressionSuite # <-- add here\n```\n\nChoose the group that best matches the area your test covers:\n\n| Group | Covers |\n| ------------- | ---------------------------------------------------------- |\n| `fuzz` | Fuzz testing and data generation |\n| `shuffle` | Shuffle operators and related exchange behavior |\n| `parquet` | Parquet read/write and native reader tests |\n| `csv` | CSV native read tests |\n| `exec` | Execution operators, joins, aggregates, plan rules, TPC-\\* |\n| `expressions` | Expression evaluation, casts, and SQL file tests |\n| `sql` | SQL-level behavior tests |\n\n**Important:** The suite lists in both workflow files must stay in sync. A separate CI check\n(`.github/workflows/pr_missing_suites.yml`) runs `dev/ci/check-suites.py` on every pull request.\nIt scans for all `*Suite.scala` files in the repository and verifies that each one appears in both\nworkflow files. If any suite is missing, this check will fail and block the PR.\n\n### Pre-PR Summary\n\n```sh\nmake format # Format code\nmake # Build everything and update generated docs\nmake test # Run tests (optional but recommended)\n```\n\n## How to format `.md` document\n\nWe are using `prettier` to format `.md` files.\n\nYou can either use `npm i -g prettier` to install it globally or use `npx` to run it as a standalone binary. Using `npx` required a working node environment. Upgrading to the latest prettier is recommended (by adding `--upgrade` to the `npm` command).\n\n```bash\n$ prettier --version\n2.3.0\n```\n\nAfter you've confirmed your prettier version, you can format all the `.md` files:\n\n```bash\nnpx prettier \"**/*.md\" --write\n```\n" }, { "path": "docs/source/contributor-guide/expression-audit-log.md", "content": "\n\n# Expression Audit Log\n\nThis document tracks which Comet expressions have been audited against their Spark\nimplementations for correctness and test coverage.\n\nEach audit compares the Comet implementation against the Spark source code for the listed\nversions, reviews existing test coverage, identifies gaps, and adds missing tests where needed.\n\n## Audited Expressions\n\n| Expression | Spark Versions Checked | Date | Findings |\n| -------------- | ---------------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `array_insert` | 3.4.3, 3.5.8, 4.0.1 | 2026-04-02 | No behavioral differences across Spark versions. Fixed `nullable()` metadata bug (did not account for `pos_expr`). Added SQL tests for multiple types (string, boolean, double, float, long, short, byte), literal arguments, null handling, negative indices, out-of-bounds padding, special float values (NaN, Infinity), multibyte UTF-8, and legacy negative index mode. Known incompatibility: pos=0 error message differs from Spark's `INVALID_INDEX_OF_ZERO`. |\n" }, { "path": "docs/source/contributor-guide/ffi.md", "content": "\n\n# Arrow FFI Usage in Comet\n\n## Overview\n\nComet uses the [Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html) for zero-copy data transfer in two directions:\n\n1. **JVM → Native**: Native code pulls batches from JVM using `CometBatchIterator`\n2. **Native → JVM**: JVM pulls batches from native code using `CometExecIterator`\n\nThe following diagram shows an example of the end-to-end flow for a query stage.\n\n![Diagram of Comet Data Flow](/_static/images/comet-dataflow.svg)\n\nBoth scenarios use the same FFI mechanism but have different ownership semantics and memory management implications.\n\n## Arrow FFI Basics\n\nThe Arrow C Data Interface defines two C structures:\n\n- `ArrowArray`: Contains pointers to data buffers and metadata\n- `ArrowSchema`: Contains type information\n\n### Key Characteristics\n\n- **Zero-copy**: Data buffers can be shared across language boundaries without copying\n- **Ownership transfer**: Clear semantics for who owns and must free the data\n- **Release callbacks**: Custom cleanup functions for proper resource management\n\n## JVM → Native Data Flow (ScanExec)\n\n### Architecture\n\nWhen native code needs data from the JVM, it uses `ScanExec` which calls into `CometBatchIterator`:\n\n```\n┌─────────────────┐\n│ Spark/Scala │\n│ CometExecIter │\n└────────┬────────┘\n │ produces batches\n ▼\n┌─────────────────┐\n│ CometBatchIter │ ◄─── JNI call from native\n│ (JVM side) │\n└────────┬────────┘\n │ Arrow FFI\n │ (transfers ArrowArray/ArrowSchema pointers)\n ▼\n┌─────────────────┐\n│ ScanExec │\n│ (Rust/native) │\n└────────┬────────┘\n │\n ▼\n┌─────────────────┐\n│ DataFusion │\n│ operators │\n└─────────────────┘\n```\n\n### FFI Transfer Process\n\nThe data transfer happens in `ScanExec::get_next()`:\n\n```rust\n// 1. Allocate FFI structures on native side (Rust heap)\nfor _ in 0..num_cols {\n let arrow_array = Rc::new(FFI_ArrowArray::empty());\n let arrow_schema = Rc::new(FFI_ArrowSchema::empty());\n let array_ptr = Rc::into_raw(arrow_array) as i64;\n let schema_ptr = Rc::into_raw(arrow_schema) as i64;\n // Store pointers...\n}\n\n// 2. Call JVM to populate FFI structures\nlet num_rows: i32 = unsafe {\n jni_call!(env, comet_batch_iterator(iter).next(array_obj, schema_obj) -> i32)?\n};\n\n// 3. Import data from FFI structures\nfor i in 0..num_cols {\n let array_data = ArrayData::from_spark((array_ptr, schema_ptr))?;\n let array = make_array(array_data);\n // ... process array\n}\n```\n\n### Memory Layout\n\nWhen a batch is transferred from JVM to native:\n\n```\nJVM Heap: Native Memory:\n┌──────────────────┐ ┌──────────────────┐\n│ ColumnarBatch │ │ FFI_ArrowArray │\n│ ┌──────────────┐ │ │ ┌──────────────┐ │\n│ │ ArrowBuf │─┼──────────────>│ │ buffers[0] │ │\n│ │ (off-heap) │ │ │ │ (pointer) │ │\n│ └──────────────┘ │ │ └──────────────┘ │\n└──────────────────┘ └──────────────────┘\n │ │\n │ │\nOff-heap Memory: │\n┌──────────────────┐ <──────────────────────┘\n│ Actual Data │\n│ (e.g., int32[]) │\n└──────────────────┘\n```\n\n**Key Point**: The actual data buffers can be off-heap, but the `ArrowArray` and `ArrowSchema` wrapper objects are **always allocated on the JVM heap**.\n\n### Wrapper Object Lifecycle\n\nWhen arrays are created in the JVM and passed to native code, the JVM creates the array data off-heap and creates\nwrapper objects `ArrowArray` and `ArrowSchema` on-heap. These wrapper objects can consume significant memory over\ntime.\n\n```\nPer batch overhead on JVM heap:\n- ArrowArray object: ~100 bytes\n- ArrowSchema object: ~100 bytes\n- Per column: ~200 bytes\n- 100 columns × 1000 batches = ~20 MB of wrapper objects\n```\n\nWhen native code pulls batches from the JVM, the JVM wrapper objects are kept alive until the native code drops\nall references to the arrays.\n\nWhen operators such as `SortExec` fetch many batches and buffer them in native code, the number of wrapper objects\nin Java on-heap memory keeps growing until the batches are released in native code at the end of the sort operation.\n\n### Ownership Transfer\n\nThe Arrow C data interface supports ownership transfer by registering callbacks in the C struct that is passed over\nthe JNI boundary for the function to delete the array data. For example, the `ArrowArray` struct has:\n\n```c\n// Release callback\nvoid (*release)(struct ArrowArray*);\n```\n\nComet currently does not always follow best practice around ownership transfer because there are some cases where\nComet JVM code will retain references to arrays after passing them to native code and may mutate the underlying\nbuffers. There is an `arrow_ffi_safe` flag in the protocol buffer definition of `Scan` that indicates whether\nownership is being transferred according to the Arrow C data interface specification.\n\n```protobuf\nmessage Scan {\n repeated spark.spark_expression.DataType fields = 1;\n // The source of the scan (e.g. file scan, broadcast exchange, shuffle, etc). This\n // is purely for informational purposes when viewing native query plans in\n // debug mode.\n string source = 2;\n // Whether native code can assume ownership of batches that it receives\n bool arrow_ffi_safe = 3;\n}\n```\n\n#### When ownership is NOT transferred to native:\n\nIf the data originates from a scan that uses mutable buffers\nthen ownership is not transferred to native and the JVM may re-use the underlying buffers in the future.\n\nIt is critical that the native code performs a deep copy of the arrays if the arrays are to be buffered by\noperators such as `SortExec` or `ShuffleWriterExec`, otherwise data corruption is likely to occur.\n\n#### When ownership IS transferred to native:\n\nWhen ownership is transferred, it is safe to buffer batches in native. However, JVM wrapper objects will not be\nreleased until the native batches are dropped. This can lead to OOM or GC pressure if there is not enough Java\nheap memory configured.\n\n## Native → JVM Data Flow (CometExecIterator)\n\n### Architecture\n\nWhen JVM needs results from native execution:\n\n```\n┌─────────────────┐\n│ DataFusion Plan │\n│ (native) │\n└────────┬────────┘\n │ produces RecordBatch\n ▼\n┌─────────────────┐\n│ CometExecIter │\n│ (Rust/native) │\n└────────┬────────┘\n │ Arrow FFI\n │ (transfers ArrowArray/ArrowSchema pointers)\n ▼\n┌─────────────────┐\n│ CometExecIter │ ◄─── JNI call from Spark\n│ (Scala side) │\n└────────┬────────┘\n │\n ▼\n┌─────────────────┐\n│ Spark Actions │\n│ (collect, etc) │\n└─────────────────┘\n```\n\n### FFI Transfer Process\n\nThe transfer happens in `CometExecIterator::getNextBatch()`:\n\n```scala\n// Scala side\ndef getNextBatch(): ColumnarBatch = {\n val batchHandle = Native.getNextBatch(nativeHandle)\n\n // Import from FFI structures\n val vectors = (0 until schema.length).map { i =>\n val array = Array.empty[Long](1)\n val schemaPtr = Array.empty[Long](1)\n\n // Get FFI pointers from native\n Native.exportVector(batchHandle, i, array, schemaPtr)\n\n // Import into Arrow Java\n Data.importVector(allocator, array(0), schemaPtr(0))\n }\n\n new ColumnarBatch(vectors.toArray, numRows)\n}\n```\n\n```rust\n// Native side (simplified)\n#[no_mangle]\npub extern \"system\" fn Java_..._getNextBatch(\n env: JNIEnv,\n handle: jlong,\n) -> jlong {\n let context = get_exec_context(handle)?;\n let batch = context.stream.next().await?;\n\n // Store batch and return handle\n let batch_handle = Box::into_raw(Box::new(batch)) as i64;\n batch_handle\n}\n\n#[no_mangle]\npub extern \"system\" fn Java_..._exportVector(\n env: JNIEnv,\n batch_handle: jlong,\n col_idx: jint,\n array_ptr: jlongArray,\n schema_ptr: jlongArray,\n) {\n let batch = get_batch(batch_handle)?;\n let array = batch.column(col_idx);\n\n // Export to FFI structures\n let (array_ffi, schema_ffi) = to_ffi(array.to_data())?;\n\n // Write pointers back to JVM\n env.set_long_array_region(array_ptr, 0, &[array_ffi as i64])?;\n env.set_long_array_region(schema_ptr, 0, &[schema_ffi as i64])?;\n}\n```\n\n### Wrapper Object Lifecycle (Native → JVM)\n\n```\nTime Native Memory JVM Heap Off-heap/Native\n────────────────────────────────────────────────────────────────────────\nt0 RecordBatch produced - Data in native\n in DataFusion\n\nt1 FFI_ArrowArray created - Data in native\n FFI_ArrowSchema created\n (native heap)\n\nt2 Pointers exported to JVM ArrowBuf created Data in native\n (wraps native ptr)\n\nt3 FFI structures kept alive Spark processes Data in native\n via batch handle ColumnarBatch ✓ Valid\n\nt4 Batch handle released ArrowBuf freed Data freed\n Release callback runs (triggers native (via release\n release callback) callback)\n```\n\n**Key Difference from JVM → Native**:\n\n- Native code controls lifecycle through batch handle\n- JVM creates `ArrowBuf` wrappers that point to native memory\n- Release callback ensures proper cleanup when JVM is done\n- No GC pressure issue because native allocator manages the data\n\n### Release Callbacks\n\nCritical for proper cleanup:\n\n```rust\n// Native release callback (simplified)\nextern \"C\" fn release_batch(array: *mut FFI_ArrowArray) {\n if !array.is_null() {\n unsafe {\n // Free the data buffers\n for buffer in (*array).buffers {\n drop(Box::from_raw(buffer));\n }\n // Free the array structure itself\n drop(Box::from_raw(array));\n }\n }\n}\n```\n\nWhen JVM is done with the data:\n\n```java\n// ArrowBuf.close() triggers the release callback\narrowBuf.close(); // → calls native release_batch()\n```\n\n## Memory Ownership Rules\n\n### JVM → Native\n\n| Scenario | `arrow_ffi_safe` | Ownership | Action Required |\n| ------------------ | ---------------- | ----------- | -------------------------------------- |\n| Temporary scan | `false` | JVM keeps | **Must deep copy** to avoid corruption |\n| Ownership transfer | `true` | Native owns | Copy only to unpack dictionaries |\n\n### Native → JVM\n\n| Scenario | Ownership | Action Required |\n| --------- | -------------------------------- | ---------------------------------------------------------- |\n| All cases | Native allocates, JVM references | JVM must call `close()` to trigger native release callback |\n\n## Further Reading\n\n- [Arrow C Data Interface Specification](https://arrow.apache.org/docs/format/CDataInterface.html)\n- [Arrow Java FFI Implementation](https://github.com/apache/arrow/tree/main/java/c)\n- [Arrow Rust FFI Implementation](https://docs.rs/arrow/latest/arrow/ffi/)\n" }, { "path": "docs/source/contributor-guide/iceberg-spark-tests.md", "content": "\n\n# Running Iceberg Spark Tests\n\nRunning Apache Iceberg's Spark tests with Comet enabled is a good way to ensure that Comet produces the same\nresults as Spark when reading Iceberg tables. To enable this, we apply diff files to the Apache Iceberg source\ncode so that Comet is loaded when we run the tests.\n\nHere is an overview of the changes that the diffs make to Iceberg:\n\n- Configure Comet as a dependency and set the correct version in `libs.versions.toml` and `build.gradle`\n- Delete upstream Comet reader classes that reference legacy Comet APIs removed in [#3739]. These classes were\n added upstream in [apache/iceberg#15674] and depend on Comet's old Iceberg Java integration. Since Comet now\n uses a native Iceberg scan, these classes fail to compile and must be removed.\n- Configure test base classes (`TestBase`, `ExtensionsTestBase`, `ScanTestBase`, etc.) to load the Comet Spark\n plugin and shuffle manager\n\n[#3739]: https://github.com/apache/datafusion-comet/pull/3739\n[apache/iceberg#15674]: https://github.com/apache/iceberg/pull/15674\n\n## 1. Install Comet\n\nRun `make release` in Comet to install the Comet JAR into the local Maven repository, specifying the Spark version.\n\n```shell\nPROFILES=\"-Pspark-3.5\" make release\n```\n\n## 2. Clone Iceberg and Apply Diff\n\nClone Apache Iceberg locally and apply the diff file from Comet against the matching tag.\n\n```shell\ngit clone git@github.com:apache/iceberg.git apache-iceberg\ncd apache-iceberg\ngit checkout apache-iceberg-1.8.1\ngit apply ../datafusion-comet/dev/diffs/iceberg/1.8.1.diff\n```\n\n## 3. Run Iceberg Spark Tests\n\n```shell\nENABLE_COMET=true ./gradlew -DsparkVersions=3.5 -DscalaVersion=2.13 -DflinkVersions= -DkafkaVersions= \\\n :iceberg-spark:iceberg-spark-3.5_2.13:test \\\n -Pquick=true -x javadoc\n```\n\nThe three Gradle targets tested in CI are:\n\n| Gradle Target | What It Covers |\n| --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `iceberg-spark-:test` | Core read/write paths (Parquet, Avro, ORC, vectorized), scan operations, filtering, bloom filters, runtime filtering, deletion handling, structured streaming, DDL/DML (create/alter/drop, writes, deletes), filter and aggregate pushdown, actions (snapshot expiration, file rewriting, orphan cleanup, table migration), serialization, and data format conversions. |\n| `iceberg-spark-extensions-:test` | SQL extensions: stored procedures (migrate, snapshot, cherrypick, rollback, rewrite-data-files, rewrite-manifests, expire-snapshots, remove-orphan-files, etc.), row-level operations (copy-on-write and merge-on-read update/delete/merge), DDL extensions (branches, tags, alter schema, partition fields), changelog tables/views, metadata tables, and views. |\n| `iceberg-spark-runtime-:integrationTest` | A single smoke test (`SmokeTest.java`) that validates the shaded runtime JAR. The `spark-runtime` module has no main source — it packages Iceberg and all dependencies into a shaded uber-JAR. The smoke test exercises basic create, insert, merge, query, partition field, and sort order operations to confirm the shaded JAR works end-to-end. |\n\n## Updating Diffs\n\nTo update a diff (e.g. after modifying test configuration), apply the existing diff, make changes, then\nregenerate:\n\n```shell\ncd apache-iceberg\ngit reset --hard apache-iceberg-1.8.1 && git clean -fd\ngit apply ../datafusion-comet/dev/diffs/iceberg/1.8.1.diff\n\n# Make changes, then run spotless to fix formatting\n./gradlew spotlessApply\n\n# Stage any new or deleted files, then generate the diff\ngit add -A\ngit diff apache-iceberg-1.8.1 > ../datafusion-comet/dev/diffs/iceberg/1.8.1.diff\n```\n\nRepeat for each Iceberg version (1.8.1, 1.9.1, 1.10.0). The file contents differ between versions, so each\ndiff must be generated against its own tag.\n\n## Running Tests in CI\n\nThe `iceberg_spark_test.yml` workflow applies these diffs and runs the three Gradle targets above against\neach Iceberg version. The test matrix covers Spark 3.4 and 3.5 across Iceberg 1.8.1, 1.9.1, and 1.10.0\nwith Java 11 and 17. The workflow runs on all pull requests and pushes to the main branch.\n" }, { "path": "docs/source/contributor-guide/index.md", "content": "\n\n# Comet Contributor Guide\n\n```{toctree}\n:maxdepth: 2\n:caption: Contributor Guide\n\nGetting Started \nComet Plugin Overview \nArrow FFI \nJVM Shuffle \nNative Shuffle \nParquet Scans \nDevelopment Guide \nDebugging Guide \nBenchmarking Guide \nAdding a New Operator \nAdding a New Expression \nTracing \nProfiling \nSpark SQL Tests \nIceberg Spark Tests \nSQL File Tests \nBug Triage \nRoadmap \nRelease Process \nGithub and Issue Tracker \n```\n" }, { "path": "docs/source/contributor-guide/jvm_shuffle.md", "content": "\n\n# JVM Shuffle\n\nThis document describes Comet's JVM-based columnar shuffle implementation (`CometColumnarShuffle`), which\nwrites shuffle data in Arrow IPC format using JVM code with native encoding. For the fully native\nalternative, see [Native Shuffle](native_shuffle.md).\n\n## Overview\n\nComet provides two shuffle implementations:\n\n- **CometNativeShuffle** (`CometExchange`): Fully native shuffle using Rust. Takes columnar input directly\n from Comet native operators and performs partitioning in native code.\n- **CometColumnarShuffle** (`CometColumnarExchange`): JVM-based shuffle that operates on rows internally,\n buffers `UnsafeRow`s in memory pages, and uses native code (via JNI) to encode them to Arrow IPC format.\n Uses Spark's partitioner for partition assignment. Can accept either row-based or columnar input\n (columnar input is converted to rows via `ColumnarToRowExec`).\n\nThe JVM shuffle is selected via `CometShuffleDependency.shuffleType`.\n\n## When JVM Shuffle is Used\n\nJVM shuffle (`CometColumnarExchange`) is used instead of native shuffle (`CometExchange`) in the following cases:\n\n1. **Shuffle mode is explicitly set to \"jvm\"**: When `spark.comet.exec.shuffle.mode` is set to `jvm`.\n\n2. **Child plan is not a Comet native operator**: When the child plan is a Spark row-based operator\n (not a `CometPlan`), JVM shuffle is the only option since native shuffle requires columnar input\n from Comet operators.\n\n3. **Unsupported partition key types**: For `HashPartitioning` and `RangePartitioning`, native shuffle\n only supports primitive types as partition keys. Complex types (struct, array, map) cannot be used\n as partition keys in native shuffle and will fall back to JVM columnar shuffle. Note that complex types are\n fully supported as data columns in both implementations.\n\n## Input Handling\n\n### Spark Row-Based Input\n\nWhen the child plan is a Spark row-based operator, `CometColumnarExchange` calls `child.execute()` which\nreturns an `RDD[InternalRow]`. The rows flow directly to the JVM shuffle writers.\n\n### Comet Columnar Input\n\nWhen the child plan is a Comet native operator (e.g., `CometHashAggregate`) but JVM shuffle is selected\n(due to shuffle mode setting or unsupported partitioning), `CometColumnarExchange` still calls\n`child.execute()`. Comet operators implement `doExecute()` by wrapping themselves with `ColumnarToRowExec`:\n\n```scala\n// In CometExec base class\noverride def doExecute(): RDD[InternalRow] =\n ColumnarToRowExec(this).doExecute()\n```\n\nThis means the data path becomes:\n\n```\nComet Native (columnar) → ColumnarToRowExec → rows → JVM Shuffle → Arrow IPC → columnar\n```\n\nThis is less efficient than native shuffle which avoids the columnar-to-row conversion:\n\n```\nComet Native (columnar) → Native Shuffle → Arrow IPC → columnar\n```\n\n### Why Use Spark's Partitioner?\n\nJVM shuffle uses row-based input so it can leverage Spark's existing partitioner infrastructure\n(`partitioner.getPartition(key)`). This allows Comet to support all of Spark's partitioning schemes\nwithout reimplementing them in Rust. Native shuffle, by contrast, serializes the partitioning scheme\nto protobuf and implements the partitioning logic in native code.\n\n## Architecture\n\n```\n┌─────────────────────────────────────────────────────────────────────────┐\n│ CometShuffleManager │\n│ - Extends Spark's ShuffleManager │\n│ - Routes to appropriate writer/reader based on ShuffleHandle type │\n└─────────────────────────────────────────────────────────────────────────┘\n │\n ┌────────────────────────┼────────────────────────┐\n ▼ ▼ ▼\n┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐\n│ CometBypassMerge- │ │ CometUnsafe- │ │ CometNative- │\n│ SortShuffleWriter │ │ ShuffleWriter │ │ ShuffleWriter │\n│ (hash-based) │ │ (sort-based) │ │ (fully native) │\n└─────────────────────┘ └─────────────────────┘ └─────────────────────┘\n │ │\n ▼ ▼\n┌─────────────────────┐ ┌─────────────────────┐\n│ CometDiskBlock- │ │ CometShuffleExternal│\n│ Writer │ │ Sorter │\n└─────────────────────┘ └─────────────────────┘\n │ │\n └────────────┬───────────┘\n ▼\n ┌─────────────────────┐\n │ SpillWriter │\n │ (native encoding │\n │ via JNI) │\n └─────────────────────┘\n```\n\n## Key Classes\n\n### Shuffle Manager\n\n| Class | Location | Description |\n| ------------------------ | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ |\n| `CometShuffleManager` | `.../shuffle/CometShuffleManager.scala` | Entry point. Extends Spark's `ShuffleManager`. Selects writer/reader based on handle type. Delegates non-Comet shuffles to `SortShuffleManager`. |\n| `CometShuffleDependency` | `.../shuffle/CometShuffleDependency.scala` | Extends `ShuffleDependency`. Contains `shuffleType` (`CometColumnarShuffle` or `CometNativeShuffle`) and schema info. |\n\n### Shuffle Handles\n\n| Handle | Writer Strategy |\n| ----------------------------------- | --------------------------------------------------------- |\n| `CometBypassMergeSortShuffleHandle` | Hash-based: one file per partition, merged at end |\n| `CometSerializedShuffleHandle` | Sort-based: records sorted by partition ID, single output |\n| `CometNativeShuffleHandle` | Fully native shuffle |\n\nSelection logic in `CometShuffleManager.shouldBypassMergeSort()`:\n\n- Uses bypass if partitions < threshold AND partitions × cores ≤ max threads\n- Otherwise uses sort-based to avoid OOM from many concurrent writers\n\n### Writers\n\n| Class | Location | Description |\n| ----------------------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |\n| `CometBypassMergeSortShuffleWriter` | `.../shuffle/CometBypassMergeSortShuffleWriter.java` | Hash-based writer. Creates one `CometDiskBlockWriter` per partition. Supports async writes. |\n| `CometUnsafeShuffleWriter` | `.../shuffle/CometUnsafeShuffleWriter.java` | Sort-based writer. Uses `CometShuffleExternalSorter` to buffer and sort records, then merges spill files. |\n| `CometDiskBlockWriter` | `.../shuffle/CometDiskBlockWriter.java` | Buffers rows in memory pages for a single partition. Spills to disk via native encoding. Used by bypass writer. |\n| `CometShuffleExternalSorter` | `.../shuffle/sort/CometShuffleExternalSorter.java` | Buffers records across all partitions, sorts by partition ID, spills sorted data. Used by unsafe writer. |\n| `SpillWriter` | `.../shuffle/SpillWriter.java` | Base class for spill logic. Manages memory pages and calls `Native.writeSortedFileNative()` for Arrow IPC encoding. |\n\n### Reader\n\n| Class | Location | Description |\n| ------------------------------ | ------------------------------------------------ | -------------------------------------------------------------------------------------------------- |\n| `CometBlockStoreShuffleReader` | `.../shuffle/CometBlockStoreShuffleReader.scala` | Fetches shuffle blocks via `ShuffleBlockFetcherIterator`. Decodes Arrow IPC to `ColumnarBatch`. |\n| `NativeBatchDecoderIterator` | `.../shuffle/NativeBatchDecoderIterator.scala` | Reads compressed Arrow IPC batches from input stream. Calls `Native.decodeShuffleBlock()` via JNI. |\n\n## Data Flow\n\n### Write Path\n\n1. `ShuffleWriteProcessor` calls `CometShuffleManager.getWriter()`\n2. Writer receives `Iterator[Product2[K, V]]` where V is `UnsafeRow`\n3. Rows are serialized and buffered in off-heap memory pages\n4. When memory threshold or batch size is reached, `SpillWriter.doSpilling()` is called\n5. Native code (`Native.writeSortedFileNative()`) converts rows to Arrow arrays and writes IPC format\n6. For bypass writer: partition files are concatenated into final output\n7. For sort writer: spill files are merged\n\n### Read Path\n\n1. `CometBlockStoreShuffleReader.read()` creates `ShuffleBlockFetcherIterator`\n2. For each block, `NativeBatchDecoderIterator` reads the IPC stream\n3. Native code (`Native.decodeShuffleBlock()`) decompresses and decodes to Arrow arrays\n4. Arrow FFI imports arrays as `ColumnarBatch`\n\n## Memory Management\n\n- `CometShuffleMemoryAllocator`: Custom allocator for off-heap memory pages\n- Memory is allocated in pages; when allocation fails, writers spill to disk\n- `CometDiskBlockWriter` coordinates spilling across all partition writers (largest first)\n- Async spilling is supported via `ShuffleThreadPool`\n\n## Configuration\n\n| Config | Description |\n| ----------------------------------------------- | ----------------------------------- |\n| `spark.comet.columnar.shuffle.async.enabled` | Enable async spill writes |\n| `spark.comet.columnar.shuffle.async.thread.num` | Threads per writer for async |\n| `spark.comet.columnar.shuffle.batch.size` | Rows per Arrow batch |\n| `spark.comet.columnar.shuffle.spill.threshold` | Row count threshold for spill |\n| `spark.comet.exec.shuffle.compression.codec` | Compression codec (zstd, lz4, etc.) |\n" }, { "path": "docs/source/contributor-guide/native_shuffle.md", "content": "\n\n# Native Shuffle\n\nThis document describes Comet's native shuffle implementation (`CometNativeShuffle`), which performs\nshuffle operations entirely in Rust code for maximum performance. For the JVM-based alternative,\nsee [JVM Shuffle](jvm_shuffle.md).\n\n## Overview\n\nNative shuffle takes columnar input directly from Comet native operators and performs partitioning,\nencoding, and writing in native Rust code. This avoids the columnar-to-row-to-columnar conversion\noverhead that JVM shuffle incurs.\n\n```\nComet Native (columnar) → Native Shuffle → Arrow IPC → columnar\n```\n\nCompare this to JVM shuffle's data path:\n\n```\nComet Native (columnar) → ColumnarToRowExec → rows → JVM Shuffle → Arrow IPC → columnar\n```\n\n## When Native Shuffle is Used\n\nNative shuffle (`CometExchange`) is selected when all of the following conditions are met:\n\n1. **Shuffle mode allows native**: `spark.comet.exec.shuffle.mode` is `native` or `auto`.\n\n2. **Child plan is a Comet native operator**: The child must be a `CometPlan` that produces\n columnar output. Row-based Spark operators require JVM shuffle.\n\n3. **Supported partitioning type**: Native shuffle supports:\n - `HashPartitioning`\n - `RangePartitioning`\n - `SinglePartition`\n - `RoundRobinPartitioning`\n\n4. **Supported partition key types**: For `HashPartitioning` and `RangePartitioning`, partition\n keys must be primitive types. Complex types (struct, array, map) as partition keys require\n JVM shuffle. Note that complex types are fully supported as data columns in native shuffle.\n\n## Architecture\n\n```\n┌─────────────────────────────────────────────────────────────────────────────┐\n│ CometShuffleManager │\n│ - Routes to CometNativeShuffleWriter for CometNativeShuffleHandle │\n└─────────────────────────────────────────────────────────────────────────────┘\n │\n ▼\n┌─────────────────────────────────────────────────────────────────────────────┐\n│ CometNativeShuffleWriter │\n│ - Constructs protobuf operator plan │\n│ - Invokes native execution via CometExec.getCometIterator() │\n└─────────────────────────────────────────────────────────────────────────────┘\n │\n ▼ (JNI)\n┌─────────────────────────────────────────────────────────────────────────────┐\n│ ShuffleWriterExec (Rust) │\n│ - DataFusion ExecutionPlan │\n│ - Orchestrates partitioning and writing │\n└─────────────────────────────────────────────────────────────────────────────┘\n │ │\n ▼ ▼\n┌───────────────────────────────────┐ ┌───────────────────────────────────┐\n│ MultiPartitionShuffleRepartitioner │ │ SinglePartitionShufflePartitioner │\n│ (hash/range partitioning) │ │ (single partition case) │\n└───────────────────────────────────┘ └───────────────────────────────────┘\n │\n ▼\n┌───────────────────────────────────┐\n│ ShuffleBlockWriter │\n│ (Arrow IPC + compression) │\n└───────────────────────────────────┘\n │\n ▼\n ┌─────────────────┐\n │ Data + Index │\n │ Files │\n └─────────────────┘\n```\n\n## Key Classes\n\n### Scala Side\n\n| Class | Location | Description |\n| ------------------------------ | ------------------------------------------------ | --------------------------------------------------------------------------------------------- |\n| `CometShuffleExchangeExec` | `.../shuffle/CometShuffleExchangeExec.scala` | Physical plan node. Validates types and partitioning, creates `CometShuffleDependency`. |\n| `CometNativeShuffleWriter` | `.../shuffle/CometNativeShuffleWriter.scala` | Implements `ShuffleWriter`. Builds protobuf plan and invokes native execution. |\n| `CometShuffleDependency` | `.../shuffle/CometShuffleDependency.scala` | Extends `ShuffleDependency`. Holds shuffle type, schema, and range partition bounds. |\n| `CometBlockStoreShuffleReader` | `.../shuffle/CometBlockStoreShuffleReader.scala` | Reads shuffle blocks via `ShuffleBlockFetcherIterator`. Decodes Arrow IPC to `ColumnarBatch`. |\n| `NativeBatchDecoderIterator` | `.../shuffle/NativeBatchDecoderIterator.scala` | Reads compressed Arrow IPC from input stream. Calls native decode via JNI. |\n\n### Rust Side\n\n| File | Location | Description |\n| ----------------------- | ------------------------------------ | ------------------------------------------------------------------------------------ |\n| `shuffle_writer.rs` | `native/core/src/execution/shuffle/` | `ShuffleWriterExec` plan and partitioners. Main shuffle logic. |\n| `codec.rs` | `native/core/src/execution/shuffle/` | `ShuffleBlockWriter` for Arrow IPC encoding with compression. Also handles decoding. |\n| `comet_partitioning.rs` | `native/core/src/execution/shuffle/` | `CometPartitioning` enum defining partition schemes (Hash, Range, Single). |\n\n## Data Flow\n\n### Write Path\n\n1. **Plan construction**: `CometNativeShuffleWriter` builds a protobuf operator plan containing:\n - A scan operator reading from the input iterator\n - A `ShuffleWriter` operator with partitioning config and compression codec\n\n2. **Native execution**: `CometExec.getCometIterator()` executes the plan in Rust.\n\n3. **Partitioning**: `ShuffleWriterExec` receives batches and routes to the appropriate partitioner:\n - `MultiPartitionShuffleRepartitioner`: For hash/range/round-robin partitioning\n - `SinglePartitionShufflePartitioner`: For single partition (simpler path)\n\n4. **Buffering and spilling**: The partitioner buffers rows per partition. When memory pressure\n exceeds the threshold, partitions spill to temporary files.\n\n5. **Encoding**: `ShuffleBlockWriter` encodes each partition's data as compressed Arrow IPC:\n - Writes compression type header\n - Writes field count header\n - Writes compressed IPC stream\n\n6. **Output files**: Two files are produced:\n - **Data file**: Concatenated partition data\n - **Index file**: Array of 8-byte little-endian offsets marking partition boundaries\n\n7. **Commit**: Back in JVM, `CometNativeShuffleWriter` reads the index file to get partition\n lengths and commits via Spark's `IndexShuffleBlockResolver`.\n\n### Read Path\n\n1. `CometBlockStoreShuffleReader` fetches shuffle blocks via `ShuffleBlockFetcherIterator`.\n\n2. For each block, `NativeBatchDecoderIterator`:\n - Reads the 8-byte compressed length header\n - Reads the 8-byte field count header\n - Reads the compressed IPC data\n - Calls `Native.decodeShuffleBlock()` via JNI\n\n3. Native code decompresses and deserializes the Arrow IPC stream.\n\n4. Arrow FFI transfers the `RecordBatch` to JVM as a `ColumnarBatch`.\n\n## Partitioning\n\n### Hash Partitioning\n\nNative shuffle implements Spark-compatible hash partitioning:\n\n- Uses Murmur3 hash function with seed 42 (matching Spark)\n- Computes hash of partition key columns\n- Applies modulo by partition count: `partition_id = hash % num_partitions`\n\n### Range Partitioning\n\nFor range partitioning:\n\n1. Spark's `RangePartitioner` samples data and computes partition boundaries on the driver.\n2. Boundaries are serialized to the native plan.\n3. Native code converts sort key columns to comparable row format.\n4. Binary search (`partition_point`) determines which partition each row belongs to.\n\n### Single Partition\n\nThe simplest case: all rows go to partition 0. Uses `SinglePartitionShufflePartitioner` which\nsimply concatenates batches to reach the configured batch size.\n\n### Round Robin Partitioning\n\nComet implements round robin partitioning using hash-based assignment for determinism:\n\n1. Computes a Murmur3 hash of columns (using seed 42)\n2. Assigns partitions directly using the hash: `partition_id = hash % num_partitions`\n\nThis approach guarantees determinism across retries, which is critical for fault tolerance.\nHowever, unlike true round robin which cycles through partitions row-by-row, hash-based\nassignment only provides even distribution when the data has sufficient variation in the\nhashed columns. Data with low cardinality or identical values may result in skewed partition\nsizes.\n\n## Memory Management\n\nNative shuffle uses DataFusion's memory management with spilling support:\n\n- **Memory pool**: Tracks memory usage across the shuffle operation.\n- **Spill threshold**: When buffered data exceeds the threshold, partitions spill to disk.\n- **Per-partition spilling**: Each partition has its own spill file. Multiple spills for a\n partition are concatenated when writing the final output.\n- **Scratch space**: Reusable buffers for partition ID computation to reduce allocations.\n\nThe `MultiPartitionShuffleRepartitioner` manages:\n\n- `PartitionBuffer`: In-memory buffer for each partition\n- `SpillFile`: Temporary file for spilled data\n- Memory tracking via `MemoryConsumer` trait\n\n## Compression\n\nNative shuffle supports multiple compression codecs configured via\n`spark.comet.exec.shuffle.compression.codec`:\n\n| Codec | Description |\n| -------- | ------------------------------------------------------ |\n| `zstd` | Zstandard compression. Best ratio, configurable level. |\n| `lz4` | LZ4 compression. Fast with good ratio. |\n| `snappy` | Snappy compression. Fastest, lower ratio. |\n| `none` | No compression. |\n\nThe compression codec is applied uniformly to all partitions. Each partition's data is\nindependently compressed, allowing parallel decompression during reads.\n\n## Configuration\n\n| Config | Default | Description |\n| ------------------------------------------------- | ------- | ---------------------------------------- |\n| `spark.comet.exec.shuffle.enabled` | `true` | Enable Comet shuffle |\n| `spark.comet.exec.shuffle.mode` | `auto` | Shuffle mode: `native`, `jvm`, or `auto` |\n| `spark.comet.exec.shuffle.compression.codec` | `zstd` | Compression codec |\n| `spark.comet.exec.shuffle.compression.zstd.level` | `1` | Zstd compression level |\n| `spark.comet.shuffle.write.buffer.size` | `1MB` | Write buffer size |\n| `spark.comet.columnar.shuffle.batch.size` | `8192` | Target rows per batch |\n\n## Comparison with JVM Shuffle\n\n| Aspect | Native Shuffle | JVM Shuffle |\n| ------------------- | -------------------------------------- | --------------------------------- |\n| Input format | Columnar (direct from Comet operators) | Row-based (via ColumnarToRowExec) |\n| Partitioning logic | Rust implementation | Spark's partitioner |\n| Supported schemes | Hash, Range, Single, RoundRobin | Hash, Range, Single, RoundRobin |\n| Partition key types | Primitives only (Hash, Range) | Any type |\n| Performance | Higher (no format conversion) | Lower (columnar→row→columnar) |\n| Writer variants | Single path | Bypass (hash) and sort-based |\n\nSee [JVM Shuffle](jvm_shuffle.md) for details on the JVM-based implementation.\n" }, { "path": "docs/source/contributor-guide/parquet_scans.md", "content": "\n\n# Comet Parquet Scan Implementations\n\nComet currently has two distinct implementations of the Parquet scan operator.\n\n| Scan Implementation | Notes |\n| ----------------------- | ---------------------- |\n| `native_datafusion` | Fully native scan |\n| `native_iceberg_compat` | Hybrid JVM/native scan |\n\nThe configuration property\n`spark.comet.scan.impl` is used to select an implementation. The default setting is `spark.comet.scan.impl=auto`, which\nattempts to use `native_datafusion` first, and falls back to Spark if the scan cannot be converted\n(e.g., due to unsupported features). Most users should not need to change this setting.\nHowever, it is possible to force Comet to use a particular implementation for all scan operations by setting\nthis configuration property to one of the following implementations. For example: `--conf spark.comet.scan.impl=native_datafusion`.\n\nThe following features are not supported by either scan implementation, and Comet will fall back to Spark in these scenarios:\n\n- `ShortType` columns, by default. When reading Parquet files written by systems other than Spark that contain\n columns with the logical type `UINT_8` (unsigned 8-bit integers), Comet may produce different results than Spark.\n Spark maps `UINT_8` to `ShortType`, but Comet's Arrow-based readers respect the unsigned type and read the data as\n unsigned rather than signed. Since Comet cannot distinguish `ShortType` columns that came from `UINT_8` versus\n signed `INT16`, by default Comet falls back to Spark when scanning Parquet files containing `ShortType` columns.\n This behavior can be disabled by setting `spark.comet.scan.unsignedSmallIntSafetyCheck=false`. Note that `ByteType`\n columns are always safe because they can only come from signed `INT8`, where truncation preserves the signed value.\n- Default values that are nested types (e.g., maps, arrays, structs). Literal default values are supported.\n- Spark's Datasource V2 API. When `spark.sql.sources.useV1SourceList` does not include `parquet`, Spark uses the\n V2 API for Parquet scans. The DataFusion-based implementations only support the V1 API.\n- Spark metadata columns (e.g., `_metadata.file_path`)\n- No support for Dynamic Partition Pruning (DPP)\n\nThe following shared limitation may produce incorrect results without falling back to Spark:\n\n- No support for datetime rebasing. When reading Parquet files containing dates or timestamps written before\n Spark 3.0 (which used a hybrid Julian/Gregorian calendar), dates/timestamps will be read as if they were\n written using the Proleptic Gregorian calendar. This may produce incorrect results for dates before\n October 15, 1582.\n\nThe `native_datafusion` scan has some additional limitations, mostly related to Parquet metadata. All of these\ncause Comet to fall back to Spark (including when using `auto` mode). Note that the `native_datafusion` scan\nrequires `spark.comet.exec.enabled=true` because the scan node must be wrapped by `CometExecRule`.\n\n- No support for row indexes\n- No support for reading Parquet field IDs\n- No support for `input_file_name()`, `input_file_block_start()`, or `input_file_block_length()` SQL functions.\n The `native_datafusion` scan does not use Spark's `FileScanRDD`, so these functions cannot populate their values.\n- No support for `ignoreMissingFiles` or `ignoreCorruptFiles` being set to `true`\n- Duplicate field names in case-insensitive mode (e.g., a Parquet file with both `B` and `b` columns)\n are detected at read time and raise a `SparkRuntimeException` with error class `_LEGACY_ERROR_TEMP_2093`,\n matching Spark's behavior.\n\nThe `native_iceberg_compat` scan has the following additional limitation that may produce incorrect results\nwithout falling back to Spark:\n\n- Some Spark configuration values are hard-coded to their defaults rather than respecting user-specified values.\n This may produce incorrect results when non-default values are set. The affected configurations are\n `spark.sql.parquet.binaryAsString`, `spark.sql.parquet.int96AsTimestamp`, `spark.sql.caseSensitive`,\n `spark.sql.parquet.inferTimestampNTZ.enabled`, and `spark.sql.legacy.parquet.nanosAsLong`. See\n [issue #1816](https://github.com/apache/datafusion-comet/issues/1816) for more details.\n\n## S3 Support\n\nThe `native_datafusion` and `native_iceberg_compat` Parquet scan implementations completely offload data loading\nto native code. They use the [`object_store` crate](https://crates.io/crates/object_store) to read data from S3 and\nsupport configuring S3 access using standard [Hadoop S3A configurations](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#General_S3A_Client_configuration) by translating them to\nthe `object_store` crate's format.\n\nThis implementation maintains compatibility with existing Hadoop S3A configurations, so existing code will\ncontinue to work as long as the configurations are supported and can be translated without loss of functionality.\n\n#### Additional S3 Configuration Options\n\nBeyond credential providers, the `native_datafusion` and `native_iceberg_compat` implementations support additional\nS3 configuration options:\n\n| Option | Description |\n| ------------------------------- | -------------------------------------------------------------------------------------------------- |\n| `fs.s3a.endpoint` | The endpoint of the S3 service |\n| `fs.s3a.endpoint.region` | The AWS region for the S3 service. If not specified, the region will be auto-detected. |\n| `fs.s3a.path.style.access` | Whether to use path style access for the S3 service (true/false, defaults to virtual hosted style) |\n| `fs.s3a.requester.pays.enabled` | Whether to enable requester pays for S3 requests (true/false) |\n\nAll configuration options support bucket-specific overrides using the pattern `fs.s3a.bucket.{bucket-name}.{option}`.\n\n#### Examples\n\nThe following examples demonstrate how to configure S3 access with the `native_datafusion` and `native_iceberg_compat`\nParquet scan implementations using different authentication methods.\n\n**Example 1: Simple Credentials**\n\nThis example shows how to access a private S3 bucket using an access key and secret key. The `fs.s3a.aws.credentials.provider` configuration can be omitted since `org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider` is included in Hadoop S3A's default credential provider chain.\n\n```shell\n$SPARK_HOME/bin/spark-shell \\\n...\n--conf spark.comet.scan.impl=native_datafusion \\\n--conf spark.hadoop.fs.s3a.access.key=my-access-key \\\n--conf spark.hadoop.fs.s3a.secret.key=my-secret-key\n...\n```\n\n**Example 2: Assume Role with Web Identity Token**\n\nThis example demonstrates using an assumed role credential to access a private S3 bucket, where the base credential for assuming the role is provided by a web identity token credentials provider.\n\n```shell\n$SPARK_HOME/bin/spark-shell \\\n...\n--conf spark.comet.scan.impl=native_datafusion \\\n--conf spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider \\\n--conf spark.hadoop.fs.s3a.assumed.role.arn=arn:aws:iam::123456789012:role/my-role \\\n--conf spark.hadoop.fs.s3a.assumed.role.session.name=my-session \\\n--conf spark.hadoop.fs.s3a.assumed.role.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider\n...\n```\n\n#### Limitations\n\nThe S3 support of `native_datafusion` and `native_iceberg_compat` has the following limitations:\n\n1. **Partial Hadoop S3A configuration support**: Not all Hadoop S3A configurations are currently supported. Only the configurations listed in the tables above are translated and applied to the underlying `object_store` crate.\n\n2. **Custom credential providers**: Custom implementations of AWS credential providers are not supported. The implementation only supports the standard credential providers listed in the table above. We are planning to add support for custom credential providers through a JNI-based adapter that will allow calling Java credential providers from native code. See [issue #1829](https://github.com/apache/datafusion-comet/issues/1829) for more details.\n" }, { "path": "docs/source/contributor-guide/plugin_overview.md", "content": "\n\n# Comet Plugin Architecture\n\n## Overview\n\nThe Comet plugin enhances Spark SQL by introducing optimized query execution and shuffle mechanisms leveraging\nnative code. It integrates with Spark's plugin framework and extension API to replace or extend Spark's\ndefault behavior.\n\n---\n\n# Plugin Components\n\n## Comet SQL Plugin\n\nThe entry point to Comet is the org.apache.spark.CometPlugin class, which is registered in Spark using the following\nconfiguration:\n\n```\n--conf spark.plugins=org.apache.spark.CometPlugin\n```\n\nThe plugin is loaded on the Spark driver and does not provide executor-side plugins.\n\nThe plugin will update the current `SparkConf` with the extra configuration provided by Comet, such as executor memory\nconfiguration.\n\nThe plugin also registers `CometSparkSessionExtensions` with Spark's extension API.\n\n## CometSparkSessionExtensions\n\nOn initialization, this class registers two physical plan optimization rules with Spark: `CometScanRule`\nand `CometExecRule`. These rules run whenever a query stage is being planned during Adaptive Query Execution, and\nrun once for the entire plan when Adaptive Query Execution is disabled.\n\n### CometScanRule\n\n`CometScanRule` replaces any Parquet scans with Comet operators. There are different paths for Spark v1 and v2 data sources.\n\nWhen reading from Parquet v1 data sources, Comet replaces `FileSourceScanExec` with a `CometScanExec`, and for v2\ndata sources, `BatchScanExec` is replaced with `CometBatchScanExec`. In both cases, Comet replaces Spark's Parquet\nreader with a custom vectorized Parquet reader. This is similar to Spark's vectorized Parquet reader used by the v2\nParquet data source but leverages native code for decoding Parquet row groups directly into Arrow format.\n\nComet only supports a subset of data types and will fall back to Spark's scan if unsupported types\nexist. Comet can still accelerate the rest of the query execution in this case because `CometSparkToColumnarExec` will\nconvert the output from Spark's scan to Arrow arrays. Note that both `spark.comet.exec.enabled=true` and\n`spark.comet.convert.parquet.enabled=true` must be set to enable this conversion.\n\nRefer to the [Supported Spark Data Types](https://datafusion.apache.org/comet/user-guide/datatypes.html) section\nin the contributor guide to see a list of currently supported data types.\n\n### CometExecRule\n\nThis rule traverses bottom-up from the original Spark plan and attempts to replace each operator with a Comet equivalent.\nFor example, a `ProjectExec` will be replaced by `CometProjectExec`.\n\nWhen replacing a node, various checks are performed to determine if Comet can support the operator and its expressions.\nIf an operator, expression, or data type is not supported by Comet then the reason will be stored in a tag on the\nunderlying Spark node and the plan will not be converted.\n\nComet does not support partially replacing subsets of the plan within a query stage because this would involve adding\ntransitions to convert between row-based and columnar data between Spark operators and Comet operators and the overhead\nof this could outweigh the benefits of running parts of the query stage natively in Comet.\n\n## Query Execution\n\nOnce the plan has been transformed, any consecutive native Comet operators are combined into a `CometNativeExec` which contains\na protocol buffer serialized version of the plan (the serialization code can be found in `QueryPlanSerde`).\n\nSpark serializes the physical plan and sends it to the executors when executing tasks. The executors deserialize the\nplan and invoke it.\n\nWhen `CometNativeExec` is invoked, it will pass the serialized protobuf plan into\n`Native.createPlan`, which invokes the native code via JNI, where the plan is then deserialized.\n\nIn the native code there is a `PhysicalPlanner` struct (in `planner.rs`) which converts the deserialized plan into an\nApache DataFusion `ExecutionPlan`. In some cases, Comet provides specialized physical operators and expressions to\noverride the DataFusion versions to ensure compatibility with Apache Spark.\n\nThe leaf nodes in the physical plan are always `ScanExec` and each of these operators will make a JNI call to\n`CometBatchIterator.next()` to fetch the next input batch. The input could be a Comet native Parquet scan,\na Spark exchange, or another native plan.\n\n`CometNativeExec` creates a `CometExecIterator` and applies this iterator to the input RDD\npartitions. Each call to `CometExecIterator.next()` will invoke `Native.executePlan`. Once the plan finishes\nexecuting, the resulting Arrow batches are imported into the JVM using Arrow FFI.\n\n## Shuffle\n\nComet integrates with Spark's shuffle mechanism, optimizing both shuffle writes and reads. Comet's shuffle manager\nmust be registered with Spark using the following configuration:\n\n```\n--conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager\n```\n\n### Shuffle Writes\n\nFor shuffle writes, a `ShuffleMapTask` runs in the executors. This task contains a `ShuffleDependency` that is\nbroadcast to all of the executors. It then passes the input RDD to `ShuffleWriteProcessor.write()` which\nrequests a `ShuffleWriter` from the shuffle manager, and this is where it gets a Comet shuffle writer.\n\n`ShuffleWriteProcessor` then invokes the dependency RDD and fetches rows/batches and passes them to Comet's\nshuffle writer, which writes batches to disk in Arrow IPC format.\n\nAs a result, we cannot avoid having one native plan to produce the shuffle input and another native plan for\nwriting the batches to the shuffle file.\n\n### Shuffle Reads\n\nFor shuffle reads a `ShuffledRDD` requests a `ShuffleReader` from the shuffle manager. Comet provides a\n`CometBlockStoreShuffleReader` which is implemented in JVM and fetches blocks from Spark and then creates an\n`ArrowReaderIterator` to process the blocks using Arrow's `StreamReader` for decoding IPC batches.\n" }, { "path": "docs/source/contributor-guide/profiling.md", "content": "\n\n# Profiling\n\nThis guide covers profiling tools and techniques for Comet development. Because Comet\nspans JVM (Spark) and native (Rust) code, choosing the right profiler depends on what\nyou are investigating.\n\n## Choosing a Profiling Tool\n\n| Tool | JVM Frames | Native (Rust) Frames | Install Required | Best For |\n| ------------------------------------------------------------------------------ | ---------- | -------------------- | ---------------- | ------------------------------------------------------------------------------ |\n| [async-profiler](https://github.com/async-profiler/async-profiler) | Yes | Yes | Yes | End-to-end Comet profiling with unified JVM + native flame graphs |\n| [Java Flight Recorder (JFR)](https://docs.oracle.com/en/java/javase/17/jfapi/) | Yes | No | No (JDK 11+) | GC pressure, allocations, thread contention, I/O — any JVM-level investigation |\n| [cargo-flamegraph](https://github.com/flamegraph-rs/flamegraph) | No | Yes | Yes | Isolated Rust micro-benchmarks without a JVM |\n\n**Recommendation:** Use **async-profiler** when profiling Spark queries with Comet enabled —\nit is the only tool that shows both JVM and native frames in a single flame graph.\nUse **JFR** when you need rich JVM event data (GC, locks, I/O) without installing anything.\nUse **cargo-flamegraph** when working on native code in isolation via `cargo bench`.\n\n## Profiling with async-profiler\n\n[async-profiler](https://github.com/async-profiler/async-profiler) captures JVM and\nnative code in the same flame graph by using Linux `perf_events` or macOS `dtrace`.\nThis makes it ideal for profiling Comet, where hot paths cross the JNI boundary between\nSpark and Rust.\n\n### Installation\n\nDownload a release from the [async-profiler releases page](https://github.com/async-profiler/async-profiler/releases):\n\n```shell\n# Linux x64\nwget https://github.com/async-profiler/async-profiler/releases/download/v3.0/async-profiler-3.0-linux-x64.tar.gz\nmkdir -p $HOME/opt/async-profiler\ntar xzf async-profiler-3.0-linux-x64.tar.gz -C $HOME/opt/async-profiler --strip-components=1\nexport ASYNC_PROFILER_HOME=$HOME/opt/async-profiler\n```\n\nOn macOS, download the appropriate `macos` archive instead.\n\n### Attaching to a running Spark application\n\nUse the `asprof` launcher to attach to a running JVM by PID:\n\n```shell\n# Start CPU profiling for 30 seconds, output an HTML flame graph\n$ASYNC_PROFILER_HOME/bin/asprof -d 30 -f flamegraph.html \n\n# Wall-clock profiling\n$ASYNC_PROFILER_HOME/bin/asprof -e wall -d 30 -f flamegraph.html \n\n# Start profiling (no duration limit), then stop later\n$ASYNC_PROFILER_HOME/bin/asprof start -e cpu \n# ... run your query ...\n$ASYNC_PROFILER_HOME/bin/asprof stop -f flamegraph.html \n```\n\nFind the Spark driver/executor PID with `jps` or `pgrep -f SparkSubmit`.\n\n### Passing profiler flags via spark-submit\n\nYou can also attach async-profiler as a Java agent at JVM startup:\n\n```shell\nspark-submit \\\n --conf \"spark.driver.extraJavaOptions=-agentpath:$ASYNC_PROFILER_HOME/lib/libasyncProfiler.so=start,event=cpu,file=driver.html,tree\" \\\n --conf \"spark.executor.extraJavaOptions=-agentpath:$ASYNC_PROFILER_HOME/lib/libasyncProfiler.so=start,event=cpu,file=executor.html,tree\" \\\n ...\n```\n\nNote: If the executor is distributed then `executor.html` will be written on the remote node.\n\n### Choosing an event type\n\n| Event | When to use |\n| ------- | --------------------------------------------------------------------------------------------------------- |\n| `cpu` | Default. Shows where CPU cycles are spent. Use for compute-bound queries. |\n| `wall` | Wall-clock time including blocked/waiting threads. Use to find JNI boundary overhead and I/O stalls. |\n| `alloc` | Heap allocation profiling. Use to find JVM allocation hotspots around Arrow FFI and columnar conversions. |\n| `lock` | Lock contention. Use when threads appear to spend time waiting on synchronized blocks or locks. |\n\n### Output formats\n\n| Format | Flag | Description |\n| ---------------- | ------------------ | -------------------------------------------------- |\n| HTML flame graph | `-f out.html` | Interactive flame graph (default and most useful). |\n| JFR | `-f out.jfr` | Viewable in JDK Mission Control or IntelliJ. |\n| Collapsed stacks | `-f out.collapsed` | For use with Brendan Gregg's FlameGraph scripts. |\n| Text summary | `-o text` | Flat list of hot methods. |\n\n### Platform notes\n\n**Linux:** Set `perf_event_paranoid` to allow profiling:\n\n```shell\nsudo sysctl kernel.perf_event_paranoid=1 # or 0 / -1 for full access\nsudo sysctl kernel.kptr_restrict=0 # optional: enable kernel symbols\n```\n\n**macOS:** async-profiler uses `dtrace` on macOS, which requires running as root or\nwith SIP (System Integrity Protection) adjustments. Native Rust frames may not be fully\nresolved on macOS; Linux is recommended for the most complete flame graphs.\n\n### Integrated benchmark profiling\n\nThe TPC benchmark scripts in `benchmarks/tpc/` have built-in async-profiler support via\nthe `--async-profiler` flag. See [benchmarks/tpc/README.md](https://github.com/apache/datafusion-comet/blob/main/benchmarks/tpc/README.md)\nfor details.\n\n## Profiling with Java Flight Recorder\n\n[Java Flight Recorder (JFR)](https://docs.oracle.com/en/java/javase/17/jfapi/) is built\ninto JDK 11+ and collects detailed JVM runtime data with very low overhead. It does not\nsee native Rust frames, but is excellent for diagnosing GC pressure, thread contention,\nI/O latency, and JVM-level allocation patterns.\n\n### Adding JFR flags to spark-submit\n\n```shell\nspark-submit \\\n --conf \"spark.driver.extraJavaOptions=-XX:StartFlightRecording=duration=120s,filename=driver.jfr\" \\\n --conf \"spark.executor.extraJavaOptions=-XX:StartFlightRecording=duration=120s,filename=executor.jfr\" \\\n ...\n```\n\nFor continuous recording without a fixed duration:\n\n```shell\n--conf \"spark.driver.extraJavaOptions=-XX:StartFlightRecording=disk=true,maxsize=500m,filename=driver.jfr\"\n```\n\nYou can also start and stop recording dynamically using `jcmd`:\n\n```shell\njcmd JFR.start name=profile\n# ... run your query ...\njcmd JFR.stop name=profile filename=recording.jfr\n```\n\n### Viewing recordings\n\n- **[JDK Mission Control (JMC)](https://jdk.java.net/jmc/)** — the most comprehensive viewer.\n Shows flame graphs, GC timeline, thread activity, I/O, and allocation hot spots.\n- **IntelliJ IDEA** — open `.jfr` files directly in the built-in profiler\n (Run → Open Profiler Snapshot).\n- **`jfr` CLI** — quick summaries from the command line: `jfr summary driver.jfr`\n\n### Useful JFR events for Comet debugging\n\n| Event | What it shows |\n| ------------------------------------------------------------------- | --------------------------------------------------------------------------- |\n| `jdk.GCPhasePause` | GC pause durations — helps identify memory pressure from Arrow allocations. |\n| `jdk.ObjectAllocationInNewTLAB` / `jdk.ObjectAllocationOutsideTLAB` | Allocation hot spots. |\n| `jdk.JavaMonitorWait` / `jdk.ThreadPark` | Thread contention and lock waits. |\n| `jdk.FileRead` / `jdk.FileWrite` / `jdk.SocketRead` | I/O latency. |\n| `jdk.ExecutionSample` | CPU sampling (method profiling, similar to a flame graph). |\n\n### Integrated benchmark profiling\n\nThe TPC benchmark scripts support `--jfr` for automatic JFR recording during benchmark\nruns. See [benchmarks/tpc/README.md](https://github.com/apache/datafusion-comet/blob/main/benchmarks/tpc/README.md) for details.\n\n## Profiling Native Code with cargo-flamegraph\n\nFor profiling Rust code in isolation — without a JVM — use `cargo bench` with\n[cargo-flamegraph](https://github.com/flamegraph-rs/flamegraph).\n\n### Running micro benchmarks with cargo bench\n\nWhen implementing a new operator or expression, it is good practice to add a new microbenchmark under `core/benches`.\n\nIt is often easiest to copy an existing benchmark and modify it for the new operator or expression. It is also\nnecessary to add a new section to the `Cargo.toml` file, such as:\n\n```toml\n[[bench]]\nname = \"shuffle_writer\"\nharness = false\n```\n\nThese benchmarks are useful for comparing performance between releases or between feature branches and the\nmain branch to help prevent regressions in performance when adding new features or fixing bugs.\n\nIndividual benchmarks can be run by name with the following command.\n\n```shell\ncargo bench shuffle_writer\n```\n\nHere is some sample output from running this command.\n\n```\n Running benches/shuffle_writer.rs (target/release/deps/shuffle_writer-e37b59e37879cce7)\nGnuplot not found, using plotters backend\nshuffle_writer/shuffle_writer\n time: [2.0880 ms 2.0989 ms 2.1118 ms]\nFound 9 outliers among 100 measurements (9.00%)\n 3 (3.00%) high mild\n 6 (6.00%) high severe\n```\n\n### Profiling with cargo-flamegraph\n\nInstall cargo-flamegraph:\n\n```shell\ncargo install flamegraph\n```\n\nFollow the instructions in [cargo-flamegraph](https://github.com/flamegraph-rs/flamegraph) for your platform for\nrunning flamegraph.\n\nHere is a sample command for running `cargo-flamegraph` on MacOS.\n\n```shell\ncargo flamegraph --root --bench shuffle_writer\n```\n\nThis will produce output similar to the following.\n\n```\ndtrace: system integrity protection is on, some features will not be available\n\ndtrace: description 'profile-997 ' matched 1 probe\nGnuplot not found, using plotters backend\nTesting shuffle_writer/shuffle_writer\nSuccess\n\ndtrace: pid 66402 has exited\nwriting flamegraph to \"flamegraph.svg\"\n```\n\nThe generated flamegraph can now be opened in a browser that supports svg format.\n\nHere is the flamegraph for this example:\n\n![flamegraph](../_static/images/flamegraph.png)\n\n## Tips for Profiling Comet\n\n### Use wall-clock profiling to spot JNI boundary overhead\n\nWhen profiling Comet with async-profiler, `wall` mode is often more revealing than `cpu`\nbecause it captures time spent crossing the JNI boundary, waiting for native results,\nand blocked on I/O — none of which show up in CPU-only profiles.\n\n```shell\n$ASYNC_PROFILER_HOME/bin/asprof -e wall -d 60 -f wall-profile.html \n```\n\n### Use alloc profiling around Arrow FFI\n\nJVM allocation profiling can identify hotspots in the Arrow FFI path where temporary\nobjects are created during data transfer between JVM and native code:\n\n```shell\n$ASYNC_PROFILER_HOME/bin/asprof -e alloc -d 60 -f alloc-profile.html \n```\n\nLook for allocations in `CometExecIterator`, `CometBatchIterator`, and Arrow vector\nclasses.\n\n### Isolate Rust-only performance issues\n\nIf a flame graph shows the hot path is entirely within native code, switch to\n`cargo-flamegraph` to get better symbol resolution and avoid JVM noise:\n\n```shell\ncd native\ncargo flamegraph --root --bench \n```\n\n### Correlating JVM and native frames\n\nIn async-profiler flame graphs, native Rust frames appear below JNI entry points like\n`Java_org_apache_comet_Native_*`. Look for these transition points to understand how\ntime is split between Spark's JVM code and Comet's native execution.\n" }, { "path": "docs/source/contributor-guide/release_process.md", "content": "\n\n# Apache DataFusion Comet: Release Process\n\nThis documentation explains the release process for Apache DataFusion Comet. Some preparation tasks can be\nperformed by any contributor, while certain release tasks can only be performed by a DataFusion Project Management\nCommittee (PMC) member.\n\n## Checklist\n\nThe following is a quick-reference checklist for the full release process. See the detailed sections below for\ninstructions on each step.\n\n- [ ] Release preparation: review expression support status and user guide\n- [ ] Create release branch\n- [ ] Generate release documentation\n- [ ] Update Maven version in release branch\n- [ ] Update version in main for next development cycle\n- [ ] Generate the change log and create PR against main\n- [ ] Cherry-pick the change log commit into the release branch\n- [ ] Build the jars\n- [ ] Tag the release candidate\n- [ ] Update documentation for the new release\n- [ ] Publish Maven artifacts to staging\n- [ ] Create the release candidate tarball\n- [ ] Start the email voting thread\n- [ ] Once the vote passes:\n - [ ] Publish source tarball\n - [ ] Create GitHub release\n - [ ] Promote Maven artifacts to production\n - [ ] Push the release tag\n - [ ] Close the vote and announce the release\n- [ ] Post release:\n - [ ] Register the release with Apache Reporter\n - [ ] Delete old RCs and releases from SVN\n - [ ] Write a blog post\n\n## Release Preparation\n\nBefore starting the release process, review the user guide to ensure it accurately reflects the current state of the\nproject:\n\n- Review the supported expressions and operators lists in the user guide. Verify that any expressions added since\n the last release are included and that their support status is accurate.\n- Spot-check the support status of individual expressions by running tests or queries to confirm they work as\n documented.\n- Look for any expressions that may have regressed or changed behavior since the last release and update the\n documentation accordingly.\n\nIt is also recommended to run benchmarks (such as TPC-H and TPC-DS) comparing performance against the previous\nrelease to check for regressions. See the\n[Comet Benchmarking Guide](benchmarking.md) for instructions.\n\nThese are tasks where agentic coding tools can be particularly helpful — for example, scanning the codebase for\nnewly registered expressions and cross-referencing them against the documented list, or generating test queries to\nverify expression support status.\n\nAny issues found should be addressed before creating the release branch.\n\n## Creating the Release Candidate\n\nThis part of the process can be performed by any committer.\n\nHere are the steps, using the 0.13.0 release as an example.\n\n### Create Release Branch\n\nThis document assumes that GitHub remotes are set up as follows:\n\n```shell\n$ git remote -v\napache\tgit@github.com:apache/datafusion-comet.git (fetch)\napache\tgit@github.com:apache/datafusion-comet.git (push)\norigin\tgit@github.com:yourgithubid/datafusion-comet.git (fetch)\norigin\tgit@github.com:yourgithubid/datafusion-comet.git (push)\n```\n\nCreate a release branch from the latest commit in main and push to the `apache` repo:\n\n```shell\ngit fetch apache\ngit checkout main\ngit reset --hard apache/main\ngit checkout -b branch-0.13\ngit push apache branch-0.13\n```\n\n### Generate Release Documentation\n\nGenerate the documentation content for this release. The docs on `main` contain only template markers,\nso we need to generate the actual content (config tables, compatibility matrices) for the release branch:\n\n```shell\n./dev/generate-release-docs.sh\ngit add docs/source/user-guide/latest/\ngit commit -m \"Generate docs for 0.13.0 release\"\ngit push apache branch-0.13\n```\n\nThis freezes the documentation to reflect the configs and expressions available in this release.\n\n### Update Maven Version\n\nUpdate the `pom.xml` files in the release branch to update the Maven version from `0.13.0-SNAPSHOT` to `0.13.0`.\n\nThere is no need to update the Rust crate versions because they will already be `0.13.0`.\n\n### Update Version in main\n\nCreate a PR against the main branch to prepare for developing the next release:\n\n- Update the Rust crate version to `0.14.0`.\n- Update the Maven version to `0.14.0-SNAPSHOT` (both in the `pom.xml` files and also in the diff files\n under `dev/diffs`).\n\n### Generate the Change Log\n\nGenerate a change log to cover changes between the previous release and the release branch HEAD by running\nthe provided `dev/release/generate-changelog.py`.\n\nIt is recommended that you set up a virtual Python environment and then install the dependencies:\n\n```shell\ncd dev/release\npython3 -m venv venv\nsource venv/bin/activate\npip3 install -r requirements.txt\n```\n\nTo generate the changelog, set the `GITHUB_TOKEN` environment variable to a valid token and then run the script\nproviding two commit ids or tags followed by the version number of the release being created. The following\nexample generates a change log of all changes between the previous version and the current release branch HEAD revision.\n\n```shell\nexport GITHUB_TOKEN=\npython3 generate-changelog.py 0.12.0 HEAD 0.13.0 > ../changelog/0.13.0.md\n```\n\nCreate a PR against the _main_ branch to add this change log and once this is approved and merged, cherry-pick the\ncommit into the release branch.\n\n### Build the jars\n\n#### Setup to do the build\n\nThe build process requires Docker. Download the latest Docker Desktop from https://www.docker.com/products/docker-desktop/.\nIf you have multiple docker contexts running switch to the context of the Docker Desktop. For example -\n\n```shell\n$ docker context ls\nNAME DESCRIPTION DOCKER ENDPOINT ERROR\ndefault Current DOCKER_HOST based configuration unix:///var/run/docker.sock\ndesktop-linux Docker Desktop unix:///Users/parth/.docker/run/docker.sock\nmy_custom_context * tcp://192.168.64.2:2376\n\n$ docker context use desktop-linux\n```\n\n#### Run the build script\n\nThe `build-release-comet.sh` script will create a docker image for each architecture and use the image\nto build the platform specific binaries. These builder images are created every time this script is run.\nThe script optionally allows overriding of the repository and branch to build the binaries from (Note that\nthe local git repo is not used in the building of the binaries, but it is used to build the final uber jar).\n\n```shell\nUsage: build-release-comet.sh [options]\n\nThis script builds comet native binaries inside a docker image. The image is named\n\"comet-rm\" and will be generated by this script\n\nOptions are:\n\n-r [repo] : git repo (default: https://github.com/apache/datafusion-comet.git)\n-b [branch] : git branch (default: release)\n-t [tag] : tag for the spark-rm docker image to use for building (default: \"latest\").\n```\n\nExample:\n\n```shell\ncd dev/release && ./build-release-comet.sh && cd ../..\n```\n\n#### Build output\n\nThe build output is installed to a temporary local maven repository. The build script will print the name of the\nrepository location at the end. This location will be required at the time of deploying the artifacts to a staging\nrepository\n\n### Tag the Release Candidate\n\nEnsure that the Maven version update and changelog cherry-pick have been pushed to the release branch before tagging.\n\nTag the release branch with `0.13.0-rc1` and push to the `apache` repo\n\n```shell\ngit fetch apache\ngit checkout branch-0.13\ngit reset --hard apache/branch-0.13\ngit tag 0.13.0-rc1\ngit push apache 0.13.0-rc1\n```\n\nNote that pushing a release candidate tag will trigger a GitHub workflow that will build a Docker image and publish\nit to GitHub Container Registry at https://github.com/apache/datafusion-comet/pkgs/container/datafusion-comet\n\n### Publishing Documentation\n\nIn `docs` directory:\n\n- Update `docs/source/index.rst` and add a new navigation menu link for the new release in the section `_toc.user-guide-links-versioned`\n- Add a new line to `build.sh` to delete the locally cloned `comet-*` branch for the new release e.g. `comet-0.13`\n- Update the main method in `generate-versions.py`:\n\n```python\n latest_released_version = \"0.13.0\"\n previous_versions = [\"0.11.0\", \"0.12.0\"]\n```\n\nTest the documentation build locally, following the instructions in `docs/README.md`.\n\nOnce verified, create a PR against the main branch with these documentation changes. After merging, the docs will be\ndeployed to https://datafusion.apache.org/comet/ by the documentation publishing workflow.\n\nNote that the download links in the installation guide will not work until the release is finalized, but having the\ndocumentation available could be useful for anyone testing out the release candidate during the voting period.\n\n## Publishing the Release Candidate\n\nThis part of the process can mostly only be performed by a PMC member.\n\n### Publish the maven artifacts\n\n#### Setup maven\n\n##### One time project setup\n\nSetting up your project in the ASF Nexus Repository from here: https://infra.apache.org/publishing-maven-artifacts.html\n\n##### Release Manager Setup\n\nSet up your development environment from here: https://infra.apache.org/publishing-maven-artifacts.html\n\n##### Build and publish a release candidate to nexus.\n\nThe script `publish-to-maven.sh` will publish the artifacts created by the `build-release-comet.sh` script.\nThe artifacts will be signed using the gpg key of the release manager and uploaded to the maven staging repository.\n\nNote that installed GPG keys can be listed with `gpg --list-keys`. The gpg key is a 40 character hex string.\n\nNote: This script needs `xmllint` to be installed. On macOS xmllint is available by default.\n\nOn Ubuntu `apt-get install -y libxml2-utils`\n\nOn RedHat `yum install -y xmlstarlet`\n\n```shell\n./dev/release/publish-to-maven.sh -h\nusage: publish-to-maven.sh options\n\nPublish signed artifacts to Maven.\n\nOptions\n-u ASF_USERNAME - Username of ASF committer account\n-r LOCAL_REPO - path to temporary local maven repo (created and written to by 'build-release-comet.sh')\n\nThe following will be prompted for -\nASF_PASSWORD - Password of ASF committer account\nGPG_KEY - GPG key used to sign release artifacts\nGPG_PASSPHRASE - Passphrase for GPG key\n```\n\nexample\n\n```shell\n./dev/release/publish-to-maven.sh -u release_manager_asf_id -r /tmp/comet-staging-repo-VsYOX\nASF Password :\nGPG Key (Optional):\nGPG Passphrase :\nCreating Nexus staging repository\n...\n```\n\nIn the Nexus repository UI (https://repository.apache.org/) locate and verify the artifacts in\nstaging (https://central.sonatype.org/publish/release/#locate-and-examine-your-staging-repository).\n\nThe script closes the staging repository but does not release it. Releasing to Maven Central is a manual step\nperformed only after the vote passes (see [Publishing Maven Artifacts](#publishing-maven-artifacts) below).\n\nNote that the Maven artifacts are always published under the final release version (e.g. `0.13.0`), not the RC\nversion — the `-rc1` / `-rc2` suffix only appears in the git tag and the source tarball in SVN. Because the script\ncreates a new staging repository on each run, re-staging the same version for a subsequent RC is supported as long\nas no staging repository for that version has been released to Maven Central.\n\n### Create the Release Candidate Tarball\n\nThe `create-tarball.sh` script creates a signed source tarball and uploads it to the dev subversion repository.\n\n#### Prerequisites\n\nBefore running this script, ensure you have:\n\n1. A GPG key set up for signing, with your public key uploaded to https://pgp.mit.edu/\n2. Apache SVN credentials (you must be logged into the Apache SVN server)\n3. The `requests` Python package installed (`pip3 install requests`)\n\n#### Run the script\n\nRun the create-tarball script on the release candidate tag (`0.13.0-rc1`):\n\n```shell\n./dev/release/create-tarball.sh 0.13.0 1\n```\n\nThis will generate an email template for starting the vote.\n\n### Start an Email Voting Thread\n\nSend the email that is generated in the previous step to `dev@datafusion.apache.org`.\n\nThe verification procedure for voters is documented in\n[Verifying Release Candidates](https://github.com/apache/datafusion-comet/blob/main/dev/release/verifying-release-candidates.md).\nVoters can also use the `dev/release/verify-release-candidate.sh` script to assist with verification:\n\n```shell\n./dev/release/verify-release-candidate.sh 0.13.0 1\n```\n\n### If the Vote Fails\n\nIf the vote does not pass, address the issues raised, increment the release candidate number, and repeat from\nthe [Tag the Release Candidate](#tag-the-release-candidate) step. For example, the next attempt would be tagged\n`0.13.0-rc2`.\n\nBefore staging the next RC, drop the previous RC's staging repository in the\n[Nexus UI](https://repository.apache.org/#stagingRepositories) by selecting it and clicking \"Drop\". This avoids\nleaving multiple closed staging repositories for the same version and prevents accidentally releasing the wrong\none when the vote eventually passes. The Maven version (e.g. `0.13.0`) is shared across all RCs, so each run of\n`publish-to-maven.sh` creates a new staging repository for the same GAV — only one of them should ever be\nreleased to Maven Central.\n\n## Publishing Binary Releases\n\nOnce the vote passes, we can publish the source and binary releases.\n\n### Publishing Source Tarball\n\nRun the release-tarball script to move the tarball to the release subversion repository.\n\n```shell\n./dev/release/release-tarball.sh 0.13.0 1\n```\n\n### Create a release in the GitHub repository\n\nGo to https://github.com/apache/datafusion-comet/releases and create a release for the release tag, and paste the\nchangelog in the description.\n\n### Publishing Maven Artifacts\n\nPromote the Maven artifacts from staging to production by visiting https://repository.apache.org/#stagingRepositories\nand selecting the staging repository and then clicking the \"release\" button.\n\n### Push a release tag to the repo\n\nPush a release tag (`0.13.0`) to the `apache` repository.\n\n```shell\ngit fetch apache\ngit checkout 0.13.0-rc1\ngit tag 0.13.0\ngit push apache 0.13.0\n```\n\nNote that pushing a release tag will trigger a GitHub workflow that will build a Docker image and publish\nit to GitHub Container Registry at https://github.com/apache/datafusion-comet/pkgs/container/datafusion-comet\n\nReply to the vote thread to close the vote and announce the release. The announcement email should include:\n\n- The release version\n- A link to the release notes / changelog\n- A link to the download page or Maven coordinates\n- Thanks to everyone who contributed and voted\n\n## Post Release\n\n### Register the release\n\nRegister the release with the [Apache Reporter Service](https://reporter.apache.org/addrelease.html?datafusion) using\na version such as `COMET-0.13.0`.\n\n### Delete old RCs and Releases\n\nSee the ASF documentation on [when to archive](https://www.apache.org/legal/release-policy.html#when-to-archive)\nfor more information.\n\n#### Deleting old release candidates from `dev` svn\n\nRelease candidates should be deleted once the release is published.\n\nGet a list of DataFusion Comet release candidates:\n\n```shell\nsvn ls https://dist.apache.org/repos/dist/dev/datafusion | grep comet\n```\n\nDelete a release candidate:\n\n```shell\nsvn delete -m \"delete old DataFusion Comet RC\" https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-comet-0.13.0-rc1/\n```\n\n#### Deleting old releases from `release` svn\n\nOnly the latest release should be available. Delete old releases after publishing the new release.\n\nGet a list of DataFusion releases:\n\n```shell\nsvn ls https://dist.apache.org/repos/dist/release/datafusion | grep comet\n```\n\nDelete a release:\n\n```shell\nsvn delete -m \"delete old DataFusion Comet release\" https://dist.apache.org/repos/dist/release/datafusion/datafusion-comet-0.12.0\n```\n\n### Write a blog post\n\nWriting a blog post about the release is a great way to generate more interest in the project. We typically create a\nGoogle document where the community can collaborate on a blog post. Once the content is agreed then a PR can be\ncreated against the [datafusion-site](https://github.com/apache/datafusion-site) repository to add the blog post. Any\ncontributor can drive this process.\n" }, { "path": "docs/source/contributor-guide/roadmap.md", "content": "\n\n# Comet Roadmap\n\nComet is an open-source project and contributors are welcome to work on any issues at any time, but we find it\nhelpful to have a roadmap for some of the major items that require coordination between contributors.\n\n## Major Initiatives\n\n### Iceberg Integration\n\nReads of Iceberg tables with Parquet data files are fully native and enabled by default, powered by a scan operator\nbacked by Iceberg-rust ([#2528]). We anticipate major improvements in the next few releases, including bringing Iceberg table format V3 features (_e.g._,\nencryption) to the reader.\n\n[#2528]: https://github.com/apache/datafusion-comet/pull/2528\n\n### Spark 4.0 Support\n\nComet has experimental support for Spark 4.0, but there is more work to do ([#1637]), such as enabling\nmore Spark SQL tests and fully implementing ANSI support ([#313]) for all supported expressions.\n\n[#313]: https://github.com/apache/datafusion-comet/issues/313\n[#1637]: https://github.com/apache/datafusion-comet/issues/1637\n\n### Dynamic Partition Pruning\n\nIceberg table scans support Dynamic Partition Pruning (DPP) filters generated by Spark's `PlanDynamicPruningFilters`\noptimizer rule ([#3349]). However, we still need to bring this functionality to our Parquet reader. Furthermore,\nSpark's `PlanAdaptiveDynamicPruningFilters` optimizer rule runs after Comet's rules, so DPP with Adaptive Query\nExecution requires a redesign of Comet's plan translation. We are focused on implementing DPP to keep Comet competitive\nwith benchmarks that benefit from this feature like TPC-DS. This effort can be tracked at [#3510].\n\n[#3349]: https://github.com/apache/datafusion-comet/pull/3349\n[#3510]: https://github.com/apache/datafusion-comet/issues/3510\n\n## Ongoing Improvements\n\nIn addition to the major initiatives above, we have the following ongoing areas of work:\n\n- Adding support for more Spark expressions\n- Moving more expressions to the `datafusion-spark` crate in the core DataFusion repository\n- Performance tuning\n- Nested type support improvements\n" }, { "path": "docs/source/contributor-guide/spark-sql-tests.md", "content": "\n\n# Running Spark SQL Tests\n\nRunning Apache Spark's SQL tests with Comet enabled is a good way to ensure that Comet produces the same\nresults as that version of Spark. To enable this, we apply some changes to the Apache Spark source code so that\nComet is enabled when we run the tests.\n\nHere is an overview of the changes that we need to make to Spark:\n\n- Update the pom.xml to add a dependency on Comet\n- Modify SparkSession to load the Comet extension\n- Modify TestHive to load Comet\n- Modify SQLTestUtilsBase to load Comet when `ENABLE_COMET` environment variable exists\n\nHere are the steps involved in running the Spark SQL tests with Comet, using Spark 3.4.3 for this example.\n\n## 1. Install Comet\n\nRun `make release` in Comet to install the Comet JAR into the local Maven repository, specifying the Spark version.\n\n```shell\nPROFILES=\"-Pspark-3.4\" make release\n```\n\n## 2. Clone Spark and Apply Diff\n\nClone Apache Spark locally and apply the diff file from Comet.\n\nNote: this is a shallow clone of a tagged Spark commit and is not suitable for general Spark development.\n\n```shell\ngit clone -b 'v3.4.3' --single-branch --depth 1 git@github.com:apache/spark.git apache-spark\ncd apache-spark\ngit apply ../datafusion-comet/dev/diffs/3.4.3.diff\n```\n\n## 3. Run Spark SQL Tests\n\n### Use the following commands to run the Spark SQL test suite locally.\n\nOptionally, enable Comet fallback logging, so that all fallback reasons are logged at `WARN` level.\n\n```shell\nexport ENABLE_COMET_LOG_FALLBACK_REASONS=true\n```\n\n```shell\nENABLE_COMET=true ENABLE_COMET_ONHEAP=true build/sbt catalyst/test\nENABLE_COMET=true ENABLE_COMET_ONHEAP=true build/sbt \"sql/testOnly * -- -l org.apache.spark.tags.ExtendedSQLTest -l org.apache.spark.tags.SlowSQLTest\"\nENABLE_COMET=true ENABLE_COMET_ONHEAP=true build/sbt \"sql/testOnly * -- -n org.apache.spark.tags.ExtendedSQLTest\"\nENABLE_COMET=true ENABLE_COMET_ONHEAP=true build/sbt \"sql/testOnly * -- -n org.apache.spark.tags.SlowSQLTest\"\nENABLE_COMET=true ENABLE_COMET_ONHEAP=true build/sbt \"hive/testOnly * -- -l org.apache.spark.tags.ExtendedHiveTest -l org.apache.spark.tags.SlowHiveTest\"\nENABLE_COMET=true ENABLE_COMET_ONHEAP=true build/sbt \"hive/testOnly * -- -n org.apache.spark.tags.ExtendedHiveTest\"\nENABLE_COMET=true ENABLE_COMET_ONHEAP=true build/sbt \"hive/testOnly * -- -n org.apache.spark.tags.SlowHiveTest\"\n```\n\n### Steps to run individual test suites through SBT\n\n1. Open SBT with Comet enabled\n\n```shell\nENABLE_COMET=true ENABLE_COMET_ONHEAP=true sbt -J-Xmx4096m -Dspark.test.includeSlowTests=true\n```\n\n2. Run individual tests (Below code runs test named `SPARK-35568` in the `spark-sql` module)\n\n```shell\n sql/testOnly org.apache.spark.sql.DynamicPartitionPruningV1SuiteAEOn -- -z \"SPARK-35568\"\n```\n\n### Steps to run individual test suites in IntelliJ IDE\n\n1. Add below configuration in VM Options for your test case (apache-spark repository)\n\n```shell\n-Dspark.comet.enabled=true -Dspark.comet.debug.enabled=true -Dspark.plugins=org.apache.spark.CometPlugin -DXmx4096m -Dspark.executor.heartbeatInterval=20000 -Dspark.network.timeout=10000 --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED\n```\n\n2. Set `ENABLE_COMET=true` in environment variables\n ![img.png](img.png)\n3. After the above tests are configured, spark tests can be run with debugging enabled on spark/comet code. Note that Comet is added as a dependency and the classes are readonly while debugging from Spark. Any new changes to Comet are to be built and deployed locally through the command (`PROFILES=\"-Pspark-3.4\" make release`)\n\n## Creating a diff file for a new Spark version\n\nOnce Comet has support for a new Spark version, we need to create a diff file that can be applied to that version\nof Apache Spark to enable Comet when running tests. This is a highly manual process and the process can\nvary depending on the changes in the new version of Spark, but here is a general guide to the process.\n\nWe typically start by applying a patch from a previous version of Spark. For example, when enabling the tests\nfor Spark version 3.5.6 we may start by applying the existing diff for 3.5.5 first.\n\n```shell\ncd git/apache/spark\ngit checkout v3.5.6\ngit apply --reject --whitespace=fix ../datafusion-comet/dev/diffs/3.5.5.diff\n```\n\nAny changes that cannot be cleanly applied will instead be written out to reject files. For example, the above\ncommand generated the following files.\n\n```shell\nfind . -name \"*.rej\"\n./pom.xml.rej\n./sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala.rej\n./sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala.rej\n./sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala.rej\n./sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala.rej\n./sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala.rej\n./sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SchemaPruningSuite.scala.rej\n./sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/binaryfile/BinaryFileFormatSuite.scala.rej\n./sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRebaseDatetimeSuite.scala.rej\n./sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaSuite.scala.rej\n./sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala.rej\n./sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala.rej\n./sql/core/src/test/scala/org/apache/spark/sql/sources/CreateTableAsSelectSuite.scala.rej\n./sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala.rej\n./sql/core/src/test/scala/org/apache/spark/sql/sources/DisableUnnecessaryBucketedScanSuite.scala.rej\n./sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala.rej\n./sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala.rej\n./sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala.rej\n```\n\nThe changes in these reject files need to be applied manually.\n\nOne method is to use the [wiggle](https://github.com/neilbrown/wiggle) command (`brew install wiggle` on Mac).\n\nFor example:\n\n```shell\nwiggle --replace ./sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala ./sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala.rej\n```\n\n## Generating The Diff File\n\nThe diff file can be generated using the `git diff` command. It may be necessary to set the `core.abbrev`\nconfiguration setting to use 11 digits hashes for consistency with existing diff files.\n\nNote that there is an `IgnoreComet.scala` that is not part of the Spark codebase, and therefore needs to be added\nusing `git add` before generating the diff.\n\n```shell\ngit config core.abbrev 11;\ngit add sql/core/src/test/scala/org/apache/spark/sql/IgnoreComet.scala\ngit diff v3.5.6 > ../datafusion-comet/dev/diffs/3.5.6.diff\n```\n\n## Running Tests in CI\n\nThe easiest way to run the tests is to create a PR against Comet and let CI run the tests. When working with a\nnew Spark version, the `spark_sql_test.yaml` and `spark_sql_test_ansi.yaml` files will need updating with the\nnew version.\n" }, { "path": "docs/source/contributor-guide/sql-file-tests.md", "content": "\n\n# SQL File Tests\n\n`CometSqlFileTestSuite` is a test suite that automatically discovers `.sql` test files and\nruns each query through both Spark and Comet, comparing results. This provides a lightweight\nway to add expression and operator test coverage without writing Scala test code.\n\n## Running the tests\n\nRun all SQL file tests:\n\n```shell\n./mvnw test -Dsuites=\"org.apache.comet.CometSqlFileTestSuite\" -Dtest=none\n```\n\nRun a single test file by adding the file name (without `.sql` extension) after the suite name:\n\n```shell\n./mvnw test -Dsuites=\"org.apache.comet.CometSqlFileTestSuite create_named_struct\" -Dtest=none\n```\n\nThis uses ScalaTest's substring matching, so the argument must match part of the test name.\nTest names follow the pattern `sql-file: expressions//.sql []`.\n\n## Test file location\n\nSQL test files live under:\n\n```\nspark/src/test/resources/sql-tests/expressions/\n```\n\nFiles are organized into category subdirectories:\n\n```\nexpressions/\n aggregate/ -- avg, sum, count, min_max, ...\n array/ -- array_contains, array_append, get_array_item, ...\n bitwise/\n cast/\n conditional/ -- case_when, coalesce, if_expr, ...\n datetime/ -- date_add, date_diff, unix_timestamp, ...\n decimal/\n hash/\n map/ -- get_map_value, map_keys, map_values, ...\n math/ -- abs, ceil, floor, round, sqrt, ...\n misc/ -- width_bucket, scalar_subquery, ...\n string/ -- concat, like, substring, lower, upper, ...\n struct/ -- create_named_struct, get_struct_field, ...\n```\n\nThe test suite recursively discovers all `.sql` files in these directories. Each file becomes\none or more ScalaTest test cases.\n\n## File format\n\nA test file consists of SQL comments, directives, statements, and queries separated by blank\nlines. Here is a minimal example:\n\n```sql\nstatement\nCREATE TABLE test_abs(v double) USING parquet\n\nstatement\nINSERT INTO test_abs VALUES (1.5), (-2.5), (0.0), (NULL)\n\nquery\nSELECT abs(v) FROM test_abs\n```\n\n### Directives\n\nDirectives are SQL comments at the top of the file that configure how the test runs.\n\n#### `Config`\n\nSets a Spark SQL config for all queries in the file.\n\n```sql\n-- Config: spark.sql.ansi.enabled=true\n```\n\n#### `ConfigMatrix`\n\nRuns the entire file once per combination of values. Multiple `ConfigMatrix` lines produce a\ncross product of all combinations.\n\n```sql\n-- ConfigMatrix: spark.sql.optimizer.inSetConversionThreshold=100,0\n```\n\nThis generates two test cases:\n\n```\nsql-file: expressions/conditional/in_set.sql [spark.sql.optimizer.inSetConversionThreshold=100]\nsql-file: expressions/conditional/in_set.sql [spark.sql.optimizer.inSetConversionThreshold=0]\n```\n\nOnly add a `ConfigMatrix` directive when there is a real reason to run the test under\nmultiple configurations. Do not add `ConfigMatrix` directives speculatively.\n\n#### `MinSparkVersion`\n\nSkips the file when running on a Spark version older than the specified version.\n\n```sql\n-- MinSparkVersion: 3.5\n```\n\n### Statements\n\nA `statement` block executes DDL or DML and does not check results. Use this for `CREATE TABLE`\nand `INSERT` commands. Table names are automatically extracted for cleanup after the test.\n\n```sql\nstatement\nCREATE TABLE my_table(x int, y double) USING parquet\n\nstatement\nINSERT INTO my_table VALUES (1, 2.0), (3, 4.0), (NULL, NULL)\n```\n\n### Queries\n\nA `query` block executes a SELECT and compares results between Spark and Comet. The query\nmode controls how results are validated.\n\n#### `query` (default mode)\n\nChecks that the query runs natively on Comet (not falling back to Spark) and that results\nmatch Spark exactly.\n\n```sql\nquery\nSELECT abs(v) FROM test_abs\n```\n\n#### `query spark_answer_only`\n\nOnly checks that Comet results match Spark. Does not assert that the query runs natively.\nUse this for expressions that Comet may not fully support yet but should still produce\ncorrect results.\n\n```sql\nquery spark_answer_only\nSELECT some_expression(v) FROM test_table\n```\n\n#### `query tolerance=`\n\nChecks results with a numeric tolerance. Useful for floating-point functions where small\ndifferences are acceptable.\n\n```sql\nquery tolerance=0.0001\nSELECT cos(v) FROM test_trig\n```\n\n#### `query expect_fallback()`\n\nAsserts that the query falls back to Spark and verifies the fallback reason contains the\ngiven string.\n\n```sql\nquery expect_fallback(unsupported expression)\nSELECT unsupported_func(v) FROM test_table\n```\n\n#### `query ignore()`\n\nSkips the query entirely. Use this for queries that hit known bugs. The reason should be a\nlink to the tracking GitHub issue.\n\n```sql\n-- Comet bug: space(-1) causes native crash\nquery ignore(https://github.com/apache/datafusion-comet/issues/3326)\nSELECT space(n) FROM test_space WHERE n < 0\n```\n\n#### `query expect_error()`\n\nAsserts that both Spark and Comet throw an exception containing the given pattern. Use this\nfor ANSI mode tests where invalid operations should throw errors.\n\n```sql\n-- Config: spark.sql.ansi.enabled=true\n\n-- integer overflow should throw in ANSI mode\nquery expect_error(ARITHMETIC_OVERFLOW)\nSELECT 2147483647 + 1\n\n-- division by zero should throw in ANSI mode\nquery expect_error(DIVIDE_BY_ZERO)\nSELECT 1 / 0\n\n-- array out of bounds should throw in ANSI mode\nquery expect_error(INVALID_ARRAY_INDEX)\nSELECT array(1, 2, 3)[10]\n```\n\n## Adding a new test\n\n1. Create a `.sql` file under the appropriate subdirectory in\n `spark/src/test/resources/sql-tests/expressions/`. Create a new subdirectory if no\n existing category fits.\n\n2. Add the Apache license header as a SQL comment.\n\n3. Add a `ConfigMatrix` directive only if the test needs to run under multiple configurations\n (e.g., testing behavior that varies with a specific Spark config). Do not add `ConfigMatrix`\n directives speculatively.\n\n4. Create tables and insert test data using `statement` blocks. Include edge cases such as\n `NULL`, boundary values, and negative numbers.\n\n5. Add `query` blocks for each expression or behavior to test. Use the default `query` mode\n when you expect Comet to run the expression natively. Use `query spark_answer_only` when\n native execution is not yet expected.\n\n6. Run the tests to verify:\n\n ```shell\n ./mvnw test -Dsuites=\"org.apache.comet.CometSqlFileTestSuite\" -Dtest=none\n ```\n\n### Tips for writing thorough tests\n\n#### Cover all combinations of literal and column arguments\n\nComet often uses different code paths for literal values versus column references. Tests\nshould exercise both. For a function with multiple arguments, test every useful combination.\n\nFor a single-argument function, test both a column reference and a literal:\n\n```sql\n-- column argument (reads from Parquet, goes through columnar evaluation)\nquery\nSELECT ascii(s) FROM test_ascii\n\n-- literal arguments\nquery\nSELECT ascii('A'), ascii(''), ascii(NULL)\n```\n\nFor a multi-argument function like `concat_ws(sep, str1, str2, ...)`, test with the separator\nas a column versus a literal, and similarly for the other arguments:\n\n```sql\n-- all columns\nquery\nSELECT concat_ws(sep, a, b) FROM test_table\n\n-- literal separator, column values\nquery\nSELECT concat_ws(',', a, b) FROM test_table\n\n-- all literals\nquery\nSELECT concat_ws(',', 'hello', 'world')\n```\n\n**Note on constant folding:** Normally Spark constant-folds all-literal expressions during\nplanning, so Comet would never see them. However, `CometSqlFileTestSuite` automatically\ndisables constant folding (by excluding `ConstantFolding` from the optimizer rules), so\nall-literal queries are evaluated by Comet's native engine. This means you can use the\ndefault `query` mode for all-literal cases and they will be tested natively just like\ncolumn-based queries.\n\n#### Cover edge cases\n\nInclude edge-case values in your test data. The exact cases depend on the function, but\ncommon ones include:\n\n- **NULL values** -- every test should include NULLs\n- **Empty strings** -- for string functions\n- **Zero, negative, and very large numbers** -- for numeric functions\n- **Boundary values** -- `INT_MIN`, `INT_MAX`, `NaN`, `Infinity`, `-Infinity` for numeric\n types\n- **Special characters and multibyte UTF-8** -- for string functions (e.g. `'é'`, `'中文'`,\n `'\\t'`)\n- **Empty arrays/maps** -- for collection functions\n- **Single-element and multi-element collections** -- for aggregate and collection functions\n\n#### One file per expression\n\nKeep each `.sql` file focused on a single expression or a small group of closely related\nexpressions. This makes failures easy to locate and keeps files readable.\n\n#### Use comments to label sections\n\nAdd SQL comments before query blocks to describe what aspect of the expression is being\ntested. This helps reviewers and future maintainers understand the intent:\n\n```sql\n-- literal separator with NULL values\nquery\nSELECT concat_ws(',', NULL, 'b', 'c')\n\n-- empty separator\nquery\nSELECT concat_ws('', a, b, c) FROM test_table\n```\n\n### Using agentic coding tools\n\nWriting thorough SQL test files is a task well suited to agentic coding tools such as\nClaude Code. You can point the tool at an existing test file as an example, describe the\nexpression you want to test, and ask it to generate a complete `.sql` file covering all\nargument combinations and edge cases. This is significantly faster than writing the\ncombinatorial test cases by hand and helps ensure nothing is missed.\n\nFor example:\n\n```\nRead the test file spark/src/test/resources/sql-tests/expressions/string/ascii.sql\nand the documentation in docs/source/contributor-guide/sql-file-tests.md.\nThen write a similar test file for the `reverse` function, covering column arguments,\nliteral arguments, NULLs, empty strings, and multibyte characters.\n```\n\n## Handling test failures\n\nWhen a query fails due to a known Comet bug:\n\n1. File a GitHub issue describing the problem.\n2. Change the query mode to `ignore(...)` with a link to the issue.\n3. Optionally add a SQL comment above the query explaining the problem.\n\n```sql\n-- GetArrayItem returns incorrect results with dynamic index\nquery ignore(https://github.com/apache/datafusion-comet/issues/3332)\nSELECT arr[idx] FROM test_get_array_item\n```\n\nWhen the bug is fixed, remove the `ignore(...)` and restore the original query mode.\n\n## Architecture\n\nThe test infrastructure consists of two Scala files:\n\n- **`SqlFileTestParser`** (`spark/src/test/scala/org/apache/comet/SqlFileTestParser.scala`) --\n Parses `.sql` files into a `SqlTestFile` data structure containing directives, statements,\n and queries.\n\n- **`CometSqlFileTestSuite`** (`spark/src/test/scala/org/apache/comet/CometSqlFileTestSuite.scala`) --\n Discovers test files at suite initialization time, generates ScalaTest test cases for each\n file and config combination, and executes them using `CometTestBase` assertion methods.\n\nTables created in test files are automatically cleaned up after each test.\n" }, { "path": "docs/source/contributor-guide/sql_error_propagation.md", "content": "\n\n# ANSI SQL Error Propagation in Comet\n\n## Overview\n\nApache Comet is a native query accelerator for Apache Spark. It runs SQL expressions in\n**Rust** (via Apache DataFusion) instead of the JVM, which is much faster. But there's a\ncatch: when something goes wrong — say, a divide-by-zero or a type cast failure — the error\nneeds to travel from Rust code all the way back to the Spark/Scala/Java world as a proper\nSpark exception with the right type, the right message, and even a pointer to the exact\ncharacter in the original SQL query where the error happened.\n\nThis document explains the end-to-end error-propagation pipeline.\n\n---\n\n## The Big Picture\n\n![Error propagation pipeline overview](./error_pipeline_overview.svg)\n\n```\nSQL Query (Spark/Scala)\n │\n │ 1. Spark serializes the plan + query context into Protobuf\n ▼\n Protobuf bytes ──────────────────────────────────► JNI boundary\n │\n │ 2. Rust deserializes the plan and registers query contexts\n ▼\n Native execution (DataFusion / Rust)\n │\n │ 3. An error occurs (e.g. divide by zero)\n ▼\n SparkError (Rust enum)\n │\n │ 4. Error is wrapped with SQL location context\n ▼\n SparkErrorWithContext\n │\n │ 5. Serialized to JSON string\n ▼\n JSON string ◄──────────────────────────────────── JNI boundary\n │\n │ 6. Thrown as CometQueryExecutionException\n ▼\n CometExecIterator.scala catches it\n │\n │ 7. JSON is parsed, proper Spark exception is reconstructed\n ▼\n Spark exception (e.g. ArithmeticException with DIVIDE_BY_ZERO errorClass)\n │\n │ 8. Spark displays it to the user with SQL location pointer\n ▼\n User sees:\n [DIVIDE_BY_ZERO] Division by zero.\n == SQL (line 1, position 8) ==\n SELECT a/b FROM t\n ^^^\n```\n\n---\n\n## Step 1: Spark Serializes Query Context into Protobuf\n\nWhen Spark compiles a SQL query, it parses it and attaches _origin_ information to every\nexpression — the line number, column offset, and the full SQL text.\n\n`QueryPlanSerde.scala` is the Scala code that converts Spark's physical execution plan into\na Protobuf binary that gets sent to the Rust side. It extracts origin information from each\nexpression and encodes it alongside the expression in the Protobuf payload.\n\n### The `extractQueryContext` function\n\n```scala\n// spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala\n\nprivate def extractQueryContext(expr: Expression): Option[ExprOuterClass.QueryContext] = {\n val contexts = expr.origin.getQueryContext // Spark stores context in expr.origin\n if (contexts != null && contexts.length > 0) {\n val ctx = contexts(0)\n ctx match {\n case sqlCtx: SQLQueryContext =>\n val builder = ExprOuterClass.QueryContext.newBuilder()\n .setSqlText(sqlCtx.sqlText.getOrElse(\"\")) // full SQL text\n .setStartIndex(sqlCtx.originStartIndex.getOrElse(...)) // char offset of expression start\n .setStopIndex(sqlCtx.originStopIndex.getOrElse(...)) // char offset of expression end\n .setLine(sqlCtx.line.getOrElse(0))\n .setStartPosition(sqlCtx.startPosition.getOrElse(0))\n // ...\n Some(builder.build())\n }\n }\n}\n```\n\nThen, for **every single expression** converted to Protobuf, a unique numeric ID and the\ncontext are attached:\n\n```scala\n.map { protoExpr =>\n val builder = protoExpr.toBuilder\n builder.setExprId(nextExprId()) // unique ID (monotonically increasing counter)\n extractQueryContext(expr).foreach { ctx =>\n builder.setQueryContext(ctx) // attach the SQL location info\n }\n builder.build()\n}\n```\n\n### The Protobuf schema\n\n```protobuf\n// native/proto/src/proto/expr.proto\n\nmessage Expr {\n optional uint64 expr_id = 89; // unique ID for each expression\n optional QueryContext query_context = 90; // SQL location info\n // ... actual expression type ...\n}\n\nmessage QueryContext {\n string sql_text = 1; // \"SELECT a/b FROM t\"\n int32 start_index = 2; // 7 (0-based character index of 'a')\n int32 stop_index = 3; // 9 (0-based character index of 'b', inclusive)\n int32 line = 4; // 1 (1-based line number)\n int32 start_position = 5; // 7 (0-based column position)\n optional string object_type = 6; // e.g. \"VIEW\"\n optional string object_name = 7; // e.g. \"v1\"\n}\n```\n\n---\n\n## Step 2: Rust Deserializes the Plan and Registers Query Contexts\n\nOn the Rust side, `PhysicalPlanner` in `planner.rs` converts the Protobuf into DataFusion's\nphysical plan. A `QueryContextMap` — a global registry — maps expression IDs to their SQL\ncontext.\n\n### `QueryContextMap` (`native/spark-expr/src/query_context.rs`)\n\n```rust\npub struct QueryContextMap {\n contexts: RwLock>>,\n}\n\nimpl QueryContextMap {\n pub fn register(&self, expr_id: u64, context: QueryContext) { ... }\n pub fn get(&self, expr_id: u64) -> Option> { ... }\n}\n```\n\nThis is basically a lookup table: \"for expression #42, the SQL context is: text=`SELECT a/b\nFROM t`, characters 7–9, line 1, column 7\".\n\n### The `PhysicalPlanner` registers contexts during plan creation\n\n```rust\n// native/core/src/execution/planner.rs\n\npub struct PhysicalPlanner {\n query_context_registry: Arc,\n // ...\n}\n\npub(crate) fn create_expr(&self, spark_expr: &Expr, ...) {\n // 1. If this expression has a query context, register it\n if let (Some(expr_id), Some(ctx_proto)) =\n (spark_expr.expr_id, spark_expr.query_context.as_ref()) {\n let query_ctx = QueryContext::new(\n ctx_proto.sql_text.clone(),\n ctx_proto.start_index,\n ctx_proto.stop_index,\n ...\n );\n self.query_context_registry.register(expr_id, query_ctx);\n }\n\n // 2. When building specific expressions (Cast, CheckOverflow, etc.),\n // look up the context and pass it to the expression\n ExprStruct::Cast(expr) => {\n let query_context = spark_expr.expr_id.and_then(|id| {\n self.query_context_registry.get(id)\n });\n Ok(Arc::new(Cast::new(child, datatype, options, spark_expr.expr_id, query_context)))\n }\n}\n```\n\n---\n\n## Step 3: An Error Occurs During Native Execution\n\nDuring query execution, a Rust expression might encounter something like division by zero.\n\n### The `SparkError` enum (`native/spark-expr/src/error.rs`)\n\nThis enum contains one variant for every kind of error Spark can produce, with exactly the\nsame error message format as Spark:\n\n```rust\npub enum SparkError {\n #[error(\"[DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor \\\n being 0 and return NULL instead. If necessary set \\\n \\\"spark.sql.ansi.enabled\\\" to \\\"false\\\" to bypass this error.\")]\n DivideByZero,\n\n #[error(\"[CAST_INVALID_INPUT] The value '{value}' of the type \\\"{from_type}\\\" \\\n cannot be cast to \\\"{to_type}\\\" because it is malformed. ...\")]\n CastInvalidValue { value: String, from_type: String, to_type: String },\n\n // ... 30+ more variants matching Spark's error codes ...\n}\n```\n\nWhen a divide-by-zero happens, the arithmetic expression creates:\n\n```rust\nreturn Err(DataFusionError::External(Box::new(SparkError::DivideByZero)));\n```\n\n---\n\n## Step 4: Error Gets Wrapped with SQL Context\n\nThe expression wrappers (`CheckedBinaryExpr`, `CheckOverflow`, `Cast`) catch the\n`SparkError` and attach the SQL context using `SparkErrorWithContext`:\n\n```rust\n// native/core/src/execution/expressions/arithmetic.rs\n// CheckedBinaryExpr wraps arithmetic operations\n\nimpl PhysicalExpr for CheckedBinaryExpr {\n fn evaluate(&self, batch: &RecordBatch) -> Result<...> {\n match self.child.evaluate(batch) {\n Err(DataFusionError::External(e)) if self.query_context.is_some() => {\n if let Some(spark_error) = e.downcast_ref::() {\n // Wrap the error with SQL location info\n let wrapped = SparkErrorWithContext::with_context(\n spark_error.clone(),\n Arc::clone(self.query_context.as_ref().unwrap()),\n );\n return Err(DataFusionError::External(Box::new(wrapped)));\n }\n Err(DataFusionError::External(e))\n }\n other => other,\n }\n }\n}\n```\n\n### `SparkErrorWithContext` (`native/spark-expr/src/error.rs`)\n\n```rust\npub struct SparkErrorWithContext {\n pub error: SparkError, // the actual error\n pub context: Option>, // optional SQL location\n}\n```\n\n---\n\n## Step 5: Error Is Serialized to JSON\n\nWhen DataFusion propagates the error all the way up through the execution engine and it\nreaches the JNI boundary, `throw_exception()` in `errors.rs` is called. It detects the\n`SparkErrorWithContext` type and calls `.to_json()` on it:\n\n```rust\n// native/core/src/errors.rs\n\nfn throw_exception(env: &mut JNIEnv, error: &CometError, ...) {\n match error {\n CometError::DataFusion {\n source: DataFusionError::External(e), ..\n } => {\n if let Some(spark_err_ctx) = e.downcast_ref::() {\n // Has SQL context → throw with JSON payload\n let json = spark_err_ctx.to_json();\n env.throw_new(\"org/apache/comet/exceptions/CometQueryExecutionException\", json)\n } else if let Some(spark_err) = e.downcast_ref::() {\n // No SQL context → throw with JSON payload (no context field)\n throw_spark_error_as_json(env, spark_err)\n }\n }\n // ...\n }\n}\n```\n\nThe JSON looks like this for a divide-by-zero in `SELECT a/b FROM t`:\n\n```json\n{\n \"errorType\": \"DivideByZero\",\n \"errorClass\": \"DIVIDE_BY_ZERO\",\n \"params\": {},\n \"context\": {\n \"sqlText\": \"SELECT a/b FROM t\",\n \"startIndex\": 7,\n \"stopIndex\": 9,\n \"line\": 1,\n \"startPosition\": 7,\n \"objectType\": null,\n \"objectName\": null\n },\n \"summary\": \"== SQL (line 1, position 8) ==\\nSELECT a/b FROM t\\n ^^^\"\n}\n```\n\n---\n\n## Step 6: Java Receives `CometQueryExecutionException`\n\nOn the Java side, a thin exception class carries the JSON string as its message:\n\n```java\n// common/src/main/java/org/apache/comet/exceptions/CometQueryExecutionException.java\n\npublic final class CometQueryExecutionException extends CometNativeException {\n public CometQueryExecutionException(String jsonMessage) {\n super(jsonMessage);\n }\n\n public boolean isJsonMessage() {\n String msg = getMessage();\n return msg != null && msg.trim().startsWith(\"{\") && msg.trim().endsWith(\"}\");\n }\n}\n```\n\n---\n\n## Step 7: Scala Converts JSON Back to a Real Spark Exception\n\n`CometExecIterator.scala` is the Scala code that drives the native execution. Every time it\ncalls into the native engine for the next batch of data, it catches\n`CometQueryExecutionException` and converts it:\n\n```scala\n// spark/src/main/scala/org/apache/comet/CometExecIterator.scala\n\ntry {\n nativeUtil.getNextBatch(...)\n} catch {\n case e: CometQueryExecutionException =>\n logError(s\"Native execution for task $taskAttemptId failed\", e)\n throw SparkErrorConverter.convertToSparkException(e) // converts JSON to real exception\n}\n```\n\n### `SparkErrorConverter.scala` parses the JSON\n\n```scala\n// spark/src/main/scala/org/apache/comet/SparkErrorConverter.scala\n\ndef convertToSparkException(e: CometQueryExecutionException): Throwable = {\n val json = parse(e.getMessage)\n val errorJson = json.extract[ErrorJson]\n\n // Reconstruct Spark's SQLQueryContext from the embedded context\n val sparkContext: Array[QueryContext] = errorJson.context match {\n case Some(ctx) =>\n Array(SQLQueryContext(\n sqlText = Some(ctx.sqlText),\n line = Some(ctx.line),\n startPosition = Some(ctx.startPosition),\n originStartIndex = Some(ctx.startIndex),\n originStopIndex = Some(ctx.stopIndex),\n originObjectType = ctx.objectType,\n originObjectName = ctx.objectName))\n case None => Array.empty\n }\n\n // Delegate to version-specific shim\n convertErrorType(errorJson.errorType, errorClass, params, sparkContext, summary)\n}\n```\n\n### `ShimSparkErrorConverter` calls the real Spark API\n\nBecause Spark's `QueryExecutionErrors` API changes between Spark versions (3.4, 3.5, 4.0),\nthere is a separate implementation per version (in `spark-3.4/`, `spark-3.5/`, `spark-4.0/`).\n\n![Shim pattern for per-version Spark API bridging](./shim_pattern.svg)\n\n```scala\n// spark/src/main/spark-3.5/org/apache/spark/sql/comet/shims/ShimSparkErrorConverter.scala\n\ndef convertErrorType(errorType: String, errorClass: String,\n params: Map[String, Any], context: Array[QueryContext], summary: String): Option[Throwable] = {\n\n errorType match {\n case \"DivideByZero\" =>\n Some(QueryExecutionErrors.divideByZeroError(sqlCtx(context)))\n // This is the REAL Spark method that creates the ArithmeticException\n // with the SQL context pointer. The error message will include\n // \"== SQL (line 1, position 8) ==\" etc.\n\n case \"CastInvalidValue\" =>\n Some(QueryExecutionErrors.castingCauseOverflowError(...))\n\n // ... all other error types ...\n\n case _ => None // fallback to generic SparkException\n }\n}\n```\n\nThe Spark 3.5 shim vs the Spark 4.0 shim differ in subtle API ways:\n\n```scala\n// 3.5: binaryArithmeticCauseOverflowError does NOT take functionName\ncase \"BinaryArithmeticOverflow\" =>\n Some(QueryExecutionErrors.binaryArithmeticCauseOverflowError(\n params(\"value1\").toString.toShort,\n params(\"symbol\").toString,\n params(\"value2\").toString.toShort))\n\n// 4.0: overloadedMethod takes functionName parameter\ncase \"BinaryArithmeticOverflow\" =>\n Some(QueryExecutionErrors.binaryArithmeticCauseOverflowError(\n params(\"value1\").toString.toShort,\n params(\"symbol\").toString,\n params(\"value2\").toString.toShort,\n params(\"functionName\").toString)) // extra param in 4.0\n```\n\n---\n\n## Step 8: The User Sees a Proper Spark Error\n\nThe final exception that propagates out of Spark looks exactly like what native Spark would\nproduce for the same error, including the ANSI error code and the SQL pointer:\n\n```\norg.apache.spark.SparkArithmeticException: [DIVIDE_BY_ZERO] Division by zero.\nUse `try_divide` to tolerate divisor being 0 and return NULL instead.\nIf necessary set \"spark.sql.ansi.enabled\" to \"false\" to bypass this error.\n\n== SQL (line 1, position 8) ==\nSELECT a/b FROM t\n ^^^\n```\n\n---\n\n## Key Data Structures: A Summary\n\n| Structure | Language | File | Purpose |\n| ------------------------------ | -------- | ----------------------------------- | --------------------------------------------------- |\n| `QueryContext` (proto) | Protobuf | `expr.proto` | Wire format for SQL location info |\n| `QueryContext` (Rust) | Rust | `query_context.rs` | In-memory SQL location info |\n| `QueryContextMap` | Rust | `query_context.rs` | Registry: expr_id → QueryContext |\n| `SparkError` | Rust | `error.rs` | Typed Rust enum matching all Spark error variants |\n| `SparkErrorWithContext` | Rust | `error.rs` | SparkError + optional QueryContext |\n| `CometQueryExecutionException` | Java | `CometQueryExecutionException.java` | JNI transport: carries JSON string |\n| `SparkErrorConverter` | Scala | `SparkErrorConverter.scala` | Parses JSON, creates real Spark exception |\n| `ShimSparkErrorConverter` | Scala | `ShimSparkErrorConverter.scala` | Per-Spark-version calls to `QueryExecutionErrors.*` |\n\n---\n\n## Why JSON? Why Not Throw the Right Exception Directly?\n\nJNI does not support throwing arbitrary Java exception subclasses from Rust directly — you\ncan only provide a class name and a string message. The class name is fixed (always\n`CometQueryExecutionException`), but the string payload can carry any structured data.\n\nJSON was chosen because:\n\n1. It is self-describing — the receiver can parse it without knowing the structure in advance.\n2. It is easy to add new fields without breaking old parsers.\n3. It maps cleanly to Scala's case class extraction (`json.extract[ErrorJson]`).\n4. All the typed information (error class, parameters, SQL context) can be round-tripped\n perfectly.\n\nThe alternative — throwing different Java exception classes from Rust — would require a\nseparate JNI throw path for each of the 30+ error types, which would be much harder to\nmaintain.\n\n---\n\n## Why `SparkError` Has Its Own `error_class()` Method\n\nEach `SparkError` variant knows its own ANSI error class code (e.g. `\"DIVIDE_BY_ZERO\"`,\n`\"CAST_INVALID_INPUT\"`). This is used both:\n\n- In the JSON payload's `\"errorClass\"` field (so the Java side can pass it to\n `SparkException(errorClass = ...)` as a fallback)\n- In the legacy `exception_class()` method that maps to the right Java exception class (e.g.\n `\"java/lang/ArithmeticException\"`)\n\n---\n\n## Diagrams\n\n### End-to-End Pipeline\n\n![Error propagation pipeline overview](./error_pipeline_overview.svg)\n\n### QueryContext Journey: From SQL Text to Error Pointer\n\n![QueryContext journey from SQL parser to error message](./query_context_journey.svg)\n\n### Shim Pattern: Per-Version Spark API Bridging\n\n![Shim pattern for per-version Spark API bridging](./shim_pattern.svg)\n\n---\n\n_This document and its diagrams were written by [Claude](https://claude.ai) (Anthropic)._\n" }, { "path": "docs/source/contributor-guide/tracing.md", "content": "\n\n# Tracing\n\nTracing can be enabled by setting `spark.comet.tracing.enabled=true`.\n\nWith this feature enabled, each Spark executor will write a JSON event log file in\nChrome's [Trace Event Format]. The file will be written to the executor's current working\ndirectory with the filename `comet-event-trace.json`.\n\n[Trace Event Format]: https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/preview?tab=t.0#heading=h.yr4qxyxotyw\n\nAdditionally, enabling the `jemalloc` feature will enable tracing of native memory allocations.\n\n```shell\nmake release COMET_FEATURES=\"jemalloc\"\n```\n\nExample output:\n\n```json\n{ \"name\": \"decodeShuffleBlock\", \"cat\": \"PERF\", \"ph\": \"B\", \"pid\": 1, \"tid\": 5, \"ts\": 10109225730 },\n{ \"name\": \"decodeShuffleBlock\", \"cat\": \"PERF\", \"ph\": \"E\", \"pid\": 1, \"tid\": 5, \"ts\": 10109228835 },\n{ \"name\": \"decodeShuffleBlock\", \"cat\": \"PERF\", \"ph\": \"B\", \"pid\": 1, \"tid\": 5, \"ts\": 10109245928 },\n{ \"name\": \"decodeShuffleBlock\", \"cat\": \"PERF\", \"ph\": \"E\", \"pid\": 1, \"tid\": 5, \"ts\": 10109248843 },\n{ \"name\": \"executePlan\", \"cat\": \"PERF\", \"ph\": \"E\", \"pid\": 1, \"tid\": 5, \"ts\": 10109350935 },\n{ \"name\": \"getNextBatch[JVM] stage=2\", \"cat\": \"PERF\", \"ph\": \"E\", \"pid\": 1, \"tid\": 5, \"ts\": 10109367116 },\n{ \"name\": \"getNextBatch[JVM] stage=2\", \"cat\": \"PERF\", \"ph\": \"B\", \"pid\": 1, \"tid\": 5, \"ts\": 10109479156 },\n```\n\nTraces can be viewed with [Perfetto UI].\n\n[Perfetto UI]: https://ui.perfetto.dev\n\nExample trace visualization:\n\n![tracing](../_static/images/tracing.png)\n\n## Analyzing Memory Usage\n\nThe `analyze_trace` tool parses a trace log and compares jemalloc usage against the sum of per-thread\nComet memory pool reservations. This is useful for detecting untracked native memory growth where jemalloc\nallocations exceed what the memory pools account for.\n\nBuild and run:\n\n```shell\ncd native\ncargo run --bin analyze_trace -- /path/to/comet-event-trace.json\n```\n\nThe tool reads counter events from the trace log. Because tracing logs metrics per thread, `jemalloc_allocated`\nis a process-wide value (the same global allocation reported from whichever thread logs it), while\n`thread_NNN_comet_memory_reserved` values are per-thread pool reservations that are summed to get the total\ntracked memory.\n\nSample output:\n\n```\n=== Comet Trace Memory Analysis ===\n\nCounter events parsed: 193104\nThreads with memory pools: 8\nPeak jemalloc allocated: 3068.2 MB\nPeak pool total: 2864.6 MB\nPeak excess (jemalloc - pool): 364.6 MB\n\nWARNING: jemalloc exceeded pool reservation at 138 sampled points:\n\n Time (us) jemalloc pool_total excess\n--------------------------------------------------------------\n 179578 210.8 MB 0.1 MB 210.7 MB\n 429663 420.5 MB 145.1 MB 275.5 MB\n 1304969 2122.5 MB 1797.2 MB 325.2 MB\n 21974838 407.0 MB 42.3 MB 364.6 MB\n 33543599 5.5 MB 0.1 MB 5.3 MB\n\n--- Final per-thread pool reservations ---\n\n thread_60_comet_memory_reserved: 0.0 MB\n thread_95_comet_memory_reserved: 0.0 MB\n thread_96_comet_memory_reserved: 0.0 MB\n ...\n\n Total: 0.0 MB\n```\n\nSome excess is expected (jemalloc metadata, fragmentation, non-pool allocations like Arrow IPC buffers).\nLarge or growing excess may indicate memory that is not being tracked by the pool.\n\n## Definition of Labels\n\n| Label | Meaning |\n| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |\n| jvm_heap_used | JVM heap memory usage of live objects for the executor process |\n| jemalloc_allocated | Native memory usage for the executor process (requires `jemalloc` feature) |\n| thread_NNN_comet_memory_reserved | Memory reserved by Comet's DataFusion memory pool (summed across all contexts on the thread). NNN is the Rust thread ID. |\n| thread_NNN_comet_jvm_shuffle | Off-heap memory allocated by Comet for columnar shuffle. NNN is the Rust thread ID. |\n" }, { "path": "docs/source/index.md", "content": "\n\n# Comet Accelerator for Apache Spark and Apache Iceberg\n\n\n

\n \n Star\n \n Fork\n

\n\nApache DataFusion Comet is a high-performance accelerator for Apache Spark, built on top of the powerful\n[Apache DataFusion] query engine. Comet is designed to significantly enhance the\nperformance of Apache Spark workloads while leveraging commodity hardware and seamlessly integrating with the\nSpark ecosystem without requiring any code changes.\n\nComet also accelerates Apache Iceberg, when performing Parquet scans from Spark.\n\n[Apache DataFusion]: https://datafusion.apache.org\n\n## Run Spark Queries at DataFusion Speeds\n\nComet delivers a performance speedup for many queries, enabling faster data processing and shorter time-to-insights.\n\nThe following chart shows the time it takes to run the 22 TPC-H queries against 100 GB of data in Parquet format\nusing a single executor with 8 cores. See the [Comet Benchmarking Guide](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html)\nfor details of the environment used for these benchmarks.\n\nWhen using Comet, the overall run time is reduced from 687 seconds to 302 seconds, a 2.2x speedup.\n\n![](_static/images/benchmark-results/0.11.0/tpch_allqueries.png)\n\nHere is a breakdown showing relative performance of Spark and Comet for each TPC-H query.\n\n![](_static/images/benchmark-results/0.11.0/tpch_queries_compare.png)\n\nThese benchmarks can be reproduced in any environment using the documentation in the\n[Comet Benchmarking Guide](/contributor-guide/benchmarking.md). We encourage\nyou to run your own benchmarks.\n\n## Use Commodity Hardware\n\nComet leverages commodity hardware, eliminating the need for costly hardware upgrades or\nspecialized hardware accelerators, such as GPUs or FPGA. By maximizing the utilization of commodity hardware, Comet\nensures cost-effectiveness and scalability for your Spark deployments.\n\n## Spark Compatibility\n\nComet aims for 100% compatibility with all supported versions of Apache Spark, allowing you to integrate Comet into\nyour existing Spark deployments and workflows seamlessly. With no code changes required, you can immediately harness\nthe benefits of Comet's acceleration capabilities without disrupting your Spark applications.\n\n## Tight Integration with Apache DataFusion\n\nComet tightly integrates with the core Apache DataFusion project, leveraging its powerful execution engine. With\nseamless interoperability between Comet and DataFusion, you can achieve optimal performance and efficiency in your\nSpark workloads.\n\n## Active Community\n\nComet boasts a vibrant and active community of developers, contributors, and users dedicated to advancing the\ncapabilities of Apache DataFusion and accelerating the performance of Apache Spark.\n\n## Getting Started\n\nTo get started with Apache DataFusion Comet, follow the\n[installation instructions](https://datafusion.apache.org/comet/user-guide/installation.html). Join the\n[DataFusion Slack and Discord channels](https://datafusion.apache.org/contributor-guide/communication.html) to connect\nwith other users, ask questions, and share your experiences with Comet.\n\nFollow [Apache DataFusion Comet Overview](https://datafusion.apache.org/comet/about/index.html) to get more detailed information\n\n## Contributing\n\nWe welcome contributions from the community to help improve and enhance Apache DataFusion Comet. Whether it's fixing\nbugs, adding new features, writing documentation, or optimizing performance, your contributions are invaluable in\nshaping the future of Comet. Check out our\n[contributor guide](https://datafusion.apache.org/comet/contributor-guide/contributing.html) to get started.\n\n```{toctree}\n:maxdepth: 1\n:caption: Index\n:hidden:\n\nComet Overview \nUser Guide \nContributor Guide \nASF Links \n```\n" }, { "path": "docs/source/user-guide/index.md", "content": "\n\n# Comet User Guide\n\n```{toctree}\n:maxdepth: 2\n:caption: User Guides\n\n0.15.0-SNAPSHOT \n0.14.x <0.14/index>\n0.13.x <0.13/index>\n0.12.x <0.12/index>\n0.11.x <0.11/index>\n```\n" }, { "path": "docs/source/user-guide/latest/compatibility.md", "content": "\n\n# Compatibility Guide\n\nComet aims to provide consistent results with the version of Apache Spark that is being used.\n\nThis guide offers information about areas of functionality where there are known differences.\n\n## Parquet\n\nComet has the following limitations when reading Parquet files:\n\n- Comet does not support reading decimals encoded in binary format.\n- No support for default values that are nested types (e.g., maps, arrays, structs). Literal default values are supported.\n\n## ANSI Mode\n\nComet will fall back to Spark for the following expressions when ANSI mode is enabled. These expressions can be enabled by setting\n`spark.comet.expression.EXPRNAME.allowIncompatible=true`, where `EXPRNAME` is the Spark expression class name. See\nthe [Comet Supported Expressions Guide](expressions.md) for more information on this configuration setting.\n\n- Average (supports all numeric inputs except decimal types)\n- Cast (in some cases)\n\nThere is an [epic](https://github.com/apache/datafusion-comet/issues/313) where we are tracking the work to fully implement ANSI support.\n\n## Floating-point Number Comparison\n\nSpark normalizes NaN and zero for floating point numbers for several cases. See `NormalizeFloatingNumbers` optimization rule in Spark.\nHowever, one exception is comparison. Spark does not normalize NaN and zero when comparing values\nbecause they are handled well in Spark (e.g., `SQLOrderingUtil.compareFloats`). But the comparison\nfunctions of arrow-rs used by DataFusion do not normalize NaN and zero (e.g., [arrow::compute::kernels::cmp::eq](https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/fn.eq.html#)).\nSo Comet adds additional normalization expression of NaN and zero for comparisons, and may still have differences\nto Spark in some cases, especially when the data contains both positive and negative zero. This is likely an edge\ncase that is not of concern for many users. If it is a concern, setting `spark.comet.exec.strictFloatingPoint=true`\nwill make relevant operations fall back to Spark.\n\n## Incompatible Expressions\n\nExpressions that are not 100% Spark-compatible will fall back to Spark by default and can be enabled by setting\n`spark.comet.expression.EXPRNAME.allowIncompatible=true`, where `EXPRNAME` is the Spark expression class name. See\nthe [Comet Supported Expressions Guide](expressions.md) for more information on this configuration setting.\n\n### Array Expressions\n\n- **ArraysOverlap**: Inconsistent behavior when arrays contain NULL values.\n [#3645](https://github.com/apache/datafusion-comet/issues/3645),\n [#2036](https://github.com/apache/datafusion-comet/issues/2036)\n- **ArrayUnion**: Sorts input arrays before performing the union, while Spark preserves the order of the first array\n and appends unique elements from the second.\n [#3644](https://github.com/apache/datafusion-comet/issues/3644)\n- **SortArray**: Nested arrays with `Struct` or `Null` child values are not supported natively and will fall back to Spark.\n\n### Date/Time Expressions\n\n- **Hour, Minute, Second**: Incorrectly apply timezone conversion to TimestampNTZ inputs. TimestampNTZ stores local\n time without timezone, so no conversion should be applied. These expressions work correctly with Timestamp inputs.\n [#3180](https://github.com/apache/datafusion-comet/issues/3180)\n- **TruncTimestamp (date_trunc)**: Produces incorrect results when used with non-UTC timezones. Compatible when\n timezone is UTC.\n [#2649](https://github.com/apache/datafusion-comet/issues/2649)\n\n### Struct Expressions\n\n- **StructsToJson (to_json)**: Does not support `+Infinity` and `-Infinity` for numeric types (float, double).\n [#3016](https://github.com/apache/datafusion-comet/issues/3016)\n\n## Regular Expressions\n\nComet uses the Rust regexp crate for evaluating regular expressions, and this has different behavior from Java's\nregular expression engine. Comet will fall back to Spark for patterns that are known to produce different results, but\nthis can be overridden by setting `spark.comet.expression.regexp.allowIncompatible=true`.\n\n## Window Functions\n\nComet's support for window functions is incomplete and known to be incorrect. It is disabled by default and\nshould not be used in production. The feature will be enabled in a future release. Tracking issue: [#2721](https://github.com/apache/datafusion-comet/issues/2721).\n\n## Round-Robin Partitioning\n\nComet's native shuffle implementation of round-robin partitioning (`df.repartition(n)`) is not compatible with\nSpark's implementation and is disabled by default. It can be enabled by setting\n`spark.comet.native.shuffle.partitioning.roundrobin.enabled=true`.\n\n**Why the incompatibility exists:**\n\nSpark's round-robin partitioning sorts rows by their binary `UnsafeRow` representation before assigning them to\npartitions. This ensures deterministic output for fault tolerance (task retries produce identical results).\nComet uses Arrow format internally, which has a completely different binary layout than `UnsafeRow`, making it\nimpossible to match Spark's exact partition assignments.\n\n**Comet's approach:**\n\nInstead of true round-robin assignment, Comet implements round-robin as hash partitioning on ALL columns. This\nachieves the same semantic goals:\n\n- **Even distribution**: Rows are distributed evenly across partitions (as long as the hash varies sufficiently -\n in some cases there could be skew)\n- **Deterministic**: Same input always produces the same partition assignments (important for fault tolerance)\n- **No semantic grouping**: Unlike hash partitioning on specific columns, this doesn't group related rows together\n\nThe only difference is that Comet's partition assignments will differ from Spark's. When results are sorted,\nthey will be identical to Spark. Unsorted results may have different row ordering.\n\n## Cast\n\nCast operations in Comet fall into three levels of support:\n\n- **C (Compatible)**: The results match Apache Spark\n- **I (Incompatible)**: The results may match Apache Spark for some inputs, but there are known issues where some inputs\n will result in incorrect results or exceptions. The query stage will fall back to Spark by default. Setting\n `spark.comet.expression.Cast.allowIncompatible=true` will allow all incompatible casts to run natively in Comet, but this is not\n recommended for production use.\n- **U (Unsupported)**: Comet does not provide a native version of this cast expression and the query stage will fall back to\n Spark.\n- **N/A**: Spark does not support this cast.\n\n### String to Decimal\n\nComet's native `CAST(string AS DECIMAL)` implementation matches Apache Spark's behavior,\nincluding:\n\n- Leading and trailing ASCII whitespace is trimmed before parsing.\n- Null bytes (`\\u0000`) at the start or end of a string are trimmed, matching Spark's\n `UTF8String` behavior. Null bytes embedded in the middle of a string produce `NULL`.\n- Fullwidth Unicode digits (U+FF10–U+FF19, e.g. `１２３.４５`) are treated as their ASCII\n equivalents, so `CAST('１２３.４５' AS DECIMAL(10,2))` returns `123.45`.\n- Scientific notation (e.g. `1.23E+5`) is supported.\n- Special values (`inf`, `infinity`, `nan`) produce `NULL`.\n\n### String to Timestamp\n\nComet's native `CAST(string AS TIMESTAMP)` implementation supports all timestamp formats accepted\nby Apache Spark, including ISO 8601 date-time strings, date-only strings, time-only strings\n(`HH:MM:SS`), embedded timezone offsets (e.g. `+07:30`, `GMT-01:00`, `UTC`), named timezone\nsuffixes (e.g. `Europe/Moscow`), and the full Spark timestamp year range\n(-290308 to 294247). Note that `CAST(string AS DATE)` is only compatible for years between\n262143 BC and 262142 AD due to an underlying library limitation.\n\n### Legacy Mode\n\n\n\n\n### Try Mode\n\n\n\n\n### ANSI Mode\n\n\n\n\nSee the [tracking issue](https://github.com/apache/datafusion-comet/issues/286) for more details.\n" }, { "path": "docs/source/user-guide/latest/configs.md", "content": "\n\n# Comet Configuration Settings\n\nComet provides the following configuration settings.\n\n## Scan Configuration Settings\n\n\n\n\n## Parquet Reader Configuration Settings\n\n\n\n\n## Query Execution Settings\n\n\n\n\n## Viewing Explain Plan & Fallback Reasons\n\nThese settings can be used to determine which parts of the plan are accelerated by Comet and to see why some parts of the plan could not be supported by Comet.\n\n\n\n\n## Shuffle Configuration Settings\n\n\n\n\n## Memory & Tuning Configuration Settings\n\n\n\n\n## Development & Testing Settings\n\n\n\n\n## Enabling or Disabling Individual Operators\n\n\n\n\n## Enabling or Disabling Individual Scalar Expressions\n\n\n\n\n## Enabling or Disabling Individual Aggregate Expressions\n\n\n\n" }, { "path": "docs/source/user-guide/latest/datasources.md", "content": "\n\n# Supported Spark Data Sources\n\n## File Formats\n\n### Parquet\n\nWhen `spark.comet.scan.enabled` is enabled, Parquet scans will be performed natively by Comet if all data types\nin the schema are supported. When this option is not enabled, the scan will fall back to Spark. In this case,\nenabling `spark.comet.convert.parquet.enabled` will immediately convert the data into Arrow format, allowing native\nexecution to happen after that, but the process may not be efficient.\n\n### Apache Iceberg\n\nComet accelerates Iceberg scans of Parquet files. See the [Iceberg Guide] for more information.\n\n[Iceberg Guide]: iceberg.md\n\n### CSV\n\nComet provides experimental native CSV scan support. When `spark.comet.scan.csv.v2.enabled` is enabled, CSV files\nare read natively for improved performance. This feature is experimental and performance benefits are\nworkload-dependent.\n\nAlternatively, when `spark.comet.convert.csv.enabled` is enabled, data from Spark's CSV reader is immediately\nconverted into Arrow format, allowing native execution to happen after that.\n\n### JSON\n\nComet does not provide native JSON scan, but when `spark.comet.convert.json.enabled` is enabled, data is immediately\nconverted into Arrow format, allowing native execution to happen after that.\n\n## Data Catalogs\n\n### Apache Iceberg\n\nSee the dedicated [Comet and Iceberg Guide](iceberg.md).\n\n## Supported Storages\n\nComet supports most standard storage systems, such as local file system and object storage.\n\n### HDFS\n\nApache DataFusion Comet native reader seamlessly scans files from remote HDFS for [supported formats](#supported-spark-data-sources)\n\n### Using experimental native DataFusion reader\n\nUnlike to native Comet reader the Datafusion reader fully supports nested types processing. This reader is currently experimental only\n\nTo build Comet with native DataFusion reader and remote HDFS support it is required to have a JDK installed\n\nExample:\nBuild a Comet for `spark-3.5` provide a JDK path in `JAVA_HOME`\nProvide the JRE linker path in `RUSTFLAGS`, the path can vary depending on the system. Typically JRE linker is a part of installed JDK\n\n```shell\nexport JAVA_HOME=\"/opt/homebrew/opt/openjdk@11\"\nmake release PROFILES=\"-Pspark-3.5\" COMET_FEATURES=hdfs RUSTFLAGS=\"-L $JAVA_HOME/libexec/openjdk.jdk/Contents/Home/lib/server\"\n```\n\nStart Comet with experimental reader and HDFS support as [described](installation.md/#run-spark-shell-with-comet-enabled)\nand add additional parameters\n\n```shell\n--conf spark.comet.scan.impl=native_datafusion \\\n--conf spark.hadoop.fs.defaultFS=\"hdfs://namenode:9000\" \\\n--conf spark.hadoop.dfs.client.use.datanode.hostname = true \\\n--conf dfs.client.use.datanode.hostname = true\n```\n\nQuery a struct type from Remote HDFS\n\n```shell\nspark.read.parquet(\"hdfs://namenode:9000/user/data\").show(false)\n\nroot\n |-- id: integer (nullable = true)\n |-- first_name: string (nullable = true)\n |-- personal_info: struct (nullable = true)\n | |-- firstName: string (nullable = true)\n | |-- lastName: string (nullable = true)\n | |-- ageInYears: integer (nullable = true)\n\n25/01/30 16:50:43 INFO core/src/lib.rs: Comet native library version $COMET_VERSION initialized\n== Physical Plan ==\n* CometColumnarToRow (2)\n+- CometNativeScan: (1)\n\n\n(1) CometNativeScan:\nOutput [3]: [id#0, first_name#1, personal_info#4]\nArguments: [id#0, first_name#1, personal_info#4]\n\n(2) CometColumnarToRow [codegen id : 1]\nInput [3]: [id#0, first_name#1, personal_info#4]\n\n\n25/01/30 16:50:44 INFO fs-hdfs-0.1.12/src/hdfs.rs: Connecting to Namenode (hdfs://namenode:9000)\n+---+----------+-----------------+\n|id |first_name|personal_info |\n+---+----------+-----------------+\n|2 |Jane |{Jane, Smith, 34}|\n|1 |John |{John, Doe, 28} |\n+---+----------+-----------------+\n\n\n\n```\n\nVerify the native scan type should be `CometNativeScan`.\n\nMore on [HDFS Reader](https://github.com/apache/datafusion-comet/blob/main/native/hdfs/README.md)\n\n### Local HDFS development\n\n- Configure local machine network. Add hostname to `/etc/hosts`\n\n```shell\n127.0.0.1\tlocalhost namenode datanode1 datanode2 datanode3\n::1 localhost namenode datanode1 datanode2 datanode3\n```\n\n- Start local HDFS cluster, 3 datanodes, namenode url is `namenode:9000`\n\n```shell\ndocker compose -f kube/local/hdfs-docker-compose.yml up\n```\n\n- Check the local namenode is up and running on `http://localhost:9870/dfshealth.html#tab-overview`\n- Build a project with HDFS support\n\n```shell\nJAVA_HOME=\"/opt/homebrew/opt/openjdk@11\" make release PROFILES=\"-Pspark-3.5\" COMET_FEATURES=hdfs RUSTFLAGS=\"-L /opt/homebrew/opt/openjdk@11/libexec/openjdk.jdk/Contents/Home/lib/server\"\n```\n\n- Run local test\n\n```scala\n\n withSQLConf(\n CometConf.COMET_ENABLED.key -> \"true\",\n CometConf.COMET_EXEC_ENABLED.key -> \"true\",\n CometConf.COMET_NATIVE_SCAN_IMPL.key -> CometConf.SCAN_NATIVE_DATAFUSION,\n SQLConf.USE_V1_SOURCE_LIST.key -> \"parquet\",\n \"fs.defaultFS\" -> \"hdfs://namenode:9000\",\n \"dfs.client.use.datanode.hostname\" -> \"true\") {\n val df = spark.read.parquet(\"/tmp/2\")\n df.show(false)\n df.explain(\"extended\")\n }\n }\n```\n\nOr use `spark-shell` with HDFS support as described [above](#using-experimental-native-datafusion-reader)\n\n## S3\n\n#### Root CA Certificates\n\nOne major difference between Spark and Comet is the mechanism for discovering Root\nCA Certificates. Spark uses the JVM to read CA Certificates from the Java Trust Store, but native Comet\nscans use system Root CA Certificates (typically stored\nin `/etc/ssl/certs` on Linux). These scans will not be able to interact with S3 if the Root CA Certificates are not\ninstalled.\n\n#### Supported Credential Providers\n\nAWS credential providers can be configured using the `fs.s3a.aws.credentials.provider` configuration. The following table shows the supported credential providers and their configuration options:\n\n| Credential provider | Description | Supported Options |\n| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |\n| `org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider` | Access S3 using access key and secret key | `fs.s3a.access.key`, `fs.s3a.secret.key` |\n| `org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider` | Access S3 using temporary credentials | `fs.s3a.access.key`, `fs.s3a.secret.key`, `fs.s3a.session.token` |\n| `org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider` | Access S3 using AWS STS assume role | `fs.s3a.assumed.role.arn`, `fs.s3a.assumed.role.session.name` (optional), `fs.s3a.assumed.role.credentials.provider` (optional) |\n| `org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider` | Access S3 using EC2 instance profile or ECS task credentials (tries ECS first, then IMDS) | None (auto-detected) |\n| `org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider`
`com.amazonaws.auth.AnonymousAWSCredentials`
`software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider` | Access S3 without authentication (public buckets only) | None |\n| `com.amazonaws.auth.EnvironmentVariableCredentialsProvider`
`software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider` | Load credentials from environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`) | None |\n| `com.amazonaws.auth.InstanceProfileCredentialsProvider`
`software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider` | Access S3 using EC2 instance metadata service (IMDS) | None |\n| `com.amazonaws.auth.ContainerCredentialsProvider`
`software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider`
`com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper` | Access S3 using ECS task credentials | None |\n| `com.amazonaws.auth.WebIdentityTokenCredentialsProvider`
`software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider` | Authenticate using web identity token file | None |\n\nMultiple credential providers can be specified in a comma-separated list using the `fs.s3a.aws.credentials.provider` configuration, just as Hadoop AWS supports. If `fs.s3a.aws.credentials.provider` is not configured, Hadoop S3A's default credential provider chain will be used. All configuration options also support bucket-specific overrides using the pattern `fs.s3a.bucket.{bucket-name}.{option}`.\n" }, { "path": "docs/source/user-guide/latest/datatypes.md", "content": "\n\n# Supported Spark Data Types\n\nComet supports the following Spark data types. Refer to the [Comet Compatibility Guide] for information about data\ntype support in scans and other operators.\n\n[Comet Compatibility Guide]: compatibility.md\n\n\n\n| Data Type |\n| ------------ |\n| Null |\n| Boolean |\n| Byte |\n| Short |\n| Integer |\n| Long |\n| Float |\n| Double |\n| Decimal |\n| String |\n| Binary |\n| Date |\n| Timestamp |\n| TimestampNTZ |\n| Struct |\n| Array |\n| Map |\n" }, { "path": "docs/source/user-guide/latest/expressions.md", "content": "\n\n# Supported Spark Expressions\n\nComet supports the following Spark expressions. Expressions that are marked as Spark-compatible will either run\nnatively in Comet and provide the same results as Spark, or will fall back to Spark for cases that would not\nbe compatible.\n\nAll expressions are enabled by default, but most can be disabled by setting\n`spark.comet.expression.EXPRNAME.enabled=false`, where `EXPRNAME` is the expression name as specified in\nthe following tables, such as `Length`, or `StartsWith`. See the [Comet Configuration Guide] for a full list\nof expressions that be disabled.\n\nExpressions that are not Spark-compatible will fall back to Spark by default and can be enabled by setting\n`spark.comet.expression.EXPRNAME.allowIncompatible=true`.\n\n## Conditional Expressions\n\n| Expression | SQL | Spark-Compatible? |\n| ---------- | ------------------------------------------- | ----------------- |\n| CaseWhen | `CASE WHEN expr THEN expr ELSE expr END` | Yes |\n| If | `IF(predicate_expr, true_expr, false_expr)` | Yes |\n\n## Predicate Expressions\n\n| Expression | SQL | Spark-Compatible? |\n| ------------------ | ------------- | ----------------- |\n| And | `AND` | Yes |\n| EqualTo | `=` | Yes |\n| EqualNullSafe | `<=>` | Yes |\n| GreaterThan | `>` | Yes |\n| GreaterThanOrEqual | `>=` | Yes |\n| LessThan | `<` | Yes |\n| LessThanOrEqual | `<=` | Yes |\n| In | `IN` | Yes |\n| IsNotNull | `IS NOT NULL` | Yes |\n| IsNull | `IS NULL` | Yes |\n| InSet | `IN (...)` | Yes |\n| Not | `NOT` | Yes |\n| Or | `OR` | Yes |\n\n## String Functions\n\n| Expression | Spark-Compatible? | Compatibility Notes |\n| --------------- | ----------------- | ---------------------------------------------------------------------------------------------------------- |\n| Ascii | Yes | |\n| BitLength | Yes | |\n| Chr | Yes | |\n| Concat | Yes | Only string inputs are supported |\n| ConcatWs | Yes | |\n| Contains | Yes | |\n| EndsWith | Yes | |\n| InitCap | No | Behavior is different in some cases, such as hyphenated names. |\n| Left | Yes | Length argument must be a literal value |\n| Length | Yes | |\n| Like | Yes | |\n| Lower | No | Results can vary depending on locale and character set. Requires `spark.comet.caseConversion.enabled=true` |\n| OctetLength | Yes | |\n| Reverse | Yes | |\n| RLike | No | Uses Rust regexp engine, which has different behavior to Java regexp engine |\n| StartsWith | Yes | |\n| StringInstr | Yes | |\n| StringRepeat | Yes | Negative argument for number of times to repeat causes exception |\n| StringReplace | Yes | |\n| StringLPad | Yes | |\n| StringRPad | Yes | |\n| StringSpace | Yes | |\n| StringTranslate | Yes | |\n| StringTrim | Yes | |\n| StringTrimBoth | Yes | |\n| StringTrimLeft | Yes | |\n| StringTrimRight | Yes | |\n| Substring | Yes | |\n| Upper | No | Results can vary depending on locale and character set. Requires `spark.comet.caseConversion.enabled=true` |\n\n## JSON Functions\n\n| Expression | Spark-Compatible? | Compatibility Notes |\n| ------------- | ----------------- | --------------------------------------------------------------------------------------------- |\n| GetJsonObject | No | Spark allows single-quoted JSON and unescaped control characters which Comet does not support |\n\n## Date/Time Functions\n\n| Expression | SQL | Spark-Compatible? | Compatibility Notes |\n| -------------- | ---------------------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------- |\n| DateAdd | `date_add` | Yes | |\n| DateDiff | `datediff` | Yes | |\n| DateFormat | `date_format` | Yes | Partial support. Only specific format patterns are supported. |\n| DateSub | `date_sub` | Yes | |\n| DatePart | `date_part(field, source)` | Yes | Supported values of `field`: `year`/`month`/`week`/`day`/`dayofweek`/`dayofweek_iso`/`doy`/`quarter`/`hour`/`minute` |\n| Days | `days` | Yes | V2 partition transform. Supports DateType and TimestampType inputs. |\n| Extract | `extract(field FROM source)` | Yes | Supported values of `field`: `year`/`month`/`week`/`day`/`dayofweek`/`dayofweek_iso`/`doy`/`quarter`/`hour`/`minute` |\n| FromUnixTime | `from_unixtime` | No | Does not support format, supports only -8334601211038 <= sec <= 8210266876799 |\n| Hour | `hour` | No | Incorrectly applies timezone conversion to TimestampNTZ inputs ([#3180](https://github.com/apache/datafusion-comet/issues/3180)) |\n| LastDay | `last_day` | Yes | |\n| Minute | `minute` | No | Incorrectly applies timezone conversion to TimestampNTZ inputs ([#3180](https://github.com/apache/datafusion-comet/issues/3180)) |\n| Second | `second` | No | Incorrectly applies timezone conversion to TimestampNTZ inputs ([#3180](https://github.com/apache/datafusion-comet/issues/3180)) |\n| TruncDate | `trunc` | Yes | |\n| TruncTimestamp | `date_trunc` | No | Incorrect results in non-UTC timezones ([#2649](https://github.com/apache/datafusion-comet/issues/2649)) |\n| UnixDate | `unix_date` | Yes | |\n| UnixTimestamp | `unix_timestamp` | Yes | |\n| Year | `year` | Yes | |\n| Month | `month` | Yes | |\n| DayOfMonth | `day`/`dayofmonth` | Yes | |\n| DayOfWeek | `dayofweek` | Yes | |\n| WeekDay | `weekday` | Yes | |\n| DayOfYear | `dayofyear` | Yes | |\n| WeekOfYear | `weekofyear` | Yes | |\n| Quarter | `quarter` | Yes | |\n\n## Math Expressions\n\n| Expression | SQL | Spark-Compatible? | Compatibility Notes |\n| -------------- | --------- | ----------------- | --------------------------------- |\n| Abs | `abs` | Yes | |\n| Acos | `acos` | Yes | |\n| Add | `+` | Yes | |\n| Asin | `asin` | Yes | |\n| Atan | `atan` | Yes | |\n| Atan2 | `atan2` | Yes | |\n| BRound | `bround` | Yes | |\n| Ceil | `ceil` | Yes | |\n| Cos | `cos` | Yes | |\n| Cosh | `cosh` | Yes | |\n| Cot | `cot` | Yes | |\n| Divide | `/` | Yes | |\n| Exp | `exp` | Yes | |\n| Expm1 | `expm1` | Yes | |\n| Floor | `floor` | Yes | |\n| Hex | `hex` | Yes | |\n| IntegralDivide | `div` | Yes | |\n| IsNaN | `isnan` | Yes | |\n| Log | `log` | Yes | |\n| Log2 | `log2` | Yes | |\n| Log10 | `log10` | Yes | |\n| Multiply | `*` | Yes | |\n| Pow | `power` | Yes | |\n| Rand | `rand` | Yes | |\n| Randn | `randn` | Yes | |\n| Remainder | `%` | Yes | |\n| Round | `round` | Yes | |\n| Signum | `signum` | Yes | |\n| Sin | `sin` | Yes | |\n| Sinh | `sinh` | Yes | |\n| Sqrt | `sqrt` | Yes | |\n| Subtract | `-` | Yes | |\n| Tan | `tan` | Yes | |\n| Tanh | `tanh` | Yes | |\n| TryAdd | `try_add` | Yes | Only integer inputs are supported |\n| TryDivide | `try_div` | Yes | Only integer inputs are supported |\n| TryMultiply | `try_mul` | Yes | Only integer inputs are supported |\n| TrySubtract | `try_sub` | Yes | Only integer inputs are supported |\n| UnaryMinus | `-` | Yes | |\n| Unhex | `unhex` | Yes | |\n\n## Hashing Functions\n\n| Expression | Spark-Compatible? |\n| ----------- | ----------------- |\n| Md5 | Yes |\n| Murmur3Hash | Yes |\n| Sha1 | Yes |\n| Sha2 | Yes |\n| XxHash64 | Yes |\n\n## Bitwise Expressions\n\n| Expression | SQL | Spark-Compatible? |\n| ------------ | ---- | ----------------- |\n| BitwiseAnd | `&` | Yes |\n| BitwiseCount | | Yes |\n| BitwiseGet | | Yes |\n| BitwiseOr | `\\|` | Yes |\n| BitwiseNot | `~` | Yes |\n| BitwiseXor | `^` | Yes |\n| ShiftLeft | `<<` | Yes |\n| ShiftRight | `>>` | Yes |\n\n## Aggregate Expressions\n\n| Expression | SQL | Spark-Compatible? | Compatibility Notes |\n| ------------- | ---------- | ------------------------- | ---------------------------------------------------------------- |\n| Average | | Yes, except for ANSI mode | |\n| BitAndAgg | | Yes | |\n| BitOrAgg | | Yes | |\n| BitXorAgg | | Yes | |\n| BoolAnd | `bool_and` | Yes | |\n| BoolOr | `bool_or` | Yes | |\n| Corr | | Yes | |\n| Count | | Yes | |\n| CovPopulation | | Yes | |\n| CovSample | | Yes | |\n| First | | No | This function is not deterministic. Results may not match Spark. |\n| Last | | No | This function is not deterministic. Results may not match Spark. |\n| Max | | Yes | |\n| Min | | Yes | |\n| StddevPop | | Yes | |\n| StddevSamp | | Yes | |\n| Sum | | Yes, except for ANSI mode | |\n| VariancePop | | Yes | |\n| VarianceSamp | | Yes | |\n\n## Window Functions\n\n```{warning}\nWindow support is disabled by default due to known correctness issues. Tracking issue: [#2721](https://github.com/apache/datafusion-comet/issues/2721).\n```\n\nComet supports using the following aggregate functions within window contexts with PARTITION BY and ORDER BY clauses.\n\n| Expression | Spark-Compatible? | Compatibility Notes |\n| ---------- | ----------------- | ------------------- |\n| Count | Yes | |\n| Max | Yes | |\n| Min | Yes | |\n| Sum | Yes | |\n\n**Note:** Dedicated window functions such as `rank`, `dense_rank`, `row_number`, `lag`, `lead`, `ntile`, `cume_dist`, `percent_rank`, and `nth_value` are not currently supported and will fall back to Spark.\n\n## Array Expressions\n\n| Expression | Spark-Compatible? | Compatibility Notes |\n| -------------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| ArrayAppend | Yes | |\n| ArrayCompact | No | |\n| ArrayContains | Yes | |\n| ArrayDistinct | Yes | |\n| ArrayExcept | No | |\n| ArrayFilter | Yes | Only supports case where function is `IsNotNull` |\n| ArrayInsert | No | |\n| ArrayIntersect | No | |\n| ArrayJoin | No | |\n| ArrayMax | Yes | |\n| ArrayMin | Yes | |\n| ArrayRemove | Yes | |\n| ArrayRepeat | No | |\n| ArrayUnion | No | Behaves differently than spark. Comet sorts the input arrays before performing the union, while Spark preserves the order of the first array and appends unique elements from the second. |\n| ArraysOverlap | No | |\n| CreateArray | Yes | |\n| ElementAt | Yes | Input must be an array. Map inputs are not supported. |\n| Flatten | Yes | |\n| GetArrayItem | Yes | |\n\n## Map Expressions\n\n| Expression | Spark-Compatible? |\n| ------------- | ----------------- |\n| GetMapValue | Yes |\n| MapKeys | Yes |\n| MapEntries | Yes |\n| MapValues | Yes |\n| MapFromArrays | Yes |\n\n## Struct Expressions\n\n| Expression | Spark-Compatible? | Compatibility Notes |\n| -------------------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------- |\n| CreateNamedStruct | Yes | |\n| GetArrayStructFields | Yes | |\n| GetStructField | Yes | |\n| JsonToStructs | No | Partial support. Requires explicit schema. |\n| StructsToJson | No | Does not support Infinity/-Infinity for numeric types ([#3016](https://github.com/apache/datafusion-comet/issues/3016)) |\n\n## Conversion Expressions\n\n| Expression | Spark-Compatible | Compatibility Notes |\n| ---------- | ------------------------ | ------------------------------------------------------------------------------------------- |\n| Cast | Depends on specific cast | See the [Comet Compatibility Guide] for list of supported cast expressions and known issues |\n\n## SortOrder\n\n| Expression | Spark-Compatible? | Compatibility Notes |\n| ---------- | ----------------- | ------------------- |\n| NullsFirst | Yes | |\n| NullsLast | Yes | |\n| Ascending | Yes | |\n| Descending | Yes | |\n\n## Other\n\n| Expression | Spark-Compatible? | Compatibility Notes |\n| ---------------------------- | ----------------- | --------------------------------------------------------------------------- |\n| Alias | Yes | |\n| AttributeReference | Yes | |\n| BloomFilterMightContain | Yes | |\n| Coalesce | Yes | |\n| CheckOverflow | Yes | |\n| KnownFloatingPointNormalized | Yes | |\n| Literal | Yes | |\n| MakeDecimal | Yes | |\n| MonotonicallyIncreasingID | Yes | |\n| NormalizeNaNAndZero | Yes | |\n| PromotePrecision | Yes | |\n| RegExpReplace | No | Uses Rust regexp engine, which has different behavior to Java regexp engine |\n| ScalarSubquery | Yes | |\n| SparkPartitionID | Yes | |\n| ToPrettyString | Yes | |\n| UnscaledValue | Yes | |\n\n[Comet Configuration Guide]: configs.md\n[Comet Compatibility Guide]: compatibility.md\n" }, { "path": "docs/source/user-guide/latest/iceberg.md", "content": "\n\n# Accelerating Apache Iceberg Parquet Scans using Comet\n\n## Native Reader\n\nComet's native Iceberg reader relies on reflection to extract `FileScanTask`s from Iceberg, which are\nthen serialized to Comet's native execution engine (see\n[PR #2528](https://github.com/apache/datafusion-comet/pull/2528)).\n\nThe example below uses Spark's package downloader to retrieve Comet 0.14.0 and Iceberg\n1.8.1, but Comet has been tested with Iceberg 1.5, 1.7, 1.8, 1.9, and 1.10. The native Iceberg\nreader is enabled by default. To disable it, set `spark.comet.scan.icebergNative.enabled=false`.\n\n```shell\n$SPARK_HOME/bin/spark-shell \\\n --packages org.apache.datafusion:comet-spark-spark3.5_2.12:0.14.0,org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1,org.apache.iceberg:iceberg-core:1.8.1 \\\n --repositories https://repo1.maven.org/maven2/ \\\n --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \\\n --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkCatalog \\\n --conf spark.sql.catalog.spark_catalog.type=hadoop \\\n --conf spark.sql.catalog.spark_catalog.warehouse=/tmp/warehouse \\\n --conf spark.plugins=org.apache.spark.CometPlugin \\\n --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \\\n --conf spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions \\\n --conf spark.comet.explainFallback.enabled=true \\\n --conf spark.memory.offHeap.enabled=true \\\n --conf spark.memory.offHeap.size=2g\n```\n\n### Tuning\n\nComet’s native Iceberg reader supports fetching multiple files in parallel to hide I/O latency with the\nconfig `spark.comet.scan.icebergNative.dataFileConcurrencyLimit`. This value defaults to 1 to\nmaintain test behavior on Iceberg Java tests without `ORDER BY` clauses, but we suggest increasing it to\nvalues between 2 and 8 based on your workload.\n\n### Supported features\n\nThe native Iceberg reader supports the following features:\n\n**Table specifications:**\n\n- Iceberg table spec v1 and v2 (v3 will fall back to Spark)\n\n**Schema and data types:**\n\n- All primitive types including UUID\n- Complex types: arrays, maps, and structs\n- Schema evolution (adding and dropping columns)\n\n**Time travel and branching:**\n\n- `VERSION AS OF` queries to read historical snapshots\n- Branch reads for accessing named branches\n\n**Delete handling (Merge-On-Read tables):**\n\n- Positional deletes\n- Equality deletes\n- Mixed delete types\n\n**Filter pushdown:**\n\n- Equality and comparison predicates (`=`, `!=`, `>`, `>=`, `<`, `<=`)\n- Logical operators (`AND`, `OR`)\n- NULL checks (`IS NULL`, `IS NOT NULL`)\n- `IN` and `NOT IN` list operations\n- `BETWEEN` operations\n\n**Partitioning:**\n\n- Standard partitioning with partition pruning\n- Date partitioning with `days()` transform\n- Bucket partitioning\n- Truncate transform\n- Hour transform\n\n**Storage:**\n\n- Local filesystem\n- Hadoop Distributed File System (HDFS)\n- S3-compatible storage (AWS S3, MinIO)\n\n### REST Catalog\n\nComet's native Iceberg reader also supports REST catalogs. The following example shows how to\nconfigure Spark to use a REST catalog with Comet's native Iceberg scan:\n\n```shell\n$SPARK_HOME/bin/spark-shell \\\n --packages org.apache.datafusion:comet-spark-spark3.5_2.12:0.14.0,org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1,org.apache.iceberg:iceberg-core:1.8.1 \\\n --repositories https://repo1.maven.org/maven2/ \\\n --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \\\n --conf spark.sql.catalog.rest_cat=org.apache.iceberg.spark.SparkCatalog \\\n --conf spark.sql.catalog.rest_cat.catalog-impl=org.apache.iceberg.rest.RESTCatalog \\\n --conf spark.sql.catalog.rest_cat.uri=http://localhost:8181 \\\n --conf spark.sql.catalog.rest_cat.warehouse=/tmp/warehouse \\\n --conf spark.plugins=org.apache.spark.CometPlugin \\\n --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \\\n --conf spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions \\\n --conf spark.comet.explainFallback.enabled=true \\\n --conf spark.memory.offHeap.enabled=true \\\n --conf spark.memory.offHeap.size=2g\n```\n\nNote that REST catalogs require explicit namespace creation before creating tables:\n\n```scala\nscala> spark.sql(\"CREATE NAMESPACE rest_cat.db\")\nscala> spark.sql(\"CREATE TABLE rest_cat.db.test_table (id INT, name STRING) USING iceberg\")\nscala> spark.sql(\"INSERT INTO rest_cat.db.test_table VALUES (1, 'Alice'), (2, 'Bob')\")\nscala> spark.sql(\"SELECT * FROM rest_cat.db.test_table\").show()\n```\n\n### Current limitations\n\nThe following scenarios will fall back to Spark's native Iceberg reader:\n\n- Iceberg table spec v3 scans\n- Iceberg writes (reads are accelerated, writes use Spark)\n- Tables backed by Avro or ORC data files (only Parquet is accelerated)\n- Tables partitioned on `BINARY` or `DECIMAL` (with precision >28) columns\n- Scans with residual filters using `truncate`, `bucket`, `year`, `month`, `day`, or `hour`\n transform functions (partition pruning still works, but row-level filtering of these\n transforms falls back)\n" }, { "path": "docs/source/user-guide/latest/index.rst", "content": ".. Licensed to the Apache Software Foundation (ASF) under one\n.. or more contributor license agreements. See the NOTICE file\n.. distributed with this work for additional information\n.. regarding copyright ownership. The ASF licenses this file\n.. to you under the Apache License, Version 2.0 (the\n.. \"License\"); you may not use this file except in compliance\n.. with the License. You may obtain a copy of the License at\n\n.. http://www.apache.org/licenses/LICENSE-2.0\n\n.. Unless required by applicable law or agreed to in writing,\n.. software distributed under the License is distributed on an\n.. \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n.. KIND, either express or implied. See the License for the\n.. specific language governing permissions and limitations\n.. under the License.\n\n.. image:: /_static/images/DataFusionComet-Logo-Light.png\n :alt: DataFusion Comet Logo\n\n================================\nComet $COMET_VERSION User Guide\n================================\n\n.. _toc.user-guide-links-$COMET_VERSION:\n.. toctree::\n :maxdepth: 1\n :caption: Comet $COMET_VERSION User Guide\n\n Installing Comet \n Building From Source \n Supported Data Sources \n Supported Data Types \n Supported Operators \n Supported Expressions \n Configuration Settings \n Compatibility Guide \n Tuning Guide \n Metrics Guide \n Iceberg Guide \n Kubernetes Guide \n" }, { "path": "docs/source/user-guide/latest/installation.md", "content": "\n\n# Installing DataFusion Comet\n\n## Prerequisites\n\nMake sure the following requirements are met and software installed on your machine.\n\n### Supported Operating Systems\n\n- Linux\n- Apple macOS (Intel and Apple Silicon)\n\n### Supported Spark Versions\n\nComet $COMET_VERSION supports the following versions of Apache Spark.\n\nWe recommend only using Comet with Spark versions where we currently have both Comet and Spark tests enabled in CI.\nOther versions may work well enough for development and evaluation purposes.\n\n| Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark SQL Tests in CI |\n| ------------- | ------------ | ------------- | ----------------- | --------------------- |\n| 3.4.3 | 11/17 | 2.12/2.13 | Yes | Yes |\n| 3.5.5 | 11/17 | 2.12/2.13 | Yes | No |\n| 3.5.6 | 11/17 | 2.12/2.13 | Yes | No |\n| 3.5.7 | 11/17 | 2.12/2.13 | Yes | Yes |\n| 3.5.8 | 11/17 | 2.12/2.13 | Yes | Yes |\n\nNote that we do not test the full matrix of supported Java and Scala versions in CI for every Spark version.\n\nExperimental support is provided for the following versions of Apache Spark and is intended for development/testing\nuse only and should not be used in production yet.\n\n| Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark SQL Tests in CI |\n| ------------- | ------------ | ------------- | ----------------- | --------------------- |\n| 4.0.1 | 17 | 2.13 | Yes | Yes |\n\nNote that Comet may not fully work with proprietary forks of Apache Spark such as the Spark versions offered by\nCloud Service Providers.\n\n## Using a Published JAR File\n\nComet jar files are available in [Maven Central](https://central.sonatype.com/namespace/org.apache.datafusion) for amd64 and arm64 architectures for Linux. For Apple macOS, it\nis currently necessary to build from source.\n\nFor performance reasons, published Comet jar files target baseline CPUs available in modern data centers. For example,\nthe amd64 build uses the `x86-64-v3` target that adds CPU instructions (_e.g._, AVX2) common after 2013. Similarly, the\narm64 build uses the `neoverse-n1` target, which is a common baseline for ARM cores found in AWS (Graviton2+), GCP, and\nAzure after 2019. If the Comet library fails for SIGILL (illegal instruction), please open an issue on the GitHub\nrepository describing your environment, and [build from source] for your target architecture.\n\nHere are the direct links for downloading the Comet $COMET_VERSION jar file. Note that these links are not valid if\nyou are viewing the documentation for the latest `SNAPSHOT` release, or for the latest release while it is still going\nthrough the release vote. Release candidate jars can be found [here](https://repository.apache.org/#nexus-search;quick~org.apache.datafusion).\n\n- [Comet plugin for Spark 3.4 / Scala 2.12](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.4_2.12/$COMET_VERSION/comet-spark-spark3.4_2.12-$COMET_VERSION.jar)\n- [Comet plugin for Spark 3.4 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.4_2.13/$COMET_VERSION/comet-spark-spark3.4_2.13-$COMET_VERSION.jar)\n- [Comet plugin for Spark 3.5 / Scala 2.12](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.5_2.12/$COMET_VERSION/comet-spark-spark3.5_2.12-$COMET_VERSION.jar)\n- [Comet plugin for Spark 3.5 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.5_2.13/$COMET_VERSION/comet-spark-spark3.5_2.13-$COMET_VERSION.jar)\n- [Comet plugin for Spark 4.0 / Scala 2.13 (Experimental)](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark4.0_2.13/$COMET_VERSION/comet-spark-spark4.0_2.13-$COMET_VERSION.jar)\n\n## Building from source\n\nRefer to the [Building from source] guide for instructions from building Comet from source, either from official\nsource releases, or from the latest code in the GitHub repository.\n\n[Building from source]: source.md\n\n## Deploying to Kubernetes\n\nSee the [Comet Kubernetes Guide](kubernetes.md) guide.\n\n## Run Spark Shell with Comet enabled\n\nMake sure `SPARK_HOME` points to the same Spark version as Comet was built for.\n\n```shell\nexport COMET_JAR=spark/target/comet-spark-spark3.5_2.12-$COMET_VERSION.jar\n\n$SPARK_HOME/bin/spark-shell \\\n --jars $COMET_JAR \\\n --conf spark.driver.extraClassPath=$COMET_JAR \\\n --conf spark.executor.extraClassPath=$COMET_JAR \\\n --conf spark.plugins=org.apache.spark.CometPlugin \\\n --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \\\n --conf spark.comet.explainFallback.enabled=true \\\n --conf spark.memory.offHeap.enabled=true \\\n --conf spark.memory.offHeap.size=16g\n```\n\n### Verify Comet enabled for Spark SQL query\n\nCreate a test Parquet source\n\n```scala\nscala> (0 until 10).toDF(\"a\").write.mode(\"overwrite\").parquet(\"/tmp/test\")\n```\n\nQuery the data from the test source and check:\n\n- INFO message shows the native Comet library has been initialized.\n- The query plan reflects Comet operators being used for this query instead of Spark ones\n\n```scala\nscala> spark.read.parquet(\"/tmp/test\").createOrReplaceTempView(\"t1\")\nscala> spark.sql(\"select * from t1 where a > 5\").explain\nINFO src/lib.rs: Comet native library initialized\n== Physical Plan ==\n *(1) ColumnarToRow\n +- CometFilter [a#14], (isnotnull(a#14) AND (a#14 > 5))\n +- CometScan parquet [a#14] Batched: true, DataFilters: [isnotnull(a#14), (a#14 > 5)],\n Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/tmp/test], PartitionFilters: [],\n PushedFilters: [IsNotNull(a), GreaterThan(a,5)], ReadSchema: struct\n```\n\nWith the configuration `spark.comet.explainFallback.enabled=true`, Comet will log any reasons that prevent a plan from\nbeing executed natively.\n\n```scala\nscala> Seq(1,2,3,4).toDF(\"a\").write.parquet(\"/tmp/test.parquet\")\nWARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute some parts of this plan natively because:\n - LocalTableScan is not supported\n - WriteFiles is not supported\n - Execute InsertIntoHadoopFsRelationCommand is not supported\n```\n\n## Additional Configuration\n\nDepending on your deployment mode you may also need to set the driver & executor class path(s) to\nexplicitly contain Comet otherwise Spark may use a different class-loader for the Comet components than its internal\ncomponents which will then fail at runtime. For example:\n\n```\n--driver-class-path spark/target/comet-spark-spark3.5_2.12-$COMET_VERSION.jar\n```\n\nSome cluster managers may require additional configuration, see \n\n### Memory tuning\n\nIn addition to Apache Spark memory configuration parameters, Comet introduces additional parameters to configure memory\nallocation for native execution. See [Comet Memory Tuning](./tuning.md) for details.\n" }, { "path": "docs/source/user-guide/latest/kubernetes.md", "content": "\n\n# Comet Kubernetes Support\n\n## Comet Docker Images\n\nRun the following command from the root of this repository to build the Comet Docker image, or use a [published\nDocker image](https://hub.docker.com/r/apache/datafusion-comet).\n\n```shell\ndocker build -t apache/datafusion-comet -f kube/Dockerfile .\n```\n\n## Example Spark Submit\n\nThe exact syntax will vary depending on the Kubernetes distribution, but an example `spark-submit` command can be\nfound [here](https://github.com/apache/datafusion-comet/tree/main/benchmarks).\n\n## Helm chart\n\nInstall helm Spark operator for Kubernetes\n\n```bash\n# Add the Helm repository\nhelm repo add spark-operator https://kubeflow.github.io/spark-operator\nhelm repo update\n\n# Install the operator into the spark-operator namespace and wait for deployments to be ready\nhelm install spark-operator spark-operator/spark-operator --namespace spark-operator --create-namespace --wait\n```\n\nCheck the operator is deployed\n\n```bash\nhelm status --namespace spark-operator spark-operator\n\nNAME: my-release\nNAMESPACE: spark-operator\nSTATUS: deployed\nREVISION: 1\nTEST SUITE: None\n```\n\nCreate example Spark application file `spark-pi.yaml`\n\n```bash\napiVersion: sparkoperator.k8s.io/v1beta2\nkind: SparkApplication\nmetadata:\n name: spark-pi\n namespace: default\nspec:\n type: Scala\n mode: cluster\n image: apache/datafusion-comet:0.7.0-spark3.5.5-scala2.12-java11\n imagePullPolicy: IfNotPresent\n mainClass: org.apache.spark.examples.SparkPi\n mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.5.jar\n sparkConf:\n \"spark.executor.extraClassPath\": \"/opt/spark/jars/comet-spark-spark3.5_2.12-0.7.0.jar\"\n \"spark.driver.extraClassPath\": \"/opt/spark/jars/comet-spark-spark3.5_2.12-0.7.0.jar\"\n \"spark.plugins\": \"org.apache.spark.CometPlugin\"\n \"spark.comet.enabled\": \"true\"\n \"spark.comet.exec.enabled\": \"true\"\n \"spark.comet.exec.shuffle.enabled\": \"true\"\n \"spark.comet.exec.shuffle.mode\": \"auto\"\n \"spark.shuffle.manager\": \"org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager\"\n sparkVersion: 3.5.6\n driver:\n labels:\n version: 3.5.6\n cores: 1\n coreLimit: 1200m\n memory: 512m\n serviceAccount: spark-operator-spark\n executor:\n labels:\n version: 3.5.6\n instances: 1\n cores: 1\n coreLimit: 1200m\n memory: 512m\n```\n\nRefer to [Comet builds](#comet-docker-images)\n\nRun Apache Spark application with Comet enabled\n\n```bash\nkubectl apply -f spark-pi.yaml\nsparkapplication.sparkoperator.k8s.io/spark-pi created\n```\n\nCheck application status\n\n```bash\nkubectl get sparkapp spark-pi\n\nNAME STATUS ATTEMPTS START FINISH AGE\nspark-pi RUNNING 1 2025-03-18T21:19:48Z 65s\n```\n\nTo check more runtime details\n\n```bash\nkubectl describe sparkapplication spark-pi\n\n....\nEvents:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n Normal SparkApplicationSubmitted 8m15s spark-application-controller SparkApplication spark-pi was submitted successfully\n Normal SparkDriverRunning 7m18s spark-application-controller Driver spark-pi-driver is running\n Normal SparkExecutorPending 7m11s spark-application-controller Executor [spark-pi-68732195ab217303-exec-1] is pending\n Normal SparkExecutorRunning 7m10s spark-application-controller Executor [spark-pi-68732195ab217303-exec-1] is running\n Normal SparkExecutorCompleted 7m5s spark-application-controller Executor [spark-pi-68732195ab217303-exec-1] completed\n Normal SparkDriverCompleted 7m4s spark-application-controller Driver spark-pi-driver completed\n\n```\n\nGet Driver Logs\n\n```bash\nkubectl logs spark-pi-driver\n```\n\nMore info on [Kube Spark operator](https://www.kubeflow.org/docs/components/spark-operator/getting-started/)\n" }, { "path": "docs/source/user-guide/latest/metrics.md", "content": "\n\n# Comet Metrics\n\n## Spark SQL Metrics\n\nSet `spark.comet.metrics.detailed=true` to see all available Comet metrics.\n\n### CometScanExec\n\n| Metric | Description |\n| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `scan time` | Total time to scan a Parquet file. This is not comparable to the same metric in Spark because Comet's scan metric is more accurate. Although both Comet and Spark measure the time in nanoseconds, Spark rounds this time to the nearest millisecond per batch and Comet does not. |\n\n### Exchange\n\nComet adds some additional metrics:\n\n| Metric | Description |\n| ------------------------------- | ------------------------------------------------------------- |\n| `native shuffle time` | Total time in native code excluding any child operators. |\n| `repartition time` | Time to repartition batches. |\n| `memory pool time` | Time interacting with memory pool. |\n| `encoding and compression time` | Time to encode batches in IPC format and compress using ZSTD. |\n\n## Native Metrics\n\nSetting `spark.comet.explain.native.enabled=true` will cause native plans to be logged in each executor. Metrics are\nlogged for each native plan (and there is one plan per task, so this is very verbose).\n\nHere is a guide to some of the native metrics.\n\n### ScanExec\n\n| Metric | Description |\n| ----------------- | --------------------------------------------------------------------------------------------------- |\n| `elapsed_compute` | Total time spent in this operator, fetching batches from a JVM iterator. |\n| `jvm_fetch_time` | Time spent in the JVM fetching input batches to be read by this `ScanExec` instance. |\n| `arrow_ffi_time` | Time spent using Arrow FFI to create Arrow batches from the memory addresses returned from the JVM. |\n\n### ShuffleWriterExec\n\n| Metric | Description |\n| ----------------- | ------------------------------------------------------------- |\n| `elapsed_compute` | Total time excluding any child operators. |\n| `repart_time` | Time to repartition batches. |\n| `ipc_time` | Time to encode batches in IPC format and compress using ZSTD. |\n| `mempool_time` | Time interacting with memory pool. |\n| `write_time` | Time spent writing bytes to disk. |\n" }, { "path": "docs/source/user-guide/latest/operators.md", "content": "\n\n# Supported Spark Operators\n\nThe following Spark operators are currently replaced with native versions. Query stages that contain any operators\nnot supported by Comet will fall back to regular Spark execution.\n\n| Operator | Spark-Compatible? | Compatibility Notes |\n| --------------------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------ |\n| BatchScanExec | Yes | Supports Parquet files and Apache Iceberg Parquet scans. See the [Comet Compatibility Guide] for more information. |\n| BroadcastExchangeExec | Yes | |\n| BroadcastHashJoinExec | Yes | |\n| ExpandExec | Yes | |\n| FileSourceScanExec | Yes | Supports Parquet files. See the [Comet Compatibility Guide] for more information. |\n| FilterExec | Yes | |\n| GenerateExec | Yes | Supports `explode` generator only. |\n| GlobalLimitExec | Yes | |\n| HashAggregateExec | Yes | |\n| InsertIntoHadoopFsRelationCommand | No | Experimental support for native Parquet writes. Disabled by default. |\n| LocalLimitExec | Yes | |\n| LocalTableScanExec | No | Experimental and disabled by default. |\n| ObjectHashAggregateExec | Yes | Supports a limited number of aggregates, such as `bloom_filter_agg`. |\n| ProjectExec | Yes | |\n| ShuffleExchangeExec | Yes | |\n| ShuffledHashJoinExec | Yes | |\n| SortExec | Yes | |\n| SortMergeJoinExec | Yes | |\n| UnionExec | Yes | |\n| WindowExec | No | Disabled by default due to known correctness issues. |\n\n[Comet Compatibility Guide]: compatibility.md\n" }, { "path": "docs/source/user-guide/latest/source.md", "content": "\n\n# Building Comet From Source\n\nIt is sometimes preferable to build from source for a specific platform.\n\n## Using a Published Source Release\n\nOfficial source releases can be downloaded from https://dist.apache.org/repos/dist/release/datafusion/\n\n```console\n# Pick the latest version\nexport COMET_VERSION=$COMET_VERSION\n# Download the tarball\ncurl -O \"https://dist.apache.org/repos/dist/release/datafusion/datafusion-comet-$COMET_VERSION/apache-datafusion-comet-$COMET_VERSION.tar.gz\"\n# Unpack\ntar -xzf apache-datafusion-comet-$COMET_VERSION.tar.gz\ncd apache-datafusion-comet-$COMET_VERSION\n```\n\nBuild\n\n```console\nmake release-nogit PROFILES=\"-Pspark-3.5\"\n```\n\n## Building from the GitHub repository\n\nClone the repository:\n\n```console\ngit clone https://github.com/apache/datafusion-comet.git\n```\n\nBuild Comet for a specific Spark version:\n\n```console\ncd datafusion-comet\nmake release PROFILES=\"-Pspark-3.5\"\n```\n\nNote that the project builds for Scala 2.12 by default but can be built for Scala 2.13 using an additional profile:\n\n```console\nmake release PROFILES=\"-Pspark-3.5 -Pscala-2.13\"\n```\n\nTo build Comet from the source distribution on an isolated environment without an access to `github.com` it is necessary to disable `git-commit-id-maven-plugin`, otherwise you will face errors that there is no access to the git during the build process. In that case you may use:\n\n```console\nmake release-nogit PROFILES=\"-Pspark-3.5\"\n```\n" }, { "path": "docs/source/user-guide/latest/tuning.md", "content": "\n\n# Comet Tuning Guide\n\nComet provides some tuning options to help you get the best performance from your queries.\n\n## Configuring Tokio Runtime\n\nComet uses a global tokio runtime per executor process using tokio's defaults of one worker thread per core and a\nmaximum of 512 blocking threads. These values can be overridden using the environment variables `COMET_WORKER_THREADS`\nand `COMET_MAX_BLOCKING_THREADS`.\n\nIt is recommended that `COMET_WORKER_THREADS` be set to the number of executor cores. This may not be necessary\nin some environments, such as Kubernetes, where the number of cores allocated to a pod will already be equal to the\nnumber of executor cores.\n\n## Memory Tuning\n\nIt is necessary to specify how much memory Comet can use in addition to memory already allocated to Spark. In some\ncases, it may be possible to reduce the amount of memory allocated to Spark so that overall memory allocation is\nthe same or lower than the original configuration. In other cases, enabling Comet may require allocating more memory\nthan before. See the [Determining How Much Memory to Allocate] section for more details.\n\n[Determining How Much Memory to Allocate]: #determining-how-much-memory-to-allocate\n\n### Configuring Comet Memory\n\nComet shares an off-heap memory pool with Spark. The size of the pool is\nspecified by `spark.memory.offHeap.size`.\n\nComet's memory accounting isn't 100% accurate and this can result in Comet using more memory than it reserves,\nleading to out-of-memory exceptions. To work around this issue, it is possible to\nset `spark.comet.exec.memoryPool.fraction` to a value less than `1.0` to restrict the amount of memory that can be\nreserved by Comet.\n\nFor more details about Spark off-heap memory mode, please refer to [Spark documentation].\n\n[Spark documentation]: https://spark.apache.org/docs/latest/configuration.html\n\nComet implements multiple memory pool implementations. The type of pool can be specified with `spark.comet.exec.memoryPool`.\n\nThe valid pool types are:\n\n- `fair_unified` (default when `spark.memory.offHeap.enabled=true` is set)\n- `greedy_unified`\n\nBoth pool types are shared across all native execution contexts within the same Spark task. When\nComet executes a shuffle, it runs two native execution contexts concurrently (e.g. one for\npre-shuffle operators and one for the shuffle writer). The shared pool ensures that the combined\nmemory usage stays within the per-task limit.\n\nThe `fair_unified` pool prevents operators from using more than an even fraction of the available memory\n(i.e. `pool_size / num_reservations`). This pool works best when you know beforehand\nthe query has multiple operators that will likely all need to spill. Sometimes it will cause spills even\nwhen there is sufficient memory in order to leave enough memory for other operators.\n\nThe `greedy_unified` pool type implements a greedy first-come first-serve limit. This pool works well for queries that do not\nneed to spill or have a single spillable operator.\n\n[shuffle]: #shuffle\n[Advanced Memory Tuning]: #advanced-memory-tuning\n\n### Determining How Much Memory to Allocate\n\nGenerally, increasing the amount of memory allocated to Comet will improve query performance by reducing the\namount of time spent spilling to disk, especially for aggregate, join, and shuffle operations. Allocating insufficient\nmemory can result in out-of-memory errors. This is no different from allocating memory in Spark and the amount of\nmemory will vary for different workloads, so some experimentation will be required.\n\nHere is a real-world example, based on running benchmarks derived from TPC-H, running on a single executor against\nlocal Parquet files using the 100 GB data set.\n\nBaseline Spark Performance\n\n- Spark completes the benchmark in 632 seconds with 8 cores and 8 GB RAM\n- With less than 8 GB RAM, performance degrades due to spilling\n- Spark can complete the benchmark with as little as 3 GB of RAM, but with worse performance (744 seconds)\n\nComet Performance\n\n- Comet requires at least 5 GB of RAM, but performance at this level\n is around 340 seconds, which is significantly faster than Spark with any amount of RAM\n- Comet running in off-heap with 8 cores completes the benchmark in 295 seconds, more than 2x faster than Spark\n- It is worth noting that running Comet with only 4 cores and 4 GB RAM completes the benchmark in 520 seconds,\n providing better performance than Spark for half the resource\n\nIt may be possible to reduce Comet's memory overhead by reducing batch sizes or increasing number of partitions.\n\n## Optimizing Sorting on Floating-Point Values\n\nSorting on floating-point data types (or complex types containing floating-point values) is not compatible with\nSpark if the data contains both zero and negative zero. This is likely an edge case that is not of concern for many users\nand sorting on floating-point data can be enabled by setting `spark.comet.expression.SortOrder.allowIncompatible=true`.\n\n## Optimizing Joins\n\nSpark often chooses `SortMergeJoin` over `ShuffledHashJoin` for stability reasons. If the build-side of a\n`ShuffledHashJoin` is very large then it could lead to OOM in Spark.\n\nVectorized query engines tend to perform better with `ShuffledHashJoin`, so for best performance it is often preferable\nto configure Comet to convert `SortMergeJoin` to `ShuffledHashJoin`. Comet does not yet provide spill-to-disk for\n`ShuffledHashJoin` so this could result in OOM. Also, `SortMergeJoin` may still be faster in some cases. It is best\nto test with both for your specific workloads.\n\nTo configure Comet to convert `SortMergeJoin` to `ShuffledHashJoin`, set `spark.comet.exec.replaceSortMergeJoin=true`.\n\n## Shuffle\n\nComet provides accelerated shuffle implementations that can be used to improve the performance of your queries.\n\nTo enable Comet shuffle, set the following configuration in your Spark configuration:\n\n```\nspark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager\nspark.comet.exec.shuffle.enabled=true\n```\n\n`spark.shuffle.manager` is a Spark static configuration which cannot be changed at runtime.\nIt must be set before the Spark context is created. You can enable or disable Comet shuffle\nat runtime by setting `spark.comet.exec.shuffle.enabled` to `true` or `false`.\nOnce it is disabled, Comet will fall back to the default Spark shuffle manager.\n\n### Shuffle Implementations\n\nComet provides two shuffle implementations: Native Shuffle and Columnar Shuffle. Comet will first try to use Native\nShuffle and if that is not possible it will try to use Columnar Shuffle. If neither can be applied, it will fall\nback to Spark for shuffle operations.\n\n#### Native Shuffle\n\nComet provides a fully native shuffle implementation, which generally provides the best performance. Native shuffle\nsupports `HashPartitioning`, `RangePartitioning` and `SinglePartitioning` but currently only supports primitive type\npartitioning keys. Columns that are not partitioning keys may contain complex types like maps, structs, and arrays.\n\n#### Columnar (JVM) Shuffle\n\nComet Columnar shuffle is JVM-based and supports `HashPartitioning`, `RoundRobinPartitioning`, `RangePartitioning`, and\n`SinglePartitioning`. This shuffle implementation supports complex data types as partitioning keys.\n\n### Shuffle Compression\n\nBy default, Spark compresses shuffle files using LZ4 compression. Comet overrides this behavior with ZSTD compression.\nCompression can be disabled by setting `spark.shuffle.compress=false`, which may result in faster shuffle times in\ncertain environments, such as single-node setups with fast NVMe drives, at the expense of increased disk space usage.\n\n## Explain Plan\n\n### Extended Explain\n\nWith Spark 4.0.0 and newer, Comet can provide extended explain plan information in the Spark UI. Currently this lists\nreasons why Comet may not have been enabled for specific operations.\nTo enable this, in the Spark configuration, set the following:\n\n```shell\n-c spark.sql.extendedExplainProviders=org.apache.comet.ExtendedExplainInfo\n```\n\nThis will add a section to the detailed plan displayed in the Spark SQL UI page.\n" }, { "path": "docs/spark_expressions_support.md", "content": "\n\n# Supported Spark Expressions\n\n### agg_funcs\n\n- [x] any\n- [x] any_value\n- [ ] approx_count_distinct\n- [ ] approx_percentile\n- [ ] array_agg\n- [x] avg\n- [x] bit_and\n- [x] bit_or\n- [x] bit_xor\n- [x] bool_and\n- [x] bool_or\n- [ ] collect_list\n- [ ] collect_set\n- [ ] corr\n- [x] count\n- [x] count_if\n- [ ] count_min_sketch\n- [x] covar_pop\n- [x] covar_samp\n- [x] every\n- [x] first\n- [x] first_value\n- [ ] grouping\n- [ ] grouping_id\n- [ ] histogram_numeric\n- [ ] kurtosis\n- [x] last\n- [x] last_value\n- [x] max\n- [ ] max_by\n- [x] mean\n- [ ] median\n- [x] min\n- [ ] min_by\n- [ ] mode\n- [ ] percentile\n- [ ] percentile_approx\n- [x] regr_avgx\n- [x] regr_avgy\n- [x] regr_count\n- [ ] regr_intercept\n- [ ] regr_r2\n- [ ] regr_slope\n- [ ] regr_sxx\n- [ ] regr_sxy\n- [ ] regr_syy\n- [ ] skewness\n- [x] some\n- [x] std\n- [x] stddev\n- [x] stddev_pop\n- [x] stddev_samp\n- [x] sum\n- [ ] try_avg\n- [ ] try_sum\n- [x] var_pop\n- [x] var_samp\n- [x] variance\n\n### array_funcs\n\n- [x] array\n- [x] array_append\n- [x] array_compact\n- [x] array_contains\n- [x] array_distinct\n- [x] array_except\n- [x] array_insert\n- [x] array_intersect\n- [x] array_join\n- [x] array_max\n- [ ] array_min\n- [ ] array_position\n- [x] array_remove\n- [x] array_repeat\n- [x] array_union\n- [x] arrays_overlap\n- [ ] arrays_zip\n- [x] element_at\n- [ ] flatten\n- [x] get\n- [ ] sequence\n- [ ] shuffle\n- [ ] slice\n- [x] sort_array\n\n### bitwise_funcs\n\n- [x] &\n- [x] ^\n- [ ] bit_count\n- [ ] bit_get\n- [ ] getbit\n- [x] shiftright\n- [ ] shiftrightunsigned\n- [x] |\n- [x] ~\n\n### collection_funcs\n\n- [ ] array_size\n- [ ] cardinality\n- [ ] concat\n- [x] reverse\n- [ ] size\n\n### conditional_funcs\n\n- [x] coalesce\n- [x] if\n- [x] ifnull\n- [ ] nanvl\n- [x] nullif\n- [x] nvl\n- [x] nvl2\n- [ ] when\n\n### conversion_funcs\n\n- [ ] bigint\n- [ ] binary\n- [ ] boolean\n- [ ] cast\n- [ ] date\n- [ ] decimal\n- [ ] double\n- [ ] float\n- [ ] int\n- [ ] smallint\n- [ ] string\n- [ ] timestamp\n- [ ] tinyint\n\n### csv_funcs\n\n- [ ] from_csv\n- [ ] schema_of_csv\n- [ ] to_csv\n\n### datetime_funcs\n\n- [ ] add_months\n- [ ] convert_timezone\n- [x] curdate\n- [x] current_date\n- [ ] current_timestamp\n- [x] current_timezone\n- [ ] date_add\n- [ ] date_diff\n- [ ] date_format\n- [x] date_from_unix_date\n- [x] date_part\n- [ ] date_sub\n- [ ] date_trunc\n- [ ] dateadd\n- [ ] datediff\n- [x] datepart\n- [ ] day\n- [ ] dayofmonth\n- [ ] dayofweek\n- [ ] dayofyear\n- [x] extract\n- [x] from_unixtime\n- [ ] from_utc_timestamp\n- [ ] hour\n- [ ] last_day\n- [ ] localtimestamp\n- [ ] make_date\n- [ ] make_dt_interval\n- [ ] make_interval\n- [ ] make_timestamp\n- [ ] make_timestamp_ltz\n- [ ] make_timestamp_ntz\n- [ ] make_ym_interval\n- [ ] minute\n- [ ] month\n- [ ] months_between\n- [ ] next_day\n- [ ] now\n- [ ] quarter\n- [ ] second\n- [ ] timestamp_micros\n- [ ] timestamp_millis\n- [ ] timestamp_seconds\n- [ ] to_date\n- [ ] to_timestamp\n- [ ] to_timestamp_ltz\n- [ ] to_timestamp_ntz\n- [ ] to_unix_timestamp\n- [ ] to_utc_timestamp\n- [ ] trunc\n- [ ] try_to_timestamp\n- [ ] unix_date\n- [ ] unix_micros\n- [ ] unix_millis\n- [ ] unix_seconds\n- [x] unix_timestamp\n- [ ] weekday\n- [ ] weekofyear\n- [ ] year\n\n### generator_funcs\n\n- [ ] explode\n- [ ] explode_outer\n- [ ] inline\n- [ ] inline_outer\n- [ ] posexplode\n- [ ] posexplode_outer\n- [ ] stack\n\n### hash_funcs\n\n- [x] crc32\n- [ ] hash\n- [x] md5\n- [ ] sha\n- [ ] sha1\n- [ ] sha2\n- [ ] xxhash64\n\n### json_funcs\n\n- [ ] from_json\n- [x] get_json_object\n- [ ] json_array_length\n- [ ] json_object_keys\n- [ ] json_tuple\n- [ ] schema_of_json\n- [ ] to_json\n\n### lambda_funcs\n\n- [ ] aggregate\n- [ ] array_sort\n- [ ] exists\n- [ ] filter\n- [ ] forall\n- [ ] map_filter\n- [ ] map_zip_with\n- [ ] reduce\n- [ ] transform\n- [ ] transform_keys\n- [ ] transform_values\n- [ ] zip_with\n\n### map_funcs\n\n- [ ] element_at\n- [ ] map\n- [ ] map_concat\n- [x] map_contains_key\n- [ ] map_entries\n- [ ] map_from_arrays\n- [ ] map_from_entries\n- [x] map_keys\n- [ ] map_values\n- [ ] str_to_map\n- [ ] try_element_at\n\n### math_funcs\n\n- [x] %\n- [x] -\n- [x] -\n- [x] -\n- [x] /\n- [x] abs\n- [x] acos\n- [ ] acosh\n- [x] asin\n- [ ] asinh\n- [x] atan\n- [x] atan2\n- [ ] atanh\n- [x] bin\n- [ ] bround\n- [ ] cbrt\n- [x] ceil\n- [x] ceiling\n- [ ] conv\n- [x] cos\n- [ ] cosh\n- [ ] cot\n- [ ] csc\n- [ ] degrees\n- [ ] div\n- [ ] e\n- [x] exp\n- [ ] expm1\n- [ ] factorial\n- [x] floor\n- [ ] greatest\n- [ ] hex\n- [ ] hypot\n- [ ] least\n- [x] ln\n- [ ] log\n- [x] log10\n- [ ] log1p\n- [x] log2\n- [x] mod\n- [x] negative\n- [ ] pi\n- [ ] pmod\n- [x] positive\n- [x] pow\n- [x] power\n- [ ] radians\n- [ ] rand\n- [ ] randn\n- [ ] random\n- [ ] rint\n- [x] round\n- [ ] sec\n- [x] shiftleft\n- [x] sign\n- [x] signum\n- [x] sin\n- [ ] sinh\n- [x] sqrt\n- [x] tan\n- [ ] tanh\n- [x] try_add\n- [x] try_divide\n- [x] try_multiply\n- [x] try_subtract\n- [x] unhex\n- [x] width_bucket\n\n### misc_funcs\n\n- [ ] aes_decrypt\n- [ ] aes_encrypt\n- [ ] assert_true\n- [x] current_catalog\n- [x] current_database\n- [x] current_schema\n- [x] current_user\n- [x] equal_null\n- [ ] input_file_block_length\n- [ ] input_file_block_start\n- [ ] input_file_name\n- [x] monotonically_increasing_id\n- [ ] raise_error\n- [x] rand\n- [x] randn\n- [x] spark_partition_id\n- [ ] typeof\n- [x] user\n- [ ] uuid\n- [ ] version\n\n### predicate_funcs\n\n- [x] !\n- [x] <\n- [x] <=\n- [x] <=>\n- [x] =\n- [x] ==\n- [x] >\n- [x] > =\n- [x] and\n- [x] ilike\n- [x] in\n- [ ] isnan\n- [x] isnotnull\n- [x] isnull\n- [x] like\n- [x] not\n- [x] or\n- [ ] regexp\n- [ ] regexp_like\n- [ ] rlike\n\n### string_funcs\n\n- [x] ascii\n- [ ] base64\n- [x] bit_length\n- [x] btrim\n- [x] char\n- [x] char_length\n- [x] character_length\n- [x] chr\n- [x] concat_ws\n- [x] contains\n- [ ] decode\n- [ ] elt\n- [ ] encode\n- [x] endswith\n- [ ] find_in_set\n- [ ] format_number\n- [ ] format_string\n- [x] initcap\n- [x] instr\n- [x] lcase\n- [ ] left\n- [x] len\n- [x] length\n- [ ] levenshtein\n- [ ] locate\n- [x] lower\n- [x] lpad\n- [x] ltrim\n- [ ] mask\n- [x] octet_length\n- [ ] overlay\n- [ ] position\n- [ ] printf\n- [ ] regexp_count\n- [ ] regexp_extract\n- [ ] regexp_extract_all\n- [ ] regexp_instr\n- [ ] regexp_replace\n- [ ] regexp_substr\n- [x] repeat\n- [x] replace\n- [ ] right\n- [x] rpad\n- [x] rtrim\n- [ ] sentences\n- [ ] soundex\n- [x] space\n- [ ] split\n- [ ] split_part\n- [x] startswith\n- [ ] substr\n- [ ] substring\n- [ ] substring_index\n- [ ] to_binary\n- [ ] to_char\n- [ ] to_number\n- [x] translate\n- [x] trim\n- [ ] try_to_binary\n- [ ] try_to_number\n- [x] ucase\n- [ ] unbase64\n- [x] upper\n\n### struct_funcs\n\n- [ ] named_struct\n- [ ] struct\n\n### url_funcs\n\n- [ ] parse_url\n- [ ] url_decode\n- [ ] url_encode\n\n### window_funcs\n\n- [ ] cume_dist\n- [ ] dense_rank\n- [ ] lag\n- [ ] lead\n- [ ] nth_value\n- [ ] ntile\n- [ ] percent_rank\n- [ ] rank\n- [ ] row_number\n\n### xml_funcs\n\n- [ ] xpath\n- [ ] xpath_boolean\n- [ ] xpath_double\n- [ ] xpath_float\n- [ ] xpath_int\n- [ ] xpath_long\n- [ ] xpath_number\n- [ ] xpath_short\n- [ ] xpath_string\n" }, { "path": "fuzz-testing/.gitignore", "content": ".idea\ntarget\nspark-warehouse\nqueries.sql\nresults*.md\ntest*.parquet" }, { "path": "fuzz-testing/README.md", "content": "\n\n# Comet Fuzz\n\nComet Fuzz is a standalone project for generating random data and queries and executing queries against Spark\nwith Comet disabled and enabled and checking for incompatibilities.\n\nAlthough it is a simple tool it has already been useful in finding many bugs.\n\nComet Fuzz is inspired by the [SparkFuzz](https://ir.cwi.nl/pub/30222) paper from Databricks and CWI.\n\n## Roadmap\n\nPlanned areas of improvement:\n\n- ANSI mode\n- Support for all data types, expressions, and operators supported by Comet\n- IF and CASE WHEN expressions\n- Complex (nested) expressions\n- Literal scalar values in queries\n- Add option to avoid grouping and sorting on floating-point columns\n- Improve join query support:\n - Support joins without join keys\n - Support composite join keys\n - Support multiple join keys\n - Support join conditions that use expressions\n\n## Usage\n\nFrom the root of the project, run `mvn install -DskipTests` to install Comet.\n\nThen build the fuzz testing jar.\n\n```shell\nmvn package\n```\n\nSet appropriate values for `SPARK_HOME`, `SPARK_MASTER`, and `COMET_JAR` environment variables and then use\n`spark-submit` to run CometFuzz against a Spark cluster.\n\n### Generating Data Files\n\n```shell\n$SPARK_HOME/bin/spark-submit \\\n --master $SPARK_MASTER \\\n --class org.apache.comet.fuzz.Main \\\n target/comet-fuzz-spark3.5_2.12-0.13.0-SNAPSHOT-jar-with-dependencies.jar \\\n data --num-files=2 --num-rows=200 --exclude-negative-zero --generate-arrays --generate-structs --generate-maps\n```\n\nThere is an optional `--exclude-negative-zero` flag for excluding `-0.0` from the generated data, which is\nsometimes useful because we already know that we often have different behavior for this edge case due to\ndifferences between Rust and Java handling of this value.\n\n### Generating Queries\n\nGenerate random queries that are based on the available test files.\n\n```shell\n$SPARK_HOME/bin/spark-submit \\\n --master $SPARK_MASTER \\\n --class org.apache.comet.fuzz.Main \\\n target/comet-fuzz-spark3.5_2.12-0.13.0-SNAPSHOT-jar-with-dependencies.jar \\\n queries --num-files=2 --num-queries=500\n```\n\nNote that the output filename is currently hard-coded as `queries.sql`\n\n### Execute Queries\n\n```shell\n$SPARK_HOME/bin/spark-submit \\\n --master $SPARK_MASTER \\\n --conf spark.memory.offHeap.enabled=true \\\n --conf spark.memory.offHeap.size=16G \\\n --conf spark.plugins=org.apache.spark.CometPlugin \\\n --conf spark.comet.enabled=true \\\n --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \\\n --conf spark.comet.exec.shuffle.enabled=true \\\n --jars $COMET_JAR \\\n --conf spark.driver.extraClassPath=$COMET_JAR \\\n --conf spark.executor.extraClassPath=$COMET_JAR \\\n --class org.apache.comet.fuzz.Main \\\n target/comet-fuzz-spark3.5_2.12-0.13.0-SNAPSHOT-jar-with-dependencies.jar \\\n run --num-files=2 --filename=queries.sql\n```\n\nNote that the output filename is currently hard-coded as `results-${System.currentTimeMillis()}.md`\n\n### Compare existing datasets\n\nTo compare a pair of existing datasets you can use a comparison tool.\nThe example below is for TPC-H queries results generated by pure Spark and Comet\n\n```shell\n$SPARK_HOME/bin/spark-submit \\\n --master $SPARK_MASTER \\\n --class org.apache.comet.fuzz.ComparisonTool\n target/comet-fuzz-spark3.5_2.12-0.13.0-SNAPSHOT-jar-with-dependencies.jar \\\n compareParquet --input-spark-folder=/tmp/tpch/spark --input-comet-folder=/tmp/tpch/comet\n```\n\nThe tool takes a pair of existing folders of the same layout and compares subfolders treating them as parquet based datasets\n" }, { "path": "fuzz-testing/pom.xml", "content": "\n\n\n 4.0.0\n\n \n org.apache.datafusion\n comet-parent-spark${spark.version.short}_${scala.binary.version}\n 0.15.0-SNAPSHOT\n ../pom.xml\n \n\n comet-fuzz-spark${spark.version.short}_${scala.binary.version}\n comet-fuzz\n http://maven.apache.org\n jar\n\n \n \n false\n \n\n \n \n org.scala-lang\n scala-library\n ${scala.version}\n provided\n \n \n org.apache.spark\n spark-sql_${scala.binary.version}\n provided\n \n \n org.scala-lang.modules\n scala-collection-compat_${scala.binary.version}\n \n \n \n \n org.apache.datafusion\n comet-spark-spark${spark.version.short}_${scala.binary.version}\n ${project.version}\n \n \n org.scala-lang.modules\n scala-collection-compat_${scala.binary.version}\n \n \n \n \n org.rogach\n scallop_${scala.binary.version}\n \n \n\n \n src/main/scala\n \n \n org.apache.maven.plugins\n maven-compiler-plugin\n ${maven-compiler-plugin.version}\n \n ${java.version}\n ${java.version}\n \n \n \n net.alchim31.maven\n scala-maven-plugin\n ${scala.plugin.version}\n \n \n \n compile\n testCompile\n \n \n \n \n \n maven-assembly-plugin\n ${maven-assembly-plugin.version}\n \n \n jar-with-dependencies\n \n \n \n \n make-assembly\n package\n \n single\n \n \n \n \n \n org.apache.maven.plugins\n maven-install-plugin\n \n true\n \n \n \n \n\n" }, { "path": "fuzz-testing/run.sh", "content": "#!/bin/bash\n#\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n# Usage: ./run.sh\n# Builds necessary JARs, generates data and queries, and runs fuzz tests for Comet Spark.\n# Environment variables:\n# SPARK_HOME - path to Spark installation\n# SPARK_MASTER - Spark master URL (default: local[*])\n# SCALA_MAJOR_VERSION - Scala major version to use (default: 2.12)\n# SPARK_MAJOR_VERSION - Spark major version to use (default: 3.5)\n# NUM_FILES - number of data files to generate (default: 2)\n# NUM_ROWS - number of rows per file (default: 200)\n# NUM_QUERIES - number of queries to generate (default: 500)\n\nset -eux\n\nDIR=\"$(cd \"$(dirname \"$0\")\" && pwd)\"\nPARENT_DIR=\"${DIR}/..\"\nMVN_CMD=\"${PARENT_DIR}/mvnw\"\nSPARK_MASTER=\"${SPARK_MASTER:-local[*]}\"\nSCALA_MAJOR_VERSION=\"${SCALA_MAJOR_VERSION:-2.12}\"\nSPARK_MAJOR_VERSION=\"${SPARK_MAJOR_VERSION:-3.5}\"\nPROFILES=\"-Pscala-${SCALA_MAJOR_VERSION},spark-${SPARK_MAJOR_VERSION}\"\nPROJECT_VERSION=$(\"${MVN_CMD}\" -f \"${DIR}/pom.xml\" -q help:evaluate -Dexpression=project.version -DforceStdout)\nCOMET_SPARK_JAR=\"${PARENT_DIR}/spark/target/comet-spark${SPARK_MAJOR_VERSION}_${SCALA_MAJOR_VERSION}-${PROJECT_VERSION}.jar\"\nCOMET_FUZZ_JAR=\"${DIR}/target/comet-fuzz-spark${SPARK_MAJOR_VERSION}_${SCALA_MAJOR_VERSION}-${PROJECT_VERSION}-jar-with-dependencies.jar\"\nNUM_FILES=\"${NUM_FILES:-2}\"\nNUM_ROWS=\"${NUM_ROWS:-200}\"\nNUM_QUERIES=\"${NUM_QUERIES:-500}\"\n\nif [ ! -f \"${COMET_SPARK_JAR}\" ]; then\n echo \"Building Comet Spark jar...\"\n pushd \"${PARENT_DIR}\"\n PROFILES=\"${PROFILES}\" make\n popd\nelse\n echo \"Building Fuzz testing jar...\"\n \"${MVN_CMD}\" -f \"${DIR}/pom.xml\" package -DskipTests \"${PROFILES}\"\nfi\n\necho \"Generating data...\"\n\"${SPARK_HOME}/bin/spark-submit\" \\\n --master \"${SPARK_MASTER}\" \\\n --class org.apache.comet.fuzz.Main \\\n \"${COMET_FUZZ_JAR}\" \\\n data --num-files=\"${NUM_FILES}\" --num-rows=\"${NUM_ROWS}\" \\\n --exclude-negative-zero \\\n --generate-arrays --generate-structs --generate-maps\n\necho \"Generating queries...\"\n\"${SPARK_HOME}/bin/spark-submit\" \\\n --master \"${SPARK_MASTER}\" \\\n --class org.apache.comet.fuzz.Main \\\n \"${COMET_FUZZ_JAR}\" \\\n queries --num-files=\"${NUM_FILES}\" --num-queries=\"${NUM_QUERIES}\"\n\necho \"Running fuzz tests...\"\n\"${SPARK_HOME}/bin/spark-submit\" \\\n --master \"${SPARK_MASTER}\" \\\n --conf spark.memory.offHeap.enabled=true \\\n --conf spark.memory.offHeap.size=16G \\\n --conf spark.plugins=org.apache.spark.CometPlugin \\\n --conf spark.comet.enabled=true \\\n --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \\\n --conf spark.comet.exec.shuffle.enabled=true \\\n --jars \"${COMET_SPARK_JAR}\" \\\n --conf spark.driver.extraClassPath=\"${COMET_SPARK_JAR}\" \\\n --conf spark.executor.extraClassPath=\"${COMET_SPARK_JAR}\" \\\n --class org.apache.comet.fuzz.Main \\\n \"${COMET_FUZZ_JAR}\" \\\n run --num-files=\"${NUM_FILES}\" --filename=\"queries.sql\"\n" }, { "path": "fuzz-testing/src/main/scala/org/apache/comet/fuzz/ComparisonTool.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.fuzz\n\nimport java.io.File\n\nimport org.rogach.scallop.{ScallopConf, ScallopOption, Subcommand}\n\nimport org.apache.spark.sql.{functions, SparkSession}\n\nclass ComparisonToolConf(arguments: Seq[String]) extends ScallopConf(arguments) {\n object compareParquet extends Subcommand(\"compareParquet\") {\n val inputSparkFolder: ScallopOption[String] =\n opt[String](required = true, descr = \"Folder with Spark produced results in Parquet format\")\n val inputCometFolder: ScallopOption[String] =\n opt[String](required = true, descr = \"Folder with Comet produced results in Parquet format\")\n val tolerance: ScallopOption[Double] =\n opt[Double](default = Some(0.000002), descr = \"Tolerance for floating point comparisons\")\n }\n addSubcommand(compareParquet)\n verify()\n}\n\nobject ComparisonTool {\n\n lazy val spark: SparkSession = SparkSession\n .builder()\n .getOrCreate()\n\n def main(args: Array[String]): Unit = {\n val conf = new ComparisonToolConf(args.toIndexedSeq)\n conf.subcommand match {\n case Some(conf.compareParquet) =>\n compareParquetFolders(\n spark,\n conf.compareParquet.inputSparkFolder(),\n conf.compareParquet.inputCometFolder(),\n conf.compareParquet.tolerance())\n\n case _ =>\n // scalastyle:off println\n println(\"Invalid subcommand\")\n // scalastyle:on println\n sys.exit(-1)\n }\n }\n\n private def compareParquetFolders(\n spark: SparkSession,\n sparkFolderPath: String,\n cometFolderPath: String,\n tolerance: Double): Unit = {\n\n val output = QueryRunner.createOutputMdFile()\n\n try {\n val sparkFolder = new File(sparkFolderPath)\n val cometFolder = new File(cometFolderPath)\n\n if (!sparkFolder.exists() || !sparkFolder.isDirectory) {\n throw new IllegalArgumentException(\n s\"Spark folder does not exist or is not a directory: $sparkFolderPath\")\n }\n\n if (!cometFolder.exists() || !cometFolder.isDirectory) {\n throw new IllegalArgumentException(\n s\"Comet folder does not exist or is not a directory: $cometFolderPath\")\n }\n\n // Get all subdirectories from the Spark folder\n val sparkSubfolders = sparkFolder\n .listFiles()\n .filter(_.isDirectory)\n .map(_.getName)\n .sorted\n\n output.write(\"# Comparing Parquet Folders\\n\\n\")\n output.write(s\"Spark folder: $sparkFolderPath\\n\")\n output.write(s\"Comet folder: $cometFolderPath\\n\")\n output.write(s\"Found ${sparkSubfolders.length} subfolders to compare\\n\\n\")\n\n // Compare each subfolder\n sparkSubfolders.foreach { subfolderName =>\n val sparkSubfolderPath = new File(sparkFolder, subfolderName)\n val cometSubfolderPath = new File(cometFolder, subfolderName)\n\n if (!cometSubfolderPath.exists() || !cometSubfolderPath.isDirectory) {\n output.write(s\"## Subfolder: $subfolderName\\n\")\n output.write(\n s\"[WARNING] Comet subfolder not found: ${cometSubfolderPath.getAbsolutePath}\\n\\n\")\n } else {\n output.write(s\"## Comparing subfolder: $subfolderName\\n\\n\")\n\n try {\n // Read Spark parquet files\n spark.conf.set(\"spark.comet.enabled\", \"false\")\n val sparkDf = spark.read.parquet(sparkSubfolderPath.getAbsolutePath)\n val sparkRows = sparkDf.orderBy(sparkDf.columns.map(functions.col): _*).collect()\n\n // Read Comet parquet files\n val cometDf = spark.read.parquet(cometSubfolderPath.getAbsolutePath)\n val cometRows = cometDf.orderBy(cometDf.columns.map(functions.col): _*).collect()\n\n // Compare the results\n if (QueryComparison.assertSameRows(sparkRows, cometRows, output, tolerance)) {\n output.write(s\"Subfolder $subfolderName: ${sparkRows.length} rows matched\\n\\n\")\n } else {\n // Output schema if dataframes are not equal\n QueryComparison.showSchema(\n output,\n sparkDf.schema.treeString,\n cometDf.schema.treeString)\n }\n } catch {\n case e: Exception =>\n output.write(\n s\"[ERROR] Failed to compare subfolder $subfolderName: ${e.getMessage}\\n\")\n val sw = new java.io.StringWriter()\n val p = new java.io.PrintWriter(sw)\n e.printStackTrace(p)\n p.close()\n output.write(s\"```\\n${sw.toString}\\n```\\n\\n\")\n }\n }\n\n output.flush()\n }\n\n output.write(\"\\n# Comparison Complete\\n\")\n output.write(s\"Compared ${sparkSubfolders.length} subfolders\\n\")\n\n } finally {\n output.close()\n }\n }\n}\n" }, { "path": "fuzz-testing/src/main/scala/org/apache/comet/fuzz/Main.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.fuzz\n\nimport scala.util.Random\n\nimport org.rogach.scallop.{ScallopConf, Subcommand}\nimport org.rogach.scallop.ScallopOption\n\nimport org.apache.spark.sql.SparkSession\n\nimport org.apache.comet.testing.{DataGenOptions, ParquetGenerator, SchemaGenOptions}\n\nclass Conf(arguments: Seq[String]) extends ScallopConf(arguments) {\n object generateData extends Subcommand(\"data\") {\n val numFiles: ScallopOption[Int] =\n opt[Int](required = true, descr = \"Number of files to generate\")\n val numRows: ScallopOption[Int] = opt[Int](required = true, descr = \"Number of rows per file\")\n val randomSeed: ScallopOption[Long] =\n opt[Long](required = false, descr = \"Random seed to use\")\n val generateArrays: ScallopOption[Boolean] =\n opt[Boolean](required = false, descr = \"Whether to generate arrays\")\n val generateStructs: ScallopOption[Boolean] =\n opt[Boolean](required = false, descr = \"Whether to generate structs\")\n val generateMaps: ScallopOption[Boolean] =\n opt[Boolean](required = false, descr = \"Whether to generate maps\")\n val excludeNegativeZero: ScallopOption[Boolean] =\n opt[Boolean](required = false, descr = \"Whether to exclude negative zero\")\n }\n addSubcommand(generateData)\n object generateQueries extends Subcommand(\"queries\") {\n val numFiles: ScallopOption[Int] =\n opt[Int](required = true, descr = \"Number of input files to use\")\n val numQueries: ScallopOption[Int] =\n opt[Int](required = true, descr = \"Number of queries to generate\")\n val randomSeed: ScallopOption[Long] =\n opt[Long](required = false, descr = \"Random seed to use\")\n }\n addSubcommand(generateQueries)\n object runQueries extends Subcommand(\"run\") {\n val filename: ScallopOption[String] =\n opt[String](required = true, descr = \"File to write queries to\")\n val numFiles: ScallopOption[Int] =\n opt[Int](required = true, descr = \"Number of input files to use\")\n val showFailedSparkQueries: ScallopOption[Boolean] =\n opt[Boolean](required = false, descr = \"Whether to show failed Spark queries\")\n }\n addSubcommand(runQueries)\n verify()\n}\n\nobject Main {\n\n lazy val spark: SparkSession = SparkSession\n .builder()\n .getOrCreate()\n\n def main(args: Array[String]): Unit = {\n val conf = new Conf(args.toIndexedSeq)\n conf.subcommand match {\n case Some(conf.generateData) =>\n val r = conf.generateData.randomSeed.toOption match {\n case Some(seed) => new Random(seed)\n case None => new Random()\n }\n for (i <- 0 until conf.generateData.numFiles()) {\n ParquetGenerator.makeParquetFile(\n r,\n spark,\n s\"test$i.parquet\",\n numRows = conf.generateData.numRows(),\n SchemaGenOptions(\n generateArray = conf.generateData.generateArrays(),\n generateStruct = conf.generateData.generateStructs(),\n generateMap = conf.generateData.generateMaps(),\n // create two columns of each primitive type so that they can be used in binary\n // expressions such as `a + b` and `a < b`\n primitiveTypes = SchemaGenOptions.defaultPrimitiveTypes ++\n SchemaGenOptions.defaultPrimitiveTypes),\n DataGenOptions(\n allowNull = true,\n generateNegativeZero = !conf.generateData.excludeNegativeZero()))\n }\n case Some(conf.generateQueries) =>\n val r = conf.generateQueries.randomSeed.toOption match {\n case Some(seed) => new Random(seed)\n case None => new Random()\n }\n QueryGen.generateRandomQueries(\n r,\n spark,\n numFiles = conf.generateQueries.numFiles(),\n conf.generateQueries.numQueries())\n case Some(conf.runQueries) =>\n QueryRunner.runQueries(\n spark,\n conf.runQueries.numFiles(),\n conf.runQueries.filename(),\n conf.runQueries.showFailedSparkQueries())\n case _ =>\n // scalastyle:off println\n println(\"Invalid subcommand\")\n // scalastyle:on println\n sys.exit(-1)\n }\n }\n}\n" }, { "path": "fuzz-testing/src/main/scala/org/apache/comet/fuzz/Meta.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.fuzz\n\nimport org.apache.spark.sql.types.DataType\nimport org.apache.spark.sql.types.DataTypes\n\nsealed trait SparkType\ncase class SparkTypeOneOf(dataTypes: Seq[SparkType]) extends SparkType\ncase object SparkBooleanType extends SparkType\ncase object SparkBinaryType extends SparkType\ncase object SparkStringType extends SparkType\ncase object SparkIntegralType extends SparkType\ncase object SparkByteType extends SparkType\ncase object SparkShortType extends SparkType\ncase object SparkIntType extends SparkType\ncase object SparkLongType extends SparkType\ncase object SparkFloatType extends SparkType\ncase object SparkDoubleType extends SparkType\ncase class SparkDecimalType(p: Int, s: Int) extends SparkType\ncase object SparkNumericType extends SparkType\ncase object SparkDateType extends SparkType\ncase object SparkTimestampType extends SparkType\ncase object SparkDateOrTimestampType extends SparkType\ncase class SparkArrayType(elementType: SparkType) extends SparkType\ncase class SparkMapType(keyType: SparkType, valueType: SparkType) extends SparkType\ncase class SparkStructType(fields: Seq[SparkType]) extends SparkType\ncase object SparkAnyType extends SparkType\n\ncase class FunctionSignature(inputTypes: Seq[SparkType], varArgs: Boolean = false)\n\ncase class Function(name: String, signatures: Seq[FunctionSignature])\n\nobject Meta {\n\n val primitiveSparkTypes: Seq[SparkType] = Seq(\n SparkBooleanType,\n SparkBinaryType,\n SparkStringType,\n SparkByteType,\n SparkShortType,\n SparkIntType,\n SparkLongType,\n SparkFloatType,\n SparkDoubleType,\n SparkDateType,\n SparkTimestampType)\n\n val dataTypes: Seq[(DataType, Double)] = Seq(\n (DataTypes.BooleanType, 0.1),\n (DataTypes.ByteType, 0.2),\n (DataTypes.ShortType, 0.2),\n (DataTypes.IntegerType, 0.2),\n (DataTypes.LongType, 0.2),\n (DataTypes.FloatType, 0.2),\n (DataTypes.DoubleType, 0.2),\n (DataTypes.createDecimalType(10, 2), 0.2),\n (DataTypes.DateType, 0.2),\n (DataTypes.TimestampType, 0.2),\n (DataTypes.TimestampNTZType, 0.2),\n (DataTypes.StringType, 0.2),\n (DataTypes.BinaryType, 0.1))\n\n private def createFunctionWithInputTypes(\n name: String,\n inputs: Seq[SparkType],\n varArgs: Boolean = false): Function = {\n Function(name, Seq(FunctionSignature(inputs, varArgs)))\n createFunctions(name, Seq(FunctionSignature(inputs, varArgs)))\n }\n\n private def createFunctions(name: String, signatures: Seq[FunctionSignature]): Function = {\n signatures.foreach { s =>\n assert(\n !s.varArgs || s.inputTypes.length == 1,\n s\"Variadic function $s must have exactly one input type\")\n }\n Function(name, signatures)\n }\n\n private def createUnaryStringFunction(name: String): Function = {\n createFunctionWithInputTypes(name, Seq(SparkStringType))\n }\n\n private def createUnaryNumericFunction(name: String): Function = {\n createFunctionWithInputTypes(name, Seq(SparkNumericType))\n }\n\n // Math expressions (corresponds to mathExpressions in QueryPlanSerde)\n val mathScalarFunc: Seq[Function] = Seq(\n createUnaryNumericFunction(\"abs\"),\n createUnaryNumericFunction(\"acos\"),\n createUnaryNumericFunction(\"asin\"),\n createUnaryNumericFunction(\"atan\"),\n createFunctionWithInputTypes(\"atan2\", Seq(SparkNumericType, SparkNumericType)),\n createUnaryNumericFunction(\"cos\"),\n createUnaryNumericFunction(\"cosh\"),\n createUnaryNumericFunction(\"exp\"),\n createUnaryNumericFunction(\"expm1\"),\n createFunctionWithInputTypes(\"log\", Seq(SparkNumericType, SparkNumericType)),\n createUnaryNumericFunction(\"log10\"),\n createUnaryNumericFunction(\"log2\"),\n createFunctionWithInputTypes(\"pow\", Seq(SparkNumericType, SparkNumericType)),\n createFunctionWithInputTypes(\"remainder\", Seq(SparkNumericType, SparkNumericType)),\n createFunctions(\n \"round\",\n Seq(\n FunctionSignature(Seq(SparkNumericType)),\n FunctionSignature(Seq(SparkNumericType, SparkIntType)))),\n createUnaryNumericFunction(\"signum\"),\n createUnaryNumericFunction(\"sin\"),\n createUnaryNumericFunction(\"sinh\"),\n createUnaryNumericFunction(\"sqrt\"),\n createUnaryNumericFunction(\"tan\"),\n createUnaryNumericFunction(\"tanh\"),\n createUnaryNumericFunction(\"cot\"),\n createUnaryNumericFunction(\"ceil\"),\n createUnaryNumericFunction(\"floor\"),\n createFunctionWithInputTypes(\"unary_minus\", Seq(SparkNumericType)))\n\n // Hash expressions (corresponds to hashExpressions in QueryPlanSerde)\n val hashScalarFunc: Seq[Function] = Seq(\n createFunctionWithInputTypes(\"md5\", Seq(SparkAnyType)),\n createFunctionWithInputTypes(\"murmur3_hash\", Seq(SparkAnyType)), // TODO variadic\n createFunctionWithInputTypes(\"sha2\", Seq(SparkAnyType, SparkIntType)))\n\n // String expressions (corresponds to stringExpressions in QueryPlanSerde)\n val stringScalarFunc: Seq[Function] = Seq(\n createUnaryStringFunction(\"ascii\"),\n createUnaryStringFunction(\"bit_length\"),\n createUnaryStringFunction(\"chr\"),\n createFunctionWithInputTypes(\n \"concat\",\n Seq(\n SparkTypeOneOf(\n Seq(\n SparkStringType,\n SparkBinaryType,\n SparkArrayType(SparkStringType),\n SparkArrayType(SparkNumericType),\n SparkArrayType(SparkBinaryType)))),\n varArgs = true),\n createFunctionWithInputTypes(\"concat_ws\", Seq(SparkStringType, SparkStringType)),\n createFunctionWithInputTypes(\"contains\", Seq(SparkStringType, SparkStringType)),\n createFunctionWithInputTypes(\"ends_with\", Seq(SparkStringType, SparkStringType)),\n createFunctionWithInputTypes(\n \"hex\",\n Seq(SparkTypeOneOf(Seq(SparkStringType, SparkBinaryType, SparkIntType, SparkLongType)))),\n createUnaryStringFunction(\"init_cap\"),\n createFunctionWithInputTypes(\"instr\", Seq(SparkStringType, SparkStringType)),\n createFunctionWithInputTypes(\n \"length\",\n Seq(SparkTypeOneOf(Seq(SparkStringType, SparkBinaryType)))),\n createFunctionWithInputTypes(\"like\", Seq(SparkStringType, SparkStringType)),\n createUnaryStringFunction(\"lower\"),\n createFunctions(\n \"lpad\",\n Seq(\n FunctionSignature(Seq(SparkStringType, SparkIntegralType)),\n FunctionSignature(Seq(SparkStringType, SparkIntegralType, SparkStringType)))),\n createUnaryStringFunction(\"ltrim\"),\n createUnaryStringFunction(\"octet_length\"),\n createFunctions(\n \"regexp_replace\",\n Seq(\n FunctionSignature(Seq(SparkStringType, SparkStringType, SparkStringType)),\n FunctionSignature(Seq(SparkStringType, SparkStringType, SparkStringType, SparkIntType)))),\n createFunctionWithInputTypes(\"repeat\", Seq(SparkStringType, SparkIntType)),\n createFunctions(\n \"replace\",\n Seq(\n FunctionSignature(Seq(SparkStringType, SparkStringType)),\n FunctionSignature(Seq(SparkStringType, SparkStringType, SparkStringType)))),\n createFunctions(\n \"reverse\",\n Seq(\n FunctionSignature(Seq(SparkStringType)),\n FunctionSignature(Seq(SparkArrayType(SparkAnyType))))),\n createFunctionWithInputTypes(\"rlike\", Seq(SparkStringType, SparkStringType)),\n createFunctions(\n \"rpad\",\n Seq(\n FunctionSignature(Seq(SparkStringType, SparkIntegralType)),\n FunctionSignature(Seq(SparkStringType, SparkIntegralType, SparkStringType)))),\n createUnaryStringFunction(\"rtrim\"),\n createFunctions(\n \"split\",\n Seq(\n FunctionSignature(Seq(SparkStringType, SparkStringType)),\n FunctionSignature(Seq(SparkStringType, SparkStringType, SparkIntType)))),\n createFunctionWithInputTypes(\"starts_with\", Seq(SparkStringType, SparkStringType)),\n createFunctionWithInputTypes(\"string_space\", Seq(SparkIntType)),\n createFunctionWithInputTypes(\"substring\", Seq(SparkStringType, SparkIntType, SparkIntType)),\n createFunctionWithInputTypes(\"translate\", Seq(SparkStringType, SparkStringType)),\n createUnaryStringFunction(\"trim\"),\n createUnaryStringFunction(\"btrim\"),\n createUnaryStringFunction(\"unhex\"),\n createUnaryStringFunction(\"upper\"),\n createFunctionWithInputTypes(\"xxhash64\", Seq(SparkAnyType)), // TODO variadic\n createFunctionWithInputTypes(\"sha1\", Seq(SparkAnyType)))\n\n // Conditional expressions (corresponds to conditionalExpressions in QueryPlanSerde)\n val conditionalScalarFunc: Seq[Function] = Seq(\n createFunctionWithInputTypes(\"if\", Seq(SparkBooleanType, SparkAnyType, SparkAnyType)))\n\n // Map expressions (corresponds to mapExpressions in QueryPlanSerde)\n val mapScalarFunc: Seq[Function] = Seq(\n createFunctionWithInputTypes(\n \"map_extract\",\n Seq(SparkMapType(SparkAnyType, SparkAnyType), SparkAnyType)),\n createFunctionWithInputTypes(\"map_keys\", Seq(SparkMapType(SparkAnyType, SparkAnyType))),\n createFunctionWithInputTypes(\"map_entries\", Seq(SparkMapType(SparkAnyType, SparkAnyType))),\n createFunctionWithInputTypes(\"map_values\", Seq(SparkMapType(SparkAnyType, SparkAnyType))),\n createFunctionWithInputTypes(\n \"map_from_arrays\",\n Seq(SparkArrayType(SparkAnyType), SparkArrayType(SparkAnyType))),\n createFunctionWithInputTypes(\n \"map_from_entries\",\n Seq(\n SparkArrayType(\n SparkStructType(Seq(\n SparkTypeOneOf(primitiveSparkTypes.filterNot(_ == SparkBinaryType)),\n SparkTypeOneOf(primitiveSparkTypes.filterNot(_ == SparkBinaryType))))))))\n\n // Predicate expressions (corresponds to predicateExpressions in QueryPlanSerde)\n val predicateScalarFunc: Seq[Function] = Seq(\n createFunctionWithInputTypes(\"and\", Seq(SparkBooleanType, SparkBooleanType)),\n createFunctionWithInputTypes(\"or\", Seq(SparkBooleanType, SparkBooleanType)),\n createFunctionWithInputTypes(\"not\", Seq(SparkBooleanType)),\n createFunctionWithInputTypes(\"in\", Seq(SparkAnyType, SparkAnyType))\n ) // TODO: variadic\n\n // Struct expressions (corresponds to structExpressions in QueryPlanSerde)\n val structScalarFunc: Seq[Function] = Seq(\n createFunctionWithInputTypes(\n \"create_named_struct\",\n Seq(SparkStringType, SparkAnyType)\n ), // TODO: variadic name/value pairs\n createFunctionWithInputTypes(\n \"get_struct_field\",\n Seq(SparkStructType(Seq(SparkAnyType)), SparkStringType)))\n\n // Bitwise expressions (corresponds to bitwiseExpressions in QueryPlanSerde)\n val bitwiseScalarFunc: Seq[Function] = Seq(\n createFunctionWithInputTypes(\"bitwise_and\", Seq(SparkIntegralType, SparkIntegralType)),\n createFunctionWithInputTypes(\"bitwise_count\", Seq(SparkIntegralType)),\n createFunctionWithInputTypes(\"bitwise_get\", Seq(SparkIntegralType, SparkIntType)),\n createFunctionWithInputTypes(\"bitwise_or\", Seq(SparkIntegralType, SparkIntegralType)),\n createFunctionWithInputTypes(\"bitwise_not\", Seq(SparkIntegralType)),\n createFunctionWithInputTypes(\"bitwise_xor\", Seq(SparkIntegralType, SparkIntegralType)),\n createFunctionWithInputTypes(\"shift_left\", Seq(SparkIntegralType, SparkIntType)),\n createFunctionWithInputTypes(\"shift_right\", Seq(SparkIntegralType, SparkIntType)))\n\n // Misc expressions (corresponds to miscExpressions in QueryPlanSerde)\n val miscScalarFunc: Seq[Function] =\n Seq(\n createFunctionWithInputTypes(\"isnan\", Seq(SparkNumericType)),\n createFunctionWithInputTypes(\"isnull\", Seq(SparkAnyType)),\n createFunctionWithInputTypes(\"isnotnull\", Seq(SparkAnyType)),\n createFunctionWithInputTypes(\"coalesce\", Seq(SparkAnyType, SparkAnyType))\n ) // TODO: variadic\n\n // Array expressions (corresponds to arrayExpressions in QueryPlanSerde)\n val arrayScalarFunc: Seq[Function] = Seq(\n createFunctionWithInputTypes(\"array_append\", Seq(SparkArrayType(SparkAnyType), SparkAnyType)),\n createFunctionWithInputTypes(\"array_compact\", Seq(SparkArrayType(SparkAnyType))),\n createFunctionWithInputTypes(\n \"array_contains\",\n Seq(SparkArrayType(SparkAnyType), SparkAnyType)),\n createFunctionWithInputTypes(\"array_distinct\", Seq(SparkArrayType(SparkAnyType))),\n createFunctionWithInputTypes(\n \"array_except\",\n Seq(SparkArrayType(SparkAnyType), SparkArrayType(SparkAnyType))),\n createFunctionWithInputTypes(\n \"array_insert\",\n Seq(SparkArrayType(SparkAnyType), SparkIntType, SparkAnyType)),\n createFunctionWithInputTypes(\n \"array_intersect\",\n Seq(SparkArrayType(SparkAnyType), SparkArrayType(SparkAnyType))),\n createFunctions(\n \"array_join\",\n Seq(\n FunctionSignature(Seq(SparkArrayType(SparkAnyType), SparkStringType)),\n FunctionSignature(Seq(SparkArrayType(SparkAnyType), SparkStringType, SparkStringType)))),\n createFunctionWithInputTypes(\"array_max\", Seq(SparkArrayType(SparkAnyType))),\n createFunctionWithInputTypes(\"array_min\", Seq(SparkArrayType(SparkAnyType))),\n createFunctionWithInputTypes(\"array_remove\", Seq(SparkArrayType(SparkAnyType), SparkAnyType)),\n createFunctionWithInputTypes(\"array_repeat\", Seq(SparkAnyType, SparkIntType)),\n createFunctionWithInputTypes(\n \"arrays_overlap\",\n Seq(SparkArrayType(SparkAnyType), SparkArrayType(SparkAnyType))),\n createFunctionWithInputTypes(\n \"array_union\",\n Seq(SparkArrayType(SparkAnyType), SparkArrayType(SparkAnyType))),\n createFunctionWithInputTypes(\"array\", Seq(SparkAnyType, SparkAnyType)), // TODO: variadic\n createFunctionWithInputTypes(\n \"element_at\",\n Seq(\n SparkTypeOneOf(\n Seq(SparkArrayType(SparkAnyType), SparkMapType(SparkAnyType, SparkAnyType))),\n SparkAnyType)),\n createFunctionWithInputTypes(\"flatten\", Seq(SparkArrayType(SparkArrayType(SparkAnyType)))),\n createFunctionWithInputTypes(\n \"get_array_item\",\n Seq(SparkArrayType(SparkAnyType), SparkIntType)))\n\n // Temporal expressions (corresponds to temporalExpressions in QueryPlanSerde)\n val temporalScalarFunc: Seq[Function] =\n Seq(\n createFunctionWithInputTypes(\"date_add\", Seq(SparkDateType, SparkIntType)),\n createFunctionWithInputTypes(\"date_sub\", Seq(SparkDateType, SparkIntType)),\n createFunctions(\n \"from_unixtime\",\n Seq(\n FunctionSignature(Seq(SparkLongType)),\n FunctionSignature(Seq(SparkLongType, SparkStringType)))),\n createFunctionWithInputTypes(\"hour\", Seq(SparkDateOrTimestampType)),\n createFunctionWithInputTypes(\"minute\", Seq(SparkDateOrTimestampType)),\n createFunctionWithInputTypes(\"second\", Seq(SparkDateOrTimestampType)),\n createFunctionWithInputTypes(\"trunc\", Seq(SparkDateOrTimestampType, SparkStringType)),\n createFunctionWithInputTypes(\"year\", Seq(SparkDateOrTimestampType)),\n createFunctionWithInputTypes(\"month\", Seq(SparkDateOrTimestampType)),\n createFunctionWithInputTypes(\"day\", Seq(SparkDateOrTimestampType)),\n createFunctionWithInputTypes(\"dayofmonth\", Seq(SparkDateOrTimestampType)),\n createFunctionWithInputTypes(\"dayofweek\", Seq(SparkDateOrTimestampType)),\n createFunctionWithInputTypes(\"weekday\", Seq(SparkDateOrTimestampType)),\n createFunctionWithInputTypes(\"dayofyear\", Seq(SparkDateOrTimestampType)),\n createFunctionWithInputTypes(\"weekofyear\", Seq(SparkDateOrTimestampType)),\n createFunctionWithInputTypes(\"quarter\", Seq(SparkDateOrTimestampType)))\n\n // Combined in same order as exprSerdeMap in QueryPlanSerde\n val scalarFunc: Seq[Function] = mathScalarFunc ++ hashScalarFunc ++ stringScalarFunc ++\n conditionalScalarFunc ++ mapScalarFunc ++ predicateScalarFunc ++\n structScalarFunc ++ bitwiseScalarFunc ++ miscScalarFunc ++ arrayScalarFunc ++\n temporalScalarFunc\n\n val aggFunc: Seq[Function] = Seq(\n createFunctionWithInputTypes(\"min\", Seq(SparkAnyType)),\n createFunctionWithInputTypes(\"max\", Seq(SparkAnyType)),\n createFunctionWithInputTypes(\"count\", Seq(SparkAnyType)),\n createUnaryNumericFunction(\"avg\"),\n createUnaryNumericFunction(\"sum\"),\n // first/last are non-deterministic and known to be incompatible with Spark\n// createFunctionWithInputTypes(\"first\", Seq(SparkAnyType)),\n// createFunctionWithInputTypes(\"last\", Seq(SparkAnyType)),\n createUnaryNumericFunction(\"var_pop\"),\n createUnaryNumericFunction(\"var_samp\"),\n createFunctionWithInputTypes(\"covar_pop\", Seq(SparkNumericType, SparkNumericType)),\n createFunctionWithInputTypes(\"covar_samp\", Seq(SparkNumericType, SparkNumericType)),\n createUnaryNumericFunction(\"stddev_pop\"),\n createUnaryNumericFunction(\"stddev_samp\"),\n createFunctionWithInputTypes(\"corr\", Seq(SparkNumericType, SparkNumericType)),\n createFunctionWithInputTypes(\"bit_and\", Seq(SparkIntegralType)),\n createFunctionWithInputTypes(\"bit_or\", Seq(SparkIntegralType)),\n createFunctionWithInputTypes(\"bit_xor\", Seq(SparkIntegralType)))\n\n val unaryArithmeticOps: Seq[String] = Seq(\"+\", \"-\")\n\n val binaryArithmeticOps: Seq[String] =\n Seq(\"+\", \"-\", \"*\", \"/\", \"%\", \"&\", \"|\", \"^\", \"<<\", \">>\", \"div\")\n\n val comparisonOps: Seq[String] = Seq(\"=\", \"<=>\", \">\", \">=\", \"<\", \"<=\")\n\n // TODO make this more comprehensive\n val comparisonTypes: Seq[SparkType] = Seq(\n SparkStringType,\n SparkBinaryType,\n SparkNumericType,\n SparkDateType,\n SparkTimestampType,\n SparkArrayType(SparkTypeOneOf(Seq(SparkStringType, SparkNumericType, SparkDateType))))\n\n}\n" }, { "path": "fuzz-testing/src/main/scala/org/apache/comet/fuzz/QueryGen.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.fuzz\n\nimport java.io.{BufferedWriter, FileWriter}\n\nimport scala.annotation.tailrec\nimport scala.collection.mutable\nimport scala.util.Random\n\nimport org.apache.spark.sql.{DataFrame, SparkSession}\nimport org.apache.spark.sql.types._\n\nobject QueryGen {\n\n def generateRandomQueries(\n r: Random,\n spark: SparkSession,\n numFiles: Int,\n numQueries: Int): Unit = {\n for (i <- 0 until numFiles) {\n spark.read.parquet(s\"test$i.parquet\").createTempView(s\"test$i\")\n }\n\n val w = new BufferedWriter(new FileWriter(\"queries.sql\"))\n\n val uniqueQueries = mutable.HashSet[String]()\n\n for (_ <- 0 until numQueries) {\n try {\n val sql = r.nextInt().abs % 8 match {\n case 0 => generateJoin(r, spark, numFiles)\n case 1 => generateAggregate(r, spark, numFiles)\n case 2 => generateScalar(r, spark, numFiles)\n case 3 => generateCast(r, spark, numFiles)\n case 4 => generateUnaryArithmetic(r, spark, numFiles)\n case 5 => generateBinaryArithmetic(r, spark, numFiles)\n case 6 => generateBinaryComparison(r, spark, numFiles)\n case _ => generateConditional(r, spark, numFiles)\n }\n if (!uniqueQueries.contains(sql)) {\n uniqueQueries += sql\n w.write(sql + \"\\n\")\n }\n } catch {\n case e: Exception =>\n // scalastyle:off\n println(s\"Failed to generate query: ${e.getMessage}\")\n }\n }\n w.close()\n }\n\n private def generateAggregate(r: Random, spark: SparkSession, numFiles: Int): String = {\n val tableName = s\"test${r.nextInt(numFiles)}\"\n val table = spark.table(tableName)\n\n val func = Utils.randomChoice(Meta.aggFunc, r)\n try {\n val signature = Utils.randomChoice(func.signatures, r)\n val args = signature.inputTypes.map(x => pickRandomColumn(r, table, x))\n\n val groupingCols = Range(0, 2).map(_ => Utils.randomChoice(table.columns, r))\n\n if (groupingCols.isEmpty) {\n s\"SELECT ${args.mkString(\", \")}, ${func.name}(${args.mkString(\", \")}) AS x \" +\n s\"FROM $tableName \" +\n s\"ORDER BY ${args.mkString(\", \")};\"\n } else {\n s\"SELECT ${groupingCols.mkString(\", \")}, ${func.name}(${args.mkString(\", \")}) \" +\n s\"FROM $tableName \" +\n s\"GROUP BY ${groupingCols.mkString(\",\")} \" +\n s\"ORDER BY ${groupingCols.mkString(\", \")};\"\n }\n } catch {\n case e: Exception =>\n throw new IllegalStateException(\n s\"Failed to generate SQL for aggregate function ${func.name}\",\n e)\n }\n }\n\n private def generateScalar(r: Random, spark: SparkSession, numFiles: Int): String = {\n val tableName = s\"test${r.nextInt(numFiles)}\"\n val table = spark.table(tableName)\n\n val func = Utils.randomChoice(Meta.scalarFunc, r)\n try {\n val signature = Utils.randomChoice(func.signatures, r)\n val args =\n if (signature.varArgs) {\n pickRandomColumns(r, table, signature.inputTypes.head)\n } else {\n signature.inputTypes.map(x => pickRandomColumn(r, table, x))\n }\n\n // Example SELECT c0, log(c0) as x FROM test0\n s\"SELECT ${args.mkString(\", \")}, ${func.name}(${args.mkString(\", \")}) AS x \" +\n s\"FROM $tableName \" +\n s\"ORDER BY ${args.mkString(\", \")};\"\n } catch {\n case e: Exception =>\n throw new IllegalStateException(\n s\"Failed to generate SQL for scalar function ${func.name}\",\n e)\n }\n }\n\n @tailrec\n private def pickRandomColumns(r: Random, df: DataFrame, targetType: SparkType): Seq[String] = {\n targetType match {\n case SparkTypeOneOf(choices) =>\n val chosenType = Utils.randomChoice(choices, r)\n pickRandomColumns(r, df, chosenType)\n case _ =>\n var columns = Set.empty[String]\n for (_ <- 0 to r.nextInt(df.columns.length)) {\n columns += pickRandomColumn(r, df, targetType)\n }\n columns.toSeq\n }\n }\n\n private def pickRandomColumn(r: Random, df: DataFrame, targetType: SparkType): String = {\n targetType match {\n case SparkAnyType =>\n Utils.randomChoice(df.schema.fields, r).name\n case SparkBooleanType =>\n select(r, df, _.dataType == BooleanType)\n case SparkByteType =>\n select(r, df, _.dataType == ByteType)\n case SparkShortType =>\n select(r, df, _.dataType == ShortType)\n case SparkIntType =>\n select(r, df, _.dataType == IntegerType)\n case SparkLongType =>\n select(r, df, _.dataType == LongType)\n case SparkFloatType =>\n select(r, df, _.dataType == FloatType)\n case SparkDoubleType =>\n select(r, df, _.dataType == DoubleType)\n case SparkDecimalType(_, _) =>\n select(r, df, _.dataType.isInstanceOf[DecimalType])\n case SparkIntegralType =>\n select(\n r,\n df,\n f =>\n f.dataType == ByteType || f.dataType == ShortType ||\n f.dataType == IntegerType || f.dataType == LongType)\n case SparkNumericType =>\n select(r, df, f => isNumeric(f.dataType))\n case SparkStringType =>\n select(r, df, _.dataType == StringType)\n case SparkBinaryType =>\n select(r, df, _.dataType == BinaryType)\n case SparkDateType =>\n select(r, df, _.dataType == DateType)\n case SparkTimestampType =>\n select(r, df, _.dataType == TimestampType)\n case SparkDateOrTimestampType =>\n select(r, df, f => f.dataType == DateType || f.dataType == TimestampType)\n case SparkTypeOneOf(choices) =>\n pickRandomColumn(r, df, Utils.randomChoice(choices, r))\n case SparkArrayType(elementType) =>\n select(\n r,\n df,\n _.dataType match {\n case ArrayType(x, _) if typeMatch(elementType, x) => true\n case _ => false\n })\n case SparkMapType(keyType, valueType) =>\n select(\n r,\n df,\n _.dataType match {\n case MapType(k, v, _) if typeMatch(keyType, k) && typeMatch(valueType, v) => true\n case _ => false\n })\n case SparkStructType(fields) =>\n select(\n r,\n df,\n _.dataType match {\n case StructType(structFields) if structFields.length == fields.length => true\n case _ => false\n })\n case _ =>\n throw new IllegalStateException(targetType.toString)\n }\n }\n\n def pickTwoRandomColumns(r: Random, df: DataFrame, targetType: SparkType): (String, String) = {\n val a = pickRandomColumn(r, df, targetType)\n val df2 = df.drop(a)\n val b = pickRandomColumn(r, df2, targetType)\n (a, b)\n }\n\n /** Select a random field that matches a predicate */\n private def select(r: Random, df: DataFrame, predicate: StructField => Boolean): String = {\n val candidates = df.schema.fields.filter(predicate)\n if (candidates.isEmpty) {\n throw new IllegalStateException(\"Failed to find suitable column\")\n }\n Utils.randomChoice(candidates, r).name\n }\n\n private def isNumeric(d: DataType): Boolean = {\n d match {\n case _: ByteType | _: ShortType | _: IntegerType | _: LongType | _: FloatType |\n _: DoubleType | _: DecimalType =>\n true\n case _ => false\n }\n }\n\n private def typeMatch(s: SparkType, d: DataType): Boolean = {\n (s, d) match {\n case (SparkAnyType, _) => true\n case (SparkBooleanType, BooleanType) => true\n case (SparkByteType, ByteType) => true\n case (SparkShortType, ShortType) => true\n case (SparkIntType, IntegerType) => true\n case (SparkLongType, LongType) => true\n case (SparkFloatType, FloatType) => true\n case (SparkDoubleType, DoubleType) => true\n case (SparkDecimalType(_, _), _: DecimalType) => true\n case (SparkIntegralType, ByteType | ShortType | IntegerType | LongType) => true\n case (SparkNumericType, _) if isNumeric(d) => true\n case (SparkStringType, StringType) => true\n case (SparkBinaryType, BinaryType) => true\n case (SparkDateType, DateType) => true\n case (SparkTimestampType, TimestampType | TimestampNTZType) => true\n case (SparkDateOrTimestampType, DateType | TimestampType | TimestampNTZType) => true\n case (SparkArrayType(elementType), ArrayType(elementDataType, _)) =>\n typeMatch(elementType, elementDataType)\n case (SparkMapType(keyType, valueType), MapType(keyDataType, valueDataType, _)) =>\n typeMatch(keyType, keyDataType) && typeMatch(valueType, valueDataType)\n case (SparkStructType(fields), StructType(structFields)) =>\n fields.length == structFields.length &&\n fields.zip(structFields.map(_.dataType)).forall { case (sparkType, dataType) =>\n typeMatch(sparkType, dataType)\n }\n case (SparkTypeOneOf(choices), _) =>\n choices.exists(choice => typeMatch(choice, d))\n case _ => false\n }\n }\n\n private def generateUnaryArithmetic(r: Random, spark: SparkSession, numFiles: Int): String = {\n val tableName = s\"test${r.nextInt(numFiles)}\"\n val table = spark.table(tableName)\n\n val op = Utils.randomChoice(Meta.unaryArithmeticOps, r)\n val a = pickRandomColumn(r, table, SparkNumericType)\n\n // Example SELECT a, -a FROM test0\n s\"SELECT $a, $op$a \" +\n s\"FROM $tableName \" +\n s\"ORDER BY $a;\"\n }\n\n private def generateBinaryArithmetic(r: Random, spark: SparkSession, numFiles: Int): String = {\n val tableName = s\"test${r.nextInt(numFiles)}\"\n val table = spark.table(tableName)\n\n val op = Utils.randomChoice(Meta.binaryArithmeticOps, r)\n val (a, b) = pickTwoRandomColumns(r, table, SparkNumericType)\n\n // Example SELECT a, b, a+b FROM test0\n s\"SELECT $a, $b, $a $op $b \" +\n s\"FROM $tableName \" +\n s\"ORDER BY $a, $b;\"\n }\n\n private def generateBinaryComparison(r: Random, spark: SparkSession, numFiles: Int): String = {\n val tableName = s\"test${r.nextInt(numFiles)}\"\n val table = spark.table(tableName)\n\n val op = Utils.randomChoice(Meta.comparisonOps, r)\n\n // pick two columns with the same type\n val opType = Utils.randomChoice(Meta.comparisonTypes, r)\n val (a, b) = pickTwoRandomColumns(r, table, opType)\n\n // Example SELECT a, b, a <=> b FROM test0\n s\"SELECT $a, $b, $a $op $b \" +\n s\"FROM $tableName \" +\n s\"ORDER BY $a, $b;\"\n }\n\n private def generateConditional(r: Random, spark: SparkSession, numFiles: Int): String = {\n val tableName = s\"test${r.nextInt(numFiles)}\"\n val table = spark.table(tableName)\n\n val op = Utils.randomChoice(Meta.comparisonOps, r)\n\n // pick two columns with the same type\n val opType = Utils.randomChoice(Meta.comparisonTypes, r)\n val (a, b) = pickTwoRandomColumns(r, table, opType)\n\n // Example SELECT a, b, IF(a <=> b, 1, 2), CASE WHEN a <=> b THEN 1 ELSE 2 END FROM test0\n s\"SELECT $a, $b, $a $op $b, IF($a $op $b, 1, 2), CASE WHEN $a $op $b THEN 1 ELSE 2 END \" +\n s\"FROM $tableName \" +\n s\"ORDER BY $a, $b;\"\n }\n\n private def generateCast(r: Random, spark: SparkSession, numFiles: Int): String = {\n val tableName = s\"test${r.nextInt(numFiles)}\"\n val table = spark.table(tableName)\n\n val toType = Utils.randomWeightedChoice(Meta.dataTypes, r).sql\n val arg = Utils.randomChoice(table.columns, r)\n\n // We test both `cast` and `try_cast` to cover LEGACY and TRY eval modes. It is not\n // recommended to run Comet Fuzz with ANSI enabled currently.\n // Example SELECT c0, cast(c0 as float), try_cast(c0 as float) FROM test0\n s\"SELECT $arg, cast($arg as $toType), try_cast($arg as $toType) \" +\n s\"FROM $tableName \" +\n s\"ORDER BY $arg;\"\n }\n\n private def generateJoin(r: Random, spark: SparkSession, numFiles: Int): String = {\n val leftTableName = s\"test${r.nextInt(numFiles)}\"\n val rightTableName = s\"test${r.nextInt(numFiles)}\"\n val leftTable = spark.table(leftTableName)\n val rightTable = spark.table(rightTableName)\n\n val leftCol = Utils.randomChoice(leftTable.columns, r)\n val rightCol = Utils.randomChoice(rightTable.columns, r)\n\n val joinTypes = Seq((\"INNER\", 0.4), (\"LEFT\", 0.3), (\"RIGHT\", 0.3))\n val joinType = Utils.randomWeightedChoice(joinTypes, r)\n\n val leftColProjection = leftTable.columns.map(c => s\"l.$c\").mkString(\", \")\n val rightColProjection = rightTable.columns.map(c => s\"r.$c\").mkString(\", \")\n \"SELECT \" +\n s\"$leftColProjection, \" +\n s\"$rightColProjection \" +\n s\"FROM $leftTableName l \" +\n s\"$joinType JOIN $rightTableName r \" +\n s\"ON l.$leftCol = r.$rightCol \" +\n \"ORDER BY \" +\n s\"$leftColProjection, \" +\n s\"$rightColProjection;\"\n }\n\n}\n" }, { "path": "fuzz-testing/src/main/scala/org/apache/comet/fuzz/QueryRunner.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.fuzz\n\nimport java.io.{BufferedWriter, FileWriter, PrintWriter, StringWriter}\n\nimport scala.collection.mutable\nimport scala.io.Source\n\nimport org.apache.spark.sql.{Row, SparkSession}\n\nimport org.apache.comet.fuzz.QueryComparison.showPlans\n\nobject QueryRunner {\n\n def createOutputMdFile(): BufferedWriter = {\n val outputFilename = s\"results-${System.currentTimeMillis()}.md\"\n // scalastyle:off println\n println(s\"Writing results to $outputFilename\")\n // scalastyle:on println\n\n new BufferedWriter(new FileWriter(outputFilename))\n }\n\n def runQueries(\n spark: SparkSession,\n numFiles: Int,\n filename: String,\n showFailedSparkQueries: Boolean = false): Unit = {\n\n var queryCount = 0\n var invalidQueryCount = 0\n var cometFailureCount = 0\n var cometSuccessCount = 0\n\n val w = createOutputMdFile()\n\n // register input files\n for (i <- 0 until numFiles) {\n val table = spark.read.parquet(s\"test$i.parquet\")\n val tableName = s\"test$i\"\n table.createTempView(tableName)\n w.write(\n s\"Created table $tableName with schema:\\n\\t\" +\n s\"${table.schema.fields.map(f => s\"${f.name}: ${f.dataType}\").mkString(\"\\n\\t\")}\\n\\n\")\n }\n\n val querySource = Source.fromFile(filename)\n try {\n querySource\n .getLines()\n .foreach(sql => {\n queryCount += 1\n try {\n // execute with Spark\n spark.conf.set(\"spark.comet.enabled\", \"false\")\n val df = spark.sql(sql)\n val sparkRows = df.collect()\n val sparkPlan = df.queryExecution.executedPlan.toString\n\n // execute with Comet\n try {\n spark.conf.set(\"spark.comet.enabled\", \"true\")\n val df = spark.sql(sql)\n val cometRows = df.collect()\n val cometPlan = df.queryExecution.executedPlan.toString\n\n var success = QueryComparison.assertSameRows(sparkRows, cometRows, output = w)\n\n // check that the plan contains Comet operators\n if (!cometPlan.contains(\"Comet\")) {\n success = false\n w.write(\"[ERROR] Comet did not accelerate any part of the plan\\n\")\n }\n\n QueryComparison.showSQL(w, sql)\n\n if (success) {\n cometSuccessCount += 1\n } else {\n // show plans for failed queries\n showPlans(w, sparkPlan, cometPlan)\n cometFailureCount += 1\n }\n\n } catch {\n case e: Exception =>\n // the query worked in Spark but failed in Comet, so this is likely a bug in Comet\n cometFailureCount += 1\n QueryComparison.showSQL(w, sql)\n w.write(\"### Spark Plan\\n\")\n w.write(s\"```\\n$sparkPlan\\n```\\n\")\n\n w.write(s\"[ERROR] Query failed in Comet: ${e.getMessage}:\\n\")\n w.write(\"```\\n\")\n val sw = new StringWriter()\n val p = new PrintWriter(sw)\n e.printStackTrace(p)\n p.close()\n w.write(s\"${sw.toString}\\n\")\n w.write(\"```\\n\")\n }\n\n // flush after every query so that results are saved in the event of the driver crashing\n w.flush()\n\n } catch {\n case e: Exception =>\n // we expect many generated queries to be invalid\n invalidQueryCount += 1\n if (showFailedSparkQueries) {\n QueryComparison.showSQL(w, sql)\n w.write(s\"Query failed in Spark: ${e.getMessage}\\n\")\n }\n }\n })\n\n w.write(\"# Summary\\n\")\n w.write(\n s\"Total queries: $queryCount; Invalid queries: $invalidQueryCount; \" +\n s\"Comet failed: $cometFailureCount; Comet succeeded: $cometSuccessCount\\n\")\n\n } finally {\n w.close()\n querySource.close()\n }\n }\n}\n\nobject QueryComparison {\n def assertSameRows(\n sparkRows: Array[Row],\n cometRows: Array[Row],\n output: BufferedWriter,\n tolerance: Double = 0.000001): Boolean = {\n if (sparkRows.length == cometRows.length) {\n var i = 0\n while (i < sparkRows.length) {\n val l = sparkRows(i)\n val r = cometRows(i)\n\n // Check the schema is equal for first row only\n if (i == 0 && l.schema != r.schema) {\n output.write(\"[ERROR] Spark produced different schema than Comet.\\n\")\n\n return false\n }\n\n assert(l.length == r.length)\n for (j <- 0 until l.length) {\n if (!same(l(j), r(j), tolerance)) {\n output.write(s\"First difference at row $i:\\n\")\n output.write(\"Spark: `\" + formatRow(l) + \"`\\n\")\n output.write(\"Comet: `\" + formatRow(r) + \"`\\n\")\n i = sparkRows.length\n\n return false\n }\n }\n i += 1\n }\n } else {\n output.write(\n s\"[ERROR] Spark produced ${sparkRows.length} rows and \" +\n s\"Comet produced ${cometRows.length} rows.\\n\")\n\n return false\n }\n\n true\n }\n\n private def same(l: Any, r: Any, tolerance: Double): Boolean = {\n if (l == null || r == null) {\n return l == null && r == null\n }\n (l, r) match {\n case (a: Float, b: Float) if a.isPosInfinity => b.isPosInfinity\n case (a: Float, b: Float) if a.isNegInfinity => b.isNegInfinity\n case (a: Float, b: Float) if a.isInfinity => b.isInfinity\n case (a: Float, b: Float) if a.isNaN => b.isNaN\n case (a: Float, b: Float) => (a - b).abs <= tolerance\n case (a: Double, b: Double) if a.isPosInfinity => b.isPosInfinity\n case (a: Double, b: Double) if a.isNegInfinity => b.isNegInfinity\n case (a: Double, b: Double) if a.isInfinity => b.isInfinity\n case (a: Double, b: Double) if a.isNaN => b.isNaN\n case (a: Double, b: Double) => (a - b).abs <= tolerance\n case (a: Array[_], b: Array[_]) =>\n a.length == b.length && a.zip(b).forall(x => same(x._1, x._2, tolerance))\n case (a: mutable.WrappedArray[_], b: mutable.WrappedArray[_]) =>\n a.length == b.length && a.zip(b).forall(x => same(x._1, x._2, tolerance))\n case (a: Row, b: Row) =>\n val aa = a.toSeq\n val bb = b.toSeq\n aa.length == bb.length && aa.zip(bb).forall(x => same(x._1, x._2, tolerance))\n case (a, b) => a == b\n }\n }\n\n private def format(value: Any): String = {\n value match {\n case null => \"NULL\"\n case v: mutable.WrappedArray[_] => s\"[${v.map(format).mkString(\",\")}]\"\n case v: Array[Byte] => s\"[${v.mkString(\",\")}]\"\n case r: Row => formatRow(r)\n case other => other.toString\n }\n }\n\n private def formatRow(row: Row): String = {\n row.toSeq.map(format).mkString(\",\")\n }\n\n def showSQL(w: BufferedWriter, sql: String, maxLength: Int = 120): Unit = {\n w.write(\"## SQL\\n\")\n w.write(\"```\\n\")\n val words = sql.split(\" \")\n val currentLine = new StringBuilder\n for (word <- words) {\n if (currentLine.length + word.length + 1 > maxLength) {\n w.write(currentLine.toString.trim)\n w.write(\"\\n\")\n currentLine.setLength(0)\n }\n currentLine.append(word).append(\" \")\n }\n if (currentLine.nonEmpty) {\n w.write(currentLine.toString.trim)\n w.write(\"\\n\")\n }\n w.write(\"```\\n\")\n }\n\n def showPlans(w: BufferedWriter, sparkPlan: String, cometPlan: String): Unit = {\n w.write(\"### Spark Plan\\n\")\n w.write(s\"```\\n$sparkPlan\\n```\\n\")\n w.write(\"### Comet Plan\\n\")\n w.write(s\"```\\n$cometPlan\\n```\\n\")\n }\n\n def showSchema(w: BufferedWriter, sparkSchema: String, cometSchema: String): Unit = {\n w.write(\"### Spark Schema\\n\")\n w.write(s\"```\\n$sparkSchema\\n```\\n\")\n w.write(\"### Comet Schema\\n\")\n w.write(s\"```\\n$cometSchema\\n```\\n\")\n }\n}\n" }, { "path": "fuzz-testing/src/main/scala/org/apache/comet/fuzz/Utils.scala", "content": "/*\n * Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements. See the NOTICE file\n * distributed with this work for additional information\n * regarding copyright ownership. The ASF licenses this file\n * to you under the Apache License, Version 2.0 (the\n * \"License\"); you may not use this file except in compliance\n * with the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\npackage org.apache.comet.fuzz\n\nimport scala.util.Random\n\nobject Utils {\n\n def randomChoice[T](list: Seq[T], r: Random): T = {\n list(r.nextInt(list.length))\n }\n\n def randomWeightedChoice[T](valuesWithWeights: Seq[(T, Double)], r: Random): T = {\n val totalWeight = valuesWithWeights.map(_._2).sum\n val randomValue = r.nextDouble() * totalWeight\n var cumulativeWeight = 0.0\n\n for ((value, weight) <- valuesWithWeights) {\n cumulativeWeight += weight\n if (cumulativeWeight >= randomValue) {\n return value\n }\n }\n\n // If for some reason the loop doesn't return, return the last value\n valuesWithWeights.last._1\n }\n\n}\n" }, { "path": "kube/Dockerfile", "content": "#\n# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n\nFROM apache/spark:3.5.8 AS builder\n\nUSER root\n\n# Installing JDK11 as the image comes with JRE\nRUN apt update \\\n && apt install -y curl \\\n && apt install -y openjdk-11-jdk \\\n && apt clean\n\nRUN apt install -y gcc-10 g++-10 cpp-10 unzip\nENV CC=\"gcc-10\"\nENV CXX=\"g++-10\"\n\nRUN PB_REL=\"https://github.com/protocolbuffers/protobuf/releases\" \\\n && curl -LO $PB_REL/download/v30.2/protoc-30.2-linux-x86_64.zip \\\n && unzip protoc-30.2-linux-x86_64.zip -d /root/.local\nENV PATH=\"$PATH:/root/.local/bin\"\n\nRUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y\nENV PATH=\"/root/.cargo/bin:${PATH}\"\nENV RUSTFLAGS=\"-C debuginfo=line-tables-only -C incremental=false\"\nENV SPARK_VERSION=3.5\nENV SCALA_VERSION=2.12\n\n# copy source files to Docker image\nRUN mkdir /comet\nWORKDIR /comet\n\n# build native code first so that this layer can be re-used\n# if only Scala code gets modified\nCOPY native /comet/native\nRUN cd native && RUSTFLAGS=\"-Ctarget-cpu=native\" cargo build --release\n\n# copy the rest of the project\nCOPY .mvn /comet/.mvn\nCOPY mvnw /comet/mvnw\nCOPY common /comet/common\nCOPY dev /comet/dev\nCOPY docs /comet/docs\nCOPY fuzz-testing /comet/fuzz-testing\nCOPY spark /comet/spark\nCOPY spark-integration /comet/spark-integration\nCOPY scalafmt.conf /comet/scalafmt.conf\nCOPY .scalafix.conf /comet/.scalafix.conf\nCOPY Makefile /comet/Makefile\nCOPY pom.xml /comet/pom.xml\n\nRUN mkdir -p /root/.m2 && \\\n echo 'centralcentralhttps://repo1.maven.org/maven2' > /root/.m2/settings.xml\n\n# Pick the JDK instead of JRE to compile Comet\nRUN cd /comet \\\n && JAVA_HOME=$(readlink -f $(which javac) | sed \"s/\\/bin\\/javac//\") make release-nogit PROFILES=\"-Pspark-$SPARK_VERSION -Pscala-$SCALA_VERSION\"\n\nFROM apache/spark:3.5.8\nENV SPARK_VERSION=3.5\nENV SCALA_VERSION=2.12\nUSER root\n\n# note the use of a wildcard in the file name so that this works with both snapshot and final release versions\nCOPY --from=builder /comet/spark/target/comet-spark-spark${SPARK_VERSION}_$SCALA_VERSION-*.jar $SPARK_HOME/jars\n" }, { "path": "kube/local/hadoop.env", "content": "#\n# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n\nCORE_CONF_fs_defaultFS=hdfs://namenode:9000\nCORE_CONF_hadoop_http_staticuser_user=root\nCORE_CONF_hadoop_proxyuser_hue_hosts=*\nCORE_CONF_hadoop_proxyuser_hue_groups=*\nCORE_CONF_io_compression_codecs=org.apache.hadoop.io.compress.SnappyCodec\nCORE_CONF_hadoop_tmp_dir=/hadoop-data\nCORE_CONF_dfs_client_use_datanode_hostname=true\nCORE_CONF_dfs_datanode_use_datanode_hostname=true\n\nHDFS_CONF_dfs_webhdfs_enabled=true\nHDFS_CONF_dfs_permissions_enabled=false\nHDFS_CONF_dfs_namenode_datanode_registration_ip___hostname___check=false\nHDFS_CONF_dfs_client_use_datanode_hostname=true\nHDFS_CONF_dfs_datanode_use_datanode_hostname=true" }, { "path": "kube/local/hdfs-docker-compose.yml", "content": "#\n# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n\nversion: \"3\"\n\nservices:\n namenode:\n image: bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8\n container_name: namenode\n restart: always\n ports:\n - 9870:9870\n - 9000:9000\n volumes:\n - /tmp/hadoop/dfs/name:/hadoop/dfs/name\n environment:\n - CLUSTER_NAME=test\n env_file:\n - hadoop.env\n\n datanode1:\n image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8\n container_name: datanode1\n hostname: datanode1\n restart: always\n ports:\n - 9866:9866\n - 9864:9864\n depends_on:\n - namenode\n volumes:\n - /tmp/hadoop/dfs/data1:/hadoop/dfs/data\n environment:\n SERVICE_PRECONDITION: \"namenode:9870\"\n env_file:\n - hadoop.env\n datanode2:\n image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8\n container_name: datanode2\n hostname: datanode2\n restart: always\n ports:\n - 9867:9866\n - 9865:9864\n depends_on:\n - namenode\n volumes:\n - /tmp/hadoop/dfs/data2:/hadoop/dfs/data\n environment:\n SERVICE_PRECONDITION: \"namenode:9870\"\n env_file:\n - hadoop.env\n datanode3:\n image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8\n container_name: datanode3\n hostname: datanode3\n restart: always\n ports:\n - 9868:9866\n - 9863:9864\n depends_on:\n - namenode\n volumes:\n - /tmp/hadoop/dfs/data3:/hadoop/dfs/data\n environment:\n SERVICE_PRECONDITION: \"namenode:9870\"\n env_file:\n - hadoop.env\n\n resourcemanager:\n image: bde2020/hadoop-resourcemanager:2.0.0-hadoop3.2.1-java8\n container_name: resourcemanager\n restart: always\n environment:\n SERVICE_PRECONDITION: \"namenode:9000 namenode:9870 datanode1:9864 datanode2:9864 datanode3:9864 datanode1:9866 datanode2:9866 datanode3:9866\"\n env_file:\n - hadoop.env\n\n nodemanager1:\n image: bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8\n container_name: nodemanager\n restart: always\n environment:\n SERVICE_PRECONDITION: \"namenode:9000 namenode:9870 datanode1:9864 datanode2:9864 datanode3:9864 datanode1:9866 datanode2:9866 datanode3:9866 resourcemanager:8088\"\n env_file:\n - hadoop.env" }, { "path": "mvnw", "content": "#!/bin/sh\n# ----------------------------------------------------------------------------\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n# ----------------------------------------------------------------------------\n\n# ----------------------------------------------------------------------------\n# Apache Maven Wrapper startup batch script, version 3.2.0\n#\n# Required ENV vars:\n# ------------------\n# JAVA_HOME - location of a JDK home dir\n#\n# Optional ENV vars\n# -----------------\n# MAVEN_OPTS - parameters passed to the Java VM when running Maven\n# e.g. to debug Maven itself, use\n# set MAVEN_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000\n# MAVEN_SKIP_RC - flag to disable loading of mavenrc files\n# ----------------------------------------------------------------------------\n\nif [ -z \"$MAVEN_SKIP_RC\" ] ; then\n\n if [ -f /usr/local/etc/mavenrc ] ; then\n . /usr/local/etc/mavenrc\n fi\n\n if [ -f /etc/mavenrc ] ; then\n . /etc/mavenrc\n fi\n\n if [ -f \"$HOME/.mavenrc\" ] ; then\n . \"$HOME/.mavenrc\"\n fi\n\nfi\n\n# OS specific support. $var _must_ be set to either true or false.\ncygwin=false;\ndarwin=false;\nmingw=false\ncase \"$(uname)\" in\n CYGWIN*) cygwin=true ;;\n MINGW*) mingw=true;;\n Darwin*) darwin=true\n # Use /usr/libexec/java_home if available, otherwise fall back to /Library/Java/Home\n # See https://developer.apple.com/library/mac/qa/qa1170/_index.html\n if [ -z \"$JAVA_HOME\" ]; then\n if [ -x \"/usr/libexec/java_home\" ]; then\n JAVA_HOME=\"$(/usr/libexec/java_home)\"; export JAVA_HOME\n else\n JAVA_HOME=\"/Library/Java/Home\"; export JAVA_HOME\n fi\n fi\n ;;\nesac\n\nif [ -z \"$JAVA_HOME\" ] ; then\n if [ -r /etc/gentoo-release ] ; then\n JAVA_HOME=$(java-config --jre-home)\n fi\nfi\n\n# For Cygwin, ensure paths are in UNIX format before anything is touched\nif $cygwin ; then\n [ -n \"$JAVA_HOME\" ] &&\n JAVA_HOME=$(cygpath --unix \"$JAVA_HOME\")\n [ -n \"$CLASSPATH\" ] &&\n CLASSPATH=$(cygpath --path --unix \"$CLASSPATH\")\nfi\n\n# For Mingw, ensure paths are in UNIX format before anything is touched\nif $mingw ; then\n [ -n \"$JAVA_HOME\" ] && [ -d \"$JAVA_HOME\" ] &&\n JAVA_HOME=\"$(cd \"$JAVA_HOME\" || (echo \"cannot cd into $JAVA_HOME.\"; exit 1); pwd)\"\nfi\n\nif [ -z \"$JAVA_HOME\" ]; then\n javaExecutable=\"$(which javac)\"\n if [ -n \"$javaExecutable\" ] && ! [ \"$(expr \"\\\"$javaExecutable\\\"\" : '\\([^ ]*\\)')\" = \"no\" ]; then\n # readlink(1) is not available as standard on Solaris 10.\n readLink=$(which readlink)\n if [ ! \"$(expr \"$readLink\" : '\\([^ ]*\\)')\" = \"no\" ]; then\n if $darwin ; then\n javaHome=\"$(dirname \"\\\"$javaExecutable\\\"\")\"\n javaExecutable=\"$(cd \"\\\"$javaHome\\\"\" && pwd -P)/javac\"\n else\n javaExecutable=\"$(readlink -f \"\\\"$javaExecutable\\\"\")\"\n fi\n javaHome=\"$(dirname \"\\\"$javaExecutable\\\"\")\"\n javaHome=$(expr \"$javaHome\" : '\\(.*\\)/bin')\n JAVA_HOME=\"$javaHome\"\n export JAVA_HOME\n fi\n fi\nfi\n\nif [ -z \"$JAVACMD\" ] ; then\n if [ -n \"$JAVA_HOME\" ] ; then\n if [ -x \"$JAVA_HOME/jre/sh/java\" ] ; then\n # IBM's JDK on AIX uses strange locations for the executables\n JAVACMD=\"$JAVA_HOME/jre/sh/java\"\n else\n JAVACMD=\"$JAVA_HOME/bin/java\"\n fi\n else\n JAVACMD=\"$(\\unset -f command 2>/dev/null; \\command -v java)\"\n fi\nfi\n\nif [ ! -x \"$JAVACMD\" ] ; then\n echo \"Error: JAVA_HOME is not defined correctly.\" >&2\n echo \" We cannot execute $JAVACMD\" >&2\n exit 1\nfi\n\nif [ -z \"$JAVA_HOME\" ] ; then\n echo \"Warning: JAVA_HOME environment variable is not set.\"\nfi\n\n# traverses directory structure from process work directory to filesystem root\n# first directory with .mvn subdirectory is considered project base directory\nfind_maven_basedir() {\n if [ -z \"$1\" ]\n then\n echo \"Path not specified to find_maven_basedir\"\n return 1\n fi\n\n basedir=\"$1\"\n wdir=\"$1\"\n while [ \"$wdir\" != '/' ] ; do\n if [ -d \"$wdir\"/.mvn ] ; then\n basedir=$wdir\n break\n fi\n # workaround for JBEAP-8937 (on Solaris 10/Sparc)\n if [ -d \"${wdir}\" ]; then\n wdir=$(cd \"$wdir/..\" || exit 1; pwd)\n fi\n # end of workaround\n done\n printf '%s' \"$(cd \"$basedir\" || exit 1; pwd)\"\n}\n\n# concatenates all lines of a file\nconcat_lines() {\n if [ -f \"$1\" ]; then\n # Remove \\r in case we run on Windows within Git Bash\n # and check out the repository with auto CRLF management\n # enabled. Otherwise, we may read lines that are delimited with\n # \\r\\n and produce $'-Xarg\\r' rather than -Xarg due to word\n # splitting rules.\n tr -s '\\r\\n' ' ' < \"$1\"\n fi\n}\n\nlog() {\n if [ \"$MVNW_VERBOSE\" = true ]; then\n printf '%s\\n' \"$1\"\n fi\n}\n\nBASE_DIR=$(find_maven_basedir \"$(dirname \"$0\")\")\nif [ -z \"$BASE_DIR\" ]; then\n exit 1;\nfi\n\nMAVEN_PROJECTBASEDIR=${MAVEN_BASEDIR:-\"$BASE_DIR\"}; export MAVEN_PROJECTBASEDIR\nlog \"$MAVEN_PROJECTBASEDIR\"\n\n##########################################################################################\n# Extension to allow automatically downloading the maven-wrapper.jar from Maven-central\n# This allows using the maven wrapper in projects that prohibit checking in binary data.\n##########################################################################################\nwrapperJarPath=\"$MAVEN_PROJECTBASEDIR/.mvn/wrapper/maven-wrapper.jar\"\nif [ -r \"$wrapperJarPath\" ]; then\n log \"Found $wrapperJarPath\"\nelse\n log \"Couldn't find $wrapperJarPath, downloading it ...\"\n\n if [ -n \"$MVNW_REPOURL\" ]; then\n wrapperUrl=\"$MVNW_REPOURL/org/apache/maven/wrapper/maven-wrapper/3.2.0/maven-wrapper-3.2.0.jar\"\n else\n wrapperUrl=\"https://repo.maven.apache.org/maven2/org/apache/maven/wrapper/maven-wrapper/3.2.0/maven-wrapper-3.2.0.jar\"\n fi\n while IFS=\"=\" read -r key value; do\n # Remove '\\r' from value to allow usage on windows as IFS does not consider '\\r' as a separator ( considers space, tab, new line ('\\n'), and custom '=' )\n safeValue=$(echo \"$value\" | tr -d '\\r')\n case \"$key\" in (wrapperUrl) wrapperUrl=\"$safeValue\"; break ;;\n esac\n done < \"$MAVEN_PROJECTBASEDIR/.mvn/wrapper/maven-wrapper.properties\"\n log \"Downloading from: $wrapperUrl\"\n\n if $cygwin; then\n wrapperJarPath=$(cygpath --path --windows \"$wrapperJarPath\")\n fi\n\n if command -v wget > /dev/null; then\n log \"Found wget ... using wget\"\n [ \"$MVNW_VERBOSE\" = true ] && QUIET=\"\" || QUIET=\"--quiet\"\n if [ -z \"$MVNW_USERNAME\" ] || [ -z \"$MVNW_PASSWORD\" ]; then\n wget $QUIET \"$wrapperUrl\" -O \"$wrapperJarPath\" || rm -f \"$wrapperJarPath\"\n else\n wget $QUIET --http-user=\"$MVNW_USERNAME\" --http-password=\"$MVNW_PASSWORD\" \"$wrapperUrl\" -O \"$wrapperJarPath\" || rm -f \"$wrapperJarPath\"\n fi\n elif command -v curl > /dev/null; then\n log \"Found curl ... using curl\"\n [ \"$MVNW_VERBOSE\" = true ] && QUIET=\"\" || QUIET=\"--silent\"\n if [ -z \"$MVNW_USERNAME\" ] || [ -z \"$MVNW_PASSWORD\" ]; then\n curl $QUIET -o \"$wrapperJarPath\" \"$wrapperUrl\" -f -L || rm -f \"$wrapperJarPath\"\n else\n curl $QUIET --user \"$MVNW_USERNAME:$MVNW_PASSWORD\" -o \"$wrapperJarPath\" \"$wrapperUrl\" -f -L || rm -f \"$wrapperJarPath\"\n fi\n else\n log \"Falling back to using Java to download\"\n javaSource=\"$MAVEN_PROJECTBASEDIR/.mvn/wrapper/MavenWrapperDownloader.java\"\n javaClass=\"$MAVEN_PROJECTBASEDIR/.mvn/wrapper/MavenWrapperDownloader.class\"\n # For Cygwin, switch paths to Windows format before running javac\n if $cygwin; then\n javaSource=$(cygpath --path --windows \"$javaSource\")\n javaClass=$(cygpath --path --windows \"$javaClass\")\n fi\n if [ -e \"$javaSource\" ]; then\n if [ ! -e \"$javaClass\" ]; then\n log \" - Compiling MavenWrapperDownloader.java ...\"\n (\"$JAVA_HOME/bin/javac\" \"$javaSource\")\n fi\n if [ -e \"$javaClass\" ]; then\n log \" - Running MavenWrapperDownloader.java ...\"\n (\"$JAVA_HOME/bin/java\" -cp .mvn/wrapper MavenWrapperDownloader \"$wrapperUrl\" \"$wrapperJarPath\") || rm -f \"$wrapperJarPath\"\n fi\n fi\n fi\nfi\n##########################################################################################\n# End of extension\n##########################################################################################\n\n# If specified, validate the SHA-256 sum of the Maven wrapper jar file\nwrapperSha256Sum=\"\"\nwhile IFS=\"=\" read -r key value; do\n case \"$key\" in (wrapperSha256Sum) wrapperSha256Sum=$value; break ;;\n esac\ndone < \"$MAVEN_PROJECTBASEDIR/.mvn/wrapper/maven-wrapper.properties\"\nif [ -n \"$wrapperSha256Sum\" ]; then\n wrapperSha256Result=false\n if command -v sha256sum > /dev/null; then\n if echo \"$wrapperSha256Sum $wrapperJarPath\" | sha256sum -c > /dev/null 2>&1; then\n wrapperSha256Result=true\n fi\n elif command -v shasum > /dev/null; then\n if echo \"$wrapperSha256Sum $wrapperJarPath\" | shasum -a 256 -c > /dev/null 2>&1; then\n wrapperSha256Result=true\n fi\n else\n echo \"Checksum validation was requested but neither 'sha256sum' or 'shasum' are available.\"\n echo \"Please install either command, or disable validation by removing 'wrapperSha256Sum' from your maven-wrapper.properties.\"\n exit 1\n fi\n if [ $wrapperSha256Result = false ]; then\n echo \"Error: Failed to validate Maven wrapper SHA-256, your Maven wrapper might be compromised.\" >&2\n echo \"Investigate or delete $wrapperJarPath to attempt a clean download.\" >&2\n echo \"If you updated your Maven version, you need to update the specified wrapperSha256Sum property.\" >&2\n exit 1\n fi\nfi\n\nMAVEN_OPTS=\"$(concat_lines \"$MAVEN_PROJECTBASEDIR/.mvn/jvm.config\") $MAVEN_OPTS\"\n\n# For Cygwin, switch paths to Windows format before running java\nif $cygwin; then\n [ -n \"$JAVA_HOME\" ] &&\n JAVA_HOME=$(cygpath --path --windows \"$JAVA_HOME\")\n [ -n \"$CLASSPATH\" ] &&\n CLASSPATH=$(cygpath --path --windows \"$CLASSPATH\")\n [ -n \"$MAVEN_PROJECTBASEDIR\" ] &&\n MAVEN_PROJECTBASEDIR=$(cygpath --path --windows \"$MAVEN_PROJECTBASEDIR\")\nfi\n\n# Provide a \"standardized\" way to retrieve the CLI args that will\n# work with both Windows and non-Windows executions.\nMAVEN_CMD_LINE_ARGS=\"$MAVEN_CONFIG $*\"\nexport MAVEN_CMD_LINE_ARGS\n\nWRAPPER_LAUNCHER=org.apache.maven.wrapper.MavenWrapperMain\n\n# shellcheck disable=SC2086 # safe args\nexec \"$JAVACMD\" \\\n $MAVEN_OPTS \\\n $MAVEN_DEBUG_OPTS \\\n -classpath \"$MAVEN_PROJECTBASEDIR/.mvn/wrapper/maven-wrapper.jar\" \\\n \"-Dmaven.multiModuleProjectDirectory=${MAVEN_PROJECTBASEDIR}\" \\\n ${WRAPPER_LAUNCHER} $MAVEN_CONFIG \"$@\"\n" }, { "path": "mvnw.cmd", "content": "@REM ----------------------------------------------------------------------------\r\n@REM Licensed to the Apache Software Foundation (ASF) under one\r\n@REM or more contributor license agreements. See the NOTICE file\r\n@REM distributed with this work for additional information\r\n@REM regarding copyright ownership. The ASF licenses this file\r\n@REM to you under the Apache License, Version 2.0 (the\r\n@REM \"License\"); you may not use this file except in compliance\r\n@REM with the License. You may obtain a copy of the License at\r\n@REM\r\n@REM http://www.apache.org/licenses/LICENSE-2.0\r\n@REM\r\n@REM Unless required by applicable law or agreed to in writing,\r\n@REM software distributed under the License is distributed on an\r\n@REM \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\r\n@REM KIND, either express or implied. See the License for the\r\n@REM specific language governing permissions and limitations\r\n@REM under the License.\r\n@REM ----------------------------------------------------------------------------\r\n\r\n@REM ----------------------------------------------------------------------------\r\n@REM Apache Maven Wrapper startup batch script, version 3.2.0\r\n@REM\r\n@REM Required ENV vars:\r\n@REM JAVA_HOME - location of a JDK home dir\r\n@REM\r\n@REM Optional ENV vars\r\n@REM MAVEN_BATCH_ECHO - set to 'on' to enable the echoing of the batch commands\r\n@REM MAVEN_BATCH_PAUSE - set to 'on' to wait for a keystroke before ending\r\n@REM MAVEN_OPTS - parameters passed to the Java VM when running Maven\r\n@REM e.g. to debug Maven itself, use\r\n@REM set MAVEN_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000\r\n@REM MAVEN_SKIP_RC - flag to disable loading of mavenrc files\r\n@REM ----------------------------------------------------------------------------\r\n\r\n@REM Begin all REM lines with '@' in case MAVEN_BATCH_ECHO is 'on'\r\n@echo off\r\n@REM set title of command window\r\ntitle %0\r\n@REM enable echoing by setting MAVEN_BATCH_ECHO to 'on'\r\n@if \"%MAVEN_BATCH_ECHO%\" == \"on\" echo %MAVEN_BATCH_ECHO%\r\n\r\n@REM set %HOME% to equivalent of $HOME\r\nif \"%HOME%\" == \"\" (set \"HOME=%HOMEDRIVE%%HOMEPATH%\")\r\n\r\n@REM Execute a user defined script before this one\r\nif not \"%MAVEN_SKIP_RC%\" == \"\" goto skipRcPre\r\n@REM check for pre script, once with legacy .bat ending and once with .cmd ending\r\nif exist \"%USERPROFILE%\\mavenrc_pre.bat\" call \"%USERPROFILE%\\mavenrc_pre.bat\" %*\r\nif exist \"%USERPROFILE%\\mavenrc_pre.cmd\" call \"%USERPROFILE%\\mavenrc_pre.cmd\" %*\r\n:skipRcPre\r\n\r\n@setlocal\r\n\r\nset ERROR_CODE=0\r\n\r\n@REM To isolate internal variables from possible post scripts, we use another setlocal\r\n@setlocal\r\n\r\n@REM ==== START VALIDATION ====\r\nif not \"%JAVA_HOME%\" == \"\" goto OkJHome\r\n\r\necho.\r\necho Error: JAVA_HOME not found in your environment. >&2\r\necho Please set the JAVA_HOME variable in your environment to match the >&2\r\necho location of your Java installation. >&2\r\necho.\r\ngoto error\r\n\r\n:OkJHome\r\nif exist \"%JAVA_HOME%\\bin\\java.exe\" goto init\r\n\r\necho.\r\necho Error: JAVA_HOME is set to an invalid directory. >&2\r\necho JAVA_HOME = \"%JAVA_HOME%\" >&2\r\necho Please set the JAVA_HOME variable in your environment to match the >&2\r\necho location of your Java installation. >&2\r\necho.\r\ngoto error\r\n\r\n@REM ==== END VALIDATION ====\r\n\r\n:init\r\n\r\n@REM Find the project base dir, i.e. the directory that contains the folder \".mvn\".\r\n@REM Fallback to current working directory if not found.\r\n\r\nset MAVEN_PROJECTBASEDIR=%MAVEN_BASEDIR%\r\nIF NOT \"%MAVEN_PROJECTBASEDIR%\"==\"\" goto endDetectBaseDir\r\n\r\nset EXEC_DIR=%CD%\r\nset WDIR=%EXEC_DIR%\r\n:findBaseDir\r\nIF EXIST \"%WDIR%\"\\.mvn goto baseDirFound\r\ncd ..\r\nIF \"%WDIR%\"==\"%CD%\" goto baseDirNotFound\r\nset WDIR=%CD%\r\ngoto findBaseDir\r\n\r\n:baseDirFound\r\nset MAVEN_PROJECTBASEDIR=%WDIR%\r\ncd \"%EXEC_DIR%\"\r\ngoto endDetectBaseDir\r\n\r\n:baseDirNotFound\r\nset MAVEN_PROJECTBASEDIR=%EXEC_DIR%\r\ncd \"%EXEC_DIR%\"\r\n\r\n:endDetectBaseDir\r\n\r\nIF NOT EXIST \"%MAVEN_PROJECTBASEDIR%\\.mvn\\jvm.config\" goto endReadAdditionalConfig\r\n\r\n@setlocal EnableExtensions EnableDelayedExpansion\r\nfor /F \"usebackq delims=\" %%a in (\"%MAVEN_PROJECTBASEDIR%\\.mvn\\jvm.config\") do set JVM_CONFIG_MAVEN_PROPS=!JVM_CONFIG_MAVEN_PROPS! %%a\r\n@endlocal & set JVM_CONFIG_MAVEN_PROPS=%JVM_CONFIG_MAVEN_PROPS%\r\n\r\n:endReadAdditionalConfig\r\n\r\nSET MAVEN_JAVA_EXE=\"%JAVA_HOME%\\bin\\java.exe\"\r\nset WRAPPER_JAR=\"%MAVEN_PROJECTBASEDIR%\\.mvn\\wrapper\\maven-wrapper.jar\"\r\nset WRAPPER_LAUNCHER=org.apache.maven.wrapper.MavenWrapperMain\r\n\r\nset WRAPPER_URL=\"https://repo.maven.apache.org/maven2/org/apache/maven/wrapper/maven-wrapper/3.2.0/maven-wrapper-3.2.0.jar\"\r\n\r\nFOR /F \"usebackq tokens=1,2 delims==\" %%A IN (\"%MAVEN_PROJECTBASEDIR%\\.mvn\\wrapper\\maven-wrapper.properties\") DO (\r\n IF \"%%A\"==\"wrapperUrl\" SET WRAPPER_URL=%%B\r\n)\r\n\r\n@REM Extension to allow automatically downloading the maven-wrapper.jar from Maven-central\r\n@REM This allows using the maven wrapper in projects that prohibit checking in binary data.\r\nif exist %WRAPPER_JAR% (\r\n if \"%MVNW_VERBOSE%\" == \"true\" (\r\n echo Found %WRAPPER_JAR%\r\n )\r\n) else (\r\n if not \"%MVNW_REPOURL%\" == \"\" (\r\n SET WRAPPER_URL=\"%MVNW_REPOURL%/org/apache/maven/wrapper/maven-wrapper/3.2.0/maven-wrapper-3.2.0.jar\"\r\n )\r\n if \"%MVNW_VERBOSE%\" == \"true\" (\r\n echo Couldn't find %WRAPPER_JAR%, downloading it ...\r\n echo Downloading from: %WRAPPER_URL%\r\n )\r\n\r\n powershell -Command \"&{\"^\r\n\t\t\"$webclient = new-object System.Net.WebClient;\"^\r\n\t\t\"if (-not ([string]::IsNullOrEmpty('%MVNW_USERNAME%') -and [string]::IsNullOrEmpty('%MVNW_PASSWORD%'))) {\"^\r\n\t\t\"$webclient.Credentials = new-object System.Net.NetworkCredential('%MVNW_USERNAME%', '%MVNW_PASSWORD%');\"^\r\n\t\t\"}\"^\r\n\t\t\"[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; $webclient.DownloadFile('%WRAPPER_URL%', '%WRAPPER_JAR%')\"^\r\n\t\t\"}\"\r\n if \"%MVNW_VERBOSE%\" == \"true\" (\r\n echo Finished downloading %WRAPPER_JAR%\r\n )\r\n)\r\n@REM End of extension\r\n\r\n@REM If specified, validate the SHA-256 sum of the Maven wrapper jar file\r\nSET WRAPPER_SHA_256_SUM=\"\"\r\nFOR /F \"usebackq tokens=1,2 delims==\" %%A IN (\"%MAVEN_PROJECTBASEDIR%\\.mvn\\wrapper\\maven-wrapper.properties\") DO (\r\n IF \"%%A\"==\"wrapperSha256Sum\" SET WRAPPER_SHA_256_SUM=%%B\r\n)\r\nIF NOT %WRAPPER_SHA_256_SUM%==\"\" (\r\n powershell -Command \"&{\"^\r\n \"$hash = (Get-FileHash \\\"%WRAPPER_JAR%\\\" -Algorithm SHA256).Hash.ToLower();\"^\r\n \"If('%WRAPPER_SHA_256_SUM%' -ne $hash){\"^\r\n \" Write-Output 'Error: Failed to validate Maven wrapper SHA-256, your Maven wrapper might be compromised.';\"^\r\n \" Write-Output 'Investigate or delete %WRAPPER_JAR% to attempt a clean download.';\"^\r\n \" Write-Output 'If you updated your Maven version, you need to update the specified wrapperSha256Sum property.';\"^\r\n \" exit 1;\"^\r\n \"}\"^\r\n \"}\"\r\n if ERRORLEVEL 1 goto error\r\n)\r\n\r\n@REM Provide a \"standardized\" way to retrieve the CLI args that will\r\n@REM work with both Windows and non-Windows executions.\r\nset MAVEN_CMD_LINE_ARGS=%*\r\n\r\n%MAVEN_JAVA_EXE% ^\r\n %JVM_CONFIG_MAVEN_PROPS% ^\r\n %MAVEN_OPTS% ^\r\n %MAVEN_DEBUG_OPTS% ^\r\n -classpath %WRAPPER_JAR% ^\r\n \"-Dmaven.multiModuleProjectDirectory=%MAVEN_PROJECTBASEDIR%\" ^\r\n %WRAPPER_LAUNCHER% %MAVEN_CONFIG% %*\r\nif ERRORLEVEL 1 goto error\r\ngoto end\r\n\r\n:error\r\nset ERROR_CODE=1\r\n\r\n:end\r\n@endlocal & set ERROR_CODE=%ERROR_CODE%\r\n\r\nif not \"%MAVEN_SKIP_RC%\"==\"\" goto skipRcPost\r\n@REM check for post script, once with legacy .bat ending and once with .cmd ending\r\nif exist \"%USERPROFILE%\\mavenrc_post.bat\" call \"%USERPROFILE%\\mavenrc_post.bat\"\r\nif exist \"%USERPROFILE%\\mavenrc_post.cmd\" call \"%USERPROFILE%\\mavenrc_post.cmd\"\r\n:skipRcPost\r\n\r\n@REM pause the script if MAVEN_BATCH_PAUSE is set to 'on'\r\nif \"%MAVEN_BATCH_PAUSE%\"==\"on\" pause\r\n\r\nif \"%MAVEN_TERMINATE_CMD%\"==\"on\" exit %ERROR_CODE%\r\n\r\ncmd /C exit /B %ERROR_CODE%\r\n" }, { "path": "native/Cargo.toml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n[workspace]\ndefault-members = [\"core\", \"spark-expr\", \"common\", \"proto\", \"jni-bridge\", \"shuffle\"]\nmembers = [\"core\", \"spark-expr\", \"common\", \"proto\", \"jni-bridge\", \"shuffle\", \"hdfs\", \"fs-hdfs\"]\nresolver = \"2\"\n\n[workspace.package]\nversion = \"0.15.0\"\nhomepage = \"https://datafusion.apache.org/comet\"\nrepository = \"https://github.com/apache/datafusion-comet\"\nauthors = [\"Apache DataFusion \"]\ndescription = \"Apache DataFusion Comet: High performance accelerator for Apache Spark\"\nreadme = \"README.md\"\nlicense = \"Apache-2.0\"\nedition = \"2021\"\n\n# Comet uses the same minimum Rust version as DataFusion\nrust-version = \"1.88\"\n\n[workspace.dependencies]\narrow = { version = \"58.1.0\", features = [\"prettyprint\", \"ffi\", \"chrono-tz\"] }\nasync-trait = { version = \"0.1\" }\nbytes = { version = \"1.11.1\" }\nparquet = { version = \"58.1.0\", default-features = false, features = [\"experimental\"] }\ndatafusion = { version = \"53.1.0\", default-features = false, features = [\"unicode_expressions\", \"crypto_expressions\", \"nested_expressions\", \"parquet\"] }\ndatafusion-datasource = { version = \"53.1.0\" }\ndatafusion-physical-expr-adapter = { version = \"53.1.0\" }\ndatafusion-spark = { version = \"53.1.0\", features = [\"core\"] }\ndatafusion-comet-spark-expr = { path = \"spark-expr\" }\ndatafusion-comet-common = { path = \"common\" }\ndatafusion-comet-jni-bridge = { path = \"jni-bridge\" }\ndatafusion-comet-proto = { path = \"proto\" }\ndatafusion-comet-shuffle = { path = \"shuffle\" }\nchrono = { version = \"0.4\", default-features = false, features = [\"clock\"] }\nchrono-tz = { version = \"0.10\" }\nfutures = \"0.3.32\"\nnum = \"0.4\"\nrand = \"0.10\"\nregex = \"1.12.3\"\nthiserror = \"2\"\nobject_store = { version = \"0.13.1\", features = [\"gcp\", \"azure\", \"aws\", \"http\"] }\nurl = \"2.2\"\naws-config = \"1.8.14\"\naws-credential-types = \"1.2.13\"\niceberg = { git = \"https://github.com/apache/iceberg-rust\", rev = \"a2f067d\" }\niceberg-storage-opendal = { git = \"https://github.com/apache/iceberg-rust\", rev = \"a2f067d\", features = [\"opendal-all\"] }\n\n[profile.release]\ndebug = true\noverflow-checks = false\nlto = \"thin\"\ncodegen-units = 1\nstrip = \"debuginfo\"\n\n# CI profile: faster compilation, same overflow behavior as release\n# Use with: cargo build --profile ci\n[profile.ci]\ninherits = \"release\"\nlto = false # Skip LTO for faster linking\ncodegen-units = 16 # Parallel codegen (faster compile, slightly larger binary)\ndebug-assertions = true\npanic = \"unwind\" # Allow panics to be caught and logged across FFI boundary\n# overflow-checks inherited as false from release\n" }, { "path": "native/README.md", "content": "\n\n# Apache DataFusion Comet Native Code\n\nThis project contains the following crates:\n\n- [core](core): Native code used by the Comet Spark plugin\n- [proto](proto): Comet protocol buffer definition for query plans\n- [spark-expr](spark-expr): Spark-compatible DataFusion operators and expressions\n" }, { "path": "native/common/Cargo.toml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n[package]\nname = \"datafusion-comet-common\"\ndescription = \"Apache DataFusion Comet: common types shared across crates\"\nversion = { workspace = true }\nhomepage = { workspace = true }\nrepository = { workspace = true }\nauthors = { workspace = true }\nreadme = { workspace = true }\nlicense = { workspace = true }\nedition = { workspace = true }\n\npublish = false\n\n[dependencies]\narrow = { workspace = true }\ndatafusion = { workspace = true }\nserde = { version = \"1.0\", features = [\"derive\"] }\nserde_json = \"1.0\"\nthiserror = { workspace = true }\n\n[lib]\nname = \"datafusion_comet_common\"\npath = \"src/lib.rs\"\n\n[[bin]]\nname = \"analyze_trace\"\npath = \"src/bin/analyze_trace.rs\"\n" }, { "path": "native/common/README.md", "content": "\n\n# datafusion-comet-common: Common Types\n\nThis crate provides common types shared across Apache DataFusion Comet crates and is maintained as part of the\n[Apache DataFusion Comet] subproject.\n\n[Apache DataFusion Comet]: https://github.com/apache/datafusion-comet/\n" }, { "path": "native/common/src/bin/analyze_trace.rs", "content": "// Licensed to the Apache Software Foundation (ASF) under one\n// or more contributor license agreements. See the NOTICE file\n// distributed with this work for additional information\n// regarding copyright ownership. The ASF licenses this file\n// to you under the Apache License, Version 2.0 (the\n// \"License\"); you may not use this file except in compliance\n// with the License. You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing,\n// software distributed under the License is distributed on an\n// \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n// KIND, either express or implied. See the License for the\n// specific language governing permissions and limitations\n// under the License.\n\n//! Analyzes a Comet chrome trace event log (`comet-event-trace.json`) and\n//! compares jemalloc usage against the sum of per-thread Comet memory pool\n//! reservations. Reports any points where jemalloc exceeds the total pool size.\n//!\n//! Usage:\n//! cargo run --bin analyze_trace -- \n\nuse serde::Deserialize;\nuse std::collections::HashMap;\nuse std::io::{BufRead, BufReader};\nuse std::{env, fs::File};\n\n/// A single Chrome trace event (only the fields we care about).\n#[derive(Deserialize)]\nstruct TraceEvent {\n name: String,\n ph: String,\n #[allow(dead_code)]\n tid: u64,\n ts: u64,\n #[serde(default)]\n args: HashMap,\n}\n\n/// Snapshot of memory state at a given timestamp.\nstruct MemorySnapshot {\n ts: u64,\n jemalloc: u64,\n pool_total: u64,\n}\n\nfn format_bytes(bytes: u64) -> String {\n const MB: f64 = 1024.0 * 1024.0;\n format!(\"{:.1} MB\", bytes as f64 / MB)\n}\n\nfn main() {\n let args: Vec = env::args().collect();\n if args.len() != 2 {\n eprintln!(\"Usage: analyze_trace \");\n std::process::exit(1);\n }\n\n let file = File::open(&args[1]).expect(\"Failed to open trace file\");\n let reader = BufReader::new(file);\n\n // Latest jemalloc value (global, not per-thread)\n let mut latest_jemalloc: u64 = 0;\n // Per-thread pool reservations: thread_NNN -> bytes\n let mut pool_by_thread: HashMap = HashMap::new();\n // Points where jemalloc exceeded pool total\n let mut violations: Vec = Vec::new();\n // Track peak values\n let mut peak_jemalloc: u64 = 0;\n let mut peak_pool_total: u64 = 0;\n let mut peak_excess: u64 = 0;\n let mut counter_events: u64 = 0;\n\n // Each line is one JSON event, possibly with a trailing comma.\n // The file starts with \"[ \" on the first event line or as a prefix.\n for line in reader.lines() {\n let line = line.expect(\"Failed to read line\");\n let trimmed = line.trim();\n\n // Skip empty lines or bare array brackets\n if trimmed.is_empty() || trimmed == \"[\" || trimmed == \"]\" {\n continue;\n }\n\n // Strip leading \"[ \" (first event) and trailing comma\n let json_str = trimmed\n .trim_start_matches(\"[ \")\n .trim_start_matches('[')\n .trim_end_matches(',');\n\n if json_str.is_empty() {\n continue;\n }\n\n // Only parse counter events (they contain \"\\\"ph\\\": \\\"C\\\"\")\n if !json_str.contains(\"\\\"ph\\\": \\\"C\\\"\") {\n continue;\n }\n\n let event: TraceEvent = match serde_json::from_str(json_str) {\n Ok(e) => e,\n Err(_) => continue,\n };\n\n if event.ph != \"C\" {\n continue;\n }\n\n counter_events += 1;\n\n if event.name == \"jemalloc_allocated\" {\n if let Some(val) = event.args.get(\"jemalloc_allocated\") {\n latest_jemalloc = val.as_u64().unwrap_or(0);\n if latest_jemalloc > peak_jemalloc {\n peak_jemalloc = latest_jemalloc;\n }\n }\n } else if event.name.contains(\"comet_memory_reserved\") {\n // Name format: thread_NNN_comet_memory_reserved\n let thread_key = event.name.clone();\n if let Some(val) = event.args.get(&event.name) {\n let bytes = val.as_u64().unwrap_or(0);\n pool_by_thread.insert(thread_key, bytes);\n }\n } else {\n // Skip jvm_heap_used and other counters\n continue;\n }\n\n // After each jemalloc or pool update, check the current state\n let pool_total: u64 = pool_by_thread.values().sum();\n if pool_total > peak_pool_total {\n peak_pool_total = pool_total;\n }\n\n if latest_jemalloc > 0 && pool_total > 0 && latest_jemalloc > pool_total {\n let excess = latest_jemalloc - pool_total;\n if excess > peak_excess {\n peak_excess = excess;\n }\n // Record violation (sample - don't record every single one)\n if violations.is_empty()\n || event.ts.saturating_sub(violations.last().unwrap().ts) > 1_000_000\n || excess == peak_excess\n {\n violations.push(MemorySnapshot {\n ts: event.ts,\n jemalloc: latest_jemalloc,\n pool_total,\n });\n }\n }\n }\n\n // Print summary\n println!(\"=== Comet Trace Memory Analysis ===\\n\");\n println!(\"Counter events parsed: {counter_events}\");\n println!(\"Threads with memory pools: {}\", pool_by_thread.len());\n println!(\"Peak jemalloc allocated: {}\", format_bytes(peak_jemalloc));\n println!(\n \"Peak pool total: {}\",\n format_bytes(peak_pool_total)\n );\n println!(\n \"Peak excess (jemalloc - pool): {}\",\n format_bytes(peak_excess)\n );\n println!();\n\n if violations.is_empty() {\n println!(\"OK: jemalloc never exceeded the total pool reservation.\");\n } else {\n println!(\n \"WARNING: jemalloc exceeded pool reservation at {} sampled points:\\n\",\n violations.len()\n );\n println!(\n \"{:>14} {:>14} {:>14} {:>14}\",\n \"Time (us)\", \"jemalloc\", \"pool_total\", \"excess\"\n );\n println!(\"{}\", \"-\".repeat(62));\n for snap in &violations {\n let excess = snap.jemalloc - snap.pool_total;\n println!(\n \"{:>14} {:>14} {:>14} {:>14}\",\n snap.ts,\n format_bytes(snap.jemalloc),\n format_bytes(snap.pool_total),\n format_bytes(excess),\n );\n }\n }\n\n // Show final per-thread pool state\n println!(\"\\n--- Final per-thread pool reservations ---\\n\");\n let mut threads: Vec<_> = pool_by_thread.iter().collect();\n threads.sort_by_key(|(k, _)| (*k).clone());\n for (thread, bytes) in &threads {\n println!(\" {thread}: {}\", format_bytes(**bytes));\n }\n println!(\"\\n Total: {}\", format_bytes(pool_by_thread.values().sum()));\n}\n" }, { "path": "native/common/src/error.rs", "content": "// Licensed to the Apache Software Foundation (ASF) under one\n// or more contributor license agreements. See the NOTICE file\n// distributed with this work for additional information\n// regarding copyright ownership. The ASF licenses this file\n// to you under the Apache License, Version 2.0 (the\n// \"License\"); you may not use this file except in compliance\n// with the License. You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing,\n// software distributed under the License is distributed on an\n// \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n// KIND, either express or implied. See the License for the\n// specific language governing permissions and limitations\n// under the License.\n\nuse arrow::error::ArrowError;\nuse datafusion::common::DataFusionError;\nuse std::sync::Arc;\n\n#[derive(thiserror::Error, Debug, Clone)]\npub enum SparkError {\n // This list was generated from the Spark code. Many of the exceptions are not yet used by Comet\n #[error(\"[CAST_INVALID_INPUT] The value '{value}' of the type \\\"{from_type}\\\" cannot be cast to \\\"{to_type}\\\" \\\n because it is malformed. Correct the value as per the syntax, or change its target type. \\\n Use `try_cast` to tolerate malformed input and return NULL instead. If necessary \\\n set \\\"spark.sql.ansi.enabled\\\" to \\\"false\\\" to bypass this error.\")]\n CastInvalidValue {\n value: String,\n from_type: String,\n to_type: String,\n },\n\n /// Like CastInvalidValue but maps to SparkDateTimeException instead of SparkNumberFormatException.\n /// Used for string → timestamp/date cast failures.\n #[error(\"[CAST_INVALID_INPUT] The value '{value}' of the type \\\"{from_type}\\\" cannot be cast to \\\"{to_type}\\\" \\\n because it is malformed. Correct the value as per the syntax, or change its target type. \\\n Use `try_cast` to tolerate malformed input and return NULL instead. If necessary \\\n set \\\"spark.sql.ansi.enabled\\\" to \\\"false\\\" to bypass this error.\")]\n InvalidInputInCastToDatetime {\n value: String,\n from_type: String,\n to_type: String,\n },\n\n #[error(\"[NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION] {value} cannot be represented as Decimal({precision}, {scale}). If necessary set \\\"spark.sql.ansi.enabled\\\" to \\\"false\\\" to bypass this error, and return NULL instead.\")]\n NumericValueOutOfRange {\n value: String,\n precision: u8,\n scale: i8,\n },\n\n #[error(\"[NUMERIC_OUT_OF_SUPPORTED_RANGE] The value {value} cannot be interpreted as a numeric since it has more than 38 digits.\")]\n NumericOutOfRange { value: String },\n\n #[error(\"[CAST_OVERFLOW] The value {value} of the type \\\"{from_type}\\\" cannot be cast to \\\"{to_type}\\\" \\\n due to an overflow. Use `try_cast` to tolerate overflow and return NULL instead. If necessary \\\n set \\\"spark.sql.ansi.enabled\\\" to \\\"false\\\" to bypass this error.\")]\n CastOverFlow {\n value: String,\n from_type: String,\n to_type: String,\n },\n\n #[error(\"[CANNOT_PARSE_DECIMAL] Cannot parse decimal.\")]\n CannotParseDecimal,\n\n #[error(\"[ARITHMETIC_OVERFLOW] {from_type} overflow. If necessary set \\\"spark.sql.ansi.enabled\\\" to \\\"false\\\" to bypass this error.\")]\n ArithmeticOverflow { from_type: String },\n\n #[error(\"[ARITHMETIC_OVERFLOW] Overflow in integral divide. Use `try_divide` to tolerate overflow and return NULL instead. If necessary set \\\"spark.sql.ansi.enabled\\\" to \\\"false\\\" to bypass this error.\")]\n IntegralDivideOverflow,\n\n #[error(\"[ARITHMETIC_OVERFLOW] Overflow in sum of decimals. If necessary set \\\"spark.sql.ansi.enabled\\\" to \\\"false\\\" to bypass this error.\")]\n DecimalSumOverflow,\n\n #[error(\"[DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set \\\"spark.sql.ansi.enabled\\\" to \\\"false\\\" to bypass this error.\")]\n DivideByZero,\n\n #[error(\"[REMAINDER_BY_ZERO] Division by zero. Use `try_remainder` to tolerate divisor being 0 and return NULL instead. If necessary set \\\"spark.sql.ansi.enabled\\\" to \\\"false\\\" to bypass this error.\")]\n RemainderByZero,\n\n #[error(\"[INTERVAL_DIVIDED_BY_ZERO] Divide by zero in interval arithmetic.\")]\n IntervalDividedByZero,\n\n #[error(\"[BINARY_ARITHMETIC_OVERFLOW] {value1} {symbol} {value2} caused overflow. Use `{function_name}` to tolerate overflow and return NULL instead. If necessary set \\\"spark.sql.ansi.enabled\\\" to \\\"false\\\" to bypass this error.\")]\n BinaryArithmeticOverflow {\n value1: String,\n symbol: String,\n value2: String,\n function_name: String,\n },\n\n #[error(\"[INTERVAL_ARITHMETIC_OVERFLOW.WITH_SUGGESTION] Interval arithmetic overflow. Use `{function_name}` to tolerate overflow and return NULL instead. If necessary set \\\"spark.sql.ansi.enabled\\\" to \\\"false\\\" to bypass this error.\")]\n IntervalArithmeticOverflowWithSuggestion { function_name: String },\n\n #[error(\"[INTERVAL_ARITHMETIC_OVERFLOW.WITHOUT_SUGGESTION] Interval arithmetic overflow. If necessary set \\\"spark.sql.ansi.enabled\\\" to \\\"false\\\" to bypass this error.\")]\n IntervalArithmeticOverflowWithoutSuggestion,\n\n #[error(\"[DATETIME_OVERFLOW] Datetime arithmetic overflow.\")]\n DatetimeOverflow,\n\n #[error(\"[INVALID_ARRAY_INDEX] The index {index_value} is out of bounds. The array has {array_size} elements. Use the SQL function get() to tolerate accessing element at invalid index and return NULL instead. If necessary set \\\"spark.sql.ansi.enabled\\\" to \\\"false\\\" to bypass this error.\")]\n InvalidArrayIndex { index_value: i32, array_size: i32 },\n\n #[error(\"[INVALID_ARRAY_INDEX_IN_ELEMENT_AT] The index {index_value} is out of bounds. The array has {array_size} elements. Use try_element_at to tolerate accessing element at invalid index and return NULL instead. If necessary set \\\"spark.sql.ansi.enabled\\\" to \\\"false\\\" to bypass this error.\")]\n InvalidElementAtIndex { index_value: i32, array_size: i32 },\n\n #[error(\"[INVALID_BITMAP_POSITION] The bit position {bit_position} is out of bounds. The bitmap has {bitmap_num_bytes} bytes ({bitmap_num_bits} bits).\")]\n InvalidBitmapPosition {\n bit_position: i64,\n bitmap_num_bytes: i64,\n bitmap_num_bits: i64,\n },\n\n #[error(\"[INVALID_INDEX_OF_ZERO] The index 0 is invalid. An index shall be either < 0 or > 0 (the first element has index 1).\")]\n InvalidIndexOfZero,\n\n #[error(\"[DUPLICATED_MAP_KEY] Cannot create map with duplicate keys: {key}.\")]\n DuplicatedMapKey { key: String },\n\n #[error(\"[NULL_MAP_KEY] Cannot use null as map key.\")]\n NullMapKey,\n\n #[error(\"[MAP_KEY_VALUE_DIFF_SIZES] The key array and value array of a map must have the same length.\")]\n MapKeyValueDiffSizes,\n\n #[error(\"[EXCEED_LIMIT_LENGTH] Cannot create a map with {size} elements which exceeds the limit {max_size}.\")]\n ExceedMapSizeLimit { size: i32, max_size: i32 },\n\n #[error(\"[COLLECTION_SIZE_LIMIT_EXCEEDED] Cannot create array with {num_elements} elements which exceeds the limit {max_elements}.\")]\n CollectionSizeLimitExceeded {\n num_elements: i64,\n max_elements: i64,\n },\n\n #[error(\"[NOT_NULL_ASSERT_VIOLATION] The field `{field_name}` cannot be null.\")]\n NotNullAssertViolation { field_name: String },\n\n #[error(\"[VALUE_IS_NULL] The value of field `{field_name}` at row {row_index} is null.\")]\n ValueIsNull { field_name: String, row_index: i32 },\n\n #[error(\"[CANNOT_PARSE_TIMESTAMP] Cannot parse timestamp: {message}. Try using `{suggested_func}` instead.\")]\n CannotParseTimestamp {\n message: String,\n suggested_func: String,\n },\n\n #[error(\"[INVALID_FRACTION_OF_SECOND] The fraction of second {value} is invalid. Valid values are in the range [0, 60]. If necessary set \\\"spark.sql.ansi.enabled\\\" to \\\"false\\\" to bypass this error.\")]\n InvalidFractionOfSecond { value: f64 },\n\n #[error(\"[INVALID_UTF8_STRING] Invalid UTF-8 string: {hex_string}.\")]\n InvalidUtf8String { hex_string: String },\n\n #[error(\"[UNEXPECTED_POSITIVE_VALUE] The {parameter_name} parameter must be less than or equal to 0. The actual value is {actual_value}.\")]\n UnexpectedPositiveValue {\n parameter_name: String,\n actual_value: i32,\n },\n\n #[error(\"[UNEXPECTED_NEGATIVE_VALUE] The {parameter_name} parameter must be greater than or equal to 0. The actual value is {actual_value}.\")]\n UnexpectedNegativeValue {\n parameter_name: String,\n actual_value: i32,\n },\n\n #[error(\"[INVALID_PARAMETER_VALUE] Invalid regex group index {group_index} in function `{function_name}`. Group count is {group_count}.\")]\n InvalidRegexGroupIndex {\n function_name: String,\n group_count: i32,\n group_index: i32,\n },\n\n #[error(\"[DATATYPE_CANNOT_ORDER] Cannot order by type: {data_type}.\")]\n DatatypeCannotOrder { data_type: String },\n\n #[error(\"[SCALAR_SUBQUERY_TOO_MANY_ROWS] Scalar subquery returned more than one row.\")]\n ScalarSubqueryTooManyRows,\n\n #[error(\"{message}\")]\n FileNotFound { message: String },\n\n #[error(\"[_LEGACY_ERROR_TEMP_2093] Found duplicate field(s) \\\"{required_field_name}\\\": [{matched_fields}] in case-insensitive mode\")]\n DuplicateFieldCaseInsensitive {\n required_field_name: String,\n matched_fields: String,\n },\n\n #[error(\"ArrowError: {0}.\")]\n Arrow(Arc),\n\n #[error(\"InternalError: {0}.\")]\n Internal(String),\n}\n\nimpl SparkError {\n /// Serialize this error to JSON format for JNI transfer\n pub fn to_json(&self) -> String {\n let error_class = self.error_class().unwrap_or(\"\");\n\n // Create a JSON structure with errorType, errorClass, and params\n match serde_json::to_string(&serde_json::json!({\n \"errorType\": self.error_type_name(),\n \"errorClass\": error_class,\n \"params\": self.params_as_json(),\n })) {\n Ok(json) => json,\n Err(e) => {\n // Fallback if serialization fails\n format!(\n \"{{\\\"errorType\\\":\\\"SerializationError\\\",\\\"message\\\":\\\"{}\\\"}}\",\n e\n )\n }\n }\n }\n\n /// Get the error type name for JSON serialization\n pub(crate) fn error_type_name(&self) -> &'static str {\n match self {\n SparkError::CastInvalidValue { .. } => \"CastInvalidValue\",\n SparkError::InvalidInputInCastToDatetime { .. } => \"InvalidInputInCastToDatetime\",\n SparkError::NumericValueOutOfRange { .. } => \"NumericValueOutOfRange\",\n SparkError::NumericOutOfRange { .. } => \"NumericOutOfRange\",\n SparkError::CastOverFlow { .. } => \"CastOverFlow\",\n SparkError::CannotParseDecimal => \"CannotParseDecimal\",\n SparkError::ArithmeticOverflow { .. } => \"ArithmeticOverflow\",\n SparkError::IntegralDivideOverflow => \"IntegralDivideOverflow\",\n SparkError::DecimalSumOverflow => \"DecimalSumOverflow\",\n SparkError::DivideByZero => \"DivideByZero\",\n SparkError::RemainderByZero => \"RemainderByZero\",\n SparkError::IntervalDividedByZero => \"IntervalDividedByZero\",\n SparkError::BinaryArithmeticOverflow { .. } => \"BinaryArithmeticOverflow\",\n SparkError::IntervalArithmeticOverflowWithSuggestion { .. } => {\n \"IntervalArithmeticOverflowWithSuggestion\"\n }\n SparkError::IntervalArithmeticOverflowWithoutSuggestion => {\n \"IntervalArithmeticOverflowWithoutSuggestion\"\n }\n SparkError::DatetimeOverflow => \"DatetimeOverflow\",\n SparkError::InvalidArrayIndex { .. } => \"InvalidArrayIndex\",\n SparkError::InvalidElementAtIndex { .. } => \"InvalidElementAtIndex\",\n SparkError::InvalidBitmapPosition { .. } => \"InvalidBitmapPosition\",\n SparkError::InvalidIndexOfZero => \"InvalidIndexOfZero\",\n SparkError::DuplicatedMapKey { .. } => \"DuplicatedMapKey\",\n SparkError::NullMapKey => \"NullMapKey\",\n SparkError::MapKeyValueDiffSizes => \"MapKeyValueDiffSizes\",\n SparkError::ExceedMapSizeLimit { .. } => \"ExceedMapSizeLimit\",\n SparkError::CollectionSizeLimitExceeded { .. } => \"CollectionSizeLimitExceeded\",\n SparkError::NotNullAssertViolation { .. } => \"NotNullAssertViolation\",\n SparkError::ValueIsNull { .. } => \"ValueIsNull\",\n SparkError::CannotParseTimestamp { .. } => \"CannotParseTimestamp\",\n SparkError::InvalidFractionOfSecond { .. } => \"InvalidFractionOfSecond\",\n SparkError::InvalidUtf8String { .. } => \"InvalidUtf8String\",\n SparkError::UnexpectedPositiveValue { .. } => \"UnexpectedPositiveValue\",\n SparkError::UnexpectedNegativeValue { .. } => \"UnexpectedNegativeValue\",\n SparkError::InvalidRegexGroupIndex { .. } => \"InvalidRegexGroupIndex\",\n SparkError::DatatypeCannotOrder { .. } => \"DatatypeCannotOrder\",\n SparkError::ScalarSubqueryTooManyRows => \"ScalarSubqueryTooManyRows\",\n SparkError::FileNotFound { .. } => \"FileNotFound\",\n SparkError::DuplicateFieldCaseInsensitive { .. } => \"DuplicateFieldCaseInsensitive\",\n SparkError::Arrow(_) => \"Arrow\",\n SparkError::Internal(_) => \"Internal\",\n }\n }\n\n /// Extract parameters as JSON value\n pub(crate) fn params_as_json(&self) -> serde_json::Value {\n match self {\n SparkError::CastInvalidValue {\n value,\n from_type,\n to_type,\n } => {\n serde_json::json!({\n \"value\": value,\n \"fromType\": from_type,\n \"toType\": to_type,\n })\n }\n SparkError::InvalidInputInCastToDatetime {\n value,\n from_type,\n to_type,\n } => {\n serde_json::json!({\n \"value\": value,\n \"fromType\": from_type,\n \"toType\": to_type,\n })\n }\n SparkError::NumericValueOutOfRange {\n value,\n precision,\n scale,\n } => {\n serde_json::json!({\n \"value\": value,\n \"precision\": precision,\n \"scale\": scale,\n })\n }\n SparkError::NumericOutOfRange { value } => {\n serde_json::json!({\n \"value\": value,\n })\n }\n SparkError::CastOverFlow {\n value,\n from_type,\n to_type,\n } => {\n serde_json::json!({\n \"value\": value,\n \"fromType\": from_type,\n \"toType\": to_type,\n })\n }\n SparkError::ArithmeticOverflow { from_type } => {\n serde_json::json!({\n \"fromType\": from_type,\n })\n }\n SparkError::BinaryArithmeticOverflow {\n value1,\n symbol,\n value2,\n function_name,\n } => {\n serde_json::json!({\n \"value1\": value1,\n \"symbol\": symbol,\n \"value2\": value2,\n \"functionName\": function_name,\n })\n }\n SparkError::IntervalArithmeticOverflowWithSuggestion { function_name } => {\n serde_json::json!({\n \"functionName\": function_name,\n })\n }\n SparkError::InvalidArrayIndex {\n index_value,\n array_size,\n } => {\n serde_json::json!({\n \"indexValue\": index_value,\n \"arraySize\": array_size,\n })\n }\n SparkError::InvalidElementAtIndex {\n index_value,\n array_size,\n } => {\n serde_json::json!({\n \"indexValue\": index_value,\n \"arraySize\": array_size,\n })\n }\n SparkError::InvalidBitmapPosition {\n bit_position,\n bitmap_num_bytes,\n bitmap_num_bits,\n } => {\n serde_json::json!({\n \"bitPosition\": bit_position,\n \"bitmapNumBytes\": bitmap_num_bytes,\n \"bitmapNumBits\": bitmap_num_bits,\n })\n }\n SparkError::DuplicatedMapKey { key } => {\n serde_json::json!({\n \"key\": key,\n })\n }\n SparkError::ExceedMapSizeLimit { size, max_size } => {\n serde_json::json!({\n \"size\": size,\n \"maxSize\": max_size,\n })\n }\n SparkError::CollectionSizeLimitExceeded {\n num_elements,\n max_elements,\n } => {\n serde_json::json!({\n \"numElements\": num_elements,\n \"maxElements\": max_elements,\n })\n }\n SparkError::NotNullAssertViolation { field_name } => {\n serde_json::json!({\n \"fieldName\": field_name,\n })\n }\n SparkError::ValueIsNull {\n field_name,\n row_index,\n } => {\n serde_json::json!({\n \"fieldName\": field_name,\n \"rowIndex\": row_index,\n })\n }\n SparkError::CannotParseTimestamp {\n message,\n suggested_func,\n } => {\n serde_json::json!({\n \"message\": message,\n \"suggestedFunc\": suggested_func,\n })\n }\n SparkError::InvalidFractionOfSecond { value } => {\n serde_json::json!({\n \"value\": value,\n })\n }\n SparkError::InvalidUtf8String { hex_string } => {\n serde_json::json!({\n \"hexString\": hex_string,\n })\n }\n SparkError::UnexpectedPositiveValue {\n parameter_name,\n actual_value,\n } => {\n serde_json::json!({\n \"parameterName\": parameter_name,\n \"actualValue\": actual_value,\n })\n }\n SparkError::UnexpectedNegativeValue {\n parameter_name,\n actual_value,\n } => {\n serde_json::json!({\n \"parameterName\": parameter_name,\n \"actualValue\": actual_value,\n })\n }\n SparkError::InvalidRegexGroupIndex {\n function_name,\n group_count,\n group_index,\n } => {\n serde_json::json!({\n \"functionName\": function_name,\n \"groupCount\": group_count,\n \"groupIndex\": group_index,\n })\n }\n SparkError::DatatypeCannotOrder { data_type } => {\n serde_json::json!({\n \"dataType\": data_type,\n })\n }\n SparkError::FileNotFound { message } => {\n serde_json::json!({\n \"message\": message,\n })\n }\n SparkError::DuplicateFieldCaseInsensitive {\n required_field_name,\n matched_fields,\n } => {\n serde_json::json!({\n \"requiredFieldName\": required_field_name,\n \"matchedOrcFields\": matched_fields,\n })\n }\n SparkError::Arrow(e) => {\n serde_json::json!({\n \"message\": e.to_string(),\n })\n }\n SparkError::Internal(msg) => {\n serde_json::json!({\n \"message\": msg,\n })\n }\n // Simple errors with no parameters\n _ => serde_json::json!({}),\n }\n }\n\n /// Returns the appropriate Spark exception class for this error\n pub fn exception_class(&self) -> &'static str {\n match self {\n // ArithmeticException\n SparkError::DivideByZero\n | SparkError::RemainderByZero\n | SparkError::IntervalDividedByZero\n | SparkError::NumericValueOutOfRange { .. }\n | SparkError::NumericOutOfRange { .. } // Comet-specific extension\n | SparkError::ArithmeticOverflow { .. }\n | SparkError::IntegralDivideOverflow\n | SparkError::DecimalSumOverflow\n | SparkError::BinaryArithmeticOverflow { .. }\n | SparkError::IntervalArithmeticOverflowWithSuggestion { .. }\n | SparkError::IntervalArithmeticOverflowWithoutSuggestion\n | SparkError::DatetimeOverflow => \"org/apache/spark/SparkArithmeticException\",\n\n // CastOverflow gets special handling with CastOverflowException\n SparkError::CastOverFlow { .. } => \"org/apache/spark/sql/comet/CastOverflowException\",\n\n // NumberFormatException (for cast invalid input errors)\n SparkError::CastInvalidValue { .. } => \"org/apache/spark/SparkNumberFormatException\",\n\n // ArrayIndexOutOfBoundsException\n SparkError::InvalidArrayIndex { .. }\n | SparkError::InvalidElementAtIndex { .. }\n | SparkError::InvalidBitmapPosition { .. }\n | SparkError::InvalidIndexOfZero => \"org/apache/spark/SparkArrayIndexOutOfBoundsException\",\n\n // RuntimeException\n SparkError::CannotParseDecimal\n | SparkError::DuplicatedMapKey { .. }\n | SparkError::NullMapKey\n | SparkError::MapKeyValueDiffSizes\n | SparkError::ExceedMapSizeLimit { .. }\n | SparkError::CollectionSizeLimitExceeded { .. }\n | SparkError::NotNullAssertViolation { .. }\n | SparkError::ValueIsNull { .. } // Comet-specific extension\n | SparkError::UnexpectedPositiveValue { .. }\n | SparkError::UnexpectedNegativeValue { .. }\n | SparkError::InvalidRegexGroupIndex { .. }\n | SparkError::ScalarSubqueryTooManyRows => \"org/apache/spark/SparkRuntimeException\",\n\n // DateTimeException\n SparkError::InvalidInputInCastToDatetime { .. }\n | SparkError::CannotParseTimestamp { .. }\n | SparkError::InvalidFractionOfSecond { .. } => \"org/apache/spark/SparkDateTimeException\",\n\n // IllegalArgumentException\n SparkError::DatatypeCannotOrder { .. }\n | SparkError::InvalidUtf8String { .. } => \"org/apache/spark/SparkIllegalArgumentException\",\n\n // FileNotFound - will be converted to SparkFileNotFoundException by the shim\n SparkError::FileNotFound { .. } => \"org/apache/spark/SparkException\",\n\n // DuplicateFieldCaseInsensitive - converted to SparkRuntimeException by the shim\n SparkError::DuplicateFieldCaseInsensitive { .. } => {\n \"org/apache/spark/SparkRuntimeException\"\n }\n\n // Generic errors\n SparkError::Arrow(_) | SparkError::Internal(_) => \"org/apache/spark/SparkException\",\n }\n }\n\n /// Returns the Spark error class code for this error\n pub(crate) fn error_class(&self) -> Option<&'static str> {\n match self {\n // Cast errors\n SparkError::CastInvalidValue { .. } => Some(\"CAST_INVALID_INPUT\"),\n SparkError::InvalidInputInCastToDatetime { .. } => Some(\"CAST_INVALID_INPUT\"),\n SparkError::CastOverFlow { .. } => Some(\"CAST_OVERFLOW\"),\n SparkError::NumericValueOutOfRange { .. } => {\n Some(\"NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION\")\n }\n SparkError::NumericOutOfRange { .. } => Some(\"NUMERIC_OUT_OF_SUPPORTED_RANGE\"),\n SparkError::CannotParseDecimal => Some(\"CANNOT_PARSE_DECIMAL\"),\n\n // Arithmetic errors\n SparkError::DivideByZero => Some(\"DIVIDE_BY_ZERO\"),\n SparkError::RemainderByZero => Some(\"REMAINDER_BY_ZERO\"),\n SparkError::IntervalDividedByZero => Some(\"INTERVAL_DIVIDED_BY_ZERO\"),\n SparkError::ArithmeticOverflow { .. } => Some(\"ARITHMETIC_OVERFLOW\"),\n SparkError::IntegralDivideOverflow => Some(\"ARITHMETIC_OVERFLOW\"),\n SparkError::DecimalSumOverflow => Some(\"ARITHMETIC_OVERFLOW\"),\n SparkError::BinaryArithmeticOverflow { .. } => Some(\"BINARY_ARITHMETIC_OVERFLOW\"),\n SparkError::IntervalArithmeticOverflowWithSuggestion { .. } => {\n Some(\"INTERVAL_ARITHMETIC_OVERFLOW\")\n }\n SparkError::IntervalArithmeticOverflowWithoutSuggestion => {\n Some(\"INTERVAL_ARITHMETIC_OVERFLOW\")\n }\n SparkError::DatetimeOverflow => Some(\"DATETIME_OVERFLOW\"),\n\n // Array index errors\n SparkError::InvalidArrayIndex { .. } => Some(\"INVALID_ARRAY_INDEX\"),\n SparkError::InvalidElementAtIndex { .. } => Some(\"INVALID_ARRAY_INDEX_IN_ELEMENT_AT\"),\n SparkError::InvalidBitmapPosition { .. } => Some(\"INVALID_BITMAP_POSITION\"),\n SparkError::InvalidIndexOfZero => Some(\"INVALID_INDEX_OF_ZERO\"),\n\n // Map/Collection errors\n SparkError::DuplicatedMapKey { .. } => Some(\"DUPLICATED_MAP_KEY\"),\n SparkError::NullMapKey => Some(\"NULL_MAP_KEY\"),\n SparkError::MapKeyValueDiffSizes => Some(\"MAP_KEY_VALUE_DIFF_SIZES\"),\n SparkError::ExceedMapSizeLimit { .. } => Some(\"EXCEED_LIMIT_LENGTH\"),\n SparkError::CollectionSizeLimitExceeded { .. } => {\n Some(\"COLLECTION_SIZE_LIMIT_EXCEEDED\")\n }\n\n // Null validation errors\n SparkError::NotNullAssertViolation { .. } => Some(\"NOT_NULL_ASSERT_VIOLATION\"),\n SparkError::ValueIsNull { .. } => Some(\"VALUE_IS_NULL\"),\n\n // DateTime errors\n SparkError::CannotParseTimestamp { .. } => Some(\"CANNOT_PARSE_TIMESTAMP\"),\n SparkError::InvalidFractionOfSecond { .. } => Some(\"INVALID_FRACTION_OF_SECOND\"),\n\n // String/UTF8 errors\n SparkError::InvalidUtf8String { .. } => Some(\"INVALID_UTF8_STRING\"),\n\n // Function parameter errors\n SparkError::UnexpectedPositiveValue { .. } => Some(\"UNEXPECTED_POSITIVE_VALUE\"),\n SparkError::UnexpectedNegativeValue { .. } => Some(\"UNEXPECTED_NEGATIVE_VALUE\"),\n\n // Regex errors\n SparkError::InvalidRegexGroupIndex { .. } => Some(\"INVALID_PARAMETER_VALUE\"),\n\n // Unsupported operation errors\n SparkError::DatatypeCannotOrder { .. } => Some(\"DATATYPE_CANNOT_ORDER\"),\n\n // Subquery errors\n SparkError::ScalarSubqueryTooManyRows => Some(\"SCALAR_SUBQUERY_TOO_MANY_ROWS\"),\n\n // File not found\n SparkError::FileNotFound { .. } => Some(\"_LEGACY_ERROR_TEMP_2055\"),\n\n // Duplicate field in case-insensitive mode\n SparkError::DuplicateFieldCaseInsensitive { .. } => Some(\"_LEGACY_ERROR_TEMP_2093\"),\n\n // Generic errors (no error class)\n SparkError::Arrow(_) | SparkError::Internal(_) => None,\n }\n }\n}\n\npub type SparkResult = Result;\n\n/// Convert decimal overflow to SparkError::NumericValueOutOfRange.\npub fn decimal_overflow_error(value: i128, precision: u8, scale: i8) -> SparkError {\n SparkError::NumericValueOutOfRange {\n value: value.to_string(),\n precision,\n scale,\n }\n}\n\n/// Wrapper that adds QueryContext to SparkError\n///\n/// This allows attaching SQL context information (query text, line/position, object name) to errors\n#[derive(Debug, Clone)]\npub struct SparkErrorWithContext {\n /// The underlying SparkError\n pub error: SparkError,\n /// Optional QueryContext for SQL location information\n pub context: Option>,\n}\n\nimpl SparkErrorWithContext {\n /// Create a SparkErrorWithContext without context\n pub fn new(error: SparkError) -> Self {\n Self {\n error,\n context: None,\n }\n }\n\n /// Create a SparkErrorWithContext with QueryContext\n pub fn with_context(error: SparkError, context: Arc) -> Self {\n Self {\n error,\n context: Some(context),\n }\n }\n\n /// Serialize to JSON including optional context field\n pub fn to_json(&self) -> String {\n let mut json_obj = serde_json::json!({\n \"errorType\": self.error.error_type_name(),\n \"errorClass\": self.error.error_class().unwrap_or(\"\"),\n \"params\": self.error.params_as_json(),\n });\n\n if let Some(ctx) = &self.context {\n // Serialize context fields\n json_obj[\"context\"] = serde_json::json!({\n \"sqlText\": ctx.sql_text.as_str(),\n \"startIndex\": ctx.start_index,\n \"stopIndex\": ctx.stop_index,\n \"objectType\": ctx.object_type,\n \"objectName\": ctx.object_name,\n \"line\": ctx.line,\n \"startPosition\": ctx.start_position,\n });\n\n // Add formatted summary\n json_obj[\"summary\"] = serde_json::json!(ctx.format_summary());\n }\n\n serde_json::to_string(&json_obj).unwrap_or_else(|e| {\n format!(\n \"{{\\\"errorType\\\":\\\"SerializationError\\\",\\\"message\\\":\\\"{}\\\"}}\",\n e\n )\n })\n }\n}\n\nimpl std::fmt::Display for SparkErrorWithContext {\n fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n write!(f, \"{}\", self.error)?;\n if let Some(ctx) = &self.context {\n write!(f, \"\\n{}\", ctx.format_summary())?;\n }\n Ok(())\n }\n}\n\nimpl std::error::Error for SparkErrorWithContext {}\n\nimpl From for SparkErrorWithContext {\n fn from(error: SparkError) -> Self {\n SparkErrorWithContext::new(error)\n }\n}\n\nimpl From for DataFusionError {\n fn from(value: SparkErrorWithContext) -> Self {\n DataFusionError::External(Box::new(value))\n }\n}\n\nimpl From for SparkError {\n fn from(value: ArrowError) -> Self {\n SparkError::Arrow(Arc::new(value))\n }\n}\n\nimpl From for DataFusionError {\n fn from(value: SparkError) -> Self {\n DataFusionError::External(Box::new(value))\n }\n}\n\n#[cfg(test)]\nmod tests {\n use super::*;\n\n #[test]\n fn test_divide_by_zero_json() {\n let error = SparkError::DivideByZero;\n let json = error.to_json();\n\n assert!(json.contains(\"\\\"errorType\\\":\\\"DivideByZero\\\"\"));\n assert!(json.contains(\"\\\"errorClass\\\":\\\"DIVIDE_BY_ZERO\\\"\"));\n\n // Verify it's valid JSON\n let parsed: serde_json::Value = serde_json::from_str(&json).unwrap();\n assert_eq!(parsed[\"errorType\"], \"DivideByZero\");\n assert_eq!(parsed[\"errorClass\"], \"DIVIDE_BY_ZERO\");\n }\n\n #[test]\n fn test_remainder_by_zero_json() {\n let error = SparkError::RemainderByZero;\n let json = error.to_json();\n\n assert!(json.contains(\"\\\"errorType\\\":\\\"RemainderByZero\\\"\"));\n assert!(json.contains(\"\\\"errorClass\\\":\\\"REMAINDER_BY_ZERO\\\"\"));\n }\n\n #[test]\n fn test_binary_overflow_json() {\n let error = SparkError::BinaryArithmeticOverflow {\n value1: \"32767\".to_string(),\n symbol: \"+\".to_string(),\n value2: \"1\".to_string(),\n function_name: \"try_add\".to_string(),\n };\n let json = error.to_json();\n\n // Verify it's valid JSON\n let parsed: serde_json::Value = serde_json::from_str(&json).unwrap();\n assert_eq!(parsed[\"errorType\"], \"BinaryArithmeticOverflow\");\n assert_eq!(parsed[\"errorClass\"], \"BINARY_ARITHMETIC_OVERFLOW\");\n assert_eq!(parsed[\"params\"][\"value1\"], \"32767\");\n assert_eq!(parsed[\"params\"][\"symbol\"], \"+\");\n assert_eq!(parsed[\"params\"][\"value2\"], \"1\");\n assert_eq!(parsed[\"params\"][\"functionName\"], \"try_add\");\n }\n\n #[test]\n fn test_invalid_array_index_json() {\n let error = SparkError::InvalidArrayIndex {\n index_value: 10,\n array_size: 3,\n };\n let json = error.to_json();\n\n let parsed: serde_json::Value = serde_json::from_str(&json).unwrap();\n assert_eq!(parsed[\"errorType\"], \"InvalidArrayIndex\");\n assert_eq!(parsed[\"errorClass\"], \"INVALID_ARRAY_INDEX\");\n assert_eq!(parsed[\"params\"][\"indexValue\"], 10);\n assert_eq!(parsed[\"params\"][\"arraySize\"], 3);\n }\n\n #[test]\n fn test_numeric_value_out_of_range_json() {\n let error = SparkError::NumericValueOutOfRange {\n value: \"999.99\".to_string(),\n precision: 5,\n scale: 2,\n };\n let json = error.to_json();\n\n let parsed: serde_json::Value = serde_json::from_str(&json).unwrap();\n assert_eq!(parsed[\"errorType\"], \"NumericValueOutOfRange\");\n assert_eq!(\n parsed[\"errorClass\"],\n \"NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION\"\n );\n assert_eq!(parsed[\"params\"][\"value\"], \"999.99\");\n assert_eq!(parsed[\"params\"][\"precision\"], 5);\n assert_eq!(parsed[\"params\"][\"scale\"], 2);\n }\n\n #[test]\n fn test_cast_invalid_value_json() {\n let error = SparkError::CastInvalidValue {\n value: \"abc\".to_string(),\n from_type: \"STRING\".to_string(),\n to_type: \"INT\".to_string(),\n };\n let json = error.to_json();\n\n let parsed: serde_json::Value = serde_json::from_str(&json).unwrap();\n assert_eq!(parsed[\"errorType\"], \"CastInvalidValue\");\n assert_eq!(parsed[\"errorClass\"], \"CAST_INVALID_INPUT\");\n assert_eq!(parsed[\"params\"][\"value\"], \"abc\");\n assert_eq!(parsed[\"params\"][\"fromType\"], \"STRING\");\n assert_eq!(parsed[\"params\"][\"toType\"], \"INT\");\n }\n\n #[test]\n fn test_duplicated_map_key_json() {\n let error = SparkError::DuplicatedMapKey {\n key: \"duplicate_key\".to_string(),\n };\n let json = error.to_json();\n\n let parsed: serde_json::Value = serde_json::from_str(&json).unwrap();\n assert_eq!(parsed[\"errorType\"], \"DuplicatedMapKey\");\n assert_eq!(parsed[\"errorClass\"], \"DUPLICATED_MAP_KEY\");\n assert_eq!(parsed[\"params\"][\"key\"], \"duplicate_key\");\n }\n\n #[test]\n fn test_null_map_key_json() {\n let error = SparkError::NullMapKey;\n let json = error.to_json();\n\n let parsed: serde_json::Value = serde_json::from_str(&json).unwrap();\n assert_eq!(parsed[\"errorType\"], \"NullMapKey\");\n assert_eq!(parsed[\"errorClass\"], \"NULL_MAP_KEY\");\n // Params should be an empty object\n assert_eq!(parsed[\"params\"], serde_json::json!({}));\n }\n\n #[test]\n fn test_error_class_mapping() {\n // Test that error_class() returns the correct error class\n assert_eq!(\n SparkError::DivideByZero.error_class(),\n Some(\"DIVIDE_BY_ZERO\")\n );\n assert_eq!(\n SparkError::RemainderByZero.error_class(),\n Some(\"REMAINDER_BY_ZERO\")\n );\n assert_eq!(\n SparkError::InvalidArrayIndex {\n index_value: 0,\n array_size: 0\n }\n .error_class(),\n Some(\"INVALID_ARRAY_INDEX\")\n );\n assert_eq!(SparkError::NullMapKey.error_class(), Some(\"NULL_MAP_KEY\"));\n }\n\n #[test]\n fn test_exception_class_mapping() {\n // Test that exception_class() returns the correct Java exception class\n assert_eq!(\n SparkError::DivideByZero.exception_class(),\n \"org/apache/spark/SparkArithmeticException\"\n );\n assert_eq!(\n SparkError::InvalidArrayIndex {\n index_value: 0,\n array_size: 0\n }\n .exception_class(),\n \"org/apache/spark/SparkArrayIndexOutOfBoundsException\"\n );\n assert_eq!(\n SparkError::NullMapKey.exception_class(),\n \"org/apache/spark/SparkRuntimeException\"\n );\n }\n}\n" }, { "path": "native/common/src/lib.rs", "content": "// Licensed to the Apache Software Foundation (ASF) under one\n// or more contributor license agreements. See the NOTICE file\n// distributed with this work for additional information\n// regarding copyright ownership. The ASF licenses this file\n// to you under the Apache License, Version 2.0 (the\n// \"License\"); you may not use this file except in compliance\n// with the License. You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing,\n// software distributed under the License is distributed on an\n// \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n// KIND, either express or implied. See the License for the\n// specific language governing permissions and limitations\n// under the License.\n\nmod error;\nmod query_context;\npub mod tracing;\nmod utils;\n\npub use error::{decimal_overflow_error, SparkError, SparkErrorWithContext, SparkResult};\npub use query_context::{create_query_context_map, QueryContext, QueryContextMap};\npub use utils::bytes_to_i128;\n" }, { "path": "native/common/src/query_context.rs", "content": "// Licensed to the Apache Software Foundation (ASF) under one\n// or more contributor license agreements. See the NOTICE file\n// distributed with this work for additional information\n// regarding copyright ownership. The ASF licenses this file\n// to you under the Apache License, Version 2.0 (the\n// \"License\"); you may not use this file except in compliance\n// with the License. You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing,\n// software distributed under the License is distributed on an\n// \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n// KIND, either express or implied. See the License for the\n// specific language governing permissions and limitations\n// under the License.\n\n//! Query execution context for error reporting\n//!\n//! This module provides QueryContext which mirrors Spark's SQLQueryContext\n//! for providing SQL text, line/position information, and error location\n//! pointers in exception messages.\n\nuse serde::{Deserialize, Serialize};\nuse std::sync::Arc;\n\n/// Based on Spark's SQLQueryContext for error reporting.\n///\n/// Contains information about where an error occurred in a SQL query,\n/// including the full SQL text, line/column positions, and object context.\n#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]\npub struct QueryContext {\n /// Full SQL query text\n #[serde(rename = \"sqlText\")]\n pub sql_text: Arc,\n\n /// Start offset in SQL text (0-based, character index)\n #[serde(rename = \"startIndex\")]\n pub start_index: i32,\n\n /// Stop offset in SQL text (0-based, character index, inclusive)\n #[serde(rename = \"stopIndex\")]\n pub stop_index: i32,\n\n /// Object type (e.g., \"VIEW\", \"Project\", \"Filter\")\n #[serde(rename = \"objectType\", skip_serializing_if = \"Option::is_none\")]\n pub object_type: Option,\n\n /// Object name (e.g., view name, column name)\n #[serde(rename = \"objectName\", skip_serializing_if = \"Option::is_none\")]\n pub object_name: Option,\n\n /// Line number in SQL query (1-based)\n pub line: i32,\n\n /// Column position within the line (0-based)\n #[serde(rename = \"startPosition\")]\n pub start_position: i32,\n}\n\nimpl QueryContext {\n #[allow(clippy::too_many_arguments)]\n pub fn new(\n sql_text: String,\n start_index: i32,\n stop_index: i32,\n object_type: Option,\n object_name: Option,\n line: i32,\n start_position: i32,\n ) -> Self {\n Self {\n sql_text: Arc::new(sql_text),\n start_index,\n stop_index,\n object_type,\n object_name,\n line,\n start_position,\n }\n }\n\n /// Convert a character index to a byte offset in the SQL text.\n /// Returns None if the character index is out of range.\n fn char_index_to_byte_offset(&self, char_index: usize) -> Option {\n self.sql_text\n .char_indices()\n .nth(char_index)\n .map(|(byte_offset, _)| byte_offset)\n }\n\n /// Generate a summary string showing SQL fragment with error location.\n /// (From SQLQueryContext.summary)\n ///\n /// Format example:\n /// ```text\n /// == SQL of VIEW v1 (line 1, position 8) ==\n /// SELECT a/b FROM t\n /// ^^^\n /// ```\n pub fn format_summary(&self) -> String {\n let start_char = self.start_index.max(0) as usize;\n // stop_index is inclusive; fragment covers [start, stop]\n let stop_char = (self.stop_index + 1).max(0) as usize;\n\n let fragment = match (\n self.char_index_to_byte_offset(start_char),\n // stop_char may equal sql_text.chars().count() (one past the end)\n self.char_index_to_byte_offset(stop_char).or_else(|| {\n if stop_char == self.sql_text.chars().count() {\n Some(self.sql_text.len())\n } else {\n None\n }\n }),\n ) {\n (Some(start_byte), Some(stop_byte)) => &self.sql_text[start_byte..stop_byte],\n _ => \"\",\n };\n\n // Build the header line\n let mut summary = String::from(\"== SQL\");\n\n if let Some(obj_type) = &self.object_type {\n if !obj_type.is_empty() {\n summary.push_str(\" of \");\n summary.push_str(obj_type);\n\n if let Some(obj_name) = &self.object_name {\n if !obj_name.is_empty() {\n summary.push(' ');\n summary.push_str(obj_name);\n }\n }\n }\n }\n\n summary.push_str(&format!(\n \" (line {}, position {}) ==\\n\",\n self.line,\n self.start_position + 1 // Convert 0-based to 1-based for display\n ));\n\n // Add the SQL text with fragment highlighted\n summary.push_str(&self.sql_text);\n summary.push('\\n');\n\n // Add caret pointer\n let caret_position = self.start_position.max(0) as usize;\n summary.push_str(&\" \".repeat(caret_position));\n // fragment.chars().count() gives the correct display width for non-ASCII\n summary.push_str(&\"^\".repeat(fragment.chars().count().max(1)));\n\n summary\n }\n\n /// Returns the SQL fragment that caused the error.\n #[cfg(test)]\n fn fragment(&self) -> String {\n let start_char = self.start_index.max(0) as usize;\n let stop_char = (self.stop_index + 1).max(0) as usize;\n\n match (\n self.char_index_to_byte_offset(start_char),\n self.char_index_to_byte_offset(stop_char).or_else(|| {\n if stop_char == self.sql_text.chars().count() {\n Some(self.sql_text.len())\n } else {\n None\n }\n }),\n ) {\n (Some(start_byte), Some(stop_byte)) => self.sql_text[start_byte..stop_byte].to_string(),\n _ => String::new(),\n }\n }\n}\n\nuse std::collections::HashMap;\nuse std::sync::RwLock;\n\n/// Map that stores QueryContext information for expressions during execution.\n///\n/// This map is populated during plan deserialization and accessed\n/// during error creation to attach SQL context to exceptions.\n#[derive(Debug)]\npub struct QueryContextMap {\n /// Map from expression ID to QueryContext\n contexts: RwLock>>,\n}\n\nimpl QueryContextMap {\n pub fn new() -> Self {\n Self {\n contexts: RwLock::new(HashMap::new()),\n }\n }\n\n /// Register a QueryContext for an expression ID.\n ///\n /// If the expression ID already exists, it will be replaced.\n ///\n /// # Arguments\n /// * `expr_id` - Unique expression identifier from protobuf\n /// * `context` - QueryContext containing SQL text and position info\n pub fn register(&self, expr_id: u64, context: QueryContext) {\n let mut contexts = self.contexts.write().unwrap();\n contexts.insert(expr_id, Arc::new(context));\n }\n\n /// Get the QueryContext for an expression ID.\n ///\n /// Returns None if no context is registered for this expression.\n ///\n /// # Arguments\n /// * `expr_id` - Expression identifier to look up\n pub fn get(&self, expr_id: u64) -> Option> {\n let contexts = self.contexts.read().unwrap();\n contexts.get(&expr_id).cloned()\n }\n\n /// Clear all registered contexts.\n ///\n /// This is typically called after plan execution completes to free memory.\n pub fn clear(&self) {\n let mut contexts = self.contexts.write().unwrap();\n contexts.clear();\n }\n\n /// Return the number of registered contexts (for debugging/testing)\n pub fn len(&self) -> usize {\n let contexts = self.contexts.read().unwrap();\n contexts.len()\n }\n\n /// Check if the map is empty\n pub fn is_empty(&self) -> bool {\n self.len() == 0\n }\n}\n\nimpl Default for QueryContextMap {\n fn default() -> Self {\n Self::new()\n }\n}\n\n/// Create a new session-scoped QueryContextMap.\n///\n/// This should be called once per SessionContext during plan creation\n/// and passed to expressions that need query context for error reporting.\npub fn create_query_context_map() -> Arc {\n Arc::new(QueryContextMap::new())\n}\n\n#[cfg(test)]\nmod tests {\n use super::*;\n\n #[test]\n fn test_query_context_creation() {\n let ctx = QueryContext::new(\n \"SELECT a/b FROM t\".to_string(),\n 7,\n 9,\n Some(\"Divide\".to_string()),\n Some(\"a/b\".to_string()),\n 1,\n 7,\n );\n\n assert_eq!(*ctx.sql_text, \"SELECT a/b FROM t\");\n assert_eq!(ctx.start_index, 7);\n assert_eq!(ctx.stop_index, 9);\n assert_eq!(ctx.object_type, Some(\"Divide\".to_string()));\n assert_eq!(ctx.object_name, Some(\"a/b\".to_string()));\n assert_eq!(ctx.line, 1);\n assert_eq!(ctx.start_position, 7);\n }\n\n #[test]\n fn test_query_context_serialization() {\n let ctx = QueryContext::new(\n \"SELECT a/b FROM t\".to_string(),\n 7,\n 9,\n Some(\"Divide\".to_string()),\n Some(\"a/b\".to_string()),\n 1,\n 7,\n );\n\n let json = serde_json::to_string(&ctx).unwrap();\n let deserialized: QueryContext = serde_json::from_str(&json).unwrap();\n\n assert_eq!(ctx, deserialized);\n }\n\n #[test]\n fn test_format_summary() {\n let ctx = QueryContext::new(\n \"SELECT a/b FROM t\".to_string(),\n 7,\n 9,\n Some(\"VIEW\".to_string()),\n Some(\"v1\".to_string()),\n 1,\n 7,\n );\n\n let summary = ctx.format_summary();\n\n assert!(summary.contains(\"== SQL of VIEW v1 (line 1, position 8) ==\"));\n assert!(summary.contains(\"SELECT a/b FROM t\"));\n assert!(summary.contains(\"^^^\")); // Three carets for \"a/b\"\n }\n\n #[test]\n fn test_format_summary_without_object() {\n let ctx = QueryContext::new(\"SELECT a/b FROM t\".to_string(), 7, 9, None, None, 1, 7);\n\n let summary = ctx.format_summary();\n\n assert!(summary.contains(\"== SQL (line 1, position 8) ==\"));\n assert!(summary.contains(\"SELECT a/b FROM t\"));\n }\n\n #[test]\n fn test_fragment() {\n let ctx = QueryContext::new(\"SELECT a/b FROM t\".to_string(), 7, 9, None, None, 1, 7);\n\n assert_eq!(ctx.fragment(), \"a/b\");\n }\n\n #[test]\n fn test_arc_string_sharing() {\n let ctx1 = QueryContext::new(\"SELECT a/b FROM t\".to_string(), 7, 9, None, None, 1, 7);\n\n let ctx2 = ctx1.clone();\n\n // Arc should share the same allocation\n assert!(Arc::ptr_eq(&ctx1.sql_text, &ctx2.sql_text));\n }\n\n #[test]\n fn test_json_with_optional_fields() {\n let ctx = QueryContext::new(\"SELECT a/b FROM t\".to_string(), 7, 9, None, None, 1, 7);\n\n let json = serde_json::to_string(&ctx).unwrap();\n\n // Should not serialize objectType and objectName when None\n assert!(!json.contains(\"objectType\"));\n assert!(!json.contains(\"objectName\"));\n }\n\n #[test]\n fn test_map_register_and_get() {\n let map = QueryContextMap::new();\n\n let ctx = QueryContext::new(\"SELECT a/b FROM t\".to_string(), 7, 9, None, None, 1, 7);\n\n map.register(1, ctx.clone());\n\n let retrieved = map.get(1).unwrap();\n assert_eq!(*retrieved.sql_text, \"SELECT a/b FROM t\");\n assert_eq!(retrieved.start_index, 7);\n }\n\n #[test]\n fn test_map_get_nonexistent() {\n let map = QueryContextMap::new();\n assert!(map.get(999).is_none());\n }\n\n #[test]\n fn test_map_clear() {\n let map = QueryContextMap::new();\n\n let ctx = QueryContext::new(\"SELECT a/b FROM t\".to_string(), 7, 9, None, None, 1, 7);\n\n map.register(1, ctx);\n assert_eq!(map.len(), 1);\n\n map.clear();\n assert_eq!(map.len(), 0);\n assert!(map.is_empty());\n }\n\n // Verify that fragment() and format_summary() correctly handle SQL text that\n // contains multi-byte characters\n\n #[test]\n fn test_fragment_non_ascii_accented() {\n // \"é\" is a 2-byte UTF-8 sequence (U+00E9).\n // SQL: \"SELECT café FROM t\"\n // 0123456789...\n // char indices: c=7, a=8, f=9, é=10, ' '=11 ... FROM = 12..\n // start_index=7, stop_index=10 should yield \"café\"\n let sql = \"SELECT café FROM t\".to_string();\n let ctx = QueryContext::new(sql, 7, 10, None, None, 1, 7);\n assert_eq!(ctx.fragment(), \"café\");\n }\n}\n" }, { "path": "native/common/src/tracing.rs", "content": "// Licensed to the Apache Software Foundation (ASF) under one\n// or more contributor license agreements. See the NOTICE file\n// distributed with this work for additional information\n// regarding copyright ownership. The ASF licenses this file\n// to you under the Apache License, Version 2.0 (the\n// \"License\"); you may not use this file except in compliance\n// with the License. You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing,\n// software distributed under the License is distributed on an\n// \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n// KIND, either express or implied. See the License for the\n// specific language governing permissions and limitations\n// under the License.\n\nuse datafusion::common::instant::Instant;\nuse std::fs::{File, OpenOptions};\nuse std::io::{BufWriter, Write};\nuse std::sync::{Arc, LazyLock, Mutex};\n\npub static RECORDER: LazyLock = LazyLock::new(Recorder::new);\n\n/// Log events using Chrome trace format JSON\n/// https://github.com/catapult-project/catapult/blob/main/tracing/README.md\npub struct Recorder {\n now: Instant,\n writer: Arc>>,\n}\n\nimpl Default for Recorder {\n fn default() -> Self {\n Self::new()\n }\n}\n\nimpl Recorder {\n pub fn new() -> Self {\n let file = OpenOptions::new()\n .create(true)\n .append(true)\n .open(\"comet-event-trace.json\")\n .expect(\"Error writing tracing\");\n\n let mut writer = BufWriter::new(file);\n\n // Write start of JSON array. Note that there is no requirement to write\n // the closing ']'.\n writer\n .write_all(\"[ \".as_bytes())\n .expect(\"Error writing tracing\");\n Self {\n now: Instant::now(),\n writer: Arc::new(Mutex::new(writer)),\n }\n }\n pub fn begin_task(&self, name: &str) {\n self.log_event(name, \"B\")\n }\n\n pub fn end_task(&self, name: &str) {\n self.log_event(name, \"E\")\n }\n\n pub fn log_memory_usage(&self, name: &str, usage_bytes: u64) {\n let json = format!(\n \"{{ \\\"name\\\": \\\"{name}\\\", \\\"cat\\\": \\\"PERF\\\", \\\"ph\\\": \\\"C\\\", \\\"pid\\\": 1, \\\"tid\\\": {}, \\\"ts\\\": {}, \\\"args\\\": {{ \\\"{name}\\\": {usage_bytes} }} }},\\n\",\n get_thread_id(),\n self.now.elapsed().as_micros()\n );\n let mut writer = self.writer.lock().unwrap();\n writer\n .write_all(json.as_bytes())\n .expect(\"Error writing tracing\");\n }\n\n fn log_event(&self, name: &str, ph: &str) {\n let json = format!(\n \"{{ \\\"name\\\": \\\"{}\\\", \\\"cat\\\": \\\"PERF\\\", \\\"ph\\\": \\\"{ph}\\\", \\\"pid\\\": 1, \\\"tid\\\": {}, \\\"ts\\\": {} }},\\n\",\n name,\n get_thread_id(),\n self.now.elapsed().as_micros()\n );\n let mut writer = self.writer.lock().unwrap();\n writer\n .write_all(json.as_bytes())\n .expect(\"Error writing tracing\");\n }\n}\n\npub fn get_thread_id() -> u64 {\n let thread_id = std::thread::current().id();\n format!(\"{thread_id:?}\")\n .trim_start_matches(\"ThreadId(\")\n .trim_end_matches(\")\")\n .parse()\n .expect(\"Error parsing thread id\")\n}\n\npub fn trace_begin(name: &str) {\n RECORDER.begin_task(name);\n}\n\npub fn trace_end(name: &str) {\n RECORDER.end_task(name);\n}\n\npub fn log_memory_usage(name: &str, value: u64) {\n RECORDER.log_memory_usage(name, value);\n}\n\npub fn with_trace(label: &str, tracing_enabled: bool, f: F) -> T\nwhere\n F: FnOnce() -> T,\n{\n if tracing_enabled {\n trace_begin(label);\n }\n\n let result = f();\n\n if tracing_enabled {\n trace_end(label);\n }\n\n result\n}\n\npub async fn with_trace_async(label: &str, tracing_enabled: bool, f: F) -> T\nwhere\n F: FnOnce() -> Fut,\n Fut: std::future::Future,\n{\n if tracing_enabled {\n trace_begin(label);\n }\n\n let result = f().await;\n\n if tracing_enabled {\n trace_end(label);\n }\n\n result\n}\n" }, { "path": "native/common/src/utils.rs", "content": "// Licensed to the Apache Software Foundation (ASF) under one\n// or more contributor license agreements. See the NOTICE file\n// distributed with this work for additional information\n// regarding copyright ownership. The ASF licenses this file\n// to you under the Apache License, Version 2.0 (the\n// \"License\"); you may not use this file except in compliance\n// with the License. You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing,\n// software distributed under the License is distributed on an\n// \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n// KIND, either express or implied. See the License for the\n// specific language governing permissions and limitations\n// under the License.\n\n/// Converts a slice of bytes to i128. The bytes are serialized in big-endian order by\n/// `BigInteger.toByteArray()` in Java.\npub fn bytes_to_i128(slice: &[u8]) -> i128 {\n let mut bytes = [0; 16];\n let mut i = 0;\n while i != 16 && i != slice.len() {\n bytes[i] = slice[slice.len() - 1 - i];\n i += 1;\n }\n\n // if the decimal is negative, we need to flip all the bits\n if (slice[0] as i8) < 0 {\n while i < 16 {\n bytes[i] = !bytes[i];\n i += 1;\n }\n }\n\n i128::from_le_bytes(bytes)\n}\n" }, { "path": "native/core/Cargo.toml", "content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n[package]\nname = \"datafusion-comet\"\nversion = { workspace = true }\nhomepage = \"https://datafusion.apache.org/comet\"\nrepository = \"https://github.com/apache/datafusion-comet\"\nauthors = [\"Apache DataFusion \"]\ndescription = \"Apache DataFusion Comet: High performance accelerator for Apache Spark\"\nreadme = \"README.md\"\nlicense = \"Apache-2.0\"\nedition = \"2021\"\ninclude = [\n \"benches/*.rs\",\n \"src/**/*.rs\",\n \"Cargo.toml\",\n]\n\n# this crate is used in the Spark plugin and does not contain public Rust APIs so we do not publish this crate\npublish = false\n\n[dependencies]\narrow = { workspace = true }\nparquet = { workspace = true, default-features = false, features = [\"experimental\", \"arrow\"] }\nfutures = { workspace = true }\nmimalloc = { version = \"*\", default-features = false, optional = true }\ntikv-jemallocator = { version = \"0.6.1\", optional = true, features = [\"disable_initial_exec_tls\"] }\ntikv-jemalloc-ctl = { version = \"0.6.1\", optional = true, features = [\"disable_initial_exec_tls\", \"stats\"] }\ntokio = { version = \"1\", features = [\"rt-multi-thread\"] }\nasync-trait = { workspace = true }\nlog = \"0.4\"\nlog4rs = \"1.4.0\"\nprost = \"0.14.3\"\njni = \"0.22.4\"\nrand = { workspace = true }\nnum = { workspace = true }\nbytes = { workspace = true }\ntempfile = \"3.26.0\"\nitertools = \"0.14.0\"\npaste = \"1.0.14\"\ndatafusion = { workspace = true, features = [\"parquet_encryption\", \"sql\"] }\ndatafusion-physical-expr-adapter = { workspace = true }\ndatafusion-datasource = { workspace = true }\ndatafusion-spark = { workspace = true }\nonce_cell = \"1.18.0\"\ndatafusion-comet-common = { workspace = true }\ndatafusion-comet-spark-expr = { workspace = true }\ndatafusion-comet-jni-bridge = { workspace = true }\ndatafusion-comet-proto = { workspace = true }\ndatafusion-comet-shuffle = { workspace = true }\nobject_store = { workspace = true }\nurl = { workspace = true }\naws-config = { workspace = true }\naws-credential-types = { workspace = true }\nparking_lot = \"0.12.5\"\ndatafusion-comet-objectstore-hdfs = { path = \"../hdfs\", optional = true, default-features = false, features = [\"hdfs\"] }\nreqwest = { version = \"0.12\", default-features = false, features = [\"rustls-tls-native-roots\", \"http2\"] }\nobject_store_opendal = { git = \"https://github.com/apache/opendal\", rev = \"6909efcdfd12b3b2ac3a76f654c35ee576811512\", package = \"object_store_opendal\", optional = true}\nhdfs-sys = {version = \"0.3\", optional = true, features = [\"hdfs_3_3\"]}\nopendal = { git = \"https://github.com/apache/opendal\", rev = \"6909efcdfd12b3b2ac3a76f654c35ee576811512\", optional = true, features = [\"services-hdfs\"] }\niceberg = { workspace = true }\niceberg-storage-opendal = { workspace = true }\nserde_json = \"1.0\"\nuuid = \"1.23.0\"\n\n[target.'cfg(target_os = \"linux\")'.dependencies]\nprocfs = \"0.18.0\"\n\n[target.'cfg(target_os = \"macos\")'.dependencies]\nhdrs = { version = \"0.3.2\", features = [\"vendored\"] }\n\n[dev-dependencies]\npprof = { version = \"0.15\", features = [\"flamegraph\"] }\ncriterion = { version = \"0.7\", features = [\"async\", \"async_tokio\", \"async_std\"] }\njni = { version = \"0.22.4\", features = [\"invocation\"] }\nlazy_static = \"1.4\"\nassertables = \"9\"\nhex = \"0.4.3\"\ndatafusion-functions-nested = { version = \"53.1.0\" }\n\n[features]\nbacktrace = [\"datafusion/backtrace\"]\ndefault = [\"hdfs-opendal\"]\nhdfs = [\"datafusion-comet-objectstore-hdfs\"]\nhdfs-opendal = [\"opendal\", \"object_store_opendal\", \"hdfs-sys\"]\njemalloc = [\"tikv-jemallocator\", \"tikv-jemalloc-ctl\"]\n\n# exclude optional packages from cargo machete verifications\n[package.metadata.cargo-machete]\nignored = [\"hdfs-sys\", \"paste\"]\n\n[lib]\nname = \"comet\"\n# \"rlib\" is for benchmarking with criterion.\ncrate-type = [\"cdylib\", \"rlib\"]\n\n[[bench]]\nname = \"parquet_read\"\nharness = false\n\n[[bench]]\nname = \"bit_util\"\nharness = false\n\n[[bench]]\nname = \"parquet_decode\"\nharness = false\n\n[[bench]]\nname = \"array_element_append\"\nharness = false\n" }, { "path": "native/core/benches/array_element_append.rs", "content": "// Licensed to the Apache Software Foundation (ASF) under one\n// or more contributor license agreements. See the NOTICE file\n// distributed with this work for additional information\n// regarding copyright ownership. The ASF licenses this file\n// to you under the Apache License, Version 2.0 (the\n// \"License\"); you may not use this file except in compliance\n// with the License. You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing,\n// software distributed under the License is distributed on an\n// \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n// KIND, either express or implied. See the License for the\n// specific language governing permissions and limitations\n// under the License.\n\n//! Micro-benchmarks for SparkUnsafeArray element iteration.\n//!\n//! This tests the low-level `append_to_builder` function which converts\n//! SparkUnsafeArray elements to Arrow array builders. This is the inner loop\n//! used when processing List/Array columns in JVM shuffle.\n\nuse arrow::array::builder::{\n Date32Builder, Float64Builder, Int32Builder, Int64Builder, TimestampMicrosecondBuilder,\n};\nuse arrow::datatypes::{DataType, TimeUnit};\nuse comet::execution::shuffle::spark_unsafe::list::{append_to_builder, SparkUnsafeArray};\nuse criterion::{criterion_group, criterion_main, BenchmarkId, Criterion};\n\nconst NUM_ELEMENTS: usize = 10000;\n\n/// Create a SparkUnsafeArray in memory with i32 elements.\n/// Layout:\n/// - 8 bytes: num_elements (i64)\n/// - null bitset: 8 bytes per 64 elements\n/// - element data: 4 bytes per element (i32)\nfn create_spark_unsafe_array_i32(num_elements: usize, with_nulls: bool) -> Vec {\n // Header size: 8 (num_elements) + ceil(num_elements/64) * 8 (null bitset)\n let null_bitset_words = num_elements.div_ceil(64);\n let header_size = 8 + null_bitset_words * 8;\n let data_size = num_elements * 4; // i32 = 4 bytes\n let total_size = header_size + data_size;\n\n let mut buffer = vec![0u8; total_size];\n\n // Write num_elements\n buffer[0..8].copy_from_slice(&(num_elements as i64).to_le_bytes());\n\n // Write null bitset (set every 10th element as null if with_nulls)\n if with_nulls {\n for i in (0..num_elements).step_by(10) {\n let word_idx = i / 64;\n let bit_idx = i % 64;\n let word_offset = 8 + word_idx * 8;\n let current_word =\n i64::from_le_bytes(buffer[word_offset..word_offset + 8].try_into().unwrap());\n let new_word = current_word | (1i64 << bit_idx);\n buffer[word_offset..word_offset + 8].copy_from_slice(&new_word.to_le_bytes());\n }\n }\n\n // Write element data\n for i in 0..num_elements {\n let offset = header_size + i * 4;\n buffer[offset..offset + 4].copy_from_slice(&(i as i32).to_le_bytes());\n }\n\n buffer\n}\n\n/// Create a SparkUnsafeArray in memory with i64 elements.\nfn create_spark_unsafe_array_i64(num_elements: usize, with_nulls: bool) -> Vec {\n let null_bitset_words = num_elements.div_ceil(64);\n let header_size = 8 + null_bitset_words * 8;\n let data_size = num_elements * 8; // i64 = 8 bytes\n let total_size = header_size + data_size;\n\n let mut buffer = vec![0u8; total_size];\n\n // Write num_elements\n buffer[0..8].copy_from_slice(&(num_elements as i64).to_le_bytes());\n\n // Write null bitset\n if with_nulls {\n for i in (0..num_elements).step_by(10) {\n let word_idx = i / 64;\n let bit_idx = i % 64;\n let word_offset = 8 + word_idx * 8;\n let current_word =\n i64::from_le_bytes(buffer[word_offset..word_offset + 8].try_into().unwrap());\n let new_word = current_word | (1i64 << bit_idx);\n buffer[word_offset..word_offset + 8].copy_from_slice(&new_word.to_le_bytes());\n }\n }\n\n // Write element data\n for i in 0..num_elements {\n let offset = header_size + i * 8;\n buffer[offset..offset + 8].copy_from_slice(&(i as i64).to_le_bytes());\n }\n\n buffer\n}\n\n/// Create a SparkUnsafeArray in memory with f64 elements.\nfn create_spark_unsafe_array_f64(num_elements: usize, with_nulls: bool) -> Vec {\n let null_bitset_words = num_elements.div_ceil(64);\n let header_size = 8 + null_bitset_words * 8;\n let data_size = num_elements * 8; // f64 = 8 bytes\n let total_size = header_size + data_size;\n\n let mut buffer = vec![0u8; total_size];\n\n // Write num_elements\n buffer[0..8].copy_from_slice(&(num_elements as i64).to_le_bytes());\n\n // Write null bitset\n if with_nulls {\n for i in (0..num_elements).step_by(10) {\n let word_idx = i / 64;\n let bit_idx = i % 64;\n let word_offset = 8 + word_idx * 8;\n let current_word =\n i64::from_le_bytes(buffer[word_offset..word_offset + 8].try_into().unwrap());\n let new_word = current_word | (1i64 << bit_idx);\n buffer[word_offset..word_offset + 8].copy_from_slice(&new_word.to_le_bytes());\n }\n }\n\n // Write element data\n for i in 0..num_elements {\n let offset = header_size + i * 8;\n buffer[offset..offset + 8].copy_from_slice(&(i as f64).to_le_bytes());\n }\n\n buffer\n}\n\nfn benchmark_array_conversion(c: &mut Criterion) {\n let mut group = c.benchmark_group(\"spark_unsafe_array_to_arrow\");\n\n // Benchmark i32 array conversion\n for with_nulls in [false, true] {\n let buffer = create_spark_unsafe_array_i32(NUM_ELEMENTS, with_nulls);\n let array = SparkUnsafeArray::new(buffer.as_ptr() as i64);\n let null_str = if with_nulls { \"with_nulls\" } else { \"no_nulls\" };\n\n group.bench_with_input(\n BenchmarkId::new(\"i32\", null_str),\n &(&array, &buffer),\n |b, (array, _buffer)| {\n b.iter(|| {\n let mut builder = Int32Builder::with_capacity(NUM_ELEMENTS);\n if with_nulls {\n append_to_builder::(&DataType::Int32, &mut builder, array).unwrap();\n } else {\n append_to_builder::(&DataType::Int32, &mut builder, array).unwrap();\n }\n builder.finish()\n });\n },\n );\n }\n\n // Benchmark i64 array conversion\n for with_nulls in [false, true] {\n let buffer = create_spark_unsafe_array_i64(NUM_ELEMENTS, with_nulls);\n let array = SparkUnsafeArray::new(buffer.as_ptr() as i64);\n let null_str = if with_nulls { \"with_nulls\" } else { \"no_nulls\" };\n\n group.bench_with_input(\n BenchmarkId::new(\"i64\", null_str),\n &(&array, &buffer),\n |b, (array, _buffer)| {\n b.iter(|| {\n let mut builder = Int64Builder::with_capacity(NUM_ELEMENTS);\n if with_nulls {\n append_to_builder::(&DataType::Int64, &mut builder, array).unwrap();\n } else {\n append_to_builder::(&DataType::Int64, &mut builder, array).unwrap();\n }\n builder.finish()\n });\n },\n );\n }\n\n // Benchmark f64 array conversion\n for with_nulls in [false, true] {\n let buffer = create_spark_unsafe_array_f64(NUM_ELEMENTS, with_nulls);\n let array = SparkUnsafeArray::new(buffer.as_ptr() as i64);\n let null_str = if with_nulls { \"with_nulls\" } else { \"no_nulls\" };\n\n group.bench_with_input(\n BenchmarkId::new(\"f64\", null_str),\n &(&array, &buffer),\n |b, (array, _buffer)| {\n b.iter(|| {\n let mut builder = Float64Builder::with_capacity(NUM_ELEMENTS);\n if with_nulls {\n append_to_builder::(&DataType::Float64, &mut builder, array).unwrap();\n } else {\n append_to_builder::(&DataType::Float64, &mut builder, array)\n .unwrap();\n }\n builder.finish()\n });\n },\n );\n }\n\n // Benchmark date32 array conversion (same memory layout as i32)\n for with_nulls in [false, true] {\n let buffer = create_spark_unsafe_array_i32(NUM_ELEMENTS, with_nulls);\n let array = SparkUnsafeArray::new(buffer.as_ptr() as i64);\n let null_str = if with_nulls { \"with_nulls\" } else { \"no_nulls\" };\n\n group.bench_with_input(\n BenchmarkId::new(\"date32\", null_str),\n &(&array, &buffer),\n |b, (array, _buffer)| {\n b.iter(|| {\n let mut builder = Date32Builder::with_capacity(NUM_ELEMENTS);\n if with_nulls {\n append_to_builder::(&DataType::Date32, &mut builder, array).unwrap();\n } else {\n append_to_builder::(&DataType::Date32, &mut builder, array).unwrap();\n }\n builder.finish()\n });\n },\n );\n }\n\n // Benchmark timestamp array conversion (same memory layout as i64)\n for with_nulls in [false, true] {\n let buffer = create_spark_unsafe_array_i64(NUM_ELEMENTS, with_nulls);\n let array = SparkUnsafeArray::new(buffer.as_ptr() as i64);\n let null_str = if with_nulls { \"with_nulls\" } else { \"no_nulls\" };\n\n group.bench_with_input(\n BenchmarkId::new(\"timestamp\", null_str),\n &(&array, &buffer),\n |b, (array, _buffer)| {\n b.iter(|| {\n let mut builder = TimestampMicrosecondBuilder::with_capacity(NUM_ELEMENTS);\n let dt = DataType::Timestamp(TimeUnit::Microsecond, None);\n if with_nulls {\n append_to_builder::(&dt, &mut builder, array).unwrap();\n } else {\n append_to_builder::(&dt, &mut builder, array).unwrap();\n }\n builder.finish()\n });\n },\n );\n }\n\n group.finish();\n}\n\nfn config() -> Criterion {\n Criterion::default()\n}\n\ncriterion_group! {\n name = benches;\n config = config();\n targets = benchmark_array_conversion\n}\ncriterion_main!(benches);\n" }, { "path": "native/core/benches/bit_util.rs", "content": "// Licensed to the Apache Software Foundation (ASF) under one\n// or more contributor license agreements. See the NOTICE file\n// distributed with this work for additional information\n// regarding copyright ownership. The ASF licenses this file\n// to you under the Apache License, Version 2.0 (the\n// \"License\"); you may not use this file except in compliance\n// with the License. You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing,\n// software distributed under the License is distributed on an\n// \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n// KIND, either express or implied. See the License for the\n// specific language governing permissions and limitations\n// under the License.\n\nuse std::{mem::size_of, time::Duration};\n\nuse rand::{rng, RngExt};\n\nuse arrow::buffer::Buffer;\nuse comet::common::bit::{\n log2, read_num_bytes_u32, read_num_bytes_u64, read_u32, read_u64, set_bits, trailing_bits,\n BitReader, BitWriter,\n};\nuse criterion::{criterion_group, criterion_main, BenchmarkId, Criterion};\nuse std::hint::black_box;\n\n/// Benchmark to measure bit_util performance.\n/// To run this benchmark:\n/// `cd core && cargo bench --bench bit_util`\n/// Results will be written to \"core/target/criterion/bit_util/\"\nfn criterion_benchmark(c: &mut Criterion) {\n let mut group = c.benchmark_group(\"bit_util\");\n\n const N: usize = 1024 * 1024;\n let mut writer: BitWriter = BitWriter::new(N * 10);\n for _ in 0..N {\n if !writer.put_vlq_int(rng().random::()) {\n break;\n }\n }\n let buffer = writer.consume();\n let buffer = Buffer::from(buffer.as_slice());\n\n // log2\n for bits in (0..64).step_by(3) {\n let x = 1u64 << bits;\n group.bench_with_input(BenchmarkId::new(\"log2\", bits), &x, |b, &x| {\n b.iter(|| log2(black_box(x)));\n });\n }\n\n // set_bits\n for offset in (0..16).step_by(3) {\n for length in (0..16).step_by(3) {\n let x = (offset, length);\n group.bench_with_input(\n BenchmarkId::new(\"set_bits\", format!(\"offset_{}_length_{}\", x.0, x.1)),\n &x,\n |b, &x| {\n b.iter(|| set_bits(&mut [0u8; 4], black_box(x.0), black_box(x.1)));\n },\n );\n }\n }\n\n // get_vlq_int\n group.bench_function(\"get_vlq_int\", |b| {\n b.iter(|| {\n let mut reader: BitReader = BitReader::new_all(buffer.slice(0));\n bench_get_vlq_int(&mut reader)\n })\n });\n\n // get_bits\n for offset in (0..32).step_by(17) {\n for num_bits in (1..5).step_by(1) {\n let x = (offset, num_bits);\n group.bench_with_input(\n BenchmarkId::new(\"get_bits\", format!(\"offset_{}_num_bits_{}\", x.0, x.1)),\n &x,\n |b, &x| {\n let mut reader: BitReader = BitReader::new_all(buffer.slice(0));\n b.iter(|| reader.get_bits(&mut [0u8; 4], black_box(x.0), black_box(x.1)));\n },\n );\n }\n }\n\n // get_aligned\n for num_bytes in (1..=size_of::()).step_by(3) {\n let x = num_bytes;\n group.bench_with_input(\n BenchmarkId::new(\"get_aligned\", format!(\"u8_num_bytes_{x}\")),\n &x,\n |b, &x| {\n let mut reader: BitReader = BitReader::new_all(buffer.slice(0));\n b.iter(|| reader.get_aligned::(black_box(x)));\n },\n );\n }\n for num_bytes in (1..=size_of::()).step_by(3) {\n let x = num_bytes;\n group.bench_with_input(\n BenchmarkId::new(\"get_aligned\", format!(\"u32_num_bytes_{x}\")),\n &x,\n |b, &x| {\n let mut reader: BitReader = BitReader::new_all(buffer.slice(0));\n b.iter(|| reader.get_aligned::(black_box(x)));\n },\n );\n }\n for num_bytes in (1..=size_of::()).step_by(3) {\n let x = num_bytes;\n group.bench_with_input(\n BenchmarkId::new(\"get_aligned\", format!(\"i32_num_bytes_{x}\")),\n &x,\n |b, &x| {\n let mut reader: BitReader = BitReader::new_all(buffer.slice(0));\n b.iter(|| reader.get_aligned::(black_box(x)));\n },\n );\n }\n\n // get_value\n for num_bytes in (1..=size_of::()).step_by(3) {\n let x = num_bytes * 8;\n group.bench_with_input(\n BenchmarkId::new(\"get_value\", format!(\"i32_num_bits_{x}\")),\n &x,\n |b, &x| {\n let mut reader: BitReader = BitReader::new_all(buffer.slice(0));\n b.iter(|| reader.get_value::(black_box(x)));\n },\n );\n }\n\n // read_num_bytes_u64\n for num_bytes in (1..=8).step_by(7) {\n let x = num_bytes;\n group.bench_with_input(\n BenchmarkId::new(\"read_num_bytes_u64\", format!(\"num_bytes_{x}\")),\n &x,\n |b, &x| {\n b.iter(|| read_num_bytes_u64(black_box(x), black_box(buffer.as_slice())));\n },\n );\n }\n\n // read_num_bytes_u32\n for num_bytes in (1..=4).step_by(3) {\n let x = num_bytes;\n group.bench_with_input(\n BenchmarkId::new(\"read_num_bytes_u32\", format!(\"num_bytes_{x}\")),\n &x,\n |b, &x| {\n b.iter(|| read_num_bytes_u32(black_box(x), black_box(buffer.as_slice())));\n },\n );\n }\n\n // trailing_bits\n for length in (0..=64).step_by(32) {\n let x = length;\n group.bench_with_input(\n BenchmarkId::new(\"trailing_bits\", format!(\"num_bits_{x}\")),\n &x,\n |b, &x| {\n b.iter(|| trailing_bits(black_box(1234567890), black_box(x)));\n },\n );\n }\n\n // read_u64\n group.bench_function(\"read_u64\", |b| {\n b.iter(|| read_u64(black_box(&[0u8; 8])));\n });\n\n // read_u32\n group.bench_function(\"read_u32\", |b| {\n b.iter(|| read_u32(black_box(&[0u8; 4])));\n });\n\n // get_u32_value\n group.bench_function(\"get_u32_value\", |b| {\n b.iter(|| {\n let mut reader: BitReader = BitReader::new_all(buffer.slice(0));\n for _ in 0..(buffer.len() * 8 / 31) {\n black_box(reader.get_u32_value(black_box(31)));\n }\n })\n });\n\n group.finish();\n}\n\nfn bench_get_vlq_int(reader: &mut BitReader) {\n while let Some(v) = reader.get_vlq_int() {\n black_box(v);\n }\n}\n\nfn config() -> Criterion {\n Criterion::default()\n .measurement_time(Duration::from_millis(500))\n .warm_up_time(Duration::from_millis(500))\n}\n\ncriterion_group! {\n name = benches;\n config = config();\n targets = criterion_benchmark\n}\ncriterion_main!(benches);\n" }, { "path": "native/core/benches/common.rs", "content": "// Licensed to the Apache Software Foundation (ASF) under one\n// or more contributor license agreements. See the NOTICE file\n// distributed with this work for additional information\n// regarding copyright ownership. The ASF licenses this file\n// to you under the Apache License, Version 2.0 (the\n// \"License\"); you may not use this file except in compliance\n// with the License. You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing,\n// software distributed under the License is distributed on an\n// \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n// KIND, either express or implied. See the License for the\n// specific language governing permissions and limitations\n// under the License.\n\nuse arrow::error::ArrowError;\nuse arrow::{\n array::{DictionaryArray, Int64Array, PrimitiveArray},\n datatypes::{ArrowPrimitiveType, Int32Type},\n};\nuse rand::{\n distr::{Distribution, StandardUniform},\n rngs::StdRng,\n RngExt, SeedableRng,\n};\nuse std::sync::Arc;\n\n/// Returns fixed seedable RNG\npub fn seedable_rng() -> StdRng {\n StdRng::seed_from_u64(42)\n}\n\npub fn create_int64_array(size: usize, null_density: f32, min: i64, max: i64) -> Int64Array {\n let mut rng = seedable_rng();\n (0..size)\n .map(|_| {\n if rng.random::() < null_density {\n None\n } else {\n Some(rng.random_range(min..max))\n }\n })\n .collect()\n}\n\n#[allow(dead_code)]\npub fn create_primitive_array(size: usize, null_density: f32) -> PrimitiveArray\nwhere\n T: ArrowPrimitiveType,\n StandardUniform: Distribution,\n{\n let mut rng = seedable_rng();\n (0..size)\n .map(|_| {\n if rng.random::() < null_density {\n None\n } else {\n Some(rng.random())\n }\n })\n .collect()\n}\n\n/// Creates a dictionary with random keys and values, with value type `T`.\n/// Note here the keys are the dictionary indices.\n#[allow(dead_code)]\npub fn create_dictionary_array(\n size: usize,\n value_size: usize,\n null_density: f32,\n) -> Result, ArrowError>\nwhere\n T: ArrowPrimitiveType,\n StandardUniform: Distribution,\n{\n // values are not null\n let values = create_primitive_array::(value_size, 0.0);\n let keys = create_primitive_array::(size, null_density)\n .iter()\n .map(|v| v.map(|w| w.abs() % (value_size as i32)))\n .collect();\n DictionaryArray::try_new(keys, Arc::new(values))\n}\n" }, { "path": "native/core/benches/parquet_decode.rs", "content": "// Licensed to the Apache Software Foundation (ASF) under one\n// or more contributor license agreements. See the NOTICE file\n// distributed with this work for additional information\n// regarding copyright ownership. The ASF licenses this file\n// to you under the Apache License, Version 2.0 (the\n// \"License\"); you may not use this file except in compliance\n// with the License. You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing,\n// software distributed under the License is distributed on an\n// \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n// KIND, either express or implied. See the License for the\n// specific language governing permissions and limitations\n// under the License.\n\nuse arrow::datatypes::ToByteSlice;\nuse comet::parquet::read::values::{copy_i32_to_i16, copy_i32_to_u16, copy_i64_to_i64};\nuse criterion::{criterion_group, criterion_main, Criterion};\n\nfn criterion_benchmark(c: &mut Criterion) {\n let num = 1000;\n let source = vec![78_i8; num * 8];\n let mut group = c.benchmark_group(\"parquet_decode\");\n group.bench_function(\"decode_i32_to_i16\", |b| {\n let mut dest: Vec = vec![b' '; num * 2];\n b.iter(|| {\n copy_i32_to_i16(source.to_byte_slice(), dest.as_mut_slice(), num);\n });\n });\n group.bench_function(\"decode_i32_to_u16\", |b| {\n let mut dest: Vec = vec![b' '; num * 4];\n b.iter(|| {\n copy_i32_to_u16(source.to_byte_slice(), dest.as_mut_slice(), num);\n });\n });\n group.bench_function(\"decode_i64_to_i64\", |b| {\n let mut dest: Vec = vec![b' '; num * 8];\n b.iter(|| {\n copy_i64_to_i64(source.to_byte_slice(), dest.as_mut_slice(), num);\n });\n });\n}\n\n// Create UTF8 batch with strings representing ints, floats, nulls\nfn config() -> Criterion {\n Criterion::default()\n}\n\ncriterion_group! {\n name = benches;\n config = config();\n targets = criterion_benchmark\n}\ncriterion_main!(benches);\n" }, { "path": "native/core/benches/parquet_read.rs", "content": "// Licensed to the Apache Software Foundation (ASF) under one\n// or more contributor license agreements. See the NOTICE file\n// distributed with this work for additional information\n// regarding copyright ownership. The ASF licenses this file\n// to you under the Apache License, Version 2.0 (the\n// \"License\"); you may not use this file except in compliance\n// with the License. You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing,\n// software distributed under the License is distributed on an\n// \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n// KIND, either express or implied. See the License for the\n// specific language governing permissions and limitations\n// under the License.\n\nmod perf;\n\nuse std::sync::Arc;\n\nuse arrow::{array::ArrayData, buffer::Buffer};\nuse comet::parquet::{read::ColumnReader, util::jni::TypePromotionInfo};\nuse criterion::{criterion_group, criterion_main, Criterion};\nuse parquet::{\n basic::{Encoding, Type as PhysicalType},\n column::page::{PageIterator, PageReader},\n data_type::Int32Type,\n schema::types::{\n ColumnDescPtr, ColumnDescriptor, ColumnPath, PrimitiveTypeBuilder, SchemaDescPtr, TypePtr,\n },\n};\n\nuse comet::parquet::util::test_common::page_util::{\n DataPageBuilder, DataPageBuilderImpl, InMemoryPageIterator,\n};\n\nuse perf::FlamegraphProfiler;\nuse rand::{prelude::StdRng, RngExt, SeedableRng};\n\nfn bench(c: &mut Criterion) {\n let expected_num_values: usize = NUM_PAGES * VALUES_PER_PAGE;\n let mut group = c.benchmark_group(\"comet_parquet_read\");\n let schema = build_test_schema();\n\n let pages = build_plain_int32_pages(schema.column(0), 0.0);\n group.bench_function(\"INT/PLAIN/NOT_NULL\", |b| {\n let t = TypePtr::new(\n PrimitiveTypeBuilder::new(\"f\", PhysicalType::INT32)\n .with_length(4)\n .build()\n .unwrap(),\n );\n b.iter(|| {\n let cd = ColumnDescriptor::new(t.clone(), 0, 0, ColumnPath::from(Vec::new()));\n let promotion_info = TypePromotionInfo::new(PhysicalType::INT32, -1, -1, 32);\n let mut column_reader = TestColumnReader::new(\n cd,\n promotion_info,\n BATCH_SIZE,\n pages.clone(),\n expected_num_values,\n );\n\n let mut total = 0;\n for batch in column_reader.by_ref() {\n total += batch.len();\n ::std::mem::forget(batch);\n }\n assert_eq!(total, expected_num_values);\n });\n });\n}\n\nfn profiled() -> Criterion {\n Criterion::default().with_profiler(FlamegraphProfiler::new(100))\n}\n\ncriterion_group! {\n name = benches;\n config = profiled();\n targets = bench\n}\ncriterion_main!(benches);\n\nfn build_test_schema() -> SchemaDescPtr {\n use parquet::schema::{parser::parse_message_type, types::SchemaDescriptor};\n let message_type = \"\n message test_schema {\n REQUIRED INT32 c1;\n OPTIONAL INT32 c2;\n }\n \";\n parse_message_type(message_type)\n .map(|t| Arc::new(SchemaDescriptor::new(Arc::new(t))))\n .unwrap()\n}\n\nfn seedable_rng() -> StdRng {\n StdRng::seed_from_u64(42)\n}\n\n// test data params\nconst NUM_PAGES: usize = 1000;\nconst VALUES_PER_PAGE: usize = 10_000;\nconst BATCH_SIZE: usize = 4096;\n\nfn build_plain_int32_pages(\n column_desc: ColumnDescPtr,\n null_density: f32,\n) -> impl PageIterator + Clone {\n let max_def_level = column_desc.max_def_level();\n let max_rep_level = column_desc.max_rep_level();\n let rep_levels = vec![0; VALUES_PER_PAGE];\n let mut rng = seedable_rng();\n let mut pages: Vec = Vec::new();\n let mut int32_value = 0;\n for _ in 0..NUM_PAGES {\n // generate page\n let mut values = Vec::with_capacity(VALUES_PER_PAGE);\n let mut def_levels = Vec::with_capacity(VALUES_PER_PAGE);\n for _ in 0..VALUES_PER_PAGE {\n let def_level = if rng.random::() < null_density {\n max_def_level - 1\n } else {\n max_def_level\n };\n if def_level == max_def_level {\n int32_value += 1;\n values.push(int32_value);\n }\n def_levels.push(def_level);\n }\n let mut page_builder =\n DataPageBuilderImpl::new(column_desc.clone(), values.len() as u32, true);\n page_builder.add_rep_levels(max_rep_level, &rep_levels);\n page_builder.add_def_levels(max_def_level, &def_levels);\n page_builder.add_values::(Encoding::PLAIN, &values);\n pages.push(page_builder.consume());\n }\n\n // Since `InMemoryPageReader` is not exposed from parquet crate, here we use\n // `InMemoryPageIterator` instead which is a Iter>.\n InMemoryPageIterator::new(vec![pages])\n}\n\nstruct TestColumnReader {\n inner: ColumnReader,\n pages: Box,\n batch_size: usize,\n total_num_values: usize,\n total_num_values_read: usize,\n first_page_loaded: bool,\n}\n\nimpl TestColumnReader {\n pub fn new(\n cd: ColumnDescriptor,\n promotion_info: TypePromotionInfo,\n batch_size: usize,\n mut page_iter: impl PageIterator + 'static,\n total_num_values: usize,\n ) -> Self {\n let reader = ColumnReader::get(cd, promotion_info, batch_size, false, false);\n let first = page_iter.next().unwrap().unwrap();\n Self {\n inner: reader,\n pages: first,\n batch_size,\n total_num_values,\n total_num_values_read: 0,\n first_page_loaded: false,\n }\n }\n\n fn load_page(&mut self) {\n if let Some(page) = self.pages.get_next_page().unwrap() {\n let num_values = page.num_values() as usize;\n let buffer = Buffer::from_slice_ref(page.buffer());\n self.inner.set_page_v1(num_values, buffer, page.encoding());\n }\n }\n}\n\nimpl Iterator for TestColumnReader {\n type Item = ArrayData;\n\n fn next(&mut self) -> Option {\n if self.total_num_values_read >= self.total_num_values {\n return None;\n }\n\n if !self.first_page_loaded {\n self.load_page();\n self.first_page_loaded = true;\n }\n\n self.inner.reset_batch();\n let total = ::std::cmp::min(\n self.batch_size,\n self.total_num_values - self.total_num_values_read,\n );\n\n let mut left = total;\n while left > 0 {\n let (num_read, _) = self.inner.read_batch(left, 0);\n if num_read < left {\n self.load_page();\n }\n left -= num_read;\n }\n self.total_num_values_read += total;\n\n Some(self.inner.current_batch().unwrap())\n }\n}\n" }, { "path": "native/core/benches/perf.rs", "content": "// Licensed to the Apache Software Foundation (ASF) under one\n// or more contributor license agreements. See the NOTICE file\n// distributed with this work for additional information\n// regarding copyright ownership. The ASF licenses this file\n// to you under the Apache License, Version 2.0 (the\n// \"License\"); you may not use this file except in compliance\n// with the License. You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing,\n// software distributed under the License is distributed on an\n// \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n// KIND, either express or implied. See the License for the\n// specific language governing permissions and limitations\n// under the License.\n\nuse std::{fs::File, os::raw::c_int, path::Path};\n\nuse criterion::profiler::Profiler;\nuse pprof::ProfilerGuard;\n\n/// A custom profiler for criterion which generates flamegraph.\n///\n/// Mostly followed this blog post: https://www.jibbow.com/posts/criterion-flamegraphs/\n/// After `cargo bench --bench -- --profile-time=