[AURON #2431] Prune Iceberg changelog tasks by metadata predicates (#2456) # Which issue does this PR close? Closes #2431 # Rationale for this change Iceberg changelog metadata values are constant per changelog task. Auron currently plans every supported `AddedRowsScanTask` in a changelog range even when a simple predicate on `_change_type`, `_commit_snapshot_id`, or `_change_ordinal` proves that a task cannot match. # What changes are included in this PR? - Associate supported metadata-only filters with a direct Iceberg changelog scan, allowing only deterministic projections between the filter and scan. - Match filter attributes to the scan by expression ID before tagging it. - Prepare extension plans in the existing pre-columnar transition pass. - Prune `AddedRowsScanTask` instances for `=` and `IN` predicates combined with `AND`. - Preserve the changelog filter tag when Spark rebuilds a `BatchScanExec` for runtime filters. - Keep `OR`, `NOT`, mixed data-column predicates, unsafe operator shapes, and non-`AddedRowsScanTask` tasks on the existing path. # Are there any user-facing changes? Yes. Eligible native Iceberg changelog scans plan and read fewer insert tasks. There are no public API or configuration changes. # How was this patch tested? - `AuronIcebergIntegrationSuite`: 45 tests passed on Spark 3.5 / Iceberg 1.10.1 with the native engine. - `./dev/reformat --check`: passed both Spark 3.5 formatting configurations and compile/test-compile across Spark 3.0-3.4 and Spark 4.0-4.1. - Regressions cover supported and unsupported predicates, aggregate/limit/nondeterministic barriers, expression-ID mismatches, pre-columnar staging, runtime-filter copies, dynamic partition pruning, and retention of non-insert tasks. # Was this patch authored or co-authored using generative AI tooling? - [x] Yes - [ ] No Generated-by: OpenAI Codex (GPT-5.6) Co-authored-by: Shilun Fan <slfan1989@apache.org> Signed-off-by: Shilun Fan <slfan1989@apache.org>
The Auron accelerator for big data engines (e.g., Spark, Flink) leverages native vectorized execution to accelerate query processing. It combines the power of the Apache DataFusion library and the scale of the distributed computing framework.
Auron takes a fully optimized physical plan from a distributed computing framework, mapping it into DataFusion's execution plan, and performs native plan computation.
The key capabilities of Auron include:
Based on the inherent well-defined extensibility of DataFusion, Auron can be easily extended to support:
We encourage you to extend DataFusion capability directly and add support in Auron with simple modifications in plan-serde and extension translation.
To build Auron from source, follow the steps below:
Auron's native execution lib is written in Rust. You need to install Rust (nightly) before compiling.
We recommend using rustup for installation.
Auron is regularly tested on JDK 8, 11, and 17, and is also tested on JDK 21.
Make sure JAVA_HOME is properly set and points to your desired version.
Check out the source code.
Build the project.
You can build Auron either locally or inside Docker using one of the supported OS images via the unified script: auron-build.sh.
Run ./auron-build.sh --help to see all available options.
After the build completes, a fat JAR with all dependencies will be generated in either the target/ directory (for local builds) or target-docker/ directory (for Docker builds), depending on the selected build mode.
This section describes how to submit and configure a Spark Job with Auron support.
Move the Auron JAR to the Spark client classpath (normally spark-xx.xx.xx/jars/).
Add the following configs to spark configuration in spark-xx.xx.xx/conf/spark-default.conf:
spark.auron.enable true spark.sql.extensions org.apache.spark.sql.auron.AuronSparkSessionExtension spark.shuffle.manager org.apache.spark.sql.execution.auron.shuffle.AuronShuffleManager spark.memory.offHeap.enabled false # suggested executor memory configuration spark.executor.memory 4g spark.executor.memoryOverhead 4096
spark-sql -f tpcds/q01.sql
TPC-DS 1TB Benchmark Results:
For methodology and additional results, please refer to benchmark documentation.
We also encourage you to benchmark Auron and share the results with us. 🤗
Mail List is the most recognized form of communication in the Apache community. Contact us through the following mailing list.
| Name | Scope | ||
|---|---|---|---|
| dev@auron.apache.org | Development-related discussions | Subscribe | Unsubscribe |
Interested in contributing to Auron? Please read our Contributing Guide for detailed information on how to get started.
Auron is licensed under the Apache 2.0 License. A copy of the license can be found here.