ci: write large actions/cache entries only on push to main An actions/cache entry is scoped to the ref that wrote it: a run restores from its own ref and from the default branch, and nothing else. So an entry written from refs/pull/*/merge is visible only to another run of the same pull request, and one written from the merge queue's gh-readonly-queue/* branch is visible to nobody, because the queue deletes that branch when it is finished with it. Both still consume the repository's shared cache budget, which is evicted least-recently-used. On 2026-09-15 the repository held 12.27 GiB across 14 entries: 9.22 GiB on one gh-readonly-queue/* branch, 3.01 GiB on refs/pull/*/merge, and nothing at all on main. Seven near-identical Linux-java-maven-* Maven repositories accounted for 9.06 GiB of it, every one of them a write-only copy. With main's cargo-ci entry evicted, all eight native builds in merge_group run 34977717057 missed their cache and paid a cold compile of 20 to 28 minutes each, 259 runner-minutes in a single run. The same build on the push run two hours later restored in 56s and compiled in 2m21s. The two cargo saves in pr_build_linux.yml carried `if: github.ref == 'refs/heads/main'` when this was written. #5963 has since replaced that guard with `if: github.event_name == 'push'`, which also excludes the scheduled run at the same sha, so both conditions are now spelled out together and check_cache_save_scope can see the ref half. Hold every other large cache to the same rule: a cache of ~/.m2/repository or of a cargo tree is restored everywhere and saved only on push to main. The bare actions/cache@vN form cannot express that, since it saves in an implicit post step no `if:` can reach, so those sites split into actions/cache/restore plus a guarded actions/cache/save. The TPC-H and TPC-DS datasets keep the read-write form: they are a few hundred MB, they are keyed on the workflow file, and a pull request editing that file would otherwise regenerate them on every run. publish_snapshot.yml keeps it too, since it runs from main on a schedule already. check_cache_save_scope in dev/ci/check-ci-config.py pins both halves, and rejects a new bare actions/cache@vN on any of those paths. Six mutations were used to confirm it fails: reverting a site to the read-write form, dropping the github.ref guard from each of four saves, and adding a fresh unguarded cargo cache. pyarrow_udf_test.yml now lists itself in its paths filter. It sits outside the ci.yml umbrella, so compute-changes.py does not route it, and an edit to the file merged without the workflow ever running -- including the cache step changed here.
Apache DataFusion Comet is a high-performance accelerator for Apache Spark. Comet keeps Spark queries Arrow-native end-to-end: operators, expressions, shuffle, and broadcast all stay in Apache Arrow columnar format, avoiding the per-row overhead of Spark's row-based engine. Within the Arrow-native pipeline, operators and expressions execute as Rust code (via the Apache DataFusion query engine) or as JVM code that operates directly on Arrow batches. Comet integrates with the Spark ecosystem without requiring any code changes.
Comet provides a ~2x speedup for TPC-DS @ SF 1000 (1TB), resulting in ~50% cost savings.
That 2x speedup gives you a choice: finish the same Spark workload in half the time on the cluster you already have, or match your current Spark performance on roughly half the resources. Either way, the gain translates directly into lower cloud bills, reduced on-prem capacity, and lower energy usage, with no changes to your existing Spark SQL, DataFrame, or PySpark code. Comet runs on commodity hardware: no GPUs, FPGAs, or other specialized accelerators are required, so the savings come from better utilization of the infrastructure you already run on.
See the Comet Benchmarking Guide for more details.
Comet accelerates Spark workloads by replacing Spark operators and expressions with high-performance implementations that process Apache Arrow columnar data directly. Most operators are powered by native Rust execution built on Apache DataFusion, while others run efficiently in the JVM on Arrow batches. This unified columnar execution model keeps processing within the Comet engine end-to-end, reducing overhead and delivering faster, more efficient query execution without reverting to Spark's traditional row-based engine.
FILTER (WHERE ...) clausesFor the authoritative lists, see the supported expressions and supported operators pages.
Comet is designed as a drop-in accelerator for Apache Spark, allowing you to integrate Comet into your existing Spark deployments and workflows seamlessly. With no code changes required, you can immediately harness the benefits of Comet's acceleration capabilities without disrupting your Spark applications.
Comet supports Apache Spark 3.4, 3.5, 4.0, and 4.1, and provides experimental support for Spark 4.2. See the installation guide for the detailed version, Java, and Scala compatibility matrix.
Install Comet by adding the jar for your Spark and Scala version to the Spark classpath and enabling the plugin. A typical configuration looks like:
export COMET_JAR=/path/to/comet-spark-spark3.5_2.12-<version>.jar $SPARK_HOME/bin/spark-shell \ --jars $COMET_JAR \ --conf spark.driver.extraClassPath=$COMET_JAR \ --conf spark.executor.extraClassPath=$COMET_JAR \ --conf spark.plugins=org.apache.spark.CometPlugin \ --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \ --conf spark.comet.explain.fallback.enabled=true \ --conf spark.memory.offHeap.enabled=true \ --conf spark.memory.offHeap.size=4g
For full installation instructions, published jar downloads, and configuration reference, see the installation guide and the configuration reference.
Join the DataFusion Slack and Discord channels to connect with other users, ask questions, and share your experiences with Comet.
We welcome contributions from the community to help improve and enhance Apache DataFusion Comet. Whether it's fixing bugs, adding new features, writing documentation, or optimizing performance, your contributions are invaluable in shaping the future of Comet. Check out our contributor guide to get started.
Apache DataFusion Comet is licensed under the Apache License 2.0. See the LICENSE.txt file for details.