refactor: simplify task-shared pool guard after review Cleanups from a quality pass over the previous commit, no behaviour change except where noted: - Use `parking_lot::Mutex` for TASK_SHARED_MEMORY_POOLS instead of hand-rolled `PoisonError` recovery over `std::sync::Mutex`. parking_lot never poisons, is already a direct dependency, and is what the sibling THREAD_MEMORY_POOLS registry and fair_pool.rs in this same module already use. Removes the `lock_pools` helper and its rationale comment. - Delete `MemoryPoolType::is_task_shared` and the `debug_assert` that was its only remaining caller. Task-sharedness is now encoded solely by which `create_memory_pool` arms return a guard, so the predicate was a second source of truth reconciled only by an assertion that compiles out in release builds. This also drops `TaskSharedPoolRef::pool_type`, which existed only to name the pool type in one warning, and with it the `Debug` derive on MemoryPoolType. - Collapse the four repeated destructure-and-rewrap task-shared arms of `create_memory_pool` behind a local `task_shared` helper paired with the existing `tracked` helper, so `Some(pool_ref)` appears once. - Use the `Entry` API in `TaskSharedPoolRef::drop` so the release path hashes the key once rather than for both `get_mut` and `remove`. - Drop the raw-pointer round trip through `get_execution_context`'s `&mut` in `releasePlan`; cast straight to `*mut ExecutionContext` for `Box::from_raw`. - State the concrete consequence of the `_task_shared_pool_ref` field ordering (a later plan in the same task would get a second full-budget pool) rather than just asserting that the ordering matters. Behaviour change: `CometExecIterator.close` now releases the native plan even when an earlier teardown step throws. Setting `closed` first removed the double-release hazard but made any throw from `currentBatch.close()`, `nativeUtil.close()` or a shuffle block iterator skip `releasePlan` entirely, which strands the very context this change set out to free. The teardown failure is still propagated, with a release failure attached as a suppressed exception.
Apache DataFusion Comet is a high-performance accelerator for Apache Spark. Comet keeps Spark queries Arrow-native end-to-end: operators, expressions, shuffle, and broadcast all stay in Apache Arrow columnar format, avoiding the per-row overhead of Spark's row-based engine. Within the Arrow-native pipeline, operators and expressions execute as Rust code (via the Apache DataFusion query engine) or as JVM code that operates directly on Arrow batches. Comet integrates with the Spark ecosystem without requiring any code changes.
Comet provides a ~2x speedup for TPC-DS @ SF 1000 (1TB), resulting in ~50% cost savings.
That 2x speedup gives you a choice: finish the same Spark workload in half the time on the cluster you already have, or match your current Spark performance on roughly half the resources. Either way, the gain translates directly into lower cloud bills, reduced on-prem capacity, and lower energy usage, with no changes to your existing Spark SQL, DataFrame, or PySpark code. Comet runs on commodity hardware: no GPUs, FPGAs, or other specialized accelerators are required, so the savings come from better utilization of the infrastructure you already run on.
See the Comet Benchmarking Guide for more details.
Comet accelerates Spark workloads by replacing Spark operators and expressions with high-performance implementations that process Apache Arrow columnar data directly. Most operators are powered by native Rust execution built on Apache DataFusion, while others run efficiently in the JVM on Arrow batches. This unified columnar execution model keeps processing within the Comet engine end-to-end, reducing overhead and delivering faster, more efficient query execution without reverting to Spark's traditional row-based engine.
FILTER (WHERE ...) clausesFor the authoritative lists, see the supported expressions and supported operators pages.
Comet is designed as a drop-in accelerator for Apache Spark, allowing you to integrate Comet into your existing Spark deployments and workflows seamlessly. With no code changes required, you can immediately harness the benefits of Comet's acceleration capabilities without disrupting your Spark applications.
Comet supports Apache Spark 3.4, 3.5, 4.0, and 4.1, and provides experimental support for Spark 4.2. See the installation guide for the detailed version, Java, and Scala compatibility matrix.
Install Comet by adding the jar for your Spark and Scala version to the Spark classpath and enabling the plugin. A typical configuration looks like:
export COMET_JAR=/path/to/comet-spark-spark3.5_2.12-<version>.jar $SPARK_HOME/bin/spark-shell \ --jars $COMET_JAR \ --conf spark.driver.extraClassPath=$COMET_JAR \ --conf spark.executor.extraClassPath=$COMET_JAR \ --conf spark.plugins=org.apache.spark.CometPlugin \ --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \ --conf spark.comet.explain.fallback.enabled=true \ --conf spark.memory.offHeap.enabled=true \ --conf spark.memory.offHeap.size=4g
For full installation instructions, published jar downloads, and configuration reference, see the installation guide and the configuration reference.
Join the DataFusion Slack and Discord channels to connect with other users, ask questions, and share your experiences with Comet.
We welcome contributions from the community to help improve and enhance Apache DataFusion Comet. Whether it's fixing bugs, adding new features, writing documentation, or optimizing performance, your contributions are invaluable in shaping the future of Comet. Check out our contributor guide to get started.
Apache DataFusion Comet is licensed under the Apache License 2.0. See the LICENSE.txt file for details.