Spark Operator Support

This page is the complete reference for how Apache Comet handles each Spark physical operator. Comet replaces supported operators with native equivalents. Comet runs whole subtrees of native operators together, so if a query stage contains an operator Comet does not support, that stage falls back to regular Spark execution. Results are unaffected.

Operators marked ✅ Supported are enabled by default. Each can be turned off individually with spark.comet.exec.OPERATOR.enabled=false (for example spark.comet.exec.sort.enabled=false), and all native execution can be turned off with spark.comet.exec.enabled=false. See the Comet Configuration Guide for the full list.

Status legend

Status	Meaning
✅ Supported	Native implementation, enabled by default; works in the common case. Some inputs or forms may fall back to Spark.
⚠️ Supported (caveats)	Experimental or disabled by default, or accelerates only a limited subset. See the Compatibility Guide.
🔜 Planned	Intended; tracked by an open issue or pull request.

Not currently planned

The following operator families fall back to Spark and are not on the current roadmap. They are omitted from the tables below and may be reconsidered based on demand:

Structured Streaming operators (StateStoreSaveExec, StateStoreRestoreExec, StreamingSymmetricHashJoinExec, and similar): Comet targets batch execution.
Cartesian / cross joins (CartesianProductExec): rare and expensive, with little acceleration benefit.
Range generation (RangeExec): niche leaf operator.
Pickled (non-Arrow) Python UDFs (BatchEvalPythonExec): Comet accelerates Arrow-based Python UDFs only (#4234).

Scans

Operator	Status	Notes
`FileSourceScanExec`	✅	Parquet only. Some types and configurations fall back. See Parquet Scan Compatibility.
`BatchScanExec`	✅	Parquet, Apache Iceberg Parquet, and CSV (native) scans. See Parquet Scan Compatibility and the Iceberg Guide.
`LocalTableScanExec`	⚠️	Disabled by default; there is no acceleration advantage and this operator is typically only used in test code. Can be opted into via config (#4393).
`InMemoryTableScanExec`	🔜	Cached / in-memory table scans fall back today.

Projection and filtering

Operator	Status	Notes
`ProjectExec`	✅
`FilterExec`	✅
`SampleExec`	⚠️	Sampling without replacement only. See Operator Compatibility.

Sorting and limiting

Operator	Status	Notes
`SortExec`	✅
`GlobalLimitExec`	✅
`LocalLimitExec`	✅
`CollectLimitExec`	✅
`TakeOrderedAndProjectExec`	✅

Aggregation

Operator	Status	Notes
`HashAggregateExec`	✅
`ObjectHashAggregateExec`	✅	Supports a limited set of aggregates, such as `bloom_filter_agg`.
`SortAggregateExec`	🔜	Falls back today; Comet currently accelerates hash aggregates.

Joins

Operator	Status	Notes
`BroadcastHashJoinExec`	✅
`ShuffledHashJoinExec`	✅
`SortMergeJoinExec`	✅
`BroadcastNestedLoopJoinExec`	✅	Falls back to Spark when the preserved side is broadcast (for example LEFT OUTER with BROADCAST on the left) (#4429).

Exchanges

Operator	Status	Notes
`ShuffleExchangeExec`	✅
`BroadcastExchangeExec`	✅

Window

Operator	Status	Notes
`WindowExec`	⚠️	Runs natively and is enabled by default. A broad set of window functions is accelerated; unsupported shapes fall back to Spark. See window function compatibility.
`WindowGroupLimitExec`	🔜	Window-based limit pushdown falls back today (#4837).

Generators and set operations

Operator	Status	Notes
`GenerateExec`	✅	Supports `explode`, `explode_outer`, `posexplode`, `posexplode_outer` over arrays. `inline` / `stack` fall back.
`ExpandExec`	✅
`UnionExec`	✅
`CoalesceExec`	✅

Writes

Operator	Status	Notes
`DataWritingCommandExec`	⚠️	Experimental native Parquet writes, disabled by default (opt-in).

Python and UDF

Operator	Status	Notes
`ArrowEvalPythonExec`, `MapInArrowExec`, `MapInPandasExec`, `FlatMapGroupsInPandasExec`	🔜	Experimental accelerated PyArrow UDF support is in progress (#4234).