Comet is an open-source project and contributors are welcome to work on any issues at any time, but we find it helpful to have a roadmap for some of the major items that require coordination between contributors.
Native window execution runs by default (spark.comet.exec.window.enabled). The ranking functions (rank, dense_rank, row_number, percent_rank, cume_dist, ntile), value functions (lag, lead, nth_value, first_value, last_value), and the count, min, max, sum, and avg aggregates are accelerated. Remaining work is to close the gaps that still fall back to Spark: statistical aggregates (stddev, variance, corr, covar) and collect_list / collect_set as window functions (#4766), GROUPS frames (#4836), RANGE frames with explicit date or decimal offsets (#4834), first_value / last_value on RANGE frames with a literal offset (#4835), non-literal lag / lead default values (#4268), and WindowGroupLimitExec (#4837). See the window function compatibility guide for the complete list of supported functions, frames, and fallback cases.
Spark supports higher-order functions on arrays and maps that take a lambda, including transform, exists, forall, aggregate, zip_with, map_filter, and map_zip_with. Comet evaluates these today through a JVM codegen-dispatch bridge (CometScalaUDF, CometBatchKernelCodegen) instead of falling back to Spark, but the lambda body is still interpreted row-at-a-time on the JVM rather than natively in DataFusion. DataFusion added native higher-order function support (array_transform, array_filter, array_any_match, etc.) that Comet does not yet use; it‘s not yet known whether their semantics are Spark-compatible. We’ll explore whether DataFusion's implementations can replace the JVM codegen-dispatch bridge to remove that round-trip and let these expressions benefit from vectorized native execution.
Beyond lambda bodies, a number of built-in Spark scalar expressions (some regular expression, JSON, and datetime functions, for example) and aggregate functions route through the same JVM codegen-dispatch bridge by default, either because their native DataFusion or datafusion-spark implementation has known semantic differences from Spark, or because no native implementation exists yet. See the compatibility guide for the current list of codegen-dispatched expressions. We're exploring closing these gaps so that more expressions run natively by default, which would reduce JVM round-trips beyond what the lambda and UDF work above already covers.
Comet landed its first Iceberg table format V3 feature (native data file decryption, #4991), and we want to add more V3 features to Comet‘s native Iceberg scans so they don’t fall back to Spark. The work is tracked phase-by-phase in #3376: detecting the V3 format and supporting new V3 data types, deletion vector reads, row lineage, and table encryption. Native deletion vector reads are prototyped in a draft PR (#4887), but are blocked on upstream iceberg-rust support, tracked in iceberg-rust #2792 and iceberg-rust #2411. Native Iceberg scans don‘t support HDFS-backed tables today: the storage scheme match in iceberg_scan.rs only handles file, s3/s3a, gs, and oss, and would need an HDFS StorageFactory upstream in iceberg-storage-opendal. We’re scoping what that work would take.
Comet already delivers substantial speedups over vanilla Spark on both benchmark suites; we publish per-query results for TPC-H and TPC-DS with each release. An independent AWS Labs benchmark comparing Comet 0.16.0 with Gluten 1.6.0 on a 3TB TPC-DS workload found that the two accelerators deliver similar overall performance. Increasing the speedup further and closing the remaining per-query gaps is an ongoing focus, tracked under #2004 (TPC-H) and #858 (TPC-DS).
A growing number of Spark-compatible expressions live in the datafusion-spark crate in the core DataFusion repository. Comet is migrating its expression implementations to that crate so that they can be shared by other DataFusion-based projects, and is wiring up the functions that crate already provides (#4150). Improvements to core DataFusion operators (joins, aggregates, window) made in support of Comet also benefit the wider ecosystem.
Comet's native hash join currently requires the build side to fit entirely in memory. Adding spill-to-disk support will allow Comet to handle larger joins without falling back to Spark, improving both reliability and performance for memory-intensive workloads.
Spark users frequently define custom UDFs in Java or Scala. Comet now dispatches scalar ScalaUDF expressions through a JVM codegen bridge (CometScalaUDF) instead of always falling back to Spark. Aggregate UDFs, table UDFs/generators, Python/Pandas UDFs, and Hive GenericUDF/SimpleUDF still fall back to Spark entirely. Extending the codegen-dispatch approach to cover these remaining categories will reduce fallbacks further and allow more queries to run end-to-end in Comet.
Comet coordinates memory between the JVM and native Rust execution through a custom memory pool. Improving memory accounting, reservation strategies, and spill integration will reduce out-of-memory errors and allow Comet to make better use of available resources, especially in multi-query and multi-task environments.
Comet has experimental support for native Parquet writes via InsertIntoHadoopFsRelationCommand, currently disabled by default. The goal is to reach correctness and performance parity with Spark's writer so it can be enabled by default (#1625).
Comet‘s native scans accelerate Iceberg reads today, but writes still run entirely through Spark’s Iceberg V2 write operator. #4322 tracks exploring native acceleration of Iceberg writes, so an ETL job could run end to end in native code, falling back only where Comet can‘t match Iceberg Java’s functionality. Draft proposals split the existing V2 write operator into separate “writer” and “committer” operators so the data-file-write portion can be planned and accelerated the same way as a native Parquet write (#4658, #4487). No design here is committed yet; we're exploring whether this approach is worth pursuing.
Comet currently supports Spark‘s built-in file formats and Iceberg, but not Delta Lake (#174). A draft PR reuses Comet’s existing native Parquet reader to accelerate scans of plain Delta tables that use neither deletion vectors nor column mapping (#4669); a separate, larger effort explores a full native Delta scan built on delta-kernel-rs as an out-of-tree contrib module (#4366), landing in inert, gated slices (#4700, #4952). Generalizing Comet‘s scan-side APIs so that Delta and other non-Iceberg data sources can plug in more easily is tracked as a Table Provider API abstraction (#4706). None of this is committed yet; we’re exploring which approach, if any, is worth pursuing.