Comet configuration keys live under spark.comet.* and are declared in spark/src/main/scala/org/apache/comet/CometConf.scala. This page describes the naming conventions those keys follow so that new configs stay consistent with the surface users already know, and documents the process for renaming an existing key without breaking deployments.
A Comet config key is a dotted path of segments. Each segment describes a scope, from broadest (the prefix) to narrowest (the specific setting).
spark.comet. — never bare comet. and never a nested prefix like spark.sql.comet..exec (execution and expressions), scan (source readers), parquet (Parquet-specific settings), shuffle (shuffle behavior), explain (plan explain/logging), metrics, tracing, debug, testing, convert (Spark → Comet source conversion), or expression / operator (per-expression / per-operator flags).camelCase, not dot- or kebab-separated. Prefer spark.comet.foo.maxThreadNum over spark.comet.foo.max.thread.num or spark.comet.foo.max-thread-num..enabled — for example spark.comet.debug.enabled, spark.comet.metrics.enabled, spark.comet.parquet.rowFilterPushdown.enabled. Action-form flags whose name is itself a verb (force..., allow...) can omit .enabled because the verb already encodes the action — for example spark.comet.exec.forceShuffledHashJoin.UDF, SHJ, SMJ, IO — either all-caps or camelCase, but do not mix within one cluster (pyarrowUdf alongside scalaUDF would be a red flag; both spell it UDF).Every config declares a Scala val in CometConf.scala. The symbol name is the UPPER_SNAKE_CASE form of the key's meaningful segments, prefixed with COMET_.
Examples:
| Key | Symbol |
|---|---|
spark.comet.enabled | COMET_ENABLED |
spark.comet.exec.forceShuffledHashJoin | COMET_FORCE_SHJ |
spark.comet.parquet.rowFilterPushdown.enabled | COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED |
The symbol name is what appears in code; the key is what appears in user configuration. The two do not have to match segment-for-segment — brevity in the symbol is fine as long as the key remains descriptive.
Every ConfigEntry must call .category(...). The category is used to route the key into the right table in the user guide's configs.md. Available categories are declared as CATEGORY_* constants at the top of CometConf.scala. If a new config does not fit an existing category, discuss adding a new one before landing the config.
Configs under spark.comet.* are stable across minor releases: users may have set them in production spark-defaults.conf files, Spark job submissions, or notebooks. Renaming a key must not silently break those deployments.
Use the withAlternative builder on ConfigBuilder to keep the old key working as a deprecated alias:
val COMET_FORCE_SHJ: ConfigEntry[Boolean] = conf(s"$COMET_EXEC_CONFIG_PREFIX.forceShuffledHashJoin") .withAlternative(s"$COMET_EXEC_CONFIG_PREFIX.replaceSortMergeJoin") .category(CATEGORY_EXEC) .doc("...") .booleanConf .createWithDefault(false)
Reading a value from the alternative logs a one-time deprecation warning per JVM per alternative key, pointing users at the current key. The primary key always wins if both are set.
withAlternative accepts multiple alternatives (checked in order), for keys that have been renamed more than once.
The rename checklist for a single config:
conf(...) call and add .withAlternative(oldKey).val (for example COMET_REPLACE_SMJ → COMET_FORCE_SHJ).docs/source/ for the old key and update to the new key. The auto-generated configs.md table will refresh on the next release-docs regeneration and does not need manual editing.CometConfSuite if the rename covers a new type of alias pattern (single-hop, multiple alternatives, etc.).Removing a deprecated alias is a follow-up step that belongs to a later major release — typically the next Comet major after the rename first ships. See the versioning policy for the timing rules.
A rename keeps behavior identical and only moves the key. A behavior change is different: the same query, over the same data, with the same explicitly set configuration, starts producing a different result or a different error. Changing a config's default value, or changing what its existing values mean, is a behavior change.
Comet's versioning policy allows a behavior change to ship in a minor release, but only when users have a documented way to opt back out. That escape hatch is a boolean config under spark.comet.legacy.*, defaulting to false, whose only job is to restore the previous behavior:
val COMET_LEGACY_FOO_BEHAVIOR: ConfigEntry[Boolean] = conf("spark.comet.legacy.fooBehavior") .category(CATEGORY_LEGACY) .doc("When true, restores the pre-1.1.0 behavior of <config>, which <describe old " + "behavior>. This config is deprecated and will be removed in a future major release.") .booleanConf .createWithDefault(false)
Naming follows the usual conventions: spark.comet.legacy. prefix, camelCase final segment, and a COMET_LEGACY_* Scala symbol. No legacy config exists yet, so the first one to land must also add the CATEGORY_LEGACY constant to CometConf.scala and a corresponding section to configs.md. The doc string must state which release changed the behavior and what the old behavior was, so that configs.md is self-explanatory without cross-referencing the upgrade guide.
The checklist for a behavior change:
CometConf.scala under CATEGORY_LEGACY, defaulting to false.Two cases do not need a legacy config:
Compatible expression or operator does not match Spark, that is a bug. Fixing it is a bug fix and may ship in any release, including a patch release. Add a legacy config only if the fix has an unusually wide blast radius, and say so in the PR description.Removing a legacy config is a major-release change, handled the same way as removing a deprecated alias.