This guide lists the behavior changes in each Comet release and the configuration setting that restores the previous behavior for each one. Read the section for every version between the release you are upgrading from and the release you are upgrading to.
A behavior change is one where the same query, run over the same data, with the same explicitly set configuration, produces a different result or a different error than it did in the previous release. Comet's versioning policy permits these in a minor release only when a spark.comet.legacy.* configuration key restores the previous behavior, so every entry below names such a key.
Two kinds of change are deliberately absent from this guide:
Compatible, that is a bug, and fixing it is a bug fix rather than a behavior change. These fixes appear in the release notes, not here. For a fix with an unusually wide blast radius the maintainers may still provide a spark.comet.legacy.* key, in which case it will be listed below.Additions, new configuration keys, and Apache Spark version support changes are recorded in the release notes and on the Spark Version Compatibility page rather than here.
Every key under spark.comet.legacy.* is deprecated from the moment it is added. Each one exists to give you time to adapt to a behavior change, and may be removed in any future major release, at which point the newer behavior becomes unconditional.
Treat setting one of these keys as a temporary measure. If you find you cannot stop relying on a legacy behavior, please open an issue describing your use case so it can be considered before the key is removed.
Comet 1.0.0 is the first release under the stable versioning policy. From this release onward, behavior changes are documented on this page along with the configuration key that reverts each one.
Changes made during the 0.x series are not recorded here. If you are upgrading from a 0.x release, review the release notes for the versions in between.