)]}' { "log": [ { "commit": "330b29cf1013351cd9d7e93d3d37a6bc0b6d956b", "tree": "fb7dd061270420fcf8e2c5dff230e59edea2832b", "parents": [ "b8432a00dd0b1642c990b51d09026af1a3a9b5d2" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Sat Oct 24 06:46:04 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Oct 24 06:46:04 2026 +0000" }, "message": "chore: start 1.3.0 development (#7167)" }, { "commit": "b8432a00dd0b1642c990b51d09026af1a3a9b5d2", "tree": "8ead57c8ab72ddecff93f97eef4ab82ffb2acbe4", "parents": [ "52ea7120b09d4803c8f9f30747ae20e257035baf" ], "author": { "name": "Liang-Chi Hsieh", "email": "viirya@gmail.com", "time": "Sat Oct 24 05:12:53 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Oct 24 05:12:53 2026 +0000" }, "message": "feat: Surface native scalar Arrow UDF errors as Spark\u0027s PythonException (#7182)\n\n* feat: Surface native scalar Arrow UDF errors as Spark\u0027s PythonException\n\nA native scalar Arrow UDF that raised reached the JVM as a\nCometNativeException carrying the Python traceback, so PySpark reported an\nUnknownException instead of pyspark.errors.PythonException.\n\nThe native UDF now reports the exception as a new\nSparkError::PythonException with the formatted traceback, which travels\nthrough the existing CometQueryExecutionException JSON channel. The 4.x\nshim builds Spark\u0027s PythonException with a per-version factory in\norg.apache.spark.sql.execution.python:\n\n- Spark 4.0/4.1 use the (msg, cause) constructor. PySpark 4.1 maps a\n PythonException cause only when one of its stack frames is in that\n package, and the factory\u0027s own frame satisfies it.\n- Spark 4.2 uses the PYTHON_EXCEPTION error condition, as Spark\u0027s runners\n do, so the scheduler fails the job with the exception itself.\n\nunwrap_arrow_external_error moves from spark-expr to the common crate so\nthe operator can reuse it.\n\nCloses #7177.\n\n* Allow CometPythonException in the Comet jar and fix a scalafix finding" }, { "commit": "52ea7120b09d4803c8f9f30747ae20e257035baf", "tree": "08fe133aba19f1b8ea0be79ac3bc45ef8911457b", "parents": [ "f8e508f0c6a2906d4c0e84832ff7aedb5e9ac5ab" ], "author": { "name": "Liang-Chi Hsieh", "email": "viirya@gmail.com", "time": "Sat Oct 24 01:03:35 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Oct 24 01:03:35 2026 +0000" }, "message": "feat: keep native Arrow UDFs on Spark\u0027s path when the embedded Python version differs (#7179)\n\nThe native path failed the task when a UDF\u0027s PySpark Python version\ndiffered from the embedded interpreter\u0027s. The embedded interpreter is\nthe libpython libcomet was linked against, so after a Python upgrade\nwithout a rebuild every native UDF query failed where Spark succeeds.\n\nExpose the embedded major.minor version through NativeBase, read with\nPy_GetVersion before the interpreter starts, and fall back to Spark in\ngetSupportLevel when a UDF\u0027s version differs. Link #6811 and #7177 from\nthe user guide\u0027s limitations paragraph.\n\nCloses #7178." }, { "commit": "f8e508f0c6a2906d4c0e84832ff7aedb5e9ac5ab", "tree": "a36a8145566385ffb83fd701a65c23416a9247d5", "parents": [ "8263260fe7d66ee9cd19bbf876a50059ab751732" ], "author": { "name": "Liang-Chi Hsieh", "email": "viirya@gmail.com", "time": "Sat Oct 24 00:59:33 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Oct 24 00:59:33 2026 +0000" }, "message": "Raise native UDF validation failures as typed exceptions (#7185)\n\n`CometNativeUDF.classifyNativeError` picked the JVM exception for a failed\n`validateLibrary` call by matching the native message text. The native\nside now raises the class itself through the error protocol:\n\n- `CometError::NativeUdfNotAllowed` -\u003e `CometNativeUdfNotAllowedException`\n for a refusal by `spark.comet.nativeUdf.enabled` / `allowedPaths`\n- `CometError::NativeUdfAbi` -\u003e `CometNativeUdfAbiException` for an ABI\n version mismatch or a missing discovery symbol\n- `CometError::NoSuchElement` -\u003e `java.util.NoSuchElementException` for a\n UDF name the library does not export\n\nLike `ShuffleSizeLimit` and `IllegalState`, the variants keep their class\nthrough DataFusion\u0027s Context/Shared/External wrappers. The JVM side\nrethrows those three and wraps every other failure in\n`CometNativeUdfLoadException` with its cause, as before. Exception classes\nand messages are unchanged.\n\nThe validation logic also moves out of the JNI export into\n`validate_native_udf_library`, which has no JNI types in its signature,\nwith unit tests that need no JVM." }, { "commit": "8263260fe7d66ee9cd19bbf876a50059ab751732", "tree": "26a37903095b86b4d96e032fad7102e6bec7b685", "parents": [ "034dceb325f8999f22a450f966c1c301251c566f" ], "author": { "name": "Liang-Chi Hsieh", "email": "viirya@gmail.com", "time": "Sat Oct 24 00:59:10 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Oct 24 00:59:10 2026 +0000" }, "message": "fix: return errors instead of panicking on malformed JNI arguments (#7186)\n\nSeveral JNI entry points converted their arguments with unwrap() or\nexpect(), so a malformed argument surfaced as a CometNativeException\ncarrying only the panic message. Return a CometError::Internal that\nnames the bad argument instead:\n\n- convert_datatype_arrays reports which column\u0027s data type failed to\n deserialize.\n- writeSortedFileNative, traceBegin, traceEnd and logMemoryUsage report\n which string argument could not be read.\n- get_execution_context rejects a null handle.\n\nThe exception class stays CometNativeException." }, { "commit": "034dceb325f8999f22a450f966c1c301251c566f", "tree": "c8b3ebc5f6fa6c039a25f28f697aac48d3601ac6", "parents": [ "62e31f9a6ff01e354e848cd721416fcb74e9a527" ], "author": { "name": "Shekhar Rajak", "email": "5774448+Shekharrajak@users.noreply.github.com", "time": "Sat Oct 24 00:05:16 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Oct 24 00:05:16 2026 +0000" }, "message": "bench: improve Iceberg TPC workflow and plan capture (#3783)\n\n* bench: improve iceberg tpc benchmarking\n\n* bench: drop empty metrics export\n\n* bench: document Iceberg plan capture" }, { "commit": "62e31f9a6ff01e354e848cd721416fcb74e9a527", "tree": "414869df7c99f2d28124f03fd15b33825e19c7c9", "parents": [ "2b1c6a3938cbe1c4b2d0aa6383e64360569f877a" ], "author": { "name": "Chao Sun", "email": "sunchao@apache.org", "time": "Sat Oct 24 00:03:36 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Oct 24 00:03:36 2026 +0000" }, "message": "fix: cap native projection input before collect limits (#6035)\n\n* fix: defer throwing literal casts to Spark at runtime\n\n* test: cover deliberate literal cast fallback and native controls\n\n* test: exercise cast fallback with shared Spark error checks\n\n* fix: preserve literal cast fallback after upstream dispatch changes\n\n* test: distinguish literal cast LIMIT regression from branch masking\n\n* fix: cap native projection input before collect limits\n\n* fix: retain filter pruning for column-only limited projections\n\n* docs: track remaining LIMIT evaluation gaps" }, { "commit": "2b1c6a3938cbe1c4b2d0aa6383e64360569f877a", "tree": "b4407ef7567e0ba20ac0e898776fe847b873f83c", "parents": [ "69111f68a5ff62904342910c445d191f53615aff" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Fri Oct 23 22:34:11 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 23 22:34:11 2026 +0000" }, "message": "fix: keep Text build sides on Spark by default and add Text conversion configs (#7123)\n\n* fix: default broadcast build side conversion to off and document invalid UTF-8 cases\n\nConverting a Text scan on a broadcast build side to Arrow decodes invalid\nUTF-8 to U+FFFD, so 1.2.0 could return different results from 1.1.0. Default\nspark.comet.convert.broadcastBuildSide.enabled to false and name the\nText-source and CAST(binary AS STRING) cases in the compatibility guide.\n\n* test: move the #6008 build-side tests into CometBroadcastBuildSideSuite\n\nCometBroadcastBuildSideSuite turns the build-side bridge on for every test, so the tests no\nlonger need the bridgeTest helper, and a test added later cannot miss the setting. The two tests\nthat failed because they were not wrapped in bridgeTest now run with the bridge on, and the five\nthat asserted nothing was bridged now exercise the bridge again. The test bodies are unchanged.\n\n* fix: keep Text build sides on Spark by default and add Text conversion configs\n\nConverting a Text scan to Arrow replaces bytes that are not valid UTF-8 with U+FFFD, so\nbridging a Text build side by default (#6786) could return different results from 1.1.0\n(#7118). The other formats that reach the bridge either define strings as UTF-8 (ORC, Avro,\nIceberg) or are decoded by Spark, so spark.comet.convert.broadcastBuildSide.enabled goes back\nto on by default, and Text gets its own configs:\n\n- spark.comet.convert.broadcastBuildSide.text.enabled (default false) bridges a Text build\n side. Without it, the scan records a fallback reason that names the config.\n- spark.comet.convert.text.enabled (default false) converts every Text scan, like the CSV,\n JSON and Parquet configs. The sparkToColumnar operator list still converts Text as before.\n\nThe compatibility guide now names Text rather than CSV and JSON, no longer says that Spark\nrequires Parquet strings to be valid UTF-8, adds Comet columnar shuffle as a path, and gives\nthe configs that keep min/max over CAST(binary AS STRING) in Spark.\n\n* test: drop the QueryStageExec import that the test move left unused\n\n* fix: convert a Text build side only with spark.comet.convert.text.enabled\n\nDrop spark.comet.convert.broadcastBuildSide.text.enabled. A Text scan, on a\nbuild side or not, is now converted only by spark.comet.convert.text.enabled,\nlike CSV, JSON and Parquet, and the build-side fallback reason names that\nconfig. Text goes through isConversionEnabled, so naming FileSourceScan or\nBatchScan in spark.comet.sparkToColumnar.supportedOperatorList still converts\nit but is deprecated.\n\nThe #6008 build-side tests read ORC instead of Text, since a global Text switch\nwould convert their probes and no-bridge assertions too.\n\n* docs: describe Text conversion through its own config only\n\nUpdate the Text, build-side and compatibility sections for the removal of\nspark.comet.convert.broadcastBuildSide.text.enabled, and note in the migration\nguide that converting Text through sparkToColumnar is deprecated.\n\n* fix: drop the unused SparkConf import and narrow the min/max workaround\n\nThe suite no longer overrides sparkConf, so its SparkConf import failed\nScalafix and the strict-warnings build. Reformat a CometExecRule comment\nthat Spotless rejected.\n\nThe three-config min/max workaround only keeps the decode out of the\nquery when the cast is the aggregate\u0027s own argument. A cast in a\nprojection below the aggregate runs natively and decodes first, so the\nguide now says so and keeps disabling Comet as the general answer." }, { "commit": "69111f68a5ff62904342910c445d191f53615aff", "tree": "5ed1dc6d7abfb63adea2e4902eb4d931c04f7722", "parents": [ "785fdbcb4cbf9057dfcfae3bd9dd45a77bb505bb" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Fri Oct 23 17:53:02 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 23 17:53:02 2026 +0000" }, "message": "fix: make fair_unified\u0027s per-consumer limit bind when tasks run concurrently (#7096)\n\n* fix: make fair_unified\u0027s per-consumer limit bind when tasks run concurrently\n\nCometFairMemoryPool limited each consumer to pool_size / registered\nconsumers, where pool_size is the executor\u0027s whole off-heap pool. Spark\nlimits each task to an even share of that pool among the running tasks,\nso with more than one task running the limit never bound. The first\nconsumer to reserve could take its task\u0027s whole share, and a consumer\nthat cannot spill, such as a sort\u0027s merge, was then refused by Spark\nand failed the task.\n\nDo what DataFusion\u0027s FairSpillPool does, over the task\u0027s share of the\noff-heap pool: limit each consumer that can spill to an even share of\nwhat the consumers that cannot spill leave of it, and let the consumers\nthat cannot spill take what Spark grants. The task\u0027s share is\nspark.memory.offHeap.size divided by the tasks running native plans,\nwhich a new RunningTasks count tracks for as long as each task holds its\ntask-shared pool, and at most memory_limit, so that the deprecated\nspark.comet.exec.memoryPool.fraction keeps limiting each task. Pass\nspark.memory.offHeap.size to createPlan for it.\n\nUpdate the memory guides, the fraction\u0027s config doc, the 1.2.0 upgrade\nnotes and the memory review skill, and drop the reason the Iceberg and\nDelta comments gave for one consumer per task, which no longer holds.\n\nCloses #7088.\nCloses #5465.\n\n* fix: hold the Iceberg fanout writer to a share and document the fair pool\u0027s limits\n\nThe fanout writer answers a refusal by closing partitions, so it now registers as a consumer\nthat can spill. Under fair_unified it closes files rather than grow until Spark refuses and\ntake the memory that a sort in its task could use. The unpartitioned and clustered writers\nstay unspillable.\n\nDocument the formula for the limits, which way each difference between Comet\u0027s task count and\nSpark\u0027s divisor pushes, the off-heap storage region that Spark subtracts, and that a sort can\nfail where a single batch exceeds its share or the consumers that cannot spill hold the task\u0027s\nshare. Say why the Delta deletion vector attach registers one consumer per call.\n\nTest the executor-wide task count, the task total against the task\u0027s share rather than the\npool, a consumer that can spill when the others hold the share, and the count after the\nregistry churn test.\n\n* fix: limit a fully ordered final aggregate as a consumer that cannot spill\n\nDataFusion 55.2 registers OrderedFinalAggregateStream as able to spill to\nwork around apache/datafusion#17334, but builds a spill path only for\npartially sorted input, so over fully sorted input a refusal from the\nfair pool\u0027s share fails the query. Limit it as a consumer that cannot\nspill, so it takes what Spark grants. Drop a test that duplicates one in\nmod.rs.\n\n* fix: hold a sort below an ordered final aggregate to its fair share\n\nApply the OrderedFinalAggregateStream exception only where try_grow\npicks its check, so the aggregate still counts in num_spill and a sort\nbelow it leaves it a share. Share the consumer name prefix with\nspill_replay.rs, define the spillable test helper once, and fold the\nmixed pool type test into the executor count test." }, { "commit": "785fdbcb4cbf9057dfcfae3bd9dd45a77bb505bb", "tree": "d2ea68ca290e828f9634d193bdcce1dbde1f8385", "parents": [ "6a6d71c52f3452ac92ebe1c30c92c8515508f59e" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Fri Oct 23 13:32:41 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 23 13:32:41 2026 +0000" }, "message": "chore: refresh the native dependencies with cargo update (#7169)" }, { "commit": "6a6d71c52f3452ac92ebe1c30c92c8515508f59e", "tree": "4d2d443fc4119a08e8efeae479cbe866f4fa702e", "parents": [ "c56f6a0209dba0600103a37fac7ca9b006639857" ], "author": { "name": "Han-Yin Chang", "email": "nick20350@gmail.com", "time": "Fri Oct 23 10:18:04 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 23 10:18:04 2026 +0000" }, "message": "fix: refuse codegen dispatch for a TRY cast that can put a null key in a map (#6178)\n\n* fix: refuse codegen dispatch for a TRY cast that can put a null key in a map\n\nThe dispatcher\u0027s map writer writes each key with no null check, so a null key comes out as the key type\u0027s default value. A TRY cast whose key cast fails produces one: the failing key becomes null, and Spark\u0027s Cast.castMap builds the map without rejecting it. map_keys(transform_values(try_cast(m AS map\u003cint,int\u003e), ...)) returned [1, 0] where Spark returns [1, NULL].\n\nArrow\u0027s map format cannot hold a null key, and Spark\u0027s own readers disagree about one, so refuse any tree holding such a TRY cast in canHandle and let Spark evaluate it. A key cast is treated as able to fail unless it is on a short list of casts that cannot throw. That list is narrower than Spark\u0027s Cast.canUpCast, which admits date and timestamp conversions that overflow for extreme values.\n\nCloses #6172.\n\n* review: refuse only trees whose output can carry a map, and catch a folded null-key literal\n\n---------\n\nCo-authored-by: Andy Grove \u003cagrove@apache.org\u003e" }, { "commit": "c56f6a0209dba0600103a37fac7ca9b006639857", "tree": "374e0c43fc20e2893ffe49ecd83de23f2608b6ec", "parents": [ "eefa4dddb394534f78f3cc63701a0a44d7dce149" ], "author": { "name": "dustin", "email": "dwsmith1983@users.noreply.github.com", "time": "Fri Oct 23 10:17:58 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 23 10:17:58 2026 +0000" }, "message": "fix: fall back when an outer hash join\u0027s streamed order cannot be checked (#7126)\n\n* fix: fall back when an outer hash join\u0027s streamed order cannot be checked\n\nAn outer hash join that runs as DataFusion\u0027s right outer join keeps its\nunmatched probe rows in place only when the probe input declares an\norder. When the streamed side\u0027s reported order could not be checked, the\nserde declared none but still converted the join, so a sort Spark had\nalready dropped above it was lost. This happened over a sorted\nlocalCheckpoint and over some caches written by an outer hash join.\n\nThe join now falls back to Spark when that order reaches it through an\ninput from the JVM or through a nested loop join. Over native operators\nin the same plan it stays native, since DataFusion keeps the order it\ncan see there.\n\n* test: cover a right outer join over a cache of a right outer join\n\nThe cache test now runs both shapes that run as DataFusion\u0027s right outer\njoin, LeftOuter built on the right and RightOuter built on the left, in\nboth cache formats. The nested loop join case notes that it only gives up\nthe native join for nothing until #6898 is fixed.\n\n* fix: say that an outer hash join with an unchecked order stays in Spark\n\nThe fallback reason still described the old behavior, where the join\nstayed native and declared no order. It now says the join stays in\nSpark, and the tests share it through one constant. The compatibility\nguide leads with when the join falls back and notes that a checkpoint\nreaches a native join only with spark.comet.convert.rdd.enabled. The two\ncheckpoint tests share their setup.\n\n* docs: keep only the comments that explain when an outer join\u0027s order is trusted\n\nDrop comments that restate the code or a test\u0027s name, and shorten the\nrest to why an order from the JVM or a nested loop join can\u0027t be used." }, { "commit": "eefa4dddb394534f78f3cc63701a0a44d7dce149", "tree": "8ab52cf385fa49592590b325db04c868a73516ee", "parents": [ "886e0a9532597322fc131027f4d1841f1e36ba66" ], "author": { "name": "Yu Hsun Hung", "email": "75130467+sam-1112@users.noreply.github.com", "time": "Fri Oct 23 09:59:38 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 23 09:59:38 2026 +0000" }, "message": "test: cover speculation and executor loss in multi-task native Iceberg writes (#6601)\n\n* test: add multi-worker failure-injection coverage for native Iceberg writes\n\n* test: fix Scalafix violations in Iceberg scheduler tests\n\n* docs: clarify native Iceberg scheduler failure tests\n\n* test: cover successful retry after native Iceberg write handoff failure\n\n* fix: harden Iceberg scheduler failure probes\n\n* test: strengthen Iceberg scheduler probe regression coverage\n\n* fix: allow ancient date fixture writes in window test\n\n* Revert \"fix: allow ancient date fixture writes in window test\"\n\nThis reverts commit 1c8750c37ee5bed68436a37920792b55e4a25c47." }, { "commit": "886e0a9532597322fc131027f4d1841f1e36ba66", "tree": "d17798684938fc5d51e4f54875dfc4c5f4d20a1d", "parents": [ "6cc7b8625aa9f33c24de221182c59bc7fb58c97e" ], "author": { "name": "Yu Hsun Hung", "email": "75130467+sam-1112@users.noreply.github.com", "time": "Fri Oct 23 09:38:28 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 23 09:38:28 2026 +0000" }, "message": "fix: scope fallback reasons to the current conversion (#7095)\n\n* fix: scope fallback reasons to the current conversion\n\n* fix: preserve aggregate preflight fallback reasons\n\n* fix: scope range shuffle fallback reasons and simplify expression journal" }, { "commit": "6cc7b8625aa9f33c24de221182c59bc7fb58c97e", "tree": "4f93ec81d1afd7fd992d4d199d56427db2472886", "parents": [ "897f257149b9218be45493b868334a2f2803ea81" ], "author": { "name": "KUAN-HAO HUANG", "email": "101171023+rich7420@users.noreply.github.com", "time": "Fri Oct 23 09:03:25 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 23 09:03:25 2026 +0000" }, "message": "refactor: share native filter and projection metrics wrapper (#7056)" }, { "commit": "897f257149b9218be45493b868334a2f2803ea81", "tree": "9be9e67999f6f9116f6dc0d570fc8bd5b932ed52", "parents": [ "333bae814b676e2c03fb49fd55d9a9c1a3f43de7" ], "author": { "name": "KUAN-HAO HUANG", "email": "101171023+rich7420@users.noreply.github.com", "time": "Fri Oct 23 09:03:23 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 23 09:03:23 2026 +0000" }, "message": "fix: preserve native sliding integer sum overflow semantics (#6069)\n\n* fix: fall back for ANSI and TRY integer sums over sliding windows\n\n* test: preserve window harness and ANSI coverage\n\n* fix: preserve native sliding integer sum overflow semantics\n\n* perf: avoid unnecessary sliding sum prefix queues" }, { "commit": "333bae814b676e2c03fb49fd55d9a9c1a3f43de7", "tree": "2b118abeaff8d0d4bcde27005f04eaf3bbdd0d3f", "parents": [ "653c7954a485666a4b7c5487616b8755bbcf643c" ], "author": { "name": "Oleks V", "email": "comphead@users.noreply.github.com", "time": "Fri Oct 23 08:55:16 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 23 08:55:16 2026 +0000" }, "message": "fix: report output rows and time of the native window operator (#7140)\n\n* fix: report output rows and time of the native window operator\n\n`CometWindowExec` declared only `dataSize` and `numPartitions`, which\nnothing sets, so the time spent in the native window never reached the\nSpark UI or the event log. Report the baseline metrics instead. When the\nplanner casts the window\u0027s results in a projection, add the window as an\nadditional native plan so that its time is reported with the\nprojection\u0027s.\n\n* test: overwrite the temp directory the window metrics test writes to\n\n* test: check the native window time is added to the cast projection\u0027s\n\nAlso update the window debugging example for the new SparkPlan construction.\n\n* test: share the scan-driving loop between the planner tests\n\n* test: leave the cast projection window to the native test" }, { "commit": "653c7954a485666a4b7c5487616b8755bbcf643c", "tree": "c76e291ad36ad28f8b8ab31975c606b7038b8b3a", "parents": [ "a81a591e582ad631f0ad536360a45bab745c720a" ], "author": { "name": "KUAN-HAO HUANG", "email": "101171023+rich7420@users.noreply.github.com", "time": "Fri Oct 23 08:32:20 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 23 08:32:20 2026 +0000" }, "message": "feat: support AtLeastNNonNulls natively (#6180)\n\n* feat: support AtLeastNNonNulls natively\n\n* refactor: drop AtLeastNNonNulls small batch threshold\n\nKeep only the bitmap counter. At 8192 rows, the row counter was slower\nin 51 of 56 measured cases on a non-idle machine.\n\n* fix: fall back when AtLeastNNonNulls would skip a shared subexpression\n\nSpark\u0027s subexpression elimination evaluates a shared subexpression for\nevery row before the predicate, so an ANSI failure in it surfaces even\non rows where AtLeastNNonNulls reaches n first. Decline native\nconversion when a child after the first n, which native evaluation may\nskip, holds such a subexpression, reusing the check behind\ncanShortCircuitNulls.\n\n* fix: preserve shared expressions across decimal promotion" }, { "commit": "a81a591e582ad631f0ad536360a45bab745c720a", "tree": "260d1dacbf35d6fd5bc10c6ee0062cae3c83d74f", "parents": [ "06cff604396f08da75e67955357188250d64b95d" ], "author": { "name": "Liang-Chi Hsieh", "email": "viirya@gmail.com", "time": "Fri Oct 23 08:10:39 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 23 08:10:39 2026 +0000" }, "message": "fix: log native panic backtraces instead of attaching them to the exception (#7042)\n\n* fix: keep each native panic\u0027s backtrace with its own thread\n\nThe panic hook stored every panic\u0027s backtrace in one process-wide slot,\nwhich the JNI boundary then took for the CometNativeException. When two\nthreads panicked at about the same time, one exception could carry the\nother thread\u0027s backtrace and the other none. Panics that Comet catches\ninternally also wrote to the slot and could be taken by another thread.\n\nKeep the backtrace in a thread-local instead: the panic hook and the\ncatch_unwind that turns a panic into a CometError run on the panicking\nthread. The hook uses try_with, since a panic inside the hook aborts and\nthe thread-local is gone while its thread is torn down.\n\nCloses #7041.\n\n* Do not attach a stale backtrace to a panic resumed from another thread\n\nDataFusion resumes the panics of its spawned tasks on the consuming thread\nwith resume_unwind, which does not run the panic hook. Such a panic then\nreached the boundary with nothing captured on its thread, or with the\nbacktrace of an earlier panic that was caught and handled there.\n\nEach boundary now forgets the panic captured on its thread before running\nthe call, and the hook records the panic message with the backtrace, so a\nbacktrace is only attached to the panic with the same message. A panic\nresumed from another thread gets no backtrace.\n\n* Log a native panic\u0027s backtrace instead of attaching it to the exception\n\nA panic can reach the JNI boundary on another thread than the one that\nraised it: DataFusion resumes the panics of its spawned tasks on the\nconsuming thread with resume_unwind, which does not run the panic hook.\nThe boundary then cannot tell which captured backtrace belongs to the\npanic it reports. Matching the panic message is not enough, since a\npanic with the same message may have been caught and handled on the\nthread during the same call.\n\nLog the backtrace with the panic message and location from the panic\nhook, on the thread that panicked, and no longer attach a backtrace to\nthe CometNativeException. This removes the per-thread capture added\nearlier in this change.\n\n* Remove the unused backtrace plumbing and format the panic report off the lock\n\nWith the backtrace logged by the panic hook, nothing passes one to the\nclassification any more. Remove the backtrace parameter of\nNativeError::from_comet_error and the classification helpers, the\nNativeError::backtrace field, the \"backtrace\" key of the payload (so its\nversion becomes 2), to_stacktrace_string and its test data, and update the\nstale comments.\n\nFormat the hook\u0027s report into a String before writing it: the backtrace\nresolves its symbols when displayed, and eprintln! holds the stderr lock\nwhile it formats its arguments. The report is built by panic_report, which\na test now checks." }, { "commit": "06cff604396f08da75e67955357188250d64b95d", "tree": "a28c862566c89cc5744d093123a1dd43d6842a8b", "parents": [ "50ab6b063ce69f651acfa6a457e87099f92b9efa" ], "author": { "name": "Liang-Chi Hsieh", "email": "viirya@gmail.com", "time": "Fri Oct 23 08:08:01 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 23 08:08:01 2026 +0000" }, "message": "feat: execute scalar Arrow Python UDFs in native Comet (#6130)\n\n* feat: execute scalar Arrow Python UDFs in native Comet\n\n* docs: clarify scalar Arrow UDF support starts in Spark 4.1\n\n* fix: align native Arrow UDF semantics and CI coverage\n\n* fix: avoid Arrow UDF polling spin and narrow schema support\n\n* Fix embedded Python symbols and verify Arrow input types\n\n* Document embedded Python build and deployment requirements\n\n* Fix native Arrow UDF environment and batch semantics\n\n* Honor Arrow Python UDF byte batch limit\n\n* Skip empty Arrow UDF batches and protect plan metadata\n\n* Fix native Arrow UDF subqueries, file fallback, and exchange reuse\n\n* Address native Arrow UDF review: archives, results, logging, metrics\n\n- Keep queries with Spark-added archives or Python worker logging on\n Spark\u0027s worker path.\n- Accept spark.executorEnv.* entries in a UDF\u0027s environment when the\n executor process already has them; PYTHONHASHSEED must stay 0.\n- Check the enable config last, so other ArrowEvalPythonExec types\n report why they are unsupported instead of suggesting the config.\n- On Spark 4.2, accept any result pyarrow.RecordBatch.from_arrays\n accepts, such as a list or NumPy array.\n- Decode invalid UTF-8 in returned string arrays, as for JVM imports.\n- Include the Python traceback in UDF errors.\n- Report output_rows, elapsed_compute and python_time metrics.\n- Document the task-kill limitation and fix the memory docs.\n- Test sliced inputs for every type, a multi-row-group Parquet scan,\n archives, and the new fallbacks.\n\n* Fix Arrow UDF tracebacks before Python 3.12 and serialize imports\n\nBefore Python 3.12, PyO3 keeps an exception\u0027s traceback apart from its\nvalue, so passing only the value to traceback.format_exception dropped\nthe traceback. Pass the type, value and traceback explicitly.\n\nConcurrent first imports of PyArrow from several task threads can hand\none thread a partially initialized pyarrow.lib on Python 3.10. Import\nPyArrow while the interpreter is initialized, before any other thread\ncan run Python, and unpickle callables one at a time under a lock that\nwaits without holding the GIL.\n\n* Start embedded Python with unbuffered stdio\n\nSpark starts its Python workers with PYTHONUNBUFFERED. The embedded\ninterpreter is never finalized, so print() output still buffered in\nsys.stdout was lost when the executor exited. Disable stdio buffering\nand leave the JVM\u0027s C streams unconfigured.\n\nA regression test runs a UDF that prints in a child process, exits\nwithout flushing, and checks that the parent reads the output.\n\n* Stop native Arrow UDF stream after its first error\n\nAfter a Python error, polling the stream again evaluated the next slice\nof the batch, and after a child error flat_map pulled the next child\nbatch, so a consumer that kept polling called Python again while the\ntask was failing. End the stream after the first error it yields.\n\nA regression test splits two child batches into four slices with a UDF\nthat raises, drains the stream, and checks that Python ran once.\n\n---------\n\nCo-authored-by: Isaac \u003cno-reply@databricks.com\u003e" }, { "commit": "50ab6b063ce69f651acfa6a457e87099f92b9efa", "tree": "c5fe98ce052491a61bf443392312abd93430309a", "parents": [ "58978d0d4561ea421e8547ec19b8e4143e0bd504" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Fri Oct 23 07:41:22 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 23 07:41:22 2026 +0000" }, "message": "perf: share one decoded copy of a broadcast among the tasks of an executor (#7016)\n\n* perf: share one decoded copy of a broadcast among the tasks of an executor\n\nEvery task that read a Comet broadcast decompressed and decoded the whole\nbroadcast for itself. DecodedBroadcastCache lets the tasks of an executor\nthat read a broadcast at the same time share one decoded copy: the first\ndecodes it into buffers the cache owns, tasks that arrive during that decode\nwait for it, and each reads vectors of its own over the shared buffers. An\nentry is freed when the last task holding it completes. A failed decode is\nnot cached, and a broadcast with dictionaries is still decoded per task.\nCometBatchRDD reads through the cache, so every broadcast consumer uses it.\nspark.comet.exec.broadcastExchange.shareDecoded.enabled turns it off.\n\nThe broadcast decode and the driver-side coalescing also read the IPC stream\nthrough a channel that reads up to 64 KiB at a time, instead of\nChannels.newChannel, which reads 8 KiB at a time.\n\n* docs: describe the shared broadcast decode accurately\n\nThe cache holds no memory the tasks would not hold anyway only for a\nbroadcast of one batch, which a native join keeps for as long as its task\nruns. List the cache\u0027s child allocator in the memory management guide,\nsay that the read channel copies through a reused array, count holds per\nread rather than per task, and drop an unused accessor.\n\n* fix: stop a broadcast read within a piece of an interrupt, and free a decode its task outlived\n\nArrow reads a whole message body with one read, which for a coalesced\nbroadcast is all of it, so the read channel now checks for an interrupt\nbefore each piece, as Channels.newChannel did, instead of once per read.\n\nA task whose completion listener runs on another thread can give back its\nhold while its read decodes the broadcast. If it was the last holder, the\nentry was freed before the decode finished, and the decode was kept where\nnothing would free it. The decode is now freed instead, and the read\ndecodes the broadcast for itself.\n\n* refactor: one batch iterator per read and shared test helpers\n\nThe shared batch iterator now keeps a single iterator, of views or of the\ntask\u0027s own decode, instead of branching on which one it has. A decode that\nkeeps nothing frees its batches through Decoded.close. The four tests that\nblock a decode share one cache, ScalaTest\u0027s eventually replaces a hand-written\nwait, UtilsSuite\u0027s new tests reuse its serialize helper, and the test of a\nstream larger than one read piece checks against MaxReadPiece.\n\n* refactor: keep the broadcast read on Channels.newChannel, and release the entry even if closing the view fails\n\n* refactor: keep the shared broadcast decode on the broadcast value\n\nThe decode that the tasks of an executor share lived in a process-wide map keyed by the\nbroadcast value\u0027s identity. Broadcast a CometBroadcastValue instead, which holds the same\nchunks and keeps the shared decode in a transient lazy field. Spark gives every task of an\nexecutor the same instance, so the state needs no map, no key and no global singleton, and\nthe holds and the release on task completion work as before.\n\nAlso drop the guard for a completion listener that runs on another thread while the task is\nstill decoding: Spark runs the listeners on the task\u0027s thread, and the views had no such\nguard, so say in the scaladoc that a read stays on that thread. Document that sharing lowers\nresident memory but not reservations, move the join-type matrix to a SQL file test, and fold\nthe free-and-decode-again test into the first." }, { "commit": "58978d0d4561ea421e8547ec19b8e4143e0bd504", "tree": "e11b4a0b810bf429278c825a705ac0a408d909b2", "parents": [ "5a50bd6f2777f3f20cefd929552d93dc6ab263c5" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Fri Oct 23 06:13:26 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 23 06:13:26 2026 +0000" }, "message": "refactor: share Parquet field-id and name resolution between root and nested fields (#6990)\n\n* refactor: share Parquet field-id and name resolution between root and nested fields\n\nThe root schema remap and the nested struct matcher each implemented Spark\u0027s\nclipParquetGroupFields resolution on their own. Both now select file fields through\none FieldResolver: the nested matcher resolves each requested field with it, and the\nroot remap turns its selections into one rename per file column, keeping the\nstrongest claim when several requested fields select the same column. The adapter\u0027s\ncreate and rewrite are split into named steps.\n\nCloses #6978.\n\n* refactor: build the field resolver once per file and share the duplicate-field errors\n\n`create` now folds each schema once and builds one `FieldResolver`, which drives the root\nremap and supplies the name collisions and id-resolved names that `rewrite` checks referenced\ncolumns against. `select` raises the duplicate field id error for both the root and nested\npaths, and one `duplicate_name_error` serves both name checks.\n\nSmaller simplifications: a placeholder-name closure instead of `FakeNameGenerator`,\n`Field::with_name` for renames, `Itertools::duplicates`, `apply` instead of a throwaway\n`transform`, and tests that call the real name fold.\n\n* test: fold the resolver\u0027s ambiguity tests into the tests that already cover them\n\nThree of the new `field_resolver.rs` tests repeated rules that existing tests\nalready reach through `match_struct_fields`. Remove them and keep what was new:\n`issue_5783_nested_name_duplicate` gains a field that differs from the\nduplicate only in case, which a case-sensitive read must resolve on its own,\nand `unrequested_duplicate_field_id_reads_fine` gains a name collision that the\nrequested field\u0027s id match never compares.\n\n* docs: note that the root duplicate field id check covers the whole data schema (#7006)" }, { "commit": "5a50bd6f2777f3f20cefd929552d93dc6ab263c5", "tree": "141423c4f8b42faae09caf51dde6e9e4734f7acd", "parents": [ "78a0e96f05f1402a0941e0c55589489404190623" ], "author": { "name": "KAZUYUKI TANIMURA", "email": "ktanimura@apple.com", "time": "Thu Oct 22 21:45:26 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 22 21:45:26 2026 +0000" }, "message": "test: cover positional struct reconciliation in native CASE WHEN and COALESCE (#7149)\n\n* test: cover positional struct reconciliation in native CASE WHEN and COALESCE\n\n* address review comments" }, { "commit": "78a0e96f05f1402a0941e0c55589489404190623", "tree": "fb0ec1fd7888e2e452e3fd4738f7575c4e4a2f72", "parents": [ "da2ff08134ba6ceeccade058aea09e7c21d9472b" ], "author": { "name": "Chao Sun", "email": "sunchao@apache.org", "time": "Thu Oct 22 20:12:18 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 22 20:12:18 2026 +0000" }, "message": "fix: preserve Spark AVG buffer and overflow semantics (#5420)\n\n* fix: preserve Spark AVG buffer and overflow semantics\n\n* test: pin decimal AVG results and shuffle writers\n\n* fix: address decimal AVG object buffers and wide division review\n\n* Keep AQE fallback fixtures on unsafe decimal AVG buffers\n\n* fix: address AVG review feedback after rebase" }, { "commit": "da2ff08134ba6ceeccade058aea09e7c21d9472b", "tree": "01a6ea1e29b8190dd4b9187e417277401dc134e6", "parents": [ "ab884e24aca7baeaa8df723e70a1bdfc3efc7943" ], "author": { "name": "Manu Zhang", "email": "OwenZhang1990@gmail.com", "time": "Wed Oct 21 14:11:44 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 21 14:11:44 2026 +0000" }, "message": "fix: extract bundled native libraries into private directories (#6987)\n\n* fix: extract bundled native libraries into private directories\n\nCreate the library exclusively inside an owner-only directory, reject unsafe temporary roots, and clean up failed extractions and JVM shutdown files.\n\nCo-authored-by: Codex \u003ccodex@openai.com\u003e\n\n* fix: select a safe extraction root for YARN deployments\n\nKeep strict ownership and permission validation, but try separately validated system temporary directories when Spark temp ancestry is unsuitable. Select the private directory before reading the library and never retry consumed resources.\n\nCo-authored-by: Codex \u003ccodex@openai.com\u003e\n\n* fix: simplify bundled native library extraction\n\nUse a private directory under java.io.tmpdir, with a comet.native.workdir override for executable storage. Remove ancestor and account checks and fixed system-root fallbacks. Consolidate regressions and isolate property tests in child JVMs; document cleanup after forced termination.\n\nCo-authored-by: Codex \u003ccodex@openai.com\u003e\n\n---------\n\nCo-authored-by: Codex \u003ccodex@openai.com\u003e" }, { "commit": "ab884e24aca7baeaa8df723e70a1bdfc3efc7943", "tree": "ac4e8e4ec2f4384d8f5c9581fa9887f0ce8ba407", "parents": [ "38d392940efeb8557c0cf4c17bc887640028d5b2" ], "author": { "name": "ChenChen Lai", "email": "72776271+0lai0@users.noreply.github.com", "time": "Wed Oct 21 12:31:21 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 21 12:31:21 2026 +0000" }, "message": "fix: fall back to Spark for negative-scale decimal casts and arithmetic (#5050)\n\n* fix: fall back to Spark for negative-scale decimal casts and arithmetic\n\n* fix: route negative-scale decimal casts through codegen dispatch, guard float/double\n\n* fix: drop obsolete negative-scale guards, make dispatcher decimal reader safe\n\n* refactor: fold the negative-scale check into the integral cast case\n\n* fix: avoid implicit numeric widening in the negative-scale cast test\n\n* style: apply spotless formatting to the negative-scale cast test\n\n* fix: address review - guard CometVector decimal reader, share the width calculation, correct the release-build claim\n\n* style: run cargo fmt\n\n---------\n\nCo-authored-by: Andy Grove \u003cagrove@apache.org\u003e" }, { "commit": "38d392940efeb8557c0cf4c17bc887640028d5b2", "tree": "405c2b9f7a700813dfcc649b6936ae132c76cefb", "parents": [ "74392a9427db2cdae9ba9a2a710985cce945e0f7" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Wed Oct 21 11:08:06 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 21 11:08:06 2026 +0000" }, "message": "perf: default the native Iceberg scan to two concurrent data files (#7132)\n\n* perf: default the native Iceberg scan to two concurrent data files\n\nDefault spark.comet.scan.icebergNative.dataFileConcurrencyLimit to 2 so a task overlaps the open of two data files. The default is read through COMET_ICEBERG_DATA_FILE_CONCURRENCY_LIMIT, which the Iceberg Spark test jobs and dev/local-ci.sh set to 1 because Iceberg\u0027s tests expect file order from queries without an ORDER BY.\n\n* test: keep file read order in Iceberg tests that depend on it" }, { "commit": "74392a9427db2cdae9ba9a2a710985cce945e0f7", "tree": "bc9220e66ecbf266035f45e2c9b7724f6f9ec891", "parents": [ "9a3d2a292a2fb63ea20d446491029c1f625da0c2" ], "author": { "name": "Oleks V", "email": "comphead@users.noreply.github.com", "time": "Tue Oct 20 23:48:35 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 20 23:48:35 2026 +0000" }, "message": "test: add window operator benchmarks (#7139)\n\n* test: add window operator benchmarks\n\nAdd a native criterion bench for the operator behind `CometWindowExec`,\nbuilt the way the planner builds it, and a JVM benchmark that compares\nSpark, Comet, and Comet with the window run in Spark. Both vary the rows\nper PARTITION BY group, the window function, the argument type, and the\ncolumns carried through the window.\n\n* test: compare window benchmark arms on Spark 3.x and pass the row count as a Long" }, { "commit": "9a3d2a292a2fb63ea20d446491029c1f625da0c2", "tree": "ee732cfd9bc43ba444611f9f4d46aec1c0052913", "parents": [ "bf49f1e0283d9c2adb582542b653cd3268f88ae5" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Tue Oct 20 21:52:14 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 20 21:52:14 2026 +0000" }, "message": "perf: load the next Parquet file\u0027s footer and page index while the current file is read (#7094)\n\n* perf: load the next Parquet file\u0027s footer and page index while the current file is read\n\nDataFusion 54 rebuilt FileStream on the morsel API, and its ScanState plans a file only after\nthe previous file\u0027s reader has returned its last batch (apache/datafusion#20916 removed the\nnext-file prefetch), so every file boundary in a native Parquet scan waits for a footer and\npage-index round trip.\n\nWith the new opt-in spark.comet.parquet.prefetchNextFileMetadata.enabled (off by default),\nEagerPageIndexReaderFactory gets the scan\u0027s file group, and each file\u0027s open starts a task that\nloads the next file\u0027s metadata through the plan\u0027s metadata cache. That file\u0027s open then waits\nfor the task instead of reading the footer itself. One load is in flight at a time, and\ndropping the scan aborts it. The flag reaches the scan through a SessionConfig extension, so\nDelta scans get it too, and the new scan_io_metadata_prefetches metric counts the loads. An open\nwith decryption properties neither waits for nor starts a prefetch.\n\n* address review: record prefetched opens as cache misses and document the prefetch\n\nA prefetched open now records the storage reads of the load it waited for, so\nscan_io_metadata_cache_hits keeps meaning \u0027served from the cache\u0027 and\nscan_io_metadata_prefetches counts the loads that overlapped a previous file.\nAn encrypted scan no longer builds the prefetch at all. The ranges test now\nopens through the factory, and the tuning guide documents the config.\n\n* address review: say that only loads which read storage count as prefetches\n\n* address review: build the next-file map in one pass from the back\n\n* perf: prefetch the next Parquet file\u0027s metadata by default\n\n* test: trim overlapping prefetch tests and merge the test object stores\n\nDrop the parquet_exec scan test that repeats the CometExecSuite test, run\nthe Scala test with the config on and off only, and fold GatedStore into\nthe recording store used by the range read tests." }, { "commit": "bf49f1e0283d9c2adb582542b653cd3268f88ae5", "tree": "0f5076cb895b08abd111a079a05f9df4c4879bd8", "parents": [ "666f86f71be1fc08a0894a8f754395e8f17b96f5" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Tue Oct 20 19:58:47 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 20 19:58:47 2026 +0000" }, "message": "chore: limit open PRs for users without write access to 3 (#7135)\n\n* chore: limit open PRs for users without write access to 3\n\n* docs: encourage draft PRs for later parts of a series instead of a single combined PR" }, { "commit": "666f86f71be1fc08a0894a8f754395e8f17b96f5", "tree": "ddc69d4fe05ee4b58ffac5a58d3ce4948110357e", "parents": [ "0b04b38002bc18e615a84c688df3ad36a25e4f33" ], "author": { "name": "Liang-Chi Hsieh", "email": "viirya@gmail.com", "time": "Tue Oct 20 19:34:30 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 20 19:34:30 2026 +0000" }, "message": "perf: sort JVM shuffle spill pointers in the sorter\u0027s scratch space (#7125)\n\n`Native.sortRowPartitionsNative` sorted a spill\u0027s packed record pointers\nwith a copy of the rdxsort-rs crate. It always ran six 4-bit passes over\nthe 24-bit partition id, and allocated two sets of 16 bucket vectors,\neach sized n/4: about 8n longs of virtual memory per call, none of it\nreserved from any memory pool, and allocated exactly when the task is\nspilling under memory pressure.\n\n`SpillSorter` builds its `ShuffleInMemorySorter` with radix sort\nenabled, so the sorter only fills half of its pointer array and keeps\nthe other half free as scratch space for Spark\u0027s `RadixSort`. That\nmemory is already acquired from the task memory manager.\n\nReplace the sort with an LSD counting sort over the partition id\u0027s\nthree bytes, as Spark\u0027s `RadixSort.sort` does: one pass counts every\nbyte, a byte that is the same in every pointer is skipped, and each\nremaining byte scatters between the records and the scratch half of the\nsame array, copying back if the result ends in the scratch half. The\nsort allocates nothing and stays stable, so its output is unchanged.\n\n`sortRowPartitionsNative` now takes the array\u0027s capacity and rejects\none with fewer than `2 * size` longs.\n\nThe new `sort_row_partitions` criterion bench, on 1M records:\n\n partitions before after\n 1 6.52 ms 0.54 ms\n 200 6.69 ms 1.17 ms\n 4000 6.94 ms 1.87 ms\n 100,000 7.42 ms 4.77 ms\n 200, 90% in one 9.08 ms 3.73 ms\n\nThe sort is a small part of a spill next to the row-to-Arrow\nconversion, so this mainly removes the untracked allocation rather than\nspeeding up a spill end to end.\n\nCo-authored-by: Isaac \u003cno-reply@databricks.com\u003e" }, { "commit": "0b04b38002bc18e615a84c688df3ad36a25e4f33", "tree": "2c2285bec1314e6e15e0aa9945f205596f2556ae", "parents": [ "39380cca813bc0b861353de2cb53af960dd2a87f" ], "author": { "name": "KAZUYUKI TANIMURA", "email": "ktanimura@apple.com", "time": "Tue Oct 20 19:29:32 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 20 19:29:32 2026 +0000" }, "message": "fix: evaluate the array_append item on rows whose array is NULL (#7025)\n\n* fix: array_append: ANSI item error is swallowed when the array is NUL\n\n* address review comments\n\n* ci: purge POM-only Maven entries that declare no packaging\n\nThe \"Purge partial Maven cache entries\" step only deleted a jar-less POM\nwhen it carried an explicit \u003cpackaging\u003ejar|bundle\u003c/packaging\u003e. A POM with\nno \u003cpackaging\u003e defaults to jar, and Arrow\u0027s POMs, among others, declare\nnone, so a POM-only entry for one of them survived the purge. Coursier\nthen treated the artifact as found locally and dropped the missing JAR,\nand sbt failed compiling sql-api with \"object memory is not a member of\npackage org.apache.arrow\" in the Spark 3.5 \"Pre-compile Spark Test\nclasses\" step.\n\nRead the packaging and default it to jar, as dev/local-ci.sh\u0027s\npurge_partial_poms already does. A jar-less \u003cpackaging\u003epom\u003c/packaging\u003e\nparent has no JAR by design and is kept.\n\n* Revert \"ci: purge POM-only Maven entries that declare no packaging\"\n\nThis reverts commit dc74c67d228e15b2f910577e370876ea5c975c5a.\n\n* address review comments" }, { "commit": "39380cca813bc0b861353de2cb53af960dd2a87f", "tree": "e39731b502325b311ff34ca07f17771e72ff46c6", "parents": [ "e4777a6587d6dd697867ef3b4280224e164cb07c" ], "author": { "name": "Chao Sun", "email": "sunchao@apache.org", "time": "Tue Oct 20 03:52:53 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 20 03:52:53 2026 +0000" }, "message": "feat: opt in to join filter pushdown through deterministic residuals (#6431)" }, { "commit": "e4777a6587d6dd697867ef3b4280224e164cb07c", "tree": "004df52d9d68f16c03a7ec4109131ef998e405fa", "parents": [ "a48c82da53ebf13f2bea9a4735bde6917211f679" ], "author": { "name": "Samarth Mukhija", "email": "139998526+Samearth17@users.noreply.github.com", "time": "Tue Oct 20 03:39:08 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 20 03:39:08 2026 +0000" }, "message": "fix: fall back for duplicate from_json fields (#6651)\n\n* fix: fall back for duplicate from_json fields\n\nSigned-off-by: Samarth Mukhija \u003c139998526+Samearth17@users.noreply.github.com\u003e\n\n* test(json): move duplicate-field fallback to SQL fixture\n\n* test(json): pin native unique-field controls\n\n---------\n\nSigned-off-by: Samarth Mukhija \u003c139998526+Samearth17@users.noreply.github.com\u003e\nCo-authored-by: Andy Grove \u003cagrove@apache.org\u003e" }, { "commit": "a48c82da53ebf13f2bea9a4735bde6917211f679", "tree": "540581865afe5b6569e1fbcb36aadf4c7539f040", "parents": [ "5a8c5c5eda300b0b0c124eda212455807042a45d" ], "author": { "name": "Yunchi Pang", "email": "yunchipang@gmail.com", "time": "Tue Oct 20 02:45:27 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 20 02:45:27 2026 +0000" }, "message": "fix: fall back when an Iceberg equality delete is keyed on a nested field (#7122)" }, { "commit": "5a8c5c5eda300b0b0c124eda212455807042a45d", "tree": "e07d98c3631c8d02d0637758215892c8e3aecc8b", "parents": [ "d834665aeab2fcbe812f638ec9f3637b248d38c4" ], "author": { "name": "Yu Hsun Hung", "email": "75130467+sam-1112@users.noreply.github.com", "time": "Tue Oct 20 02:04:45 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 20 02:04:45 2026 +0000" }, "message": "feat: enable native replace for safe inputs (#6680)\n\n* Enable native replace for safe inputs\n\n* Test native replace literal and broadcast boundaries\n\n* docs: format string expression audit tables\n\n* Fix replace routing tests for native safe inputs" }, { "commit": "d834665aeab2fcbe812f638ec9f3637b248d38c4", "tree": "2c0bd00d0bc2c80ebfb29346e488f7e6639ba933", "parents": [ "348d38f28f0e66607db7fe7e6a3a8ad6f711c9bf" ], "author": { "name": "LinSimon-901101", "email": "bcs113118@gm.ntcu.edu.tw", "time": "Tue Oct 20 02:03:54 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 20 02:03:54 2026 +0000" }, "message": "test: verify native timestamp output timezone labels (#6930)\n\n* test: cover native timestamp output timezone labels\n\n* test: cover boolean timestamp output labels" }, { "commit": "348d38f28f0e66607db7fe7e6a3a8ad6f711c9bf", "tree": "a9d4cd7dc5924f4837518daee928b59fe51f7ae3", "parents": [ "99c4873d52e47bc466d740b4023e0de7be797e28" ], "author": { "name": "Alex Huang", "email": "huangweijun1001@gmail.com", "time": "Tue Oct 20 02:03:00 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 20 02:03:00 2026 +0000" }, "message": "fix: preserve Spark nested null ordering in native sort (#6968)\n\n* fix: preserve Spark nested null ordering in native sort\n\n* test: update nested null sort and rank limit coverage\n\n* Address nested sort review feedback\n\n* Remove unused sort test import" }, { "commit": "99c4873d52e47bc466d740b4023e0de7be797e28", "tree": "82553824aeeae5b3949d45c19ff2ec48a501014c", "parents": [ "177fef162a8e8d2eb8d4e4c02f2e9edcecfd046e" ], "author": { "name": "Bill", "email": "StanceTheory1@gmail.com", "time": "Tue Oct 20 02:02:32 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 20 02:02:32 2026 +0000" }, "message": "perf: vectorize int, decimal, float and boolean casts to timestamp (#6969)\n\n* test: extend timestamp cast benchmarks\n\nCover 8192-row batches with no nulls, sparse nulls, dense nulls, and an\nall-null batch for the integer, float, decimal, and boolean casts to\ntimestamp, in legacy and ANSI modes where the mode changes the kernel.\n\nGenerated-by: Cursor Grok 4.7\n\nCo-authored-by: Bill \u003cErsa-tech@users.noreply.github.com\u003e\n\n* perf: vectorize numeric and boolean casts to timestamp\n\nReplace the per-element TimestampMicrosecondBuilder loops in\ncast_int_to_timestamp, cast_decimal_to_timestamp, cast_float_to_timestamp,\nand cast_boolean_to_timestamp with Arrow unary, try_unary, unary_opt,\nand from_unary.\n\nInteger and boolean casts are about 5-12x faster. Float casts are\n14-35% faster when nulls are rare and several times faster when most\nrows are null. Decimal with no nulls is under 10%; denser batches\nskip null slots and are faster. No benchmarked shape regressed.\n\nGenerated-by: Cursor Grok 4.7\n\nCo-authored-by: Bill \u003cErsa-tech@users.noreply.github.com\u003e\n\n* perf: use one unary kernel for decimal-to-timestamp\n\nDrop the quarter-null dispatch. unary_opt is faster on every measured\ndecimal shape, including a batch with no nulls. Trim the benches and\nedge tests to the shapes and cases the cast perf PRs keep.\n\nGenerated-by: Cursor Grok 4.7\n\nCo-authored-by: Bill \u003cErsa-tech@users.noreply.github.com\u003e\n\n* perf: note that timestamp unary kernels run on null slots\n\n`unary` and `from_unary` visit null slots. Say so, and that the\ninteger and boolean conversions are infallible for any bit pattern.\nFloat legacy uses `unary_opt`, which skips those slots.\n\nGenerated-by: Cursor Grok 4.7\n\nCo-authored-by: Bill \u003cErsa-tech@users.noreply.github.com\u003e\n\n* test: pin null-slot timestamp casts to the builder\n\nGenerated-by: Cursor Grok 4.7\n\nCo-authored-by: Bill \u003cErsa-tech@users.noreply.github.com\u003e\n\n* test: close timestamp-cast gaps from review\n\nGenerated-by: Cursor Grok 4.7\n\nCo-authored-by: Bill \u003cErsa-tech@users.noreply.github.com\u003e\n\n---------\n\nCo-authored-by: Cursor Agent \u003ccursoragent@cursor.com\u003e\nCo-authored-by: Bill \u003cErsa-tech@users.noreply.github.com\u003e" }, { "commit": "177fef162a8e8d2eb8d4e4c02f2e9edcecfd046e", "tree": "fd33b6f0ad8d8cd1e2aa8e8194d34e099928d677", "parents": [ "137c8945398d77523871876c6f939f038136d847" ], "author": { "name": "Armin Azimi", "email": "33401832+arminazimi@users.noreply.github.com", "time": "Tue Oct 20 02:02:09 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 20 02:02:09 2026 +0000" }, "message": "refactor: deduplicate split list construction (#7004)\n\nExtract finish_split_list while preserving the input validity buffers from\n#6953. Keep the LargeUtf8 output-type regression test and reuse the\nupstream array/array null-delimiter test.\n\nCloses #6983" }, { "commit": "137c8945398d77523871876c6f939f038136d847", "tree": "2800a7dbdc4e7345183ef6e5a8e4dcc0033fefe4", "parents": [ "cae599ace3bbc76fa2e29133e3a9504ff7b15d61" ], "author": { "name": "KUAN-HAO HUANG", "email": "101171023+rich7420@users.noreply.github.com", "time": "Tue Oct 20 02:00:42 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 20 02:00:42 2026 +0000" }, "message": "feat: support native wide-decimal hashing (#6548)\n\n* feat: support native wide-decimal hashing (WIP)\n\n* perf: avoid per-element slices when hashing wide decimal arrays\n\n* test: cover mixed wide-decimal shuffle joins\n\n* fix: explicitly widen wide-decimal benchmark row count" }, { "commit": "cae599ace3bbc76fa2e29133e3a9504ff7b15d61", "tree": "d53b787ed0d341c7cecc1e320fdb704a29f6bdcb", "parents": [ "3565f2f9b1351ddcd63a76dff1532ddcdcbd7d83" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Tue Oct 20 01:56:45 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 20 01:56:45 2026 +0000" }, "message": "feat: add config to disable or restrict native UDF library loading (#6775)\n\nAdd spark.comet.nativeUdf.enabled and spark.comet.nativeUdf.allowedPaths,\nenforced on the driver at registration and again on the executor before a\nplan loads a library.\n\nAlso stop the native UDF library cache holding its write lock across\ndlopen: each library now loads under its own slot lock, and a poisoned\nlock no longer disables the cache for the life of the process." }, { "commit": "3565f2f9b1351ddcd63a76dff1532ddcdcbd7d83", "tree": "88b53c67d5e286324a08684e3585db07fbe2b2dd", "parents": [ "53d886756b7ee5581608e82c8159400641bc3856" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Tue Oct 20 01:56:42 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 20 01:56:42 2026 +0000" }, "message": "fix: treat -0.0 and 0.0 as one value in arrays_overlap on Spark 4.0.5, 4.1.4 and 4.2.1 (#6989)\n\n* fix: treat -0.0 and 0.0 as one value in arrays_overlap on Spark 4.0.5, 4.1.4 and 4.2.1\n\nSpark 4.0.5, 4.1.4 and 4.2.1 normalize each flat FLOAT or DOUBLE element\nbefore arrays_overlap looks it up in its hash set of boxed values\n(SPARK-59602), so -0.0 and 0.0 are one value, as they already were on\n4.2.0, which normalizes the arguments in the plan (SPARK-54918). The\nnative kernel kept the zeros apart on every version, so on those\nreleases it returned false where Spark returns true.\n\nRegister a second variant, spark_arrays_overlap_normalized, whose flat\nfloat arms key elements by normalize_float instead of canonicalize_nan,\nand have CometArraysOverlap pick it wherever\nArraySetSupport.normalizesFloats holds. Earlier releases keep the\nexisting kernel. Both variants treat every NaN as one value and compare\nnested elements in Spark\u0027s SQL ordering.\n\nCloses #6768.\n\n* review: name the variant in arrays_overlap errors and trim the version table\n\nUse self.name() in the unsupported-type errors so spark_arrays_overlap_normalized reports itself,\nand test it on the array and scalar paths. Reduce the native docs to the two equality policies and\npoint at CometArraysOverlap.nativeFunction for the version mapping. Keep one Spark version on each\nside of the boundary in the routing test and the convert assertion." }, { "commit": "53d886756b7ee5581608e82c8159400641bc3856", "tree": "8002a56b50fb8631d75fcfb339596cbd04ef9eaa", "parents": [ "47730e38f3e94b76185b7d3b6666279689e48fc1" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Tue Oct 20 01:56:40 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 20 01:56:40 2026 +0000" }, "message": "perf: create a native plan\u0027s spill directories on its first spill (#7066)\n\n* perf: share one DiskManager per executor across native plans\n\nCloses #7060.\n\n* perf: create a native plan\u0027s spill directories on its first spill\n\nReplace the executor-wide shared DiskManager with a per-plan one whose\nDirectories mode DiskManager is built through DataFusion\u0027s\nTempFileFactory hook on the plan\u0027s first spill. A plan that never spills\nno longer creates or removes a datafusion-* directory on every local\ndisk, and spark.comet.maxTempDirectorySize stays a per-plan limit, so\nthe config and tuning docs go back to their previous text.\n\nUpdate the CometNativeSuite temp dir test: a plan that does not spill\ncreates no temp dir, a sort that spills creates one in a Spark local\ndir, and both are gone once the plans are released." }, { "commit": "47730e38f3e94b76185b7d3b6666279689e48fc1", "tree": "ed0c66f4c06e6715f7725a46d02bbcea82c532dc", "parents": [ "3355602e268819750884aa0ce0dd47002c6ea580" ], "author": { "name": "Liang-Chi Hsieh", "email": "viirya@gmail.com", "time": "Tue Oct 20 01:52:23 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 20 01:52:23 2026 +0000" }, "message": "chore: fix stddev accumulator size and resolve session libhdfs schemes once per scan rule (#7120)\n\nStddevAccumulator::size() subtracted align_of_val instead of size_of_val,\nunlike the sibling accumulators. The result was the same only because the\nstruct has a single field.\n\nCometScanRule copied the session\u0027s Hadoop conf (newHadoopConf) for every V1\nscan to read fs.comet.libhdfs.schemes. Resolve the session list once per\nrelation session per rule application and reuse it, still intersecting it\nwith each scan\u0027s own list.\n\nCo-authored-by: Isaac \u003cno-reply@databricks.com\u003e" }, { "commit": "3355602e268819750884aa0ce0dd47002c6ea580", "tree": "6988422e0b8c8a4567a818585e79c826ca7b6484", "parents": [ "0c7c5d69db8a4b491088a1e8c862e3d4cccedb20" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Mon Oct 19 16:50:12 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 19 16:50:12 2026 +0000" }, "message": "fix: keep a streamed side sorted outside the native plan in order through outer hash joins (#6835)\n\n* fix: keep a streamed side sorted outside the native plan in order through outer hash joins\n\nSpark\u0027s hash joins report the streamed side\u0027s order as their own, so Spark\ndrops a sort above the join that the order satisfies. DataFusion\u0027s hash join\nkeeps the unmatched probe rows of a right outer join in place only when the\nprobe input declares an order, and otherwise moves them after the matched\nrows of each batch. Comet builds every hash join on the build side, so\nSpark\u0027s left outer join built on the right and right outer join built on\nthe left both run as that right outer join, and a streamed side that enters\nthe native plan from the JVM, a relation sorted before it was cached for\ninstance, arrives through a ScanExec that declares no order. The join then\nreturned those rows out of order with no sort left in the plan to fix them.\n\nSend the streamed side\u0027s sort order with the join for those two shapes, and\nhave the planner declare it on the probe input through a pass-through\nSortedInputExec when the input declares none. The declaration is placed on\nthat one input rather than on every ScanExec because an order visible to the\nwhole native plan would also change how DataFusion\u0027s aggregates group. A\nsort order that cannot be expressed natively sends the join back to Spark.\n\nWith the in-memory cache on by default since #5634, the native cache scan\nreaches this with default settings.\n\n* test: cover the right outer shape, an aggregate above the join and the fallback\n\nFrom review of the hash join order declaration:\n\n- The join and cache tests also run a right outer join built on the left,\n which Comet does not swap, and compare rows as multisets, not sets.\n- A cache test groups by the key above the join, where the declared order\n puts both aggregates in sorted mode.\n- A join test covers the fallback when the streamed order cannot be\n serialized.\n- Planner tests cover declaring the order on the probe input for both\n shapes, and leaving a probe that already declares one alone.\n- SortedInputExec\u0027s doc says the declared order is visible above the join.\n\n* fix: declare the streamed order only over operators that keep a JVM input\u0027s order\n\nFrom review:\n\n- with_declared_order walks down from the probe input through operators\n that report keeping their input\u0027s order and declares only when it\n reaches a ScanExec or ShuffleScanExec. DataFusion reports a right hash\n join as keeping its probe order even when it moves unmatched rows, so\n the walk stops at any hash join that is not an inner join. The\n null-aware anti join stops it, so an order Spark reports above NOT IN\n no longer reaches the aggregates above the outer join.\n- doConvert sends the longest prefix of the streamed order that Comet can\n express, since declaring any order keeps the probe rows in place, and\n falls back only when it cannot express the first key.\n- SortedInputExec\u0027s doc names the top-k sort as another consumer, and the\n CometHashJoin comment names the null-aware anti join as the exception.\n- Tests: dwsmith1983\u0027s NOT IN repro, a residual join condition, a\n descending, nulls last streamed order with null keys, the prefix case,\n the aggregate test\u0027s join and exchange checks, and planner tests for\n the walk and the planned join type.\n\n* fix: send no streamed order when a Comet join that may move rows produces it\n\nA cache reports the order of the plan that wrote it, so the output of a\nnative null-aware anti join, an outer hash join whose probe order the\nnative planner did not declare, or a nested loop join can reach an outer\nhash join as a JVM input whose reported order does not hold. Declaring\nthat order put the aggregates above the join in sorted mode over unsorted\nrows, which split groups.\n\nCometHashJoin.doConvert now walks down the streamed side, through\nCometScanWrapper and into the plan that wrote any cache it reads, and\nsends no order when such a join produces it with no sort in between.\n\n* fix: walk an adaptive cache\u0027s final plan and keep the streamed order through semi joins\n\nReview feedback on the streamed order that a native outer hash join declares on its probe input.\n\nThe serde walk that decides whether a cache or checkpoint can be trusted now:\n- reads the final plan of an adaptive cached plan, as Spark\u0027s cache scan does, since the\n adaptive plan reports no order itself and the walk stopped there;\n- follows only the streamed side of a hash join, and counts an outer join as declared only\n when it sent its order and its probe input is a scan in its own native plan;\n- no longer trusts a leaf that reports an order unless it is a scan or a range, which covers\n an RDD checkpoint whose plan is gone.\n\nThe planner now walks through right semi, anti and mark hash joins as well as inner ones, since\nDataFusion builds their probe indices in ascending order whether or not the probe declares an\norder. Only a right outer join depends on the declaration.\n\nTests cover a cached NOT IN with AQE off and on, through Spark\u0027s cache scan and through the native\none, a checkpointed NOT IN, an outer join over a cache written by a declared outer join and by one\nover a sample, an outer join over a semi join, a NOT IN below an inner join\u0027s build side, and a\nbroadcast outer join over a sorted cache with default settings. The test that compared Comet with\nitself now compares against the rows.\n\n* docs: describe how native hash joins keep the streamed order\n\nAdd a Hash Join Output Order section to the operator compatibility page: native hash joins keep the\nstreamed side\u0027s order Spark relies on, an outer join whose first streamed sort key Comet cannot\nexpress falls back to Spark, and where the order might not hold Comet declares nothing.\n\n* fix: keep the streamed order to expressible keys and stop CometSampleExec reporting one\n\nSpark\u0027s SampleExec reports no output ordering, so CometSampleExec must not report its child\u0027s:\na cache written by a join over a sample carried an order that nothing keeps. The streamed order\nnow stops at a key whose type the native sort does not accept, as for a collated string,\nSortedInputExec keeps its input\u0027s equivalence properties, and the contributor guide says that an\noperator reporting an ordering must keep it natively.\n\n* test: cover the streamed order for decimal, string and timestamp keys\n\nAlso assert that the aggregate test\u0027s join sent the order and emits its groups sorted, and\nreplace the sample test with one for a cache that reports no order.\n\n* test: compare the key-type rows with Spark\u0027s as a multiset\n\n* test: make the aggregate test fail when the join\u0027s order is not declared\n\nWith 40 keys of 50 rows each, a join that moved its unmatched rows still fed the aggregate\nthe keys in first-seen order, so the test passed without the declaration. Add a layout of\n1000 keys with two rows each, where a batch holds alternately matched and unmatched keys, and\nassert that the serde sent the order and that each partition comes back in key order.\n\n* fix: declare the order of a native range for the joins above it\n\nSpark\u0027s RangeExec reports id ascending, or descending for a negative step, and Spark drops a sort\nabove an outer hash join on that basis. The native range is a leaf with no JVM input, so the\nplanner declared no order on the join\u0027s probe input and the join moved its unmatched rows.\nLazyMemoryExec::add_ordering leaves the cached output ordering of the plan properties stale in\nDataFusion 55, so the planner wraps the range in a SortedInputExec instead, which now reports its\ninput\u0027s metrics.\n\n* refactor: name the streamed side of a hash join once\n\n* test: cover a native range and share the cache test helpers\n\nMove partitionsInOrder, keyOrder, checkEachKeyGroupedOnce and cachedPlanOf to CometTestBase so\nboth suites use one copy, and set up the sorted cache the join tests share once.\n\n* test: drop the imports the shared test helpers left unused\n\nMoving partitionsInOrder and keyOrder to CometTestBase left\nInterpretedOrdering and SortDirection unused in CometJoinSuite, which\nscalafix and the strict Scala warnings build reject.\n\n* test: cover a streamed double order with signed zeros and NaN\n\nAn outer hash join over a streamed side sorted by a double key with\n-0.0, 0.0, NaN and the infinities keeps its order, groups each key once,\nand returns Spark\u0027s rows for a top-k above it. The planner declares the\norder on the key normalized as the native sort normalizes it, and the\ntop-k runs in a native plan of its own over the join\u0027s output.\n\n* fix: send a streamed order that starts with an array or a struct\n\nThe check on the streamed order\u0027s keys took only the scalar types that\nthe native sort accepts for a single column, so a join whose streamed\norder starts with an array or a struct went back to Spark. exprToProto\ntakes those keys, and declines a nested key under a non-default null\norder itself. The check was there for the collated string, so it now\nrejects a key that holds one, at the top level or nested, as\nsupportedSortType does for a sort.\n\n* fix: declare the order of a sorted bucketed table read by the native scan\n\nWith spark.sql.legacy.bucketedTableScan.outputOrdering on, Spark reports\na sorted bucketed table\u0027s sort columns as the scan\u0027s order when every\nbucket has one file. The serde trusted that order and sent it, but the\nplanner declared an order only over a JVM input, so an outer hash join\nover the native scan moved its unmatched rows after Spark had dropped\nthe sort above it. A scan of files reads the files of its one file group\none after the other, so the planner now counts it as a source whose\norder holds.\n\n* docs: qualify where native hash joins keep the streamed order\n\nThe operator compatibility guide now says the order is kept where Comet\ncan show that it holds, and points to #6922 for the caches it still\nleaves undeclared. The contributor guide says what a leaf that reports\nan ordering has to do natively.\n\n* fix: drop the native NOT IN order handling that the ordered-side fallback makes unreachable\n\nComet now leaves a null-aware anti join over an ordered streamed side in Spark\n(#6810), so a native NOT IN never reports an order and its rows cannot reach an\nouter hash join with a false one. Remove the walk\u0027s null-aware case, the tests that\nneeded a native NOT IN to write a cache or checkpoint, and the documentation that\nnamed NOT IN as a source of a false order." }, { "commit": "0c7c5d69db8a4b491088a1e8c862e3d4cccedb20", "tree": "c8b6f6215c05ecf24f93930cb8c1c93fd935d7a4", "parents": [ "2a2aa05404337919a214598f2873dba08e991ddb" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Mon Oct 19 16:19:01 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 19 16:19:01 2026 +0000" }, "message": "fix: serialize large-offset string and binary vectors with 32-bit offsets (#6371)\n\n* fix: serialize large-offset string and binary vectors with 32-bit offsets\n\nUtils.getFieldVector rejects LargeVarCharVector and LargeVarBinaryVector,\nso Utils.serializeBatches failed on a batch holding either, although a\nPyArrow UDF can produce them and Comet reads them elsewhere.\n\ngetBatchFieldVectorsWithProviders now writes a VarCharVector or\nVarBinaryVector copy of such a column, failing only when its data does\nnot fit 32-bit offsets. Every reader of the serialized bytes then gets\nthe Utf8 or Binary type the column\u0027s Spark type maps to, and the batches\nof one broadcast share a schema whichever offset width each arrived with.\ngetFieldVector, isSupportedFieldVector and the FFI export are unchanged.\n\n* fix: release the vectors a failed batch assembly made, and test offsets past zero\n\ngetBatchFieldVectorsWithProviders keeps the vectors it makes, the 32-bit\ncopies of large-offset columns and the materialized ConstantColumnVectors,\nand closes them if a later column fails. Only the root serializeBatches\nbuilds afterwards would have released them, so a failure left them\nallocated from CometArrowAllocator. The batch\u0027s own vectors stay with their\nowner.\n\nnarrowOffsets passes the validity length as a Long, which -Pstrict-warnings\nrequires, explains why the validity bitmap is copied from bit 0, and points\nat #6933 for large-offset children of nested columns.\n\nUtilsSuite covers offsets that start past zero, empty large-offset vectors\nwith no offset buffer, and the cleanup after a later column fails, and\nbuilds its vectors with fill throughout.\n\n* address review: release every vector a failed write made, and tidy the large-offset tests" }, { "commit": "2a2aa05404337919a214598f2873dba08e991ddb", "tree": "ba5716280ed2dd814540132edfd4549577e2f8b1", "parents": [ "4ab113090d8746d265789fb1bcf4872b0f6e1821" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Mon Oct 19 16:14:13 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 19 16:14:13 2026 +0000" }, "message": "perf: write every partition of a JVM shuffle spill in one native call (#7065)\n\n* perf: write every partition of a JVM shuffle spill in one native call\n\nThe sort-based JVM columnar shuffle writer wrote each spill, and the final\nfile, with one writeSortedFileNative JNI call per partition. Every call\ndecoded the schema, copied the row address and size arrays, opened the\nspill file, allocated batch-size builders, pre-encoded the IPC schema and\nwrote its own blocks, so a map task paid that partitions x (spills + 1)\ntimes.\n\nSpillSorter now makes one writeSortedPartitionsNative call per spill.\nNative code reads the sorter\u0027s sorted packed record pointers through a\npage table, converts every partition with one reused encoder, buffers the\nencoded blocks, and returns each partition\u0027s length and checksum. The\noutput is byte-identical to the per-partition calls, which the bypass\nmerge writer still makes.\n\nAdds a sorted_spill group to the row_columnar bench and\nCometColumnarShuffleWriteBenchmark.\n\nPart of #5905 (J3).\n\n* test: pin dictionary changes and decoded rows in the sorted spill tests\n\nThe per-spill test compared the output byte for byte only for an Int64\ncolumn, so the batch schema and block writer that the encoder now keeps\nacross partitions were never exercised when a string column switches\nbetween a dictionary and a plain array. A new test writes such a spill,\ncompares it with writing each partition on its own, and decodes it.\n\nThe SpillSorterSuite test checked partition lengths and checksums against\nthe files, which a page table mistake that read valid rows from the wrong\npage would still pass. It now decodes every partition back to its rows.\n\nAlso fixes the strict Scala warnings build, where the new benchmark\nwidened an Int row count to Long, and rewords the JVM shuffle guide\u0027s\nwrite path, which read as one native call per partition.\n\n* test: run the dictionary change test with zstd offsets as lengths\n\nWith zstd, a block writer built for a plain string column stores its\noffsets as lengths and one built for a dictionary column does not, so\nthe writer the encoder keeps across partitions changes the bytes of a\nblock. The test now runs that codec as well as lz4.\n\n* refactor: simplify the sorted row encoder and the spill metrics\n\nSortedRowEncoder kept the batch schema twice, once to reuse it and once\nnext to the block writer to notice that it changed. It now keeps the\nschema only with the block writer, compares the column types against it,\nand builds both when they change. The encoder also owns the output file:\nencode_batch writes the buffered blocks out once they reach 1 MiB, and\nfinish writes the rest and returns the encode time, so neither caller\nhas to remember the final write. Its builders are sized for the rows the\ncall encodes, at most one batch.\n\nprocess_sorted_partitions splits the record pointers into partitions\nwith chunk_by, and the benchmark groups its records the same way.\n\nSpillSorter no longer routes a spill\u0027s metrics through a throwaway\nShuffleWriteMetrics, which only existed for the per-partition calls. It\nreports the same values to the task\u0027s metrics directly.\n\nbytes is only used by tests now, so it moves to dev-dependencies. The\ntests reuse long_values and createSpillSorter, and SpillSorterSuite reads\neach spill file once.\n\n* refactor: rename SpillSorter.writeSortedFileNative and document the output buffer\n\nRename the spill-wide write to writeSortedPartitions so that\nwriteSortedFileNative only names the per-partition native call, document that\nthe encoder\u0027s output buffer is not reserved from a memory pool, and explain in\nthe benchmark\u0027s scaladoc why it is separate from CometShuffleBenchmark." }, { "commit": "4ab113090d8746d265789fb1bcf4872b0f6e1821", "tree": "ee433305298326bc96c77d3fa6dcaab013ade5e5", "parents": [ "115058d3cf70cb4e0b0c7af836bd2e5b8fe5e4f3" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Mon Oct 19 15:12:47 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 19 15:12:47 2026 +0000" }, "message": "ci: give the Delta contrib Maven cache its own key prefix (#7111)" }, { "commit": "115058d3cf70cb4e0b0c7af836bd2e5b8fe5e4f3", "tree": "c475193098cc64ad51d6267a2ba0b5075897e626", "parents": [ "e3503efa6aac65fc98a8d2cdfb26ecc70189d5d9" ], "author": { "name": "LinSimon-901101", "email": "bcs113118@gm.ntcu.edu.tw", "time": "Mon Oct 19 14:21:25 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 19 14:21:25 2026 +0000" }, "message": "fix: preserve Parquet root field IDs after nested pruning (#7081)\n\n* fix: preserve Parquet root field ids after nested pruning\n\n* test: cover filter-only field ids and keep schema lookup linear\n\n* fix: explicitly widen Parquet field IDs for strict warnings\n\n* docs: remove fixed Spark 3.4 Parquet field-ID limitation" }, { "commit": "e3503efa6aac65fc98a8d2cdfb26ecc70189d5d9", "tree": "c04b3b77dbb19685e399fa00a709dc0a69323efb", "parents": [ "80b518173d0631d31028b732b385c68f69ace326" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Mon Oct 19 14:12:04 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 19 14:12:04 2026 +0000" }, "message": "perf: start bypass shuffle partition writers on small pages that grow (#7059)\n\n* perf: start bypass shuffle partition writers on small pages that grow\n\nThe bypass-merge JVM columnar shuffle writer gives every partition its own\nCometDiskBlockWriter, and each one held a page of Spark\u0027s page size as soon\nas it buffered a row. A task\u0027s memory holds only 16 to 32 such pages, so a\nshuffle with more partitions than that made the writers spill each other\nabout one row at a time: REPARTITION(64) over 3M rows wrote 2.48 GB in\n266 s instead of 156 MB in about a second.\n\nEach partition writer now starts a batch on a 64 KiB page and doubles each\nfurther page up to the old page size, so a partition costs kilobytes until\nits rows need more, and a batch takes a logarithmic number of pages, well\nbelow the 8,192 pages Spark allows a task. The sort-based writer keeps\nSpark\u0027s page size.\n\nCloses #7058.\n\n* test: cover growing pages under memory pressure; update manyPartitions comment" }, { "commit": "80b518173d0631d31028b732b385c68f69ace326", "tree": "e7ab8c8a1801856995d82187d0b7417becfe31ca", "parents": [ "85b9e39a09a24728b5a35c15fcdcfe9cf816e23f" ], "author": { "name": "ChenChen Lai", "email": "72776271+0lai0@users.noreply.github.com", "time": "Mon Oct 19 13:07:46 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 19 13:07:46 2026 +0000" }, "message": "refactor: share window frame bound offset serialization between lower and upper bounds (#7107)" }, { "commit": "85b9e39a09a24728b5a35c15fcdcfe9cf816e23f", "tree": "198406847b9511d5672ab915f25b829317d044df", "parents": [ "aacb2e7441aa8fb02643163a379e94ffe63b7a7e" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Oct 19 12:33:37 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 19 12:33:37 2026 +0000" }, "message": "build(deps): bump actions/setup-python from 6 to 7 (#7046)\n\nBumps [actions/setup-python](https://github.com/actions/setup-python) from 6 to 7.\n- [Release notes](https://github.com/actions/setup-python/releases)\n- [Commits](https://github.com/actions/setup-python/compare/v6...v7)\n\n---\nupdated-dependencies:\n- dependency-name: actions/setup-python\n dependency-version: \u00277\u0027\n dependency-type: direct:production\n update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "aacb2e7441aa8fb02643163a379e94ffe63b7a7e", "tree": "02b2da3c7b1e667b090e3d93c82e08c0d7d308fa", "parents": [ "498956132ced6f4e45453841aeae4e56e8e98e61" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Mon Oct 19 09:12:56 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 19 09:12:56 2026 +0000" }, "message": "fix: count grouped integer SUM state in the aggregate\u0027s memory reservation (#6370)\n\nThe three SumInt*GroupsAccumulator impls returned size_of_val(self) from\nGroupsAccumulator::size(), which covers the Vec headers but not the\nper-group sums (16 bytes per group, plus has_all_nulls for Try).\nDataFusion sizes a hash aggregate\u0027s reservation from these sizes, so\ngrouped integer sums reserved far less than they held. Count the\ncapacity of the state vectors, as SumDecimalGroupsAccumulator does.\n\nThe Accumulator impl for SparkBloomFilter had the same mistake and left\nout its bit array, so include that too." }, { "commit": "498956132ced6f4e45453841aeae4e56e8e98e61", "tree": "1d63d2bd39026bdb5a099d17baeff01f8827042d", "parents": [ "4a8a9496d119341b6f8ea94959312340c353583c" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Mon Oct 19 09:12:19 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 19 09:12:19 2026 +0000" }, "message": "fix: build release jars from the same commit as the native libraries (#6377)\n\n* fix: build release jars from the same commit as the native libraries\n\nbuild-release-comet.sh built the native libraries in Docker from the -b\nbranch but built the jars in the local checkout the script was run from,\nso a release could combine code from two branches (plus any local\nchanges), and each container resolved the branch on its own.\n\nResolve -b to one commit in a fresh clone, pass that commit to both\nbuilder containers, and build the jars in the clone. The local checkout\nis no longer built or cleaned.\n\n* Build the docker images from the clone, and tag the commit that was built\n\nThe docker build context and cargo.config were still taken from the\nlocal checkout, so local edits to the Dockerfile, the container\nentrypoint or cargo.config could change the release. Take them from the\nclone at the resolved commit instead. The script itself is now the only\nfile used from the local checkout, so the docs say to run it from a\ncheckout of the release branch.\n\nThe tag step reset to the tip of the release branch, which is not the\ncommit that the jars were built from if the branch moved during the\nbuild. Tag the commit that the script prints instead, and after a failed\nvote repeat from the change log step, so that the next release candidate\nis rebuilt before it is tagged.\n\n* Drop the ls-remote ref check; the clone and checkout already reject a bad ref\n\nCreate the cleanup flag before the clone so a bad ref also removes the temporary clone." }, { "commit": "4a8a9496d119341b6f8ea94959312340c353583c", "tree": "f5d1f01463867da936fa30c6db83e90b5a663b67", "parents": [ "d0f367dc02520fe68f524db22083cc0af6e58bca" ], "author": { "name": "Parth Chandra", "email": "parthc@apache.org", "time": "Sun Oct 18 22:09:21 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Oct 18 22:09:21 2026 +0000" }, "message": "fix: fall back to Spark when Iceberg reports an output ordering (#7043)\n\n* fix: fall back to Spark when Iceberg reports an output ordering\n\nComet has no native k-way merge for sorted Iceberg data. If Iceberg\nreports an ordering, Spark will drop the Sort above the scan, and the native\nreader would return unsorted rows. Stay on Spark in that case.\n\nFallback requested in https://github.com/apache/datafusion-comet/pull/5331#pullrequestreview-5523559500\n\n* address review comments\n\n* test: clarify comments in Iceberg ordering fallback tests\n\n* docs: list the Iceberg output ordering fallback in current limitations\n\n---------\n\nCo-authored-by: Andy Grove \u003cagrove@apache.org\u003e" }, { "commit": "d0f367dc02520fe68f524db22083cc0af6e58bca", "tree": "3ba34b9d15608875afd72d2b6a0b11306cbe00fd", "parents": [ "0d0a1c6fe76d660492c40de3403e45e06abbe433" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Sun Oct 18 18:13:37 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Oct 18 18:13:37 2026 +0000" }, "message": "test: write ancient dates with CORRECTED rebase mode in the DATE RANGE window test (#7103)" }, { "commit": "0d0a1c6fe76d660492c40de3403e45e06abbe433", "tree": "8f80699c45c1088b67f6307f06497533c943a064", "parents": [ "c9f56176cfad3a78d0eb6cdf360831c44013a27b" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Sun Oct 18 17:46:40 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Oct 18 17:46:40 2026 +0000" }, "message": "fix: size the native runtime for standalone executors without spark.executor.cores (#6368)\n\n* fix: size the native runtime for standalone executors without spark.executor.cores\n\n* address review: fall back to available processors for unknown masters, skip the warning under COMET_WORKER_THREADS, share the YARN/Kubernetes check, update the contributor guide\n\n* style: apply spotless" }, { "commit": "c9f56176cfad3a78d0eb6cdf360831c44013a27b", "tree": "85646d24c0cb4e47d86ed8d7dc68c5a0a985de34", "parents": [ "4ef103624e6f5f87072ae1e46b85541fb559fda1" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Sun Oct 18 17:09:34 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Oct 18 17:09:34 2026 +0000" }, "message": "docs: consolidate Parquet and Iceberg documentation into one page each (#7090)\n\n* docs: consolidate Parquet and Iceberg documentation into one page each\n\nParquet compatibility, scan tuning and known issues were spread across\ncompatibility/scans.md, tuning/scans.md and\ncompatibility/parquet-iceberg-known-issues.md. Iceberg reads, writes and\nknown issues were in iceberg.md, iceberg-writes.md and the same\nknown-issues page.\n\nAdd parquet.md (reading, writing, tuning, known issues) and fold\niceberg-writes.md and the Iceberg known issues into iceberg.md. Move the\nobject store and credential entries to the S3 and Azure limitations in\ndatasources.md. Group Parquet, Iceberg, Delta Lake and S3 credential\nproviders under a Data Sources section of the user guide, link the\ntuning guide to the Parquet and Iceberg tuning sections, and add\nredirects for the removed pages.\n\n* docs: address review feedback on Parquet and Iceberg pages\n\n- Revert the prettier reformat of expressions.md, keeping only the link change\n- Move the field-id read entry to Parquet\u0027s Known Issues\n- Redirect removed pages to the matching section, keeping deep-link anchors\n- Point both Known Issues sections at the S3 and Azure limitations\n- Rename the S3 and Azure Limitations headings for stable anchors\n- State that the native Parquet writer is for testing only and list its scope\n- Link the duplicated scan-time error entries instead of repeating them" }, { "commit": "4ef103624e6f5f87072ae1e46b85541fb559fda1", "tree": "91e64ef9a5a43b8552f37d93f01a5c00ee93c4dd", "parents": [ "d60cc1dd5fe1223309ee1607811951b1dcd775cb" ], "author": { "name": "Jaden Fix", "email": "163031992+jadenfix@users.noreply.github.com", "time": "Sun Oct 18 15:52:09 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Oct 18 15:52:09 2026 +0000" }, "message": "fix: handle NullType struct fields in JVM columnar shuffle (#7099)\n\nThe field-major struct conversion in the JVM columnar shuffle\u0027s\nrow-to-Arrow path (`append_struct_fields_field_major` for a top-level\nstruct column and `append_nested_struct_fields_field_major` for a struct\ninside a struct) had no `DataType::Null` arm, so a struct such as\n`named_struct(\u0027x\u0027, a, \u0027n\u0027, null)` hit `unreachable!` and failed the\nquery. `append_field` already handled `Null` by appending a null per\nrow; both functions now do the same.\n\nAdds Rust unit tests that lay out Spark UnsafeRows for a top-level and a\nnested struct with a NullType field, and a Scala regression test in\nCometColumnarShuffleSuite covering round-robin and hash partitioning." }, { "commit": "d60cc1dd5fe1223309ee1607811951b1dcd775cb", "tree": "541740ffad4b51d5344375b92d56fbe008e4a965", "parents": [ "642660c2d861c0abf05de48afc5823ecc4cbc2af" ], "author": { "name": "Peter Lee", "email": "peterxcli@gmail.com", "time": "Sat Oct 17 18:29:28 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Oct 17 18:29:28 2026 +0000" }, "message": "perf: optimize JVM columnar-to-row conversion (#5496)\n\n* perf: optimize JVM columnar-to-row conversion\n\n* test: cover dictionary-encoded decimals in the direct converter\n\nBuild dictionary vectors directly, since the native scan hands the converter\nplain vectors, and check compact and wide decimals against UnsafeProjection\non both the fixed-width and the general path. State the precision/scale\nassumption that lets the decimal fast paths skip changePrecision. Move the\ntuning section to the new transitions page and document the eager\nwhole-batch conversion and the retained on-heap batch buffer.\n\n* test: avoid implicit numeric widening in CometC2RIsolatedBench\n\nDecimal.createUnsafe takes the unscaled value as a Long, and the strict\nScala warnings build rejects the implicit Int-to-Long widening of the\nmodulo results. Take the modulus by a Long literal instead.\n\n* docs: note when direct columnar-to-row conversion is slower\n\nThe tuning guide only described the direct converter\u0027s upside. Say that\nschemas with strings and no expensive decimals can get slower with it\nenabled, give the mixed long, int, double, string benchmark numbers, and\nadvise measuring a workload before turning it on.\n\n---------\n\nCo-authored-by: Andy Grove \u003cagrove@apache.org\u003e" }, { "commit": "642660c2d861c0abf05de48afc5823ecc4cbc2af", "tree": "b605cc06b139b7c0128cd85b0a2659481bb32b88", "parents": [ "14748af84c3754956a2795e90c31fb4633b91e77" ], "author": { "name": "dustin", "email": "dwsmith1983@users.noreply.github.com", "time": "Sat Oct 17 16:43:07 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Oct 17 16:43:07 2026 +0000" }, "message": "ci: label Delta pull requests with area:Delta (#6965)\n\nThe rule lists the Delta paths explicitly, since a *[Dd]elta* glob\nwould also match the Iceberg delta-write code." }, { "commit": "14748af84c3754956a2795e90c31fb4633b91e77", "tree": "4fc6e347f44e6b9d29c5e44df44dcd8bb7853ee3", "parents": [ "27f437a36246a36bdae06762ef2a6f6cb5b5ad70" ], "author": { "name": "ChenChen Lai", "email": "72776271+0lai0@users.noreply.github.com", "time": "Sat Oct 17 14:50:41 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Oct 17 14:50:41 2026 +0000" }, "message": "fix: return errors instead of panicking in native planner (#7087)" }, { "commit": "27f437a36246a36bdae06762ef2a6f6cb5b5ad70", "tree": "b16c5d85d53c31ecb93a62e5b08221de0dcc6d08", "parents": [ "a0cbb0fa06515acd5647538025a3a894be71b9bd" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Sat Oct 17 13:20:17 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Oct 17 13:20:17 2026 +0000" }, "message": "refactor: move MergeRows, Explode, aggregate, join and window planning into dedicated modules (#7014)\n\n* refactor: move MergeRows planning into dedicated module\n\n* refactor: move Explode planning into dedicated module\n\n* refactor: move hash aggregate planning into dedicated module\n\n* refactor: move join planning into dedicated module\n\n* refactor: move window planning into dedicated module\n\n* docs: point contributors at OperatorBuilder and refresh planner pointers\n\nUpdate the \u0027Update the Planner\u0027 step of the new-operator guide to describe\nadding an OperatorBuilder under planner/ and registering it, instead of adding\nan arm to create_plan. Refresh the window debugging example to match\nWindowBuilder::build, and point the make_all_fields_nullable comments at\nplanner/aggregate.rs." }, { "commit": "a0cbb0fa06515acd5647538025a3a894be71b9bd", "tree": "0cf061e9fcdeb3e490835f94bc67aa173369561c", "parents": [ "79ab485ff4529cb618618a67934953b8690db741" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Sat Oct 17 03:41:59 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Oct 17 03:41:59 2026 +0000" }, "message": "feat: bump iceberg-rust and write float and double identity partitions natively (#6851)\n\n* feat: bump iceberg-rust and write float and double identity partitions natively\n\napache/iceberg-rust#3327 gives `PrimitiveLiteral` an equality and ordering that\nfollow Java\u0027s `Float.compare` / `Double.compare`, so iceberg-rust\u0027s writers keep\n-0.0 and 0.0 partitions apart, as iceberg-java does. Move the pin from af1da4c to\n1f3bc34, the merge commit of that PR, and drop the fallback that #6531 added for\nfloat and double partition fields (#6138).\n\nTwo other upstream changes in the range need code changes:\n\n- apache/iceberg-rust#3145 makes `FileScanTaskDeleteFile`\u0027s fields private\n behind a validating builder, and holds the deletion-vector offset and size as\n u64. The planner builds delete files through the builder and rejects negative\n coordinates, and the scan sizes a delete file by rebuilding it.\n- apache/iceberg-rust#3323 fixes iceberg-rust\u0027s `day` for pre-epoch timestamps\n (apache/iceberg-rust#3315). Comet computes those partition values itself, so\n only the canary that pins iceberg-rust\u0027s values changes.\n\n* test: cover NaN and same-typed decimal identity partitions\n\nThe lifted gate also admits NaN partition values, so the signed-zero test\nwrites NaN and -NaN through both writers too. The bump changes how\niceberg-rust writes a named Avro type that two partition fields share, so a\nnew test writes two decimal identity partitions of the same type, in both\nencodings, and compares them with iceberg-java.\n\nThe detection test now uses plain tuples. Scala 2.12 flagged the adapted\nargument list as an error under -Pstrict-warnings.\n\n* refactor: share the delete-file count conversion and trim the new tests\n\n- One closure converts a delete file\u0027s record count and blob coordinates\n from the proto\u0027s int64, instead of three copies of the same block.\n- The planner test keeps the cases Comet\u0027s own conversion decides, and\n drops the two that only exercised iceberg-rust\u0027s validation messages.\n- `committedPartitions` sits after `partitionDirs`, replaces the far-date\n test\u0027s local copy of the same query, and is read once per assertion.\n- The detection test uses the suite\u0027s `createTable` table, so\n `assertSupportLevelIs` keeps its original signature.\n\n* test: check the clustered writer rejects interleaved signed zeros like iceberg-java\n\n* test: cover a missing or negative deletion vector record count; drop the fixed day difference from the guide\n\n* Update native/core/src/execution/operators/iceberg_scan.rs\n\n---------\n\nCo-authored-by: Matt Butrovich \u003cmbutrovich@users.noreply.github.com\u003e" }, { "commit": "79ab485ff4529cb618618a67934953b8690db741", "tree": "656e39dcd37286d74d5664adc7896294e473e5cb", "parents": [ "f180d65f18d058ba35528c945153cf9be146d04b" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Sat Oct 17 03:17:32 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Oct 17 03:17:32 2026 +0000" }, "message": "docs: list known Parquet and Iceberg issues and add a data source audit log (#7052)\n\n* docs: list known Parquet and Iceberg correctness issues and add a data source audit log\n\nAdd a compatibility page that lists the open issues in which a Parquet or\nIceberg read or write gives a different result from Spark without falling\nback, with links to the issues, and point to it from the compatibility\noverview, the Parquet scan page, the Iceberg guide and the Iceberg writes\npage. The datetime rebasing item moves from scans.md to the new page.\n\nCorrect the Iceberg writes page\u0027s claim that the float/double null-count\ndifference cannot change IS NULL pruning: iceberg-java\u0027s counts are the\nwrong ones (apache/iceberg#17561).\n\nStart a Data Source Audits section in the contributor guide, next to the\nexpression audits, recording the nested-type audits of the native Parquet\nand Iceberg scans and writers.\n\n* docs: add Parquet and Iceberg queries that fail where Spark succeeds\n\nAdd a section to the known issues page for the open issues in which a\nParquet or Iceberg read or write fails with an error where Spark, or\niceberg-java, succeeds: Parquet and Iceberg reads, object store and\ncredential settings that the native clients resolve differently from\nHadoop, and memory failures in the native writers.\n\nThe page becomes \"Known Issues in Parquet and Iceberg Reads and Writes\".\nThe existing sections move under \"Results that differ from Spark\" and keep\ntheir heading anchors, and the links from the other pages use the new\ntitle.\n\n* docs: drop the Spark 4.1 NullType read limitation\n\nparquet-rs accepts the UNKNOWN logical type on any physical type since\napache/arrow-rs#9855, which the parquet 59 crate includes, so reading a\nSpark 4.1 Parquet file with a NullType column no longer fails (#4199).\n\n* docs: address review feedback on the known issues page\n\nDrop #6264 and #6837, which are fixed on main. Scope the datetime rebase\nentry to the native Parquet scan and say the Delta scan rebases. Add the\nignored Parquet writer properties from #5304 and note that the row group\nfailure did not reproduce. Explain how apache/iceberg#17320 covers the\nidentity partition source case. Reword several entries, break the list of\nSpark SQL tests into one bullet each, and note that spark.comet.scan.enabled\nis a testing setting." }, { "commit": "f180d65f18d058ba35528c945153cf9be146d04b", "tree": "99e1cfde9adb41ce0e6ac4d5f45c25e891d07979", "parents": [ "d44366230aed8aedd9e7d9fca24fdfe9a668a184" ], "author": { "name": "Oleks V", "email": "comphead@users.noreply.github.com", "time": "Sat Oct 17 00:43:33 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Oct 17 00:43:33 2026 +0000" }, "message": "perf: reuse validity buffers and use unary kernels in native expressions (#6953)\n\n* perf: reuse input validity buffers instead of rebuilding null masks per row\n\nfrom_json, regexp_extract_all and the split_sql array variants appended one\nvalidity bit per row to a Vec\u003cbool\u003e or a builder, although the output is null\nexactly where the input is. Reuse the input null buffer instead.\n\narray_insert derives the same mask with NullBuffer::union of the list and\nposition nulls, and nested_extrema in array_extrema builds its indices with a\nUInt32Builder instead of collecting a Vec\u003cOption\u003cu32\u003e\u003e.\n\nAdds a test for the array/array split_sql path, which had none.\n\n* perf: use unary and try_unary for float and decimal integral casts\n\nThe float and decimal to integral cast macros built an Option or a\nResult\u003cOption\u003e per element to feed FromIterator. The legacy conversion cannot\nfail, so it now uses PrimitiveArray::unary, and the ANSI conversion uses\ntry_unary, which skips null slots and returns the first overflow in index\norder, as before. cast_int_to_int_macro already has this shape. The decimal\nmacros also compute the power of ten once per array instead of once per value.\n\nunix_timestamp had three copies of a null-count branch that unary replaces.\n\nAdds a test that a value which would overflow under a null slot neither fails\nan ANSI cast nor reaches the result.\n\n* fix: run negative-scale decimal to integral casts in Spark\n\nThe native decimal to integral casts divide by 10^scale, which overflows for a\nnegative scale (only creatable with spark.sql.legacy.allowNegativeScaleOfDecimal\u003dtrue).\nA debug build panics on any batch that has a value, and a release build wraps the\ndivisor to zero, so the hoisted divisor also failed on an all-null batch that used\nto return nulls. CometCast now reports these casts as Unsupported in every eval\nmode, so Spark\u0027s own cast runs.\n\nThe legacy decimal arms also use try_unary with a closure that always returns Ok,\nso the division only runs for valid slots.\n\nAdds a CometNativeCastSuite test with a values table and an all-null table for all\nfour integral targets, and a docs note." }, { "commit": "d44366230aed8aedd9e7d9fca24fdfe9a668a184", "tree": "a3dde9c660b6b99b261e888340a049f978a32115", "parents": [ "46b9c6afe20ccb8f43ca747d87b18f408486b364" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Fri Oct 16 22:53:34 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 16 22:53:34 2026 +0000" }, "message": "feat: support PivotFirst aggregate for the optimized PIVOT fast path (#4798)\n\n* feat: support PivotFirst aggregate for the optimized PIVOT fast path\n\nWire the Spark-internal PivotFirst aggregate through Comet so that\nPIVOT queries which trigger Spark\u0027s two-phase fast path (12+ pivot\nvalues with an aggregate whose output type is in\nPivotFirst.supportsDataType) run natively instead of falling back.\n\nScaffolded with the implement-comet-expression skill.\n\n* simplify: address /simplify review of PivotFirst\n\n- Delegate value-type gate to Spark\u0027s `PivotFirst.supportsDataType` instead\n of maintaining a parallel list in the serde.\n- Wrap the plan-time pivot map/vector in `Arc` so per-group accumulator\n init bumps a refcount instead of deep-cloning.\n- Drop the redundant `pivot_values` field from the accumulator (only the\n map and `slots.len()` are read at runtime); the accumulator now derives\n its slot count from the shared map.\n- Use the shared `format_state_name` helper for state field names.\n- Delegate `evaluate()` to `state()` so the two paths share their slot\n materialization.\n- Walk each state column backwards in `merge_batch` and stop at the first\n non-null so we build at most one `ScalarValue` per slot per merge.\n- Drop the \"pad every IN clause to 12 values\" workaround from the SQL\n tests and the CometAggregateSuite test - the fast-path gate is\n supported-data-type only, no minimum count.\n\n* fix: match Spark\u0027s float pivot key semantics in PivotFirst\n\n* fix: fall back for PivotFirst pivot columns and lists Comet cannot match\n\nSpark finds a pivot value\u0027s slot in a Scala HashMap only for atomic pivot\ncolumns, and even there a binary value matches by reference, so no row ever\nmatches before Spark 4.2. Arrays and structs (including the struct that a\npivot on several columns builds) are matched through an interpreted ordering,\nand from 4.2 so are binary values and non-default collations. The native\nlookup reproduces none of that, so those pivot columns now fall back.\n\nSpark also sizes the aggregate buffer from the distinct pivot values, so a\nlist with duplicates (including 0.0 next to -0.0) made the native partial\naggregate emit more columns than the shuffle schema declares. Such lists fall\nback too.\n\nThe SQL fixture now asserts native execution for every query that should run\nnatively, covers each pivot column type that does, and checks each fallback.\n\n* perf: add a GroupsAccumulator for PivotFirst\n\nPivotFirst only had a row Accumulator, so grouped aggregation went through\nDataFusion\u0027s GroupsAccumulatorAdapter, which visits every group\u0027s state on each\nbatch, and the accumulator built a ScalarValue for every input row.\n\nThe grouped accumulator keeps every group\u0027s slots in one flat, group-major\nbuffer with a validity bitmap, and returns them as the list without copying.\nPivot values are looked up in a typed index built once per aggregate, with\nSpark\u0027s float rules in a new float_semantics::ieee_equality_key helper. The\nungrouped accumulator shares the index. A pivot list with a repeated value is\nrejected at construction, since the serde falls back for it.\n\n* test: add a benchmark for the optimized PIVOT plan\n\nCometPivotBenchmark pivots sum(v) over 4M rows on a string column with 4,\n32 or 128 values, grouped into 1K, 100K or 1M groups, and compares Spark, Comet\nwith PivotFirst disabled (so the aggregates using it fall back while the rest\nof the plan stays native) and Comet with PivotFirst native. The query selects\nevery pivot column into the noop sink, and before timing each Comet run is\nchecked against Spark\u0027s answer and for the plan its name claims.\n\n* perf: copy only the emitted groups out of PivotFirst\u0027s state\n\n`EmitTo::First(n)` split the cell buffer with `split_off`, which leaves the\noriginal allocation with the first part and copies the rest. The emitted\nbatch therefore held the buffer of every group, and the remaining groups\nwere copied, nearly doubling live memory for a partial emit of a few\ngroups. `EmitTo::take_needed` copies out whichever part is smaller.\n\n* refactor: run ungrouped PivotFirst through the grouped accumulator\n\nThe ungrouped accumulator kept a `ScalarValue` per slot, built one with\n`ScalarValue::try_from_array` for every matching input row, and repeated\nthe grouped accumulator\u0027s update, merge and state logic. It now wraps a\n`PivotFirstGroupsAccumulator` holding one group. `evaluate` reads that\ngroup\u0027s slots without taking them, because DataFusion requires that an\naccumulator can be evaluated more than once.\n\nThat leaves native code no row-based path for a value type the grouped\naccumulator does not store. The serde therefore checks the value type\nagainst the types the native aggregate stores rather than calling\n`PivotFirst.supportsDataType`, so a type that a later Spark version adds\nfalls back instead of failing the query.\n\n* test: cover ungrouped, empty, multi-batch and collated PivotFirst inputs\n\nPivotFirst.sql gains a pivot with no grouping column, the same pivot over\nno rows, and a pivot over 20000 groups whose input is repartitioned by the\npivot column, so that each group has several partial states to merge.\nPivotFirst_collation.sql checks on Spark 4.0+ that a UTF8_LCASE pivot\ncolumn falls back.\n\nEvery native query now also asserts that PivotFirst itself ran natively.\nThat was the only thing the two CometAggregateSuite tests checked that the\nfixture did not, so they are removed. The fixture\u0027s comments no longer name\na nonexistent PivotTransformer or threshold, or a cast the float query\ndoes not make." }, { "commit": "46b9c6afe20ccb8f43ca747d87b18f408486b364", "tree": "b1ec257229123357af6334ace5d8729dfe775cee", "parents": [ "c4ca20e8b6070e1b7482edda5cf6a9629de04fbb" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Fri Oct 16 20:50:42 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 16 20:50:42 2026 +0000" }, "message": "fix: stop building maps natively when the key type holds a FLOAT or DOUBLE (#6992)\n\n* fix: stop building maps natively when the key type holds a FLOAT or DOUBLE\n\nThe native map_from_arrays and map_from_entries tell keys apart by\ntheir bits, and they report Compatible for every key type. Spark\u0027s\nArrayBasedMapBuilder finds a duplicate FLOAT or DOUBLE key in a HashMap\nof boxed values, where NaNs with different bits are one key, and a\nduplicate struct or array key in a TreeMap in its SQL ordering, where\n-0.0 equals 0.0 too. From Spark 4.0 it also normalizes a top-level key\nbefore the lookup, unless spark.sql.legacy.disableMapKeyNormalization is\nset, so the two zeros are one key and map_from_entries stores 0.0.\nComet built maps that Spark rejects with DUPLICATED_MAP_KEY and kept\n-0.0 keys that Spark stores as 0.0.\n\nUntil the builders follow these rules natively, report any key type\nthat holds a float at any depth as Incompatible, on every Spark version.\nmap_from_entries then runs through the codegen dispatcher and\nmap_from_arrays falls back to Spark, as each already does for LAST_WIN.\n\nCloses #6549.\n\n* fix: route map_from_arrays float keys through the codegen dispatcher\n\nAddress review feedback on the float map key gate:\n\n- Mix CodegenDispatchFallback into CometMapFromArrays so FLOAT or\n DOUBLE keys and LAST_WIN stay in the Comet pipeline, as they do for\n map_from_entries.\n- Use expect_error_class for the dispatched DUPLICATED_MAP_KEY queries,\n and fix the array-key zeros query to read the row that holds zeros.\n- Rename the fixtures to map_builders_float_key_gate* so they do not\n collide with the ones in #6682.\n- Correct the MapKeySupport reason, which described the lookup as\n Spark normalizing keys, and trim the repeated explanation of the\n builder rules down to MapBuilderKeySupport.floatingPointReason.\n\n* test: cover the dispatched map builders under LAST_WIN with a repeated key\n\nThe dedup-policy fixtures turn the codegen dispatcher off, so nothing ran the\ndefault LAST_WIN path with a key that really repeats. Add\nmap_builders_float_key_last_win.sql, which runs map_from_arrays and\nmap_from_entries through the dispatcher with a repeated string key, NaNs that\ndiffer only in the sign bit, and 0.0 / -0.0, comparing the kept value and the\nstored key with Spark\u0027s on every version.\n\nAlso drop the claim that Spark normalizes map keys before a lookup from the\nMapKeySupport scaladoc and the lookup fixtures. Normalization is a builder\nbehavior from Spark 4.0, and the lookup compares with nanSafeCompareDoubles on\nevery version." }, { "commit": "c4ca20e8b6070e1b7482edda5cf6a9629de04fbb", "tree": "822c55a6fec6be6eaf0e5600c86a49e0a997df73", "parents": [ "e242944ec43faf76ad9e22217f22e57f7e801ef3" ], "author": { "name": "dustin", "email": "dwsmith1983@users.noreply.github.com", "time": "Fri Oct 16 20:27:11 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 16 20:27:11 2026 +0000" }, "message": "fix: fall back to Spark for NOT IN over an ordered streamed side (#6810)\n\n* fix: sort a native hash join\u0027s output when the probe order it reports is not kept\n\nSpark reports a hash join\u0027s streamed-side ordering and may drop a sort\nabove the join on that basis. DataFusion keeps unmatched probe rows of a\nright join in order only when it sees the probe input sorted, which it\ndoes not when that order comes from outside the native plan. The new\nHashJoin output_ordering field carries the ordering, and the native\nplanner adds a sort above the join when the join does not already\nproduce it.\n\n* fix: send a hash join\u0027s reported ordering to the native planner\n\nFor LeftOuter with BuildRight and RightOuter with BuildLeft, Spark reports\nthe streamed side\u0027s ordering, and DataFusion runs the join as a right join\nthat may emit unmatched probe rows after the matched ones. The serde now\nsends that ordering so the native planner can restore it, and falls back\nto Spark when the ordering cannot be sorted natively.\n\n* test: a native sort-merge join below a hash join adds no extra sort\n\nThe ordering the merge join produces reaches the hash join through the\nswapped right join and its projection, so the planner keeps it without\nsorting again.\n\n* fix: keep the streamed order of a null-aware anti join and widen the ordering tests\n\nA null-aware anti join runs unswapped, so DataFusion builds on Spark\u0027s\nstreamed side and concatenates its batches in reverse; it now sends its\nordering too and gets the same native sort. The tests cover multi-column\nand multi-batch orderings, the expression fallback, a natively sorted\nprobe, join metrics with a sort on top, and the join class per hint. The\ncompatibility guide describes how hash joins keep the order.\n\n* test: tighten the hash join ordering tests and correct the NOT IN docs\n\nThe null-aware anti join test now checks that its build spans several\nbatches, a metric check that held for any batch size is gone, and a test\ncomment claims only what the test checks. The guides say a NOT IN join is\nsorted too, and that only a NOT IN under OR, planned as an existence\nnested loop join, falls back to Spark.\n\n* test: a null-aware anti join\u0027s native output is sorted across build batches\n\nDataFusion builds a NOT IN join on Spark\u0027s streamed side and, on its hash\npath, concatenates those batches in reverse, so the planner sorts the\noutput. The test feeds two sparse-key batches and checks the order.\n\n* docs: say only a NOT IN combined with OR skips the native existence join\n\nA plain NOT IN runs as a native null-aware anti join; only one combined\nwith another predicate through OR becomes an existence join that falls\nback to Spark.\n\n* refactor: share the planner\u0027s sort-unless-ordered helper with the sort aggregate\n\nsort_unless_ordered now takes physical sort expressions. The hash join\nbuilds them at its call site, and the sort aggregate arm calls the same\nhelper instead of its own ordering_satisfy check and SortExec wrap.\n\n* test: check the hash join ordering field for every shape that sends one\n\nThe streamed order matrix and the null-aware anti join test assert that\nthe native op carries an output ordering. The separate test keeps only\nthe inner join check, since its outer half repeated a matrix case.\n\n* test: keep the hash join probe order through runtime filter wrappers\n\nWith runtime filters on, a probe-side filter or projection becomes CometFilterExec or CometProjectionExec before the join is built. Check that both keep the sort\u0027s ordering, so an outer hash join that reports its streamed ordering still adds no sort.\n\n* fix: fall back to Spark for NOT IN over an ordered streamed side\n\nComet runs a null-aware anti join unswapped in CollectLeft mode, so\nDataFusion builds on Spark\u0027s streamed side. On its hash map path the\nbuild batches are concatenated in reverse, and the unmatched rows come\nout in reverse batch order. Spark has already dropped any sort above\nthe join because the join reports the streamed side\u0027s ordering, so\nsortWithinPartitions returns unsorted partitions and a sort aggregate\nabove the join splits groups.\n\nSorting the native output instead needed about four times the memory\npool under fair_unified, so fall back to Spark when a null-aware anti\njoin\u0027s streamed side reports an ordering, until the DataFusion fix\n(apache/datafusion#26226) reaches Comet. This drops the HashJoin\noutput_ordering field and the planner sort from this branch." }, { "commit": "e242944ec43faf76ad9e22217f22e57f7e801ef3", "tree": "4a13bf81159cc46215ae491c3dc3d40e927b3472", "parents": [ "1bad7c9e5fc6438a71e8ff666745d676e38d99af" ], "author": { "name": "dustin", "email": "dwsmith1983@users.noreply.github.com", "time": "Fri Oct 16 16:05:55 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 16 16:05:55 2026 +0000" }, "message": "fix: keep unconverted DPP filters in a scan\u0027s canonical form (#6270)\n\n* fix: keep unconverted DPP filters in a scan\u0027s canonical form\n\nAQE canonicalizes a query stage from its exchange before the stage rules\nconvert DPP placeholders. Comet dropped a DPP filter from the scan\u0027s\ncanonical form while it was still a placeholder, so exchanges over scans\nwith different DPP filters compared equal and AQE reused one for both,\nlosing the other branch\u0027s rows.\n\nThe shared helper now drops only DynamicPruningExpression(TrueLiteral), as\nFileSourceScanExec does. CometNativeScanExec.doCanonicalize drops every DPP\nfilter from originalPlan, a stale copy no DPP rule rewrites, so an unused\nDPP scan still matches its twin.\n\nCloses #6264. Part of #6133.\n\n* docs: describe how a native scan keeps its DPP identity when canonicalized\n\nThe plan identity section still said CometNativeScanExec canonicalizes\noriginalPlan while removing unused dynamic pruning filters. It now strips\nevery DPP filter from that stale copy, and the scan\u0027s DPP identity comes\nfrom the top-level partitionFilters, where only\nDynamicPruningExpression(TrueLiteral) is dropped. Say both, and point a\nnew scan at the second rule with the reason it matters for AQE reuse.\n\n* fix: strip every DPP filter from the Delta scan\u0027s originalPlan when canonicalizing\n\nCometDeltaNativeScanExec kept a stale DPP placeholder in its originalPlan\ncopy once filterUnusedDynamicPruningExpressions stopped dropping\nplaceholders, so after AQE turned the live filter into TrueLiteral the\nscan no longer matched its twin and lost SPARK-32509 exchange reuse.\nDrop every DPP filter from originalPlan, as CometNativeScanExec does;\nruntimeFilters still carry the scan\u0027s DPP identity. Adds a Delta version\nof the SPARK-32509 reuse test.\n\n* test: check that DPP scan stages on a Delta table are not reused across branches\n\nPorts the #6264 UNION ALL test to the native Delta scan, so a Delta scan\nwhose live DPP filter differs from its twin\u0027s keeps its own shuffle\nstage and its own rows.\n\n* refactor: share the scan originalPlan canonicalization and DPP reuse check\n\nCometNativeScanExec and CometDeltaNativeScanExec now call\nCometScanUtils.canonicalizeOriginalPlan instead of each stripping DPP\nfilters from originalPlan inline, and the operator guide points new\nscans at it. The no-reuse-over-a-DPP-scan assertion lives once in\nCometTestBase and the Comet, Iceberg and Delta suites use it.\n\nThe CometExecSuite test that passed on main is replaced by one where\nboth branches use the same DPP filter: the shuffles must still be\nreused as in Spark, which fails if the canonical form keeps too much.\n\n* test: share the reused exchange helpers in CometTestBase\n\nassertExchangeReuseOver and assertNoReusedExchangeOverDppScan now call one\nreusedExchangesOver, and assertSameExchangeReuseAsSpark replaces the reuse\ncount comparison written out in CometExecSuite and CometDeltaNativeScanSuite.\n\n---------\n\nCo-authored-by: Andy Grove \u003cagrove@apache.org\u003e" }, { "commit": "1bad7c9e5fc6438a71e8ff666745d676e38d99af", "tree": "1ba35835f50d3efb01a275e1555d44701beb4529", "parents": [ "3a4a15940983e4bb9a9a74c816ee51e2cf6d9e17" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Fri Oct 16 16:00:05 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 16 16:00:05 2026 +0000" }, "message": "perf: make the native Iceberg writer\u0027s dictionary pre-pass 1.8x to 3.9x faster (#6937)\n\n* perf: make the native Iceberg writer\u0027s dictionary pre-pass 1.6x to 3.1x faster\n\nDictionaryChooser replays parquet-mr\u0027s first-page dictionary decision for\nevery column of a partition before its rows reach the writer. Read each\ncolumn\u0027s values through keys typed by its Arrow type instead of byte slices\nbehind a boxed closure, walk nested columns without boxed iterators, hash\neach string or binary value once, and reuse one pre-sized set per kind of\nkey from column to column. The decisions do not change.\n\nAdd a criterion benchmark over primitive, struct, array, map and nested\ncolumns and a test that reads every leaf type. Put a valid row ahead of the\nnull list in values_under_a_null_list_are_not_counted, so that it fails when\nvalues under a null list are counted.\n\nCloses #6914.\n\n* perf: walk flat columns without recursing and simplify the dictionary pre-pass\n\nFollow-up from self-review of the DictionaryChooser rewrite:\n\n- Give each column its own set, sized for its first page, instead of reusing\n one per kind of key. Only one column\u0027s dictionary is alive at a time again,\n and no column clears a table sized for another.\n- Key strings and binaries by their bytes, hashed by the set without a length\n prefix and compared by length and last byte first, so a run of equal values\n is never hashed.\n- Read values through small closures instead of a trait with four impls.\n- Visit a flat column\u0027s rows without recursing, drop struct levels that have\n no nulls, and treat a null buffer with no nulls as none.\n- Add a sorted shape to the benchmark, and a test that values under a null\n struct are not counted.\n\n* test: check the dictionary pre-pass on sliced batches\n\nThe clustered writer hands the chooser zero-copy slices of the batches it\nsplits, so batch_boundaries_do_not_change_the_choice now also cuts the corpus\nwith RecordBatch::slice. A new test slices a batch of nested columns, null at\nevery level, at an odd offset and checks it against the same rows gathered by\ntake_record_batch, as the fanout writer gathers them.\n\nThe doc comment of Bytes now says what it saves over plain \u0026[u8] keys.\n\n* refactor: read plain sizes in the dictionary pre-pass and drop a dead branch\n\nReaders return each value\u0027s plain-encoded size, so FirstPage no longer\ncarries the width, and Path::visit splits the first level once instead of\nchecking for no levels in two places. Drop the large and fixed-size list\nhalf of reads_every_leaf_type, which a_sliced_batch_is_read_only_within_its_slice\ncovers." }, { "commit": "3a4a15940983e4bb9a9a74c816ee51e2cf6d9e17", "tree": "84e527ba99415f9c5c19462ac3ba58753062c8c6", "parents": [ "6a7e56cddb30ec6500d9350c581e32888d2ac4be" ], "author": { "name": "dustin", "email": "dwsmith1983@users.noreply.github.com", "time": "Fri Oct 16 15:06:14 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 16 15:06:14 2026 +0000" }, "message": "fix: read fs.comet.libhdfs.schemes for the scan rule from the Hadoop conf (#6838)\n\n* fix: read fs.comet.libhdfs.schemes for the scan rule from the Hadoop conf\n\nCometScanRule read the libhdfs scheme list from the SQLConf entry\nspark.hadoop.fs.comet.libhdfs.schemes, while native routes a scheme\nthrough libhdfs from the Hadoop conf the scan forwards. Spark copies\nSQLConf keys into the Hadoop conf with the spark.hadoop. prefix kept,\nso a list set only in core-site.xml made the rule fall back on a scheme\nnative reads, and a runtime SET of the prefixed key made the rule claim\na scan native could not open.\n\nThe rule now takes the list from the scan\u0027s Hadoop conf, as it already\ndoes for fs.comet.s3Compliant.schemes, defaulting to hdfs like native.\n\n* fix: read the Delta gate\u0027s libhdfs schemes from the scan Hadoop conf\n\nDeltaScanSupport.declineReason still read the list from the SQLConf\nentry, while the native Delta scan reads fs.comet.libhdfs.schemes from\nthe Hadoop conf the gate builds and hands over. It now uses\nNativeConfig.resolveLibhdfsSchemes on that conf, as CometScanRule does.\n\n* test: fold the libhdfs scheme gate tests into one table each\n\nresolveLibhdfsSchemes becomes private[comet] like its siblings. In\nCometScanSchemeFallbackSuite the write and the plan-and-rule steps are\nseparate helpers that the existing fake:// and hdfs:// tests also use,\nand the three gate cases (bare Hadoop key, reader option, runtime SET\nof the spark.hadoop. key) share one table written once. The Delta\nsuite plans the same three cases against one Delta table, without the\nrepeated readability precondition or the padded scheme list.\n\n* fix: fall back when only a reader option lists a libhdfs scheme\n\nlibhdfs builds its Hadoop configuration from the executor classpath, so\nit never sees a scan\u0027s reader options. A scheme that the scan\u0027s\nfs.comet.libhdfs.schemes lists but the session\u0027s does not would be\nrouted to libhdfs natively and fail there, for example when\nfs.\u003cscheme\u003e.impl is also a reader option.\n\nNativeConfig.resolveScanLibhdfsSchemes now treats a scheme as libhdfs\nonly when both the session and the scan list it, and reports the\nscan-only rest. CometScanRule and the Delta gate decline any root (and,\nfor Delta, any selected data file or deletion vector) whose scheme is\nscan-only, even when object_store could read it." }, { "commit": "6a7e56cddb30ec6500d9350c581e32888d2ac4be", "tree": "61922bd66cbec7bc7c265aa1684e32e2e51e1267", "parents": [ "86d2d78257df81d3e2443dab194405bef8a5c4ca" ], "author": { "name": "dustin", "email": "dwsmith1983@users.noreply.github.com", "time": "Fri Oct 16 14:54:25 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 16 14:54:25 2026 +0000" }, "message": "fix: honor subexpression elimination off in the native null short-circuit (#6986)\n\n* fix: honor subexpression elimination off in the native null short-circuit\n\ncanShortCircuitNulls kept a later argument eager whenever the operator\u0027s\nexpressions shared a subexpression it held, whatever\nspark.sql.subexpressionElimination.enabled says. With the setting off,\nSpark\u0027s projection, shuffle partition keys and TakeOrderedAndProject\nshare nothing and skip that argument on a row whose first argument is\nNULL, so Comet raised where Spark returns rows.\n\nwithSharedSubexpressions now takes an alwaysShared flag. With the\nsetting off an operator shares nothing unless it asks for that flag,\nwhich CometExecRule does for aggregates, since Spark\u0027s whole-stage\ncodegen aggregates share the subexpressions of their update expressions\nregardless of the setting. Nothing changes with the setting on.\n\n* test: drop the subexpression elimination on case that repeats an existing test\n\nWith elimination on, alwaysShared does not change the shared set, and the\nremaining assertions are the ones the default-conf test already makes." }, { "commit": "86d2d78257df81d3e2443dab194405bef8a5c4ca", "tree": "5948efebc0f8935465c5e06d85e77f5ab21e4f48", "parents": [ "c3316fbe39e597f311dd999efc012616a71d966e" ], "author": { "name": "ChenChen Lai", "email": "72776271+0lai0@users.noreply.github.com", "time": "Fri Oct 16 14:52:41 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 16 14:52:41 2026 +0000" }, "message": "feat: native RANGE window frames with explicit offset on DATE ORDER BY (#4974)\n\n* feat: native RANGE window frames with explicit offset on DATE ORDER BY\n\n* fix fmt\n\n* update documentation\n\n* Modified SQL test\n\n* fix: fall back for DATE RANGE offsets that can wrap an Int" }, { "commit": "c3316fbe39e597f311dd999efc012616a71d966e", "tree": "a2effdd1ad86e10d2ad75f706ab4a6bfe4d28bd1", "parents": [ "758ef1ceb1bff1a6df13387a5b56a18c93327e8a" ], "author": { "name": "ChenChen Lai", "email": "72776271+0lai0@users.noreply.github.com", "time": "Fri Oct 16 14:45:36 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 16 14:45:36 2026 +0000" }, "message": "refactor: simplify shuffle support predicates in CometShuffleExchangeExec (#7045)" }, { "commit": "758ef1ceb1bff1a6df13387a5b56a18c93327e8a", "tree": "ae29e97d52f33688ee39cd35ed3ac76a0980bd76", "parents": [ "0f2ae443cd88a4376c3456794889104c81668fa7" ], "author": { "name": "Oleks V", "email": "comphead@users.noreply.github.com", "time": "Fri Oct 16 14:30:16 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 16 14:30:16 2026 +0000" }, "message": "perf: read only the nested fields Spark keeps in the native Iceberg scan (#6725)\n\n* perf: read only the nested fields Spark keeps in the native Iceberg scan\n\nThe native Iceberg scan sent iceberg-rust the full table schema as the\nFileScanTask schema, so iceberg-rust read and decoded every leaf of a\nprojected struct, list, or map column even when Spark\u0027s nested schema\npruning kept only some of them. Pass the pruned scan schema instead,\nwithout metadata columns, and still add partition sources and\nequality-delete keys. Memoize those additions so tasks share one schema\nin the pool, and keep the full schema when a needed column is nested\ninside a pruned struct.\n\nTasks with deletes read with the pruned schema too, so a VERSION AS OF\nread of a column dropped after the snapshot no longer fails natively\nwhen the table has deletes. The pruned schema is built at planning time,\nwhere a reflection failure falls back to Spark, and findFieldObject now\npropagates reflection failures. A column added to a task schema resolves\nfrom the current table schema before the schema history, so it keeps its\ncurrent name and type.\n\nAdd spark.comet.scan.icebergNative.nestedSchemaPruning.enabled to turn\nit off.\n\n* feedback\n\n* Resolve appended task-schema columns from one schema\n\nTake every missing partition source and equality-delete key from one\nschema, the current one or the newest in the history that has them all\nunder names the task schema does not use. Picking a name per field\nclashed when two partition sources were renamed into each other\u0027s names.\n\nTasks that need no extra columns skip the memo lookups, and tasks\nwithout deletes skip the equality-key work.\n\n* Build one task schema per scan for native Iceberg reads\n\niceberg-rust caches an equality-delete predicate per delete file, with\nthe column names of the task that loads it first, and binds it by name\nin every task of a reader. Task schemas built per task could name a key\ndifferently across tasks, which failed the scan or, after two columns\nswapped names, deleted the wrong rows.\n\nBuild one pruned and one full task schema per scan during planning. The\nnames come from the newest schema whose top-level columns match the scan\nschema, and an equality key dropped before it takes one name from the\nschema history. A scan whose keys cannot be named that way falls back to\nSpark. Serialization now builds the task schema and the projection once\nper scan.\n\n* Check the full task schema for nested evolution\n\nThe full task schema of a VERSION AS OF read can be older than the\ntable\u0027s schema and hold a nested field that an earlier data file lacks,\nwhich the native scan cannot read. Walk it in the nested-evolution check\ntoo, so such a scan falls back to Spark.\n\n* Check the task schema the scan reads for nested evolution\n\nThe nested-evolution check now walks the schema the scan reads, the\npruned task schema while pruning applies and the full one otherwise,\nthrough one method that serialization uses too. A pruned VERSION AS OF\nread no longer falls back because of a nested field only the full\nschema holds.\n\nThe Iceberg user guide lists the new naming fallback. The SQL fixture\nadds empty lists and maps, and the pruned-read tests also check the\nanswers with pruning off.\n\n* Match field types when picking the full task schema\n\nThe full task schema was the newest schema whose top-level columns\nmatch the scan schema by id and name. For a VERSION AS OF read across a\ntype promotion, that could be a newer schema with the promoted type,\nwhich the batch adaptation after iceberg-rust cannot narrow back, so\nthe native scan failed. Pick the newest schema in which every field the\nscan reads, at any depth, has the same name and type instead.\n\nThe time travel test reads a promoted int, decimal, and nested field,\non partitioned and unpartitioned tables, with pruning on and off.\n\n* Fix the native Iceberg task schema at planning\n\n`taskSchema` read the nested schema pruning config every time it was\ncalled: during planning, where CometScanRule checks the schema, and\nagain when the scan ran and serialized it. A query planned with pruning\non and run with pruning off, such as a cached DataFrame, read the full\nschema, which the nested-evolution check never walked. Make it a val,\nbuilt with the metadata during planning.\n\nThe struct fallback test plans a VERSION AS OF read with pruning on and\nruns it with pruning off.\n\n* Simplify the native Iceberg task schema code and tests\n\nBuild the scan schema\u0027s data columns and the projected field ids in\none pass each, and name the type check\u0027s schema `fullTaskSchema` so it\nreads apart from `taskSchema`, the schema the scan reads. The promotion\nreads reuse `checkPrunedAndFullReads`, and a time travel query that\ncould not catch a wrong schema choice is gone." }, { "commit": "0f2ae443cd88a4376c3456794889104c81668fa7", "tree": "46de4be3a11a3a3d03d0c854073cd7eaad7a6ff5", "parents": [ "53d398bb82f32f251ce3e1c105af99c079ec95e3" ], "author": { "name": "dustin", "email": "dwsmith1983@users.noreply.github.com", "time": "Fri Oct 16 13:57:11 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 16 13:57:11 2026 +0000" }, "message": "fix: give each nondeterministic dispatched expression its own kernel (#6712)\n\n* fix: give each nondeterministic dispatched expression its own kernel\n\nThe codegen dispatcher caches one kernel per task for each distinct\nserialized expression, and a Nondeterministic node keeps its state in\nthat kernel. Two occurrences of the same subtree bind to the same\nordinals and serialize identically, so they shared one kernel and the\nsecond occurrence continued the first one\u0027s counter. Wrap a subtree\nthat contains a Nondeterministic node in a marker carrying an id from\na driver-wide counter before serializing it, and strip the marker on\nthe executor before compiling, so each occurrence gets its own entry\nwhile deterministic subtrees keep sharing compiled kernels.\n\n* test: cover two identical round(rand(1)) columns with ANSI off\n\nWith ANSI off, Round carries no query context, so two copies of\nround(rand(1), 2) in one projection serialize to the same bytes. Before\nthe per-occurrence tag they shared one dispatched kernel and the second\ncolumn continued the first one\u0027s random sequence. The SQL file test reads\nboth columns from one seeded partition and checks that round and rand\nboth go through the dispatcher.\n\n* test: run the round(rand) occurrence fixture in batches of two and drop the duplicate UDF test" }, { "commit": "53d398bb82f32f251ce3e1c105af99c079ec95e3", "tree": "f5e69b241dcc639f2f9a9fe9c29fd056d56384e7", "parents": [ "c230bf4e986ecfaf317c0621b13cd84931c948ae" ], "author": { "name": "DeviousCardi", "email": "115358213+DeviousCardi@users.noreply.github.com", "time": "Fri Oct 16 13:56:39 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 16 13:56:39 2026 +0000" }, "message": "fix: include try_add suggestion in ANSI SUM overflow error (#6732)\n\n* fix: include try_add suggestion in ANSI SUM overflow error\n\nSpark\u0027s integral SUM returns LONG and adds through Add in both its\nupdate and merge expressions, so an ANSI overflow is reported as\n\"long overflow\" with the \"Use \u0027try_add\u0027 ...\" suggestion. Comet\u0027s\nSumInteger raised \"integer overflow\" with no suggestion at its three\noverflow sites (ungrouped update/merge, grouped update, grouped merge).\nRaise ArithmeticOverflow { from_type: \"long\", function_name: \"try_add\" }\nat all three, and make the CometAggregateSuite SUM test compare the\nstructured error with Spark\u0027s, including the merge path.\n\nCloses #6661\n\n* test: drop literal \u0027long overflow\u0027 check from ANSI SUM overflow test\n\nSpark 4.2 normalizes the \u0027long overflow\u0027 message to \u0027overflow\u0027 in\nExecutionErrors.arithmeticOverflowError, so the literal check failed on\nthe spark-4.2 profile even though Comet\u0027s error matches Spark\u0027s. The\ngetMessageParameters comparison with Spark already covers the wording.\n\nCo-Authored-By: Claude Opus 5.5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* test: check native execution in ANSI SUM overflow tests\n\n- assertAnsiSumOverflowMatchesSpark now checks the plan ran on Comet\n operators first, so a SUM that fell back to Spark can\u0027t pass by\n comparing Spark\u0027s error with itself.\n- The GROUP BY test\u0027s ANSI overflow and underflow branches use the\n helper, so they compare the error parameters with Spark\u0027s instead of\n only checking for ARITHMETIC_OVERFLOW in both messages.\n- Drop the Rust full-message assertion; native/common/src/error.rs\n already pins that text, and the params check covers the accumulators.\n\nCo-Authored-By: Claude Opus 5.5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 5.5 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "c230bf4e986ecfaf317c0621b13cd84931c948ae", "tree": "3a288b9d16aabbb4dacade4ddc8e192c0ff711c8", "parents": [ "a45445c6ef770b2d008b63ef42478bfdbc3f1e01" ], "author": { "name": "Oleks V", "email": "comphead@users.noreply.github.com", "time": "Fri Oct 16 04:16:55 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 16 04:16:55 2026 +0000" }, "message": "feat: add nested benchmark derived from TPC-H (#7044)\n\nAdds benchmarks/nested: a spark-shell script that converts existing flat\nTPC-H data (Parquet or Iceberg) into a nested layout, where non-key,\nnon-partition columns move into a single-element array\u003cstruct\u003e, and the 22\nqueries rewritten for that schema." }, { "commit": "a45445c6ef770b2d008b63ef42478bfdbc3f1e01", "tree": "0e1a1f219677907f6516c32d533cea8e24b15a79", "parents": [ "5e7d918ef5ec64ca13f996f42868758474632798" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Thu Oct 15 22:31:46 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 22:31:46 2026 +0000" }, "message": "fix: compare a Remainder divisor with zero in Spark\u0027s ordering without ANSI (#6896)\n\nnull_if_zero_primitive built Remainder\u0027s non-ANSI zero-divisor check as a plain BinaryExpr, so\n`x % -0.0` returned NULL only because DataFusion 55\u0027s apply_cmp folds signed zeros before Arrow\u0027s\neq. #6866 removed the divisor normalization that used to hide this in Projects and Filters. The\ncheck now goes through spark_comparison, as Divide\u0027s guard does through the native comparison\nbuilder." }, { "commit": "5e7d918ef5ec64ca13f996f42868758474632798", "tree": "bef5684c2201d7cff6826a7f7b6b60687c5baa1c", "parents": [ "01709c72749708761da5b94d034d25a34312cd41" ], "author": { "name": "dustin", "email": "dwsmith1983@users.noreply.github.com", "time": "Thu Oct 15 21:27:29 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 21:27:29 2026 +0000" }, "message": "fix: read each native scan partition through the object store its files live in (#6753)\n\n* fix: refuse a native scan partition that spans object stores\n\nNative planning reads every file of a Spark partition through one object\nstore, built from the partition\u0027s first file, and keeps only each file\u0027s\npath. A partition holding files from two buckets read the second bucket\u0027s\nfiles from the first: wrong rows when a key exists in both, FILE_NOT_EXIST\notherwise. get_partitioned_files now fails with an error naming both\nstores instead.\n\nThe CSV scan built one store from the first partition\u0027s first file for\nevery partition, so partitions in another bucket were read from the wrong\none. It now builds the store from the partition it reads, bounds-checks\nthe partition index, and plans an empty partition against the local\nfilesystem.\n\nPart of #6746 and #6747.\n\n* fix: pack native scan files per object store and fall back where options differ\n\nSpark packs small files from every root path into the same partitions, so\na native Parquet or CSV scan over two buckets produced partitions that one\nobject store cannot serve. Files are now grouped by the store native would\nuse for them before Spark\u0027s bin packing (Parquet) or before serialization\n(CSV). A scan over one store is packed exactly as before. A scan over\nseveral stays native and can end up with a different number of\npartitions.\n\nNativeConfig.objectStoreKey gives the JVM the same store identity as\nnative, including libhdfs routing, and a shared fixture pins the two.\n\nA scan forwards the object store settings of one scheme, so a scan whose\nfiles span scheme families (s3a and gs, or an S3-compatible alias and\ns3a) falls back to Spark, as does a bucketed scan whose files span stores.\nThe rule checks root paths early and the serde decides on the listed\nfiles, which covers partitions stored outside the table location. A\ndeclined CSV conversion now falls back to Spark\u0027s own scan instead of\nleaving the Comet wrapper over row-based input.\n\nCloses #6746.\nCloses #6747.\n\n* test: keep the multi-store scan tests clean under strict Scala warnings\n\nPass Long bounds to range explicitly, and compute the flattened file\nlists before asserting on them, so the assertion text ScalaTest renders\nno longer trips the missing-interpolator lint.\n\n* perf: key the native object store cache on the settings each store reads\n\nThe native scan now receives the object store settings of every scheme it\nreads. Hashing the whole map gave one bucket a different cache entry, and\na second HTTP client and credential fetch, depending on what else the scan\nread. Each backend now hashes and receives only the keys it reads: fs.s3a.*\nfor S3; fs.azure.* and the comet.azure.* resolution markers for Azure; and\nfor every store the fs.comet.* scheme lists that decide routing.\n\n* fix: forward every scheme\u0027s object store settings and keep Spark\u0027s CSV partitioning\n\n- A native scan forwards the union of the object store settings over one\n path per scheme, and per bucket for an S3-compatible alias, applying each\n alias bucket\u0027s translated settings last. Scans mixing scheme families stay\n native again.\n- The fallback is keyed on the store: a bucket read through an alias next\n to s3, s3a or another alias, an alias path with no bucket next to other\n S3 paths, and bucketed scans over several stores. The older fallback for\n alias paths over several buckets is gone, since the settings are now\n translated per bucket.\n- On Spark 3.4 a CSV scan with one input partition reports SinglePartition,\n so Spark plans no exchange above it. The CSV conversion falls back when\n splitting by store would change that partition count.\n- Files are grouped by store in one pass before packing, instead of one\n pass over all files per store.\n- CometCsvNativeScanExec no longer prints its native plan, which included\n the forwarded options and their credentials, in toString and explain.\n\n* fix: translate alias settings only for schemes read through native S3\n\nA scheme listed in both fs.comet.s3Compliant.schemes and\nfs.comet.libhdfs.schemes is read through libhdfs, but a scan that also read\ns3a files of the same bucket still forwarded the alias\u0027s translated\nfs.s3a.bucket.\u003cbucket\u003e.* settings, so the native S3 store used the alias\nendpoint. Such a file now contributes no translated settings.\n\n* test: drop the native Parquet scan partition count asserts that hold by construction\n\n* fix: read the libhdfs schemes for the early multi-store check from the Hadoop conf\n\nThe rule\u0027s early check took fs.comet.libhdfs.schemes from SQLConf, while\nthe serde check, partition packing and native read it from the Hadoop\nconf. With the scheme lists only in core-site.xml, a scan over\ns3a://bucket and blob://bucket (blob routed through libhdfs) was\ndeclined as two schemes on one store, though native reads them as two\nstores.\n\n* fix: compare S3 alias settings by bucket host across a scan\n\nNativeConfig.bucketForUri now returns the host the native S3 store reads\nper-bucket settings under, without a port or user info, so the alias\nsettings for a path such as blob://bucket:9000/... are kept instead of\ndropped. The shared-alias fallback groups paths by that bucket, so\nblob://bucket:9000 next to s3a://bucket falls back the same way\nblob://bucket next to s3a://bucket does.\n\nThe multi-store tests and docs also cover the ABFS rules: a scan over\none container keeps that account\u0027s resolution, while several containers,\nseveral accounts, or ABFS next to wasb or HDFS fall back.\n\n* test: pin the registration URL to the full option hash while stores share backend options\n\nThe Delta scan\u0027s comment said a file under another authority would read\nthrough the first file\u0027s store; get_partitioned_files now rejects a\nmixed-store partition, so describe the Delta check as the earlier,\nJVM-aligned guard it is, and say where its authority comparison differs\nfrom the JVM\u0027s. Adds a test that two option maps differing only in\nanother backend\u0027s keys register under different URLs but share one\ncached store, which the Delta scan\u0027s store memo relies on." }, { "commit": "01709c72749708761da5b94d034d25a34312cd41", "tree": "e2f9d356225ee98f66926b0b525dc404d0323879", "parents": [ "2a0f1cfac7820901c50bc7bb59de8bca8bea80c3" ], "author": { "name": "Liang-Chi Hsieh", "email": "viirya@gmail.com", "time": "Thu Oct 15 20:32:03 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 20:32:03 2026 +0000" }, "message": "fix: do not fail planning on a cast of a literal that fails to evaluate (#6826)\n\n* fix: do not fail planning on a cast of a literal that fails to evaluate\n\nCometCast folded every cast of a literal on the JVM with cast.eval() while\nplanning. Spark\u0027s ConstantFolding deliberately leaves a failing cast unfolded\ninside a conditional branch, so under ANSI a branch that no row reaches, such\nas CASE WHEN id \u003d 5 THEN CAST(\u0027bad\u0027 AS BIGINT) ELSE id END, failed the whole\nquery in Comet while Spark returned rows.\n\nFold only when evaluation succeeds. Otherwise report the support level of the\nactual cast and serialize it as a regular cast, so the error is raised only\nwhen a row reaches it and matches Spark\u0027s error.\n\n* fix: reject a D/F suffix after NaN or Infinity when casting string to float\n\nparse_string_to_float stripped a trailing d/D/f/F and handed the rest to\nRust\u0027s parser, which accepts nan, inf and infinity in any case. So \u0027NaNF\u0027,\n\u0027InfinityD\u0027 or \u0027-nan\u0027 cast to a value, while Spark (Double.parseDouble plus\nCast.processFloatingPointSpecialLiterals) rejects them: CAST_INVALID_INPUT\nunder ANSI, NULL otherwise. Only allow the suffix after a decimal number, and\naccept a signed NaN only in the exact case Java does.\n\n* fix: reject a decimal string whose scale is outside the int range\n\nparse_decimal_str computed the scale as fractional digits minus the exponent\nin i32, and parse_string_to_decimal subtracted it from the target scale in\ni32. Both wrap in release builds, so \u00271e-2147483648\u0027 or \u00271.0e-2147483647\u0027\ncast to 0.00 even under ANSI. java.math.BigDecimal, which Spark parses with,\nrejects a scale outside the int range, and Spark reports CAST_INVALID_INPUT\nunder ANSI and NULL otherwise. Compute the scale in i64 and treat a scale\nthat does not fit in an int as invalid.\n\n* fix: run Spark\u0027s Cast for a cast of a literal that fails to evaluate\n\nA failed fold sent the cast to the native kernel, so the result depended on\nthe native parser rejecting exactly what Spark rejects. It does not always:\n\u00271e-2147483647\u0027 to DECIMAL(10,2) parses with an in-range scale, and Spark\nthen fails while rounding it, but the native cast returns 0.00.\n\nThe failed fold already proves that Spark\u0027s Cast throws for this literal, so\nreport it as Unsupported. CometCast is a CodegenDispatchFallback, so the\ncast runs Spark\u0027s own code through the JVM codegen dispatcher and keeps the\noperator native, or falls back to Spark when the dispatcher is disabled.\nEither way the error comes from Spark and is raised only when a row reaches\nthe cast.\n\nAlso replace \u00271e2147483648\u0027 in the decimal fixture: JDK 21 accepts that\nexponent and Spark then fails during conversion, while \u00271e2147483649\u0027 is\ninvalid on every JDK. Drop the Rust assertion that \u00271e-2147483647\u0027 parses\nto zero, which Spark does not do.\n\n* fix: keep failing a decimal rescale whose scale difference overflows an int\n\nComputing the scale adjustment in i64 removed an accidental failure: for\n\u00271e-2147483647\u0027 cast to a negative target scale, the difference to the\ntarget scale no longer wrapped, so the cast rounded to zero. Spark\u0027s\nBigDecimal.setScale fails when that difference does not fit in an int, so\ntreat it as invalid input.\n\nAlso check only the first byte after the sign when rejecting NaN and\ninfinity spellings with a D/F suffix. Rust\u0027s parser accepts no other\nnon-numeric form, and a second full scan of every numeric string slowed\nthe string to float cast.\n\n* test: check operators and error class for reached failing literal casts\n\nSwitch the reached-branch queries in cast_literal_ansi.sql to\nexpect_error_class, so they also fail if the cast stops reaching the\ndispatcher and the projection falls back to Spark, and compare the\nexception type, error class and SQLSTATE against Spark.\n\nKeep CometCast.convert as on main: getSupportLevel only reports a cast of a\nliteral as Compatible when evaluating it succeeds, so convert never sees a\nfailing one. Trim the Scala test to the raw ArithmeticException case the\nSQL framework cannot express.\n\n* Link the remaining string to decimal gap to #6966" }, { "commit": "2a0f1cfac7820901c50bc7bb59de8bca8bea80c3", "tree": "fb385836539f67a54ca5ca1e7a5b19460df7fd1c", "parents": [ "5ef55a742d9c6992179ec97e9d8f537fd92d0321" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Thu Oct 15 20:11:27 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 20:11:27 2026 +0000" }, "message": "fix: fall back to Spark for decimal AVG cases that return wrong results (#7017)\n\nNative decimal AVG has three wrong-result bugs that default settings reach.\nEach now falls back to Spark until the native fixes in #5420 land:\n\n- The native average rescales the sum in an i128 before dividing, so a large\n sum returns NULL for input precision 25 and up, such as DECIMAL(38,18) above\n 1.7e16 (#7007). CometAverage reports these as Incompatible, so\n spark.comet.expression.Average.allowIncompatible\u003dtrue still runs them\n natively.\n- At a maximum-precision sum (input precision 28 and up), the ungrouped native\n AVG skips the row that overflows instead of keeping the sum unbounded or\n latching it, so DECIMAL(38,38) values 0.6, 0.6, -0.6 average to 0, not 0.2\n (#5418). Ungrouped hash aggregates, object hash aggregates (#5509) and\n window aggregates decline it. Grouped hash aggregates latch like Spark and\n stay native." }, { "commit": "5ef55a742d9c6992179ec97e9d8f537fd92d0321", "tree": "0da9d300de1f6461d2611bbf37e282d6ed318f18", "parents": [ "d64915e1b310cfcf9aa980ac39f50963099efea8" ], "author": { "name": "LinSimon-901101", "email": "bcs113118@gm.ntcu.edu.tw", "time": "Thu Oct 15 19:46:22 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 19:46:22 2026 +0000" }, "message": "fix: report row contents to native write statistics trackers (#6383)\n\n* fix: report row contents to native write statistics trackers\n\n* fix: derive native write guidance from config key\n\n---------\n\nCo-authored-by: Andy Grove \u003cagrove@apache.org\u003e" }, { "commit": "d64915e1b310cfcf9aa980ac39f50963099efea8", "tree": "412d1132e16f34437219911713cda8c9da5b58a7", "parents": [ "dbefab760172e2a1318c94cac96263665b3833b9" ], "author": { "name": "KAZUYUKI TANIMURA", "email": "ktanimura@apple.com", "time": "Thu Oct 15 19:03:34 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 19:03:34 2026 +0000" }, "message": "fix: raise DIVIDE_BY_ZERO from native corr on a zero denominator in ANSI mode (#6830)\n\n* fix: match Spark variance updates for large nearby values\n\n* fix: match Spark statistical aggregate merges\n\n* fix: preserve Spark semantics when merging empty partials\n\n* test: cover regression aggregates merging fractional constants\n\n* fix: align scalar correlation evaluation with Spark\n\n* fix: preserve correlation underflow and NaN semantics\n\n* test: normalize NaN in correlation extreme-value assertions\n\n* fix: restore native regression aggregates after merge correction\n\n* fix: preserve Spark regr_r2 divide-by-zero semantics\n\n* fix: pass evaluation mode in regression benchmark\n\n* fix: Native corr, covariance, variance and stddev return wrong values for a constant fractional column merged from several partitions\n\n* test: compare corr\u0027s query context inside withSQLConf for Spark 3.x\n\nwithSQLConf returns Unit on Spark 3.4 and 3.5, so the expected error\ncould not be returned from the block.\n\n* fix: attach query context to regr_r2\u0027s divide-by-zero error\n\nMove ZeroDivisor into welford.rs so regr_r2 shares it with corr, and\npass the aggregate\u0027s expr_id and the query context registry to Regr.\nBoth ANSI tests now compare the error\u0027s query-context fragment with\nSpark\u0027s.\n\n* test: tighten corr ANSI coverage per review\n\nUse expect_error_class in corr.sql so the ANSI queries also check native\noperators, exception class and SQLSTATE. Keep only the ANSI branch of the\nScala test, whose LEGACY checks repeat the fractional-constants test, and\ndrop a Rust test loop over null_on_divide_by_zero that never reaches the\ncount \u003d\u003d 1 branch.\n\n---------\n\nCo-authored-by: rich7420 \u003crc910420@gmail.com\u003e" }, { "commit": "dbefab760172e2a1318c94cac96263665b3833b9", "tree": "a6ee84e0dd90e0c598596cbe6e74b00b863bf32e", "parents": [ "c61a79c44bca396e6c29cdfa9b5178dc477f4cb6" ], "author": { "name": "KUAN-HAO HUANG", "email": "101171023+rich7420@users.noreply.github.com", "time": "Thu Oct 15 18:02:13 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 18:02:13 2026 +0000" }, "message": "fix: compact oversized nested list_extract buffers (#6498)\n\n* fix: compact oversized nested list_extract buffers\n\n* test: exercise nested list extraction compaction through JVM\n\n* test: deduplicate list_extract reference gathers" }, { "commit": "c61a79c44bca396e6c29cdfa9b5178dc477f4cb6", "tree": "ab8ce689a9b7f07f2b89d67ad657174d09e01932", "parents": [ "88241e8f94ad4346f1e0700f6eb520078065aef8" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Thu Oct 15 16:24:36 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 16:24:36 2026 +0000" }, "message": "docs: document casts involving arrays, structs, and maps (#6320)\n\n* docs: document casts involving arrays, structs, and maps\n\n* docs: say which key casts the Try-mode map note affects" }, { "commit": "88241e8f94ad4346f1e0700f6eb520078065aef8", "tree": "eec799566a9da1685a0b582b6bab63851c8d0ac1", "parents": [ "f2ed550dc726e67cbd460397124bee7c76e3d32b" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Thu Oct 15 14:44:53 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 14:44:53 2026 +0000" }, "message": "build: use a single jni version across the native workspace (#6998)\n\nThe shuffle crate declared jni 0.21 while core, jni-bridge and spark-expr\ndeclared 0.22.4, so Cargo.lock held both and every build compiled a\nsecond copy of the binding. Shuffle only uses the jint and jlong aliases\nfrom jni::sys, which are i32 and i64 in both versions.\n\nDeclare jni once in [workspace.dependencies] and inherit it in the four\ncrates that use it. The lock drops jni 0.21.1, jni-sys 0.3.1 and the\ncrates only they needed (cesu8, thiserror 1.0.69, windows-sys 0.45.0 and\nits 0.42.2 dependencies). Nothing else in the lock changes." }, { "commit": "f2ed550dc726e67cbd460397124bee7c76e3d32b", "tree": "4f2022d288ff3a05af4e868e1e016ac63a975569", "parents": [ "72b90459607c2e566024c3267802a0e81df3b503" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Thu Oct 15 14:11:31 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 14:11:31 2026 +0000" }, "message": "docs: correct native Iceberg write docs that drifted from the code (#7009)\n\nThe Iceberg write guides, and the other pages that mention native\nwrites, had statements that were wrong from the start or went stale as\nthe writer changed. Each one below was checked against the code.\n\nUser guide (iceberg-writes.md):\n- Native files carry created_by \"Apache Iceberg \u003cversion\u003e (Comet)\".\n- iceberg-java writes page CRCs and parquet-rs does not. Neither writes\n page-header statistics.\n- Neither writer dictionary-encodes fixed-length binary columns under\n format v1, so that item is removed, and so is the claim that every\n other encoding matches.\n- parquet-rs closes row groups by size, both writers name files\n \u003cpartition\u003e-\u003cattempt\u003e-\u003coperation\u003e-\u003ccount\u003e with a counter per task,\n and a zstd table with no level is written at level 3.\n- The accepted-divergences intro no longer promises the path-layout\n caveat that #5651 removed.\n- Iceberg 1.5 deletes completed tasks\u0027 files when a job aborts. Only\n Iceberg 1.6 and later skip that cleanup.\n- rewrite_data_files is listed as supported and streaming writes as a\n fallback, and an impossible fallback example is replaced.\n\nContributor guide (iceberg-writes.md):\n- The Iceberg suites skip on Spark 4.2.\n- The Iceberg Spark test jobs report which writer ran each write\n (#6155), so the CI caveat points at that report.\n- A task that writes no files sends a manifest with no entries.\n- iceberg-rust no longer panics on pre-1970 timestamptz values.\n- The row lineage rule is pinned by CometIcebergWriteActionSuite.\n\nElsewhere, iceberg-spark-tests.md names the ci.yml jobs and installs\nComet for the Spark version it tests, the plan guide no longer says\nthe write operators fuse with their child, and operators.md,\ntuning/memory.md, iceberg.md, the 1.2.0 upgrade guide and the Iceberg\nwrite review skill are corrected where they overstated or omitted\nwhat the native writer does." }, { "commit": "72b90459607c2e566024c3267802a0e81df3b503", "tree": "1b30d58d226a8e446acad7d0724a76850e92429f", "parents": [ "7a42a35a2ee5fa2fc3ac01356a47b735547b3129" ], "author": { "name": "Chao Sun", "email": "sunchao@apache.org", "time": "Thu Oct 15 14:08:23 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 14:08:23 2026 +0000" }, "message": "feat: expose per-scan S3 HTTP coverage and retries (#6429)\n\n* feat: expose per-scan S3 HTTP attempts and retries\n\n* docs: format S3 HTTP metric table\n\n* Address S3 HTTP metric review feedback" }, { "commit": "7a42a35a2ee5fa2fc3ac01356a47b735547b3129", "tree": "84d828ad724319eafa8617d92e4d471d5b94bd72", "parents": [ "66c63bd307b1b11fb684e6b89e9bb220f444de7c" ], "author": { "name": "Parth Chandra", "email": "parthc@apache.org", "time": "Thu Oct 15 13:08:31 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 13:08:31 2026 +0000" }, "message": "feat: run broadcast joins natively when an unsupported scan is on the build side (#6786)\n\n* feat: run broadcast joins natively when an unsupported scan is on the build side (#6008)\n\n* fix: remove an unused broadcast build-side bridge after the transform (#6008)\n\nRound-2 review fixes for PR #6786:\n\n- Replace the never-firing revertBridgedBroadcasts (the bridge is a\n CometScanWrapper leaf, so .exists never saw it, and the rebuild from\n originalPlan mishandled CometSinkPlaceHolder, unions and native scans)\n with clearWastedBuildSideBridges: after the transform, clear\n BROADCAST_BUILD_SIDE_TAG from scans still under a plain Spark\n BroadcastExchangeExec and transform once more. A declined broadcast join\n then rebuilds its build side exactly as it would without the feature, and\n the AQE stage pass no longer re-bridges the scan.\n- Fix stale docstrings on broadcastBuildSideBridgeEnabled and isNativeBuildSide.\n- Add FULL OUTER BNLJ revert tests (filter/aggregate/union build sides), a\n Spark 3.4 DPP test, and a UNION ALL AQE DPP reuse test to CometExecSuite.\n- Add CometBroadcastBuildSideSuite (ORC and Iceberg build-side probes) and\n register it in the PR build workflows.\n\n* fix: scope the build-side tag clear to actual bridges (#6008)\n\nRound-3 review on PR #6786:\n\n- clearScanTags now stops at a CometBroadcastExchangeExec, so a nested\n native broadcast join inside a Spark-staying join\u0027s build keeps its\n bridge, and it only counts a clear (and asks for the second transform)\n when the tagged scan sits under a CometSparkToColumnarExec. A tagged but\n unbridged CSV/JSON/Parquet scan (per-format convert off) no longer forces\n a wasted second transform.\n- Recomment the 3.4 Parquet DPP test: on 3.4 the V1 fact with a DPP filter\n stays on Spark (CometScanRule.scala:199), so nothing is tagged - it covers\n the V1 fallback, while the Iceberg test covers the SKIP_COMET_BROADCAST_TAG\n path with a native fact.\n\n* test: pin the nested-bridge case and name the 3.4 scan check (#6008)\n\nOptional nits from the PR #6786 approval:\n\n- Add a Spark-staying test where a FULL OUTER nested loop join\u0027s build is a\n broadcast hash join of a Parquet table with a text table, asserting the\n inner join is a native CometBroadcastHashJoinExec that keeps its text\n bridge while the outer join stays a Spark BroadcastNestedLoopJoinExec.\n This pins clearWastedBuildSideBridges stopping at CometBroadcastExchangeExec.\n- Replace the brittle CometScanRule.scala:199 line reference in the 3.4 test\n with the name of the check (the Spark 3.5+ AQE DPP fallback in\n CometScanRule.transformV1Scan)." }, { "commit": "66c63bd307b1b11fb684e6b89e9bb220f444de7c", "tree": "a546fb27fcfb37cfc0560123760c5b4603bb43bb", "parents": [ "a230ae54d223e0b60097c12e470a87d54e802b0e" ], "author": { "name": "Oleks V", "email": "comphead@users.noreply.github.com", "time": "Thu Oct 15 03:33:44 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 03:33:44 2026 +0000" }, "message": "fix: keep the values of sliced batches when coalescing a broadcast (#6900)\n\n* fix: keep the values of sliced batches when coalescing a broadcast\n\nA slice of a native string, binary, list or map array reaches the JVM with offsets that start\npast zero, with the rows before the slice still at the front of its data. Arrow Java writes\nthose bytes into the IPC stream, and the VectorSchemaRootAppender that coalesces broadcast\nbatches assumes offsets start at zero, so it prepended them to the first value of each slice.\n\nserializeBatches now writes such a batch through a slice of its root, which rebases the\noffsets to zero and leaves the earlier bytes out.\n\n* Pass Long arguments explicitly in the UtilsSuite slice helper\n\nThe strict-warnings build rejects the implicit Int to Long widening in the `ArrowFieldNode`\nand `ArrowBuf.slice` calls, so convert the arguments explicitly.\n\n* Document that sliced variable-width arrays reach the JVM with offsets past zero\n\n`zero_offsets` handles `ArrayData::offset`, but a sliced string, binary, list or map array still\nreaches the JVM with offsets that start past zero and the rows before the slice at the front of its\ndata. Say so in the \"Array Offsets\" section, and that `serializeBatches` rebases broadcast batches\nbecause `VectorSchemaRootAppender` needs offsets that start at zero.\n\n* Close the rebased slice when writing a broadcast batch fails\n\n`splitAndTransfer` takes its own references on the buffers the slice shares with the batch, so a\nwrite that failed, as it does when a cancelled broadcast interrupts the task, left them pinned after\nthe batch closed. Close the slice in a `finally`, and test that the slice\u0027s allocator drains to zero\nafter an interrupted write." }, { "commit": "a230ae54d223e0b60097c12e470a87d54e802b0e", "tree": "c11486e39de0e7bf61607c35d2a50a1927fa778a", "parents": [ "e555dba6f5c8c62ceca8c9c6e311a51b8fbfeb70" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Wed Oct 14 21:07:18 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 14 21:07:18 2026 +0000" }, "message": "docs: add proposal to establish Apache Comet as a top-level project (#6817)\n\n* docs: add draft proposal to establish Apache Comet as a top-level project\n\nAdd the draft board resolution and supporting proposal for promoting\nComet from a DataFusion sub-project to a top-level Apache project, so\nthat the community can review it in the open. The initial PMC members\nand committers are existing DataFusion PMC members and committers; the\nproposal does not promote anyone.\n\nLink the page from the About section, and use colons instead of em\ndashes in that section\u0027s list.\n\n* docs: link the dev@ discussion thread from the proposal\n\n* docs: lead the proposal\u0027s background with the separate communities\n\nMake the largely distinct DataFusion and Comet communities the main\nreason for the proposal, and drop the comparison with the other\nDataFusion sub-projects. Mention that three of the five proposed\ninitial PMC members are ASF Members.\n\n* docs: drop the draft status from the top-level project proposal\n\nThe proposal has consensus with no objections, so present it as the\nproposal and board resolution to submit. Remove the note that the\nDataFusion PMC has not voted and the note that the scope is under\ndiscussion, and stop calling it a draft on the About page." }, { "commit": "e555dba6f5c8c62ceca8c9c6e311a51b8fbfeb70", "tree": "4f11546f395922cccafdf5572b89496fe98a90de", "parents": [ "b513921290f95ffc7021a4a9a8ae92ea507141b4" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Wed Oct 14 18:46:10 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 14 18:46:10 2026 +0000" }, "message": "feat: make native Iceberg writes opt-in for 1.2.0 (#6962)\n\nspark.comet.write.iceberg.enabled now defaults to false again, so 1.2.0\nships the native Iceberg writer as a supported, opt-in feature instead\nof turning it on for every eligible write. The rest of #6664 stays: the\nsetting is in the query execution category, it plans the split operator\nand the native writer by itself, merge-on-read writes keep Spark\u0027s\noperator, and spark.comet.exec.enabled\u003dfalse keeps Spark\u0027s write\noperator.\n\nThe Iceberg Spark test diffs and both Iceberg write benchmarks set the\nsetting themselves, so they still run the native writer.\nCometIcebergWriteActionSuite now checks that an application that sets\nneither write flag gets Spark\u0027s own operator, and that the write flag\nalone writes an eligible write natively under the split plan. The\nmerge-on-read and transition-reversion tests set the write flag\nthemselves.\n\nThe user guide, FAQ, Gluten comparison, contributor guide, roadmap and\nIceberg write review skill describe the native writer as opt-in, and\nthe 1.2.0 upgrade-guide entry covers the move from 1.1.0\u0027s testing\nsettings to the one supported setting." }, { "commit": "b513921290f95ffc7021a4a9a8ae92ea507141b4", "tree": "3f3456e6e7b8942f6d5bc1ccc30079fcd7e4b05b", "parents": [ "e4babc46bb608510707c281c72c60e8e1a6e7d9b" ], "author": { "name": "Oleks V", "email": "comphead@users.noreply.github.com", "time": "Wed Oct 14 18:28:09 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 14 18:28:09 2026 +0000" }, "message": "test: tag the Azure ABFS suites with AzureTest (#6942)\n\n* test: tag the Azure ABFS suites with AzureTest\n\nAbfsAuthParitySuite, AbfsAuthResolverSuite and AbfsScanFallbackSuite resolve\nABFS authentication through the hadoop-azure on the test classpath, so they\npass only when that hadoop-azure matches the Hadoop the build runs on. A\nbuild against a different Hadoop line, for example a vendor Hadoop with no\nmatching hadoop-azure pin, had no way to skip them: the root pom hard-codes\ntagsToExclude, so -DtagsToExclude on the command line was ignored.\n\nAdd an org.apache.comet.tags.AzureTest tag, annotate the three suites with\nit, and append a tagsToExclude property (empty by default) to the\nscalatest-maven-plugin exclusions so -DtagsToExclude\u003dorg.apache.comet.tags.AzureTest\nskips them. CI keeps running them, since every Spark profile pins a matching\nhadoop-azure.\n\n* test: tag only the AzureBlobFileSystem cross-check with AzureTest\n\nOn a mismatched classpath only the FileSystem.newInstance cross-check in\nAbfsAuthParitySuite fails. AzureBlobFileSystem links against Hadoop classes\nan older Hadoop lacks, while the resolver reaches nothing newer than\nAbfsConfiguration, so the resolver cases, AbfsAuthResolverSuite and\nAbfsScanFallbackSuite still pass there.\n\nTag just that test, with a ScalaTest Tag of the same name, so a build that\npasses -DtagsToExclude\u003dorg.apache.comet.tags.AzureTest keeps running the\nresolver cases." }, { "commit": "e4babc46bb608510707c281c72c60e8e1a6e7d9b", "tree": "a501ba44ce5a1dff2a810e0bfc3b9d8451b2c824", "parents": [ "62ba2ab19b02a6d95b55b14a468d2c6018466499" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Wed Oct 14 17:09:35 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 14 17:09:35 2026 +0000" }, "message": "perf: store shuffle offsets as lengths with zstd (#6943)\n\n* perf: store shuffle offsets as lengths with zstd\n\nArrow\u0027s offsets buffers are running sums, which compress poorly. With zstd, the shuffle block\nwriter now rewrites every offsets buffer of the record-batch body as lengths split into byte\nplanes before compressing, and flags the block in its codec tag. The reader rebuilds the offsets\nbefore Arrow decodes the body. Shuffle files with string, binary, array and map columns get 9%\nto 40% smaller, and zstd gets no slower. LZ4 and snappy blocks are unchanged: lz4_flex decodes\nthe long runs that lengths compress into slowly (PSeitz/lz4_flex#227).\n\nspark.comet.shuffle.compression.offsetsAsLengths.enabled (default true) turns it off.\n\n* refactor: decide offsets as lengths once per shuffle block writer\n\nShuffleBlockWriter::try_new_with_offsets_as_lengths replaces the builder. The writer now decides\nwhen it is built: it checks once, on an empty batch, that arrow-ipc writes the buffers the\nmapped roles describe, keeps plain offsets if not, and sets the codec-tag flag in its header.\nThat removes the per-block flag plumbing through every codec arm and the header rewrite.\n\nThe writer and reader share one buffer-range helper and one rewrite loop. The reader rebuilds\noffsets before freezing the body. The two Scala tests share a cometShuffleMetric helper.\n\n* docs: say where shuffle offsets are stored as lengths\n\nThe tuning guide and the config description now say that native shuffle and JVM shuffle blocks\nwithout dictionary-encoded columns store offsets as lengths with zstd, and that Celeborn shuffle\nblocks do not. native_shuffle.md describes the codec-tag flag, and the native and JVM shuffle\nconfig tables list spark.comet.shuffle.compression.offsetsAsLengths.enabled.\n\n* docs: list offsets_as_lengths.rs in the native shuffle guide\n\n* test: cover nulls, empty values, binary and struct columns and spilling in the offsets-as-lengths tests" }, { "commit": "62ba2ab19b02a6d95b55b14a468d2c6018466499", "tree": "7eb09da22b3da6f48375956d811f4cc99fd4ac20", "parents": [ "9e9eb4403efa24098c48b17741d11df4bc1ccb4b" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Wed Oct 14 16:24:40 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 14 16:24:40 2026 +0000" }, "message": "perf: generate the in-memory cache reader\u0027s class once per executor (#6860)\n\n* perf: generate the columnar-to-row projection once per executor\n\nCometColumnarToRowExec.doExecute built its UnsafeProjection with\nUnsafeProjection.create in every partition, which regenerates the\nprojection\u0027s Java source each time. The plan runs that path whenever it\nis outside whole-stage codegen, which a schema with more nested fields\nthan spark.sql.codegen.maxFields always is. For 100 columns nested three\nlevels deep, generating the source took about 19 ms per partition, half\nthe time of a 201-partition JVM shuffle of 1,000 rows.\n\nCache the compiled class per executor by column layout and create a new\ninstance for each partition.\n\n* ci: run CometUnsafeProjectionSuite on Linux and macOS\n\n* perf: generate the cache reader\u0027s class once per executor\n\nSpark reads Comet\u0027s in-memory cache through CachedBatchRowIterator, once\nper partition, whenever it does not read the cache as batches. That is\nalways the case for a schema with more fields than\nspark.sql.codegen.maxFields. Every call generated the reader\u0027s source\nagain; Spark\u0027s compiled class cache only saved the compile. For a\n1,500-column table that was about 18 ms per partition, and for 100\ncolumns nested three levels deep about 78 ms.\n\nKeep the class per executor by column layout, as CometUnsafeProjection\ndoes for CometColumnarToRowExec, with the cache both now use moved into\nGeneratedClassCache. Each reader gets its own batches through its own\ncopy of the references. The fallback for a method over the huge-method\nlimit builds its projection through CometUnsafeProjection.\n\n* fix: bound the generated class caches by spark.sql.codegen.cache.maxEntries\n\nGeneratedClassCache kept the 100 most recently used classes, the default\nof spark.sql.codegen.cache.maxEntries, without reading that conf, so\nlowering it to bound metaspace left Comet\u0027s two caches at 100 each. Read\nit when the cache is created, as Spark\u0027s CodeGenerator does for its own\ncache, and drop the constructor parameter and constant nothing passed." }, { "commit": "9e9eb4403efa24098c48b17741d11df4bc1ccb4b", "tree": "feb18041ba56c0c0e66dea4919c68aec266eaf7d", "parents": [ "7d8d9b1544b734c85a93cca2cc9d5380ad70accd" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Wed Oct 14 15:39:50 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 14 15:39:50 2026 +0000" }, "message": "fix: make Comet\u0027s in-memory cache opt-in again (#6963)\n\nSet spark.comet.exec.inMemoryCache.enabled back to false on every Spark\nversion. Since #5634 turned it on from Spark 3.5, sorted cached relations\nreach native joins that do not keep the order Spark relies on (#6787,\n#6898, #6922), so those wrong results are reachable with default settings.\n\nKeep Comet\u0027s cache suites and the 3.5, 4.0, 4.1 and 4.2 Spark SQL diffs\nrunning against Comet\u0027s cache format by setting the flag beside the cache\nserializer, and restore the opt-in wording in the docs." }, { "commit": "7d8d9b1544b734c85a93cca2cc9d5380ad70accd", "tree": "1213fb69b34ffb6ca7d3b17c0d746d56132ed556", "parents": [ "33f4f888a0f49c1eb933149c1f0c502a7b192748" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Wed Oct 14 02:41:32 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 14 02:41:32 2026 +0000" }, "message": "feat: normalize floats in native array_distinct and array_union (#6563)\n\n* feat: normalize floats in native array_distinct and array_union\n\nSpark 4.2.0 normalizes the arguments of array_distinct and array_union in\nthe plan (SPARK-54918), and 4.0.5, 4.1.4 and 4.2.1 normalize inside the\nexpressions instead (SPARK-59602), so those releases pass Comet the raw\ninput. DataFusion folds -0.0 into 0.0 only in a flat float array and\ncompares NaNs by their bits, which is why #5750 limited native execution\nof float elements to 4.2.0.\n\nArrays with a float at any depth now go to spark_array_distinct and\nspark_array_union, which run normalize_nested_floats on their inputs and\nthen delegate to DataFusion, as SparkArrayExtrema does. That matches every\nrelease that merges signed zeros, so float elements run natively by\ndefault on 4.0.5+, 4.1.4+ and 4.2+, and only 3.4, 3.5, 4.0.0 to 4.0.4 and\n4.1.0 to 4.1.3 fall back. Older Spark merges NaNs and nested zeros too, so\nthe opt-in now differs from it only on signed zeros.\n\nWithin 4.0 and 4.1 the default path now depends on the patch release, so\nthe 4.0/4.1 signed-zero fixture checks answers only and\nCometArrayExpressionSuite checks the routing. The 4.2+ fixture checks the\ndefault path instead of the opt-ins.\n\n* fix: keep collated struct fields out of native array_distinct and array_union\n\nThe spark_ variants normalize floats and then let DataFusion compare the\nelements, which compares strings by their bytes. Spark compares a\ncollated string under its collation, so under UTF8_LCASE two structs\nthat differ only in \u0027a\u0027 and \u0027A\u0027 are one element. Widening the gate to\n4.0.5+, 4.1.4+ and 4.2.x would have sent such elements to the native\nkernels, and 4.2.0 already did.\n\nElements that hold both a float and a non-UTF8_BINARY collated string\nnow fall back on every version. Collated strings without floats are\ntracked separately in #6470.\n\n* fix: skip array_union\u0027s right operand on rows whose left array is NULL\n\nSpark\u0027s ArrayUnion returns NULL for a NULL left array without evaluating\nthe right operand. The native function evaluates both over the whole\nbatch, so a right operand that throws, such as slice(b, 0, 1) or an ANSI\ncast, failed on rows that Spark never evaluates it for. That already\nhappened on main for every element type that runs natively, and this PR\nwould have extended it to float elements on 4.0.5+ and 4.1.4+.\n\nCometArrayUnion now guards the call with CASE WHEN \u003cleft\u003e IS NOT NULL, as\nCometElementAt does, unless the right operand is a column or a literal,\nwhich cannot throw. A nullable nondeterministic left operand, which the\nguard would evaluate twice, stays on Spark.\n\n* revert: drop the array_union NULL guard in favor of #6716\n\nThe CASE WHEN \u003cleft\u003e IS NOT NULL guard serialized the left operand twice,\nin the IS NOT NULL test and in the call, so nested unions grew\nexponentially: twelve nested array_union calls became 4,095 union nodes,\nand each level evaluated its left operand twice.\n\n#6716 adds a native NullShortCircuit that evaluates the left operand once\nand the right operand only on the rows where the left is not NULL. It\ncovers array_union for every element type, so it fixes the eager\nevaluation this guard was for without the duplication.\n\nThis reverts commit 88ec629827.\n\n* perf: normalize only the child range a sliced list references\n\nnormalize_nested_floats rebuilt a list\u0027s whole child array, but a sliced list\nkeeps its child whole, and explode hands out slices of one child array for a\nwhole input batch. Rebuild only the range the offsets reference, at every\ndepth, and rebase the offsets onto it.\n\nThe float_arrays bench gains array_distinct and array_union cases over whole\nand sliced lists.\n\n* test: cover array_union over ARRAY\u003cDOUBLE\u003e with a NULL left array\n\n#6716\u0027s array_null_short_circuit*.sql fixtures only use array\u003cint\u003e, which\nruns through DataFusion\u0027s array_union. A float element type goes through\nspark_array_union, so nothing checked that withNullShortCircuit reaches\nit. Without the guard, native evaluation raises on rows Spark skips:\nINVALID_PARAMETER_VALUE.START for slice(b, 0, 1) over a NULL array, and\nCAST_INVALID_INPUT for an ANSI cast of a malformed string.\n\nThe fixture opts ArrayUnion in so every Spark version runs the float path\nnatively, not only 4.0.5+, 4.1.4+ and 4.2+. The data has no signed zero,\nwhich earlier releases keep apart." } ], "next": "33f4f888a0f49c1eb933149c1f0c502a7b192748" }