)]}' { "log": [ { "commit": "0caba769278ee7992f018151ad9a1ee091e32827", "tree": "76de00a33b8571e34d198928d9a73a6500e50d8b", "parents": [ "188544238d613c2bb5ddfe982efcfc93bf94e36f" ], "author": { "name": "haoyangeng-db", "email": "haoyan.geng@gmail.com", "time": "Thu Jul 30 23:27:17 2026 +0800" }, "committer": { "name": "Wenchen Fan", "email": "wenchen@databricks.com", "time": "Thu Jul 30 23:27:17 2026 +0800" }, "message": "[MINOR][SQL][TEST] Add catch-all case to non-total onOtherEvent in test listeners\n\n### What changes were proposed in this pull request?\n\nAdd a catch-all `case _ \u003d\u003e` as the final arm to a few SparkListener.onOtherEvent match blocks in test suites that enumerate only the specific event type(s) they care about and omit a catch-all: PluginContainerSuite, SQLExecutionSuite (jobTags / jobGroupId tests), and SparkConnectServiceInternalServerSuite.\n\n### Why are the changes needed?\n\nSparkListenerBus.doPostEvent routes every non-built-in event to onOtherEvent, so a shared-queue listener receives all such events, not just the ones it enumerates. A match block with no `case _ \u003d\u003e` throws a scala.MatchError on every other event; ListenerBus.postToAll logs and swallows it, so tests pass but the logs are spammed with MatchError stack traces. The catch-all matches the convention the built-in listeners already follow and is always added last, so it never shadows an existing case.\n\n### How was this patch tested?\n\nThis change is test-only.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nCo-authored w/ Claude Code.\n\nCloses #57618 from haoyangeng-db/minor-sparklistener-onotherevent-catchall.\n\nAuthored-by: haoyangeng-db \u003chaoyan.geng@gmail.com\u003e\nSigned-off-by: Wenchen Fan \u003cwenchen@databricks.com\u003e\n" }, { "commit": "188544238d613c2bb5ddfe982efcfc93bf94e36f", "tree": "898959e04e537e149c50459160d6265bdfd5f7cf", "parents": [ "d5ec6583c9ce7d9bde1a2f9948ff4a563cd79f51" ], "author": { "name": "Boyang Jerry Peng", "email": "jerry.peng@databricks.com", "time": "Thu Jul 30 20:16:33 2026 +0800" }, "committer": { "name": "Wenchen Fan", "email": "wenchen@databricks.com", "time": "Thu Jul 30 20:16:33 2026 +0800" }, "message": "[SPARK-58398][CORE] Group-atomic failure and fail-fast rejection for pipelined-shuffle stage groups\n\n### What changes were proposed in this pull request?\n\nThis PR combines the two failure-handling layers of the pipelined-shuffle work into a single\nreviewable change, on top of the now-merged concurrent scheduling of a `PipelinedShuffleDependency`\ngroup (SPARK-58263 / #57341): the **group-atomic failure model** and the **fail-fast rejection of\nunsupported idioms** that together make co-scheduling safe under failure. (It builds on two merged\npredecessors -- the `PipelinedShuffleDependency` definition and type-based routing from\nSPARK-58185 / #57286, and the concurrent scheduling from SPARK-58263 / #57341 -- and targets\n`master` directly; its net-new content is the failure and fail-fast layers described below.)\n\nA pipelined shuffle is transient and incrementally read: its output is streamed to a co-scheduled\nconsumer and is never materialized as durable, re-readable map output. That has two consequences the\nscheduler must enforce.\n\n**Group-atomic failure (a pipelined group succeeds or fails as a whole).** A transient shuffle\ncannot be recomputed in isolation and its members run concurrently, so the stock \"resubmit one\nstage\" recovery does not apply -- any member failure must fail the whole group, which the caller\nthen reruns (a streaming query restarts the batch).\n\n- **Member task failure.** A pipelined group member\u0027s `TaskSet` is tagged `isPipelined` and pinned\n to `maxTaskFailures \u003d 1`, so the first task failure aborts the set instead of retrying in place,\n which routes to a whole-group abort.\n- **Executor loss.** Losing an executor running a member task is force-counted as a member failure\n (except benign `TaskKilled` / `TaskCommitDenied`), aborting the group.\n- **FetchFailed on a member.** Handled by a dedicated branch that aborts the whole group rather than\n resubmitting the map stage in isolation (a lone-stage resubmit of a transient shuffle would\n deadlock the group). The failed executor\u0027s *regular* outputs are still unregistered from the\n `MapOutputTracker`, exactly as the base handler does, for the benefit of other jobs.\n- **No transient-producer resubmit.** A pipelined `ShuffleMapStage` records its completed partitions\n locally and monotonically (never in the `MapOutputTracker`) and never flips back to unavailable, so\n losing an executor that held an already-consumed pipelined output cannot make the scheduler\n resubmit the producer -- which would otherwise hang the producer\u0027s streaming writer waiting on\n termination acks from reducers that already finished. The `TaskSetManager` \"Resubmitted\"\n re-enqueue loop likewise excludes pipelined sets.\n- **Cross-job / cross-time reuse rejected.** A transient shuffle has no retained output for a second\n job to read, so binding a pipelined producer stage to a second job fails fast.\n\n**Fail-fast on unsupported idioms (spec S9).** A pipelined group is rejected up front, before any\nstage is created (so a rejection leaves no partial scheduler state), when it uses an idiom v1 cannot\nsupport: a producer feeding **more than one consumer** (1:N fan-out needs multicast, deferred); a\n**barrier**, **statically-indeterminate**, **checksum-mismatch-retry**, or **push-merge** producer;\na **reliable RDD checkpoint** anywhere in a member\u0027s within-stage chain (it reintroduces cross-time\nreuse of a transient edge); or **members with differing resource profiles** (the gang slot check\ncompares one demand against one profile\u0027s capacity, so v1 requires a single-profile group). These\nthrow a typed `PipelinedShuffleUnsupportedException` (carrying the `PIPELINED_SHUFFLE_UNSUPPORTED`\nerror class), which `handleJobSubmitted` matches by type.\n\nMain changes:\n\n- `DAGScheduler.scala` -- the FetchFailed group-abort branch, the no-resubmit handling of a\n pipelined `ShuffleMapStage`, cross-job reuse rejection, and `checkPipelinedGroupsSupportedInRDDGraph`\n / `checkPipelinedProducerSupported` fail-fast (typed exception).\n- `TaskSetManager.scala` -- `maxTaskFailures \u003d 1` and force-counted executor loss for a pipelined\n set; exclusion from the \"Resubmitted\" re-enqueue loop.\n- `ShuffleMapStage.scala` -- monotonic local availability for a pipelined shuffle.\n- `PIPELINED_SHUFFLE_UNSUPPORTED` and `PIPELINED_SHUFFLE_CROSS_JOB_REUSE` error conditions.\n\n### Why are the changes needed?\n\nCo-scheduling a pipelined group (#57341) is only safe if failure is handled at the granularity of\nthe whole group: because the shuffle is transient and once-through, the stock per-stage resubmit\nrecovery would either deadlock the group or hang a producer\u0027s streaming writer. This PR makes any\nmember failure fail the group atomically (so the caller reruns it) and rejects up front the idioms\nwhose recovery/semantics are incompatible with a transient, concurrently-read shuffle -- turning\nwhat would be a hang or a silently-wrong schedule into a clear, immediate failure. The two layers\nare combined into one PR because they are inseparable in review: the fail-fast rejections define\nexactly which group shapes the failure model must then handle, and both are gated on the same\n`PipelinedShuffleDependency` type.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. All new behavior is gated on a job using a `PipelinedShuffleDependency`, which nothing constructs\nyet, so every existing job is scheduled and recovers exactly as before. The new error conditions\n(`PIPELINED_SHUFFLE_UNSUPPORTED`, `PIPELINED_SHUFFLE_CROSS_JOB_REUSE`) can only surface for a job\nthat uses a pipelined dependency.\n\n### How was this patch tested?\n\nNew unit tests in `DAGSchedulerSuite` and `TaskSetManagerSuite` cover:\n\n- group-atomic failure: `maxTaskFailures \u003d 1` aborting a member set on the first failure; executor\n loss force-counted (and benign `TaskKilled` / `TaskCommitDenied` not force-counted); a member\n FetchFailed aborting the whole group rather than resubmitting a single stage;\n- no transient-producer resubmit: a post-executor-loss straggler success not resubmitting the\n producer; a completed pipelined producer\u0027s availability surviving executor loss; the \"Resubmitted\"\n loop excluding a pipelined set (including the partial-producer-on-decommission case);\n- cross-job reuse rejected; a group-atomic rerun resetting per-partition commit authorization;\n- fail-fast idioms: fan-out, barrier / indeterminate / checksum / push-merge producer, a reliable\n checkpoint in a producer\u0027s or a consumer\u0027s chain (including downstream in the consumer stage), and\n a mixed-resource-profile group -- each rejected up front; and that regular-shuffle idioms are NOT\n rejected (inertness of the fail-fast for a job with no pipelined dependency).\n\nThe full `DAGSchedulerSuite` and `TaskSetManagerSuite` pass.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nCo-authored with Claude Code (Opus 4.8)\n\nCloses #57361 from jerrypeng/stack/pipelined-shuffle-pr6-failfast.\n\nAuthored-by: Boyang Jerry Peng \u003cjerry.peng@databricks.com\u003e\nSigned-off-by: Wenchen Fan \u003cwenchen@databricks.com\u003e\n" }, { "commit": "d5ec6583c9ce7d9bde1a2f9948ff4a563cd79f51", "tree": "a500c46c1c5f0f5aba11892208174121163ba7d2", "parents": [ "166dbc560441c0c39a8368a82c7b1d451ca71ea9" ], "author": { "name": "ChuckLin2025", "email": "lzequn@gmail.com", "time": "Thu Jul 30 19:49:52 2026 +0800" }, "committer": { "name": "Wenchen Fan", "email": "wenchen@databricks.com", "time": "Thu Jul 30 19:49:52 2026 +0800" }, "message": "[SPARK-58399][SQL][PYTHON] Add `collect_union` aggregate function\n\n### What changes were proposed in this pull request?\n\nThis PR adds a new aggregate function `collect_union` that takes an\narray-typed column and returns the distinct union of the elements of the\narrays across rows.\n\n`collect_union(col: array\u003cT\u003e) : array\u003cT\u003e`\n\nIt is equivalent to `array_distinct(flatten(collect_list(col)))`, but the\naggregation buffer holds only the distinct elements (a `HashSet`), so its\nsize is bounded by the element universe rather than by the number of input\nrows. This avoids buffering every row\u0027s whole array, which for a hot\ngrouping key can grow without bound.\n\nThe function is implemented as a `Collect[mutable.HashSet[Any]]`\n(sibling of `collect_set`); the only material difference is that `update`\niterates the input array and adds each non-null element, and the result\nelement type is the input array\u0027s element type. NULL input arrays and NULL\nelements are skipped, following `collect_set` semantics.\n\nAdded across the usual surfaces: Catalyst expression + registry, the Scala\nDataFrame API, and PySpark (classic + Spark Connect). Spark Connect needs\nno protocol change: the function travels as a generic `UnresolvedFunction`\nresolved against the registry.\n\n### Why are the changes needed?\n\nThere is no built-in aggregate that unions the elements of an array column\nacross rows into a single distinct array. The workaround\n`array_distinct(flatten(collect_list(arr)))` buffers every row\u0027s whole\narray before de-duplicating, which can OOM on skewed keys. `collect_union`\nde-duplicates during aggregation, keeping the buffer bounded by the\ndistinct-element universe.\n\nNote that `collect_set` cannot replace this. `collect_set(element)` over\n`explode(col)` does bound the buffer, but it stops being a plain aggregate:\neach array column must be exploded and grouped on its own and then joined\nback on the grouping keys. A query needing the distinct union of N array\ncolumns therefore pays N explodes + N joins purely to work around the\nmissing array-input aggregate. `collect_union` keeps the bounded buffer\nwhile staying an ordinary aggregate, so multiple array columns aggregate\ntogether in one GROUP BY with no join.\n\nIndustry precedent: BigQuery (GoogleSQL) already supports this style of\narray-input aggregate (`ARRAY_CONCAT_AGG`, which concatenates arrays across\nrows; distinct is then applied), whereas PostgreSQL has no dedicated\nfunction and users fall back to `array_agg(DISTINCT ...)` over `unnest(...)`\n(the analogue of the explode + `collect_set` workaround above). `collect_union`\ngives Spark a first-class, bounded-buffer form of this operation.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. It adds a new SQL function `collect_union` and the corresponding\n`functions.collect_union` in the Scala and Python DataFrame APIs.\n\n### How was this patch tested?\n\n- New `collect_union function` case in `DataFrameAggregateSuite` (distinct\n union, NULL array, NULL element, per-group, empty result). Full suite:\n 170 tests, all pass.\n- New `test_collect_union` in `python/pyspark/sql/tests/test_functions.py`\n covering `array\u003cint\u003e`, `array\u003cstring\u003e`, `array\u003cdouble\u003e`, NULL elements,\n and `array\u003cstruct\u003e` (passes end-to-end through the PySpark runtime).\n- Spark Connect parity check in `test_connect_function.py`.\n- `ExpressionsSchemaSuite` regenerated `sql-expression-schema.md`.\n\nCloses #57592 from ChuckLin2025/collect_union-oss.\n\nLead-authored-by: ChuckLin2025 \u003clzequn@gmail.com\u003e\nCo-authored-by: Zequn Lin \u003cchuck.lin@databricks.com\u003e\nSigned-off-by: Wenchen Fan \u003cwenchen@databricks.com\u003e\n" }, { "commit": "166dbc560441c0c39a8368a82c7b1d451ca71ea9", "tree": "04b2b354037c82c9b2342ee31117e7577817e135", "parents": [ "54dce691f533cb558a7c8403e90d90b8c971dc50" ], "author": { "name": "Mark Jarvin", "email": "mark.jarvin@databricks.com", "time": "Thu Jul 30 17:08:55 2026 +0800" }, "committer": { "name": "Wenchen Fan", "email": "wenchen@databricks.com", "time": "Thu Jul 30 17:08:55 2026 +0800" }, "message": "[SPARK-57158][SQL] ExplainUtils: extract operator ID assignment phase from processPlan into a private helper\n\n### What changes were proposed in this pull request?\n\nExtract the operator-ID-assignment phase of `ExplainUtils.processPlan` into a private `assignOperatorIds(plan, idMap)` helper. `processPlan` now initializes the idMap, delegates all ID assignment to `assignOperatorIds`, then performs the text-output pass over the returned subqueries and optimized-out exchanges.\n\n### Why are the changes needed?\n\n`processPlan` conflated two independent sequential phases in one ~40-line body:\n\n1. **ID assignment** — traversing the plan tree, subqueries, and adaptively-optimized-out exchanges (SPARK-42753) to populate an `IdentityHashMap` with monotonically-increasing operator IDs.\n2. **Text output** — calling `processPlanSkippingSubqueries` on each discovered subtree to format the verbose explain string.\n\nThese phases are sequential and independent: the text-output pass only begins after ID assignment is fully complete. Extracting phase 1 into `assignOperatorIds` shortens `processPlan` to its output logic and makes the boundary between the two phases explicit. No behavior change.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nAdded `ExplainUtilsSuite` covering:\n- Operator IDs assigned to all visible plan nodes are unique\n- `processPlan` sets `CODEGEN_ID_TAG` on nodes inside `WholeStageCodegenExec`\n- Thread-local `localIdMap` is restored to its prior value after `processPlan` returns\n- Subquery section is emitted in the explain output when subqueries are present\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude (Anthropic)\n\nCloses #56216 from markj-db/mark-jarvin_data/explain-utils-generate-plan-ids.\n\nAuthored-by: Mark Jarvin \u003cmark.jarvin@databricks.com\u003e\nSigned-off-by: Wenchen Fan \u003cwenchen@databricks.com\u003e\n" }, { "commit": "54dce691f533cb558a7c8403e90d90b8c971dc50", "tree": "3991d2ef67e5d282ea4a47e979ce8f6898007e0a", "parents": [ "6e8829dbccd7d8afa61a504389eb48084f0d999b" ], "author": { "name": "Stevo Mitric", "email": "stevomitric2000@gmail.com", "time": "Thu Jul 30 08:07:47 2026 +0000" }, "committer": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Thu Jul 30 08:07:47 2026 +0000" }, "message": "[SPARK-57811][SQL] Support string to nanosecond-precision timestamp coercion in comparisons and predicates\n\n### What changes were proposed in this pull request?\nAdd non-ANSI type-coercion arms so that a string compared against a nanosecond-precision timestamp column (`TIMESTAMP_NTZ(p)`/`TIMESTAMP_LTZ(p)`, p in [7, 9]) is cast to that nanos type, mirroring the existing microsecond `TimestampType` handling (`StringPromotionTypeCoercion` equality arms + `TypeCoercion.findCommonTypeForBinaryComparison`).\n\n### Why are the changes needed?\nMicros TimestampType has string-coercion arms that honor ...datetimeToString.enabled (legacy → promote to string; equality → cast to timestamp). Nanos had none, so it fell through to config-blind AtomicType promotion. This adds the arms so nanos matches TimestampType\u0027s legacy behavior.\n\n### Does this PR introduce _any_ user-facing change?\nOnly under legacy datetimeToString\u003dtrue + ANSI off: range comparisons (\u003c, BETWEEN, …) now promote to string (matching micros); equality unchanged. All other configs identical.\n\n### How was this patch tested?\nExtended existing suites and added golden file tests.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Claude Code\n\nCloses #57223 from stevomitric/stevomitric/spark-57811-nanos.\n\nAuthored-by: Stevo Mitric \u003cstevomitric2000@gmail.com\u003e\nSigned-off-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\n" }, { "commit": "6e8829dbccd7d8afa61a504389eb48084f0d999b", "tree": "a227612d5060d588e7b928ebf6dd621103008642", "parents": [ "92ad0991c1c9aaee995584bb93b7766d57b62dee" ], "author": { "name": "YangJie", "email": "yangjie01@baidu.com", "time": "Thu Jul 30 14:31:05 2026 +0800" }, "committer": { "name": "yangjie01", "email": "yangjie01@baidu.com", "time": "Thu Jul 30 14:31:05 2026 +0800" }, "message": "[SPARK-58365][SQL] Build the NOT-IN-in-disjunction join condition from the deduplicated subquery output\n\n### What changes were proposed in this pull request?\n\nIn `RewritePredicateSubquery.rewriteExistentialExprWithAttrs`, the `Not(InSubquery(...))` branch (the one handling a NOT IN nested inside a disjunction, e.g. `v \u003e 0 OR x NOT IN (...)`) calls `dedupSubqueryOnSelfJoin` to alias the subquery\u0027s attributes when they conflict with the outer plan, but then builds the IN equality conditions from the pre-dedup `sub.output` instead of the deduplicated `newSub.output`. The join\u0027s right child uses `newSub`, so the condition can reference attributes that are no longer on the right side. This changes `sub.output` to `newSub.output`, matching the three sibling branches that already do this (the plain `InSubquery` branch in the same method, and both the top-level IN and NOT IN branches in `apply`).\n\n### Why are the changes needed?\n\nWhen `dedupSubqueryOnSelfJoin` fires, it rebinds the conflicting subquery attributes to fresh exprIds. Building the condition from `sub.output` then uses the stale ids, which only exist on the outer side, so the null-aware anti-join condition collapses to trivially-true self-equalities like `id#2 \u003d id#2` and no longer references the join\u0027s right child. That is exactly the SPARK-26078 defect the `dedupSubqueryOnSelfJoin` call is there to prevent, so today that call is dead weight on this branch. Analysis-time `DeduplicateRelations` currently renews subquery exprIds before the optimizer runs, so this is not reachable from user SQL on current `master` and produces no wrong results today. It is a latent correctness hole: any future change that lets an outer/subquery exprId conflict reach this rule would silently return wrong NOT IN results, and the branch is the odd one out among four otherwise-consistent sites.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. When dedup does not fire, `newSub` is the same object as `sub`, so the change is a no-op on every plan reachable from user SQL today.\n\n### How was this patch tested?\n\nAdded a `RewriteSubquerySuite` case that builds the colliding-attribute plan directly (bypassing the analyzer\u0027s `DeduplicateRelations`, which would otherwise renew the ids) and asserts the rewritten join condition references the deduplicated right-side output. It fails on the unfixed tree (the condition is `(a#0 \u003d a#0) OR isnull((a#0 \u003d a#0))`, referencing nothing on the right) and passes with the fix.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57558 from LuciferYang/SPARK-58365-notin-dedup.\n\nAuthored-by: YangJie \u003cyangjie01@baidu.com\u003e\nSigned-off-by: yangjie01 \u003cyangjie01@baidu.com\u003e\n" }, { "commit": "92ad0991c1c9aaee995584bb93b7766d57b62dee", "tree": "096629f20e9f535f43d12656672a1d077cf9501d", "parents": [ "91b1abdd40106a51b4263d83866a4bfc861c7ae2" ], "author": { "name": "YangJie", "email": "yangjie01@baidu.com", "time": "Thu Jul 30 14:27:23 2026 +0800" }, "committer": { "name": "yangjie01", "email": "yangjie01@baidu.com", "time": "Thu Jul 30 14:27:23 2026 +0800" }, "message": "[SPARK-58373][SQL] Do not prune the from_json schema under named_struct when parse options are set\n\n### What changes were proposed in this pull request?\n\nThe `CreateNamedStruct` branch of `OptimizeCsvJsonExprs` prunes a `from_json` schema down to the fields the struct selects, but unlike the sibling `GetStructField` and `GetArrayStructFields` branches it does not require the parse options to be empty. This adds the same `options.isEmpty` guard, so the rewrite is skipped whenever any option is set.\n\n### Why are the changes needed?\n\nPruning the schema stops the parser from converting the dropped fields, so a malformed value in one of them is never reported. Under `mode\u003dFAILFAST` that turns a query that should fail into one that silently returns a row:\n\n```sql\nSELECT named_struct(\n \u0027a\u0027, from_json(value, \u0027a int, b int, c int\u0027, map(\u0027mode\u0027, \u0027FAILFAST\u0027)).a,\n \u0027b\u0027, from_json(value, \u0027a int, b int, c int\u0027, map(\u0027mode\u0027, \u0027FAILFAST\u0027)).b)\nFROM data -- value \u003d \u0027{\"a\": 1, \"b\": 2, \"c\": \"bad\"}\u0027\n```\n\nWith `spark.sql.optimizer.enableJsonExpressionOptimization\u003dfalse` this raises `MALFORMED_RECORD_IN_PARSING`, because `c` is parsed and rejected. With the optimization on (the default) the schema is pruned to `a int, b int`, `c` is skipped, and the query returns `{a: 1, b: 2}`.\n\nSPARK-32968 added this branch and SPARK-33907 added the `options.isEmpty` guard the following day, but only to the two `GetStructField`-style branches, so this one has been unguarded since 3.1.0.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. `named_struct` over a `from_json` that carries parse options now honors those options again: a malformed record fails under `FAILFAST` instead of being silently accepted. Queries whose `from_json` has no options are unaffected. Note the rewrite also collapses several `from_json` evaluations into one; that consolidation is now skipped for the options-set case as well, matching what SPARK-33907 already accepted for the sibling branches.\n\n### How was this patch tested?\n\nAdded an end-to-end `checkError` test in `JsonFunctionsSuite` that runs the query above with the optimization both on and off and asserts `MALFORMED_RECORD_IN_PARSING` either way; it fails on the unfixed tree because no exception is thrown. The bad field is a type mismatch rather than a structurally broken record: a structural malformation fails at tokenization no matter which schema is requested and would hide the pruning.\n\nAdded a plan-level test in `OptimizeJsonExprsSuite` asserting the rewrite is skipped for two different option maps (a parse mode and a formatting option, since the guard rejects any option), with an empty-options control asserting the same shape is still rewritten.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Code (Opus 5)\n\nCloses #57605 from LuciferYang/SPARK-58373-json-prune-options.\n\nAuthored-by: YangJie \u003cyangjie01@baidu.com\u003e\nSigned-off-by: yangjie01 \u003cyangjie01@baidu.com\u003e\n" }, { "commit": "91b1abdd40106a51b4263d83866a4bfc861c7ae2", "tree": "69190ff6993ab0c6631c6084de10535159cd80d8", "parents": [ "6b0c5f8d02dec66257cbfbb9287a78bf8f7e783e" ], "author": { "name": "YangJie", "email": "yangjie01@baidu.com", "time": "Thu Jul 30 14:18:29 2026 +0800" }, "committer": { "name": "yangjie01", "email": "yangjie01@baidu.com", "time": "Thu Jul 30 14:18:29 2026 +0800" }, "message": "[SPARK-58403][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_3201-3205`: `MALFORMED_EXPRESSION_INFO`\n\n### What changes were proposed in this pull request?\n\nThis PR proposes to assign a proper error condition for the legacy error conditions `_LEGACY_ERROR_TEMP_3201`, `_3202`, `_3203`, `_3204` and `_3205`, which are all thrown from the constructor of `ExpressionInfo` when the metadata describing an expression is malformed.\n\nThe five legacy conditions are folded into a single umbrella condition `MALFORMED_EXPRESSION_INFO` with five subclasses, one per validated field:\n\n| Legacy | New condition | Field |\n|---|---|---|\n| `_LEGACY_ERROR_TEMP_3201` | `MALFORMED_EXPRESSION_INFO.NOTE` | `note` |\n| `_LEGACY_ERROR_TEMP_3202` | `MALFORMED_EXPRESSION_INFO.GROUP` | `group` |\n| `_LEGACY_ERROR_TEMP_3203` | `MALFORMED_EXPRESSION_INFO.SOURCE` | `source` |\n| `_LEGACY_ERROR_TEMP_3204` | `MALFORMED_EXPRESSION_INFO.SINCE` | `since` |\n| `_LEGACY_ERROR_TEMP_3205` | `MALFORMED_EXPRESSION_INFO.DEPRECATED` | `deprecated` |\n\nThe shared umbrella message names the offending field and expression (`\u0027\u003cfieldName\u003e\u0027 is malformed in the expression [\u003cexprName\u003e]:`), and each subclass carries the field-specific detail. `fieldName` is a new message parameter, so `getMessageParameters()` for these errors now carries one extra key.\n\nThe assigned SQLSTATE is `22023` (invalid parameter value), consistent with the sibling `MALFORMED_*` conditions that validate a value against an allowed set or format (`MALFORMED_RECORD_IN_PARSING`, `MALFORMED_VARIANT`, and the `INVALID_PARAMETER_VALUE` archetype all use `22023`).\n\n**On reachability, and why these get a proper name rather than `INTERNAL_ERROR`:** none of the five throw sites is reachable from a user query. `FunctionRegistryBase.expressionInfo` reads the compile-time `ExpressionDescription` annotation, and `SessionCatalog.makeExprInfoForHiveFunction` / `SQLFunction.toExpressionInfo` pass constants (`\"\"` / `\"hive\"` / `\"sql_udf\"`). A malformed value can only come from extension or third-party code: constructing `new ExpressionInfo(...)` directly (which is what `SparkSessionExtensions.injectFunction` takes, see the example in `SparkSessionExtensionsProvider`), or calling `FunctionRegistryBase.createOrReplaceTempFunction(name, builder, source)` with an arbitrary `source`. That makes these developer-facing rather than engine-internal invariants: the person who triggers the error is the one who can fix it, so a named, actionable condition fits better than an internal error. This mirrors existing extension/configuration-author-facing conditions such as `CANNOT_LOAD_CATALOG` and `CANNOT_LOAD_FUNCTION_CLASS`, which also carry standard SQLSTATEs.\n\n### Why are the changes needed?\n\n`_LEGACY_ERROR_TEMP_*` conditions are placeholders that should be replaced with proper, named error conditions per the guideline in `common/utils/src/main/resources/error/README.md`. This is part of the ongoing effort to migrate legacy error conditions to the structured error framework.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. As described above, these conditions are only reachable by extension or third-party code that builds an `ExpressionInfo` itself, and the `_LEGACY_ERROR_TEMP_*` names were never part of the public API.\n\nFor completeness, the message text is preserved with four deliberate changes:\n\n- the `GROUP` value is now bracketed (`however, got \u003cgroup\u003e.` -\u003e `however, got [\u003cgroup\u003e].`) for consistency with the other four subclasses;\n- the split between the field and the detail moved from `.` to `:`, and the detail therefore starts with a lowercase `it should` instead of `It should`;\n- the rendered message now carries the `[MALFORMED_EXPRESSION_INFO.\u003cSUBCLASS\u003e] ` prefix, which `SparkThrowableHelper.formatErrorMessage` omits only for `_LEGACY_ERROR_`-prefixed names;\n- the rendered message now carries a ` SQLSTATE: 22023` suffix, since these five legacy entries previously had no `sqlState` at all.\n\nBefore and after, for `GROUP`:\n\n```\nOLD: \u0027group\u0027 is malformed in the expression [testName]. It should be a value in [...]; however, got invalid_group_funcs.\nNEW: [MALFORMED_EXPRESSION_INFO.GROUP] \u0027group\u0027 is malformed in the expression [testName]: it should be a value in [...]; however, got [invalid_group_funcs]. SQLSTATE: 22023\n```\n\n### How was this patch tested?\n\nUpdated the existing assertions in `ExpressionInfoSuite` to check the new conditions and parameters, and added `sqlState \u003d Some(\"22023\")` to each so the assigned SQLSTATE is pinned by a test. Ran:\n\n- `ExpressionInfoSuite` - 10/10 passed\n- `SparkThrowableSuite` - 34/34 passed (JSON validity, alphabetical ordering, mandatory SQLSTATE, round-trip)\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57604 from LuciferYang/assign-name-legacy-3201-3205.\n\nAuthored-by: YangJie \u003cyangjie01@baidu.com\u003e\nSigned-off-by: yangjie01 \u003cyangjie01@baidu.com\u003e\n" }, { "commit": "6b0c5f8d02dec66257cbfbb9287a78bf8f7e783e", "tree": "46288847461c686f267d89b7e7f1007dbd0a7ab8", "parents": [ "6c37fe363429e7537a124b96f32fdb573bd3b896" ], "author": { "name": "Parth Chandra", "email": "parthc@apple.com", "time": "Wed Jul 29 17:52:47 2026 -0700" }, "committer": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Wed Jul 29 17:52:47 2026 -0700" }, "message": "[SPARK-38954][CORE] Support delegation token renewal and distribution without Kerberos\n\n### What changes were proposed in this pull request?\n\n This PR enables Spark\u0027s existing credential distribution mechanism (driver obtains tokens, pushes them to executors, refreshes before expiry) to work without Kerberos. Today this mechanism is unconditionally gated\n on Kerberos being enabled. This change adds a single configuration switch (`spark.security.credentials.directProviders.enabled`) that lifts the Kerberos gate on token collection.\n\n Changes:\n - `HadoopDelegationTokenManager.renewalEnabled` now also returns true when the new config is enabled.\n - `obtainDelegationTokens(creds)` and `obtainTokensAndScheduleRenewal()` branch on whether Kerberos credentials are present: if yes, call providers inside `doLogin()`/`doAs()` as before; if not (but the switch is\n on), call providers directly. Providers that require Kerberos self-gate via `delegationTokensRequired` returning `false`.\n - Per-provider try/catch for failure isolation — one provider throwing does not block others.\n - `SupportsDelegationToken` gains an overridable `tokenManagerRequired()` hook so scheduler backends can activate the token manager based on the new config.\n - Hard startup failure if `spark.network.crypto.enabled` is not true when the switch is active (prevents bearer tokens over plaintext RPC).\n - WARN log when the config is set but no providers are discovered via ServiceLoader.\n - Note: Standalone clusters are not supported (followup https://issues.apache.org/jira/browse/SPARK-58329)\n\n Design document: [SPIP: Cloud Credential Refresh and Distribution Without Kerberos](https://issues.apache.org/jira/browse/SPARK-38954)\n\n ### Why are the changes needed?\n\n In non-Kerberos deployments (cloud, internal IdP, cross-account access on YARN), each executor independently authenticates against identity services, causing thundering-herd load amplification, inconsistent\n credential state, and no centralized control over credential rotation. The existing distribution infrastructure is already provider-agnostic (Kafka proves this) — only the activation gates prevent it from running\n without Kerberos.\n\n ### Does this PR introduce _any_ user-facing change?\n\n Yes. New configuration:\n\n | Config | Default | Description |\n |--------|---------|-------------|\n | `spark.security.credentials.directProviders.enabled` | `false` | Enables delegation token collection and renewal without Kerberos. When true, the manager starts even if Hadoop security is not enabled, calling all\n providers whose `delegationTokensRequired` returns true. Requires `spark.network.crypto.enabled\u003dtrue`. |\n\n Existing deployments are unaffected (config defaults to false). When Kerberos IS present, all code paths are unchanged.\n\n ### How was this patch tested?\n\n New unit test suite `NonKerberosCredentialsSuite` (7 tests):\n - `renewalEnabled` returns true/false based on config\n - Providers are called without Kerberos when config is enabled\n - Providers with `delegationTokensRequired\u003dfalse` are skipped\n - A failing provider does not prevent other providers from running\n - Individual provider can be disabled via `spark.security.credentials.\u003cservice\u003e.enabled`\n - Startup fails if `spark.network.crypto.enabled` is not true\n\n Existing `HadoopDelegationTokenManagerSuite` (4 tests) passes unmodified — verifies no regression in Kerberos deployments.\n\n ### Was this patch authored or co-authored using generative AI tooling?\n\n Yes, Co-authored by Claude Code (opus 4.6)\n\nCloses #57285 from parthchandra/cloud-credentials.\n\nAuthored-by: Parth Chandra \u003cparthc@apple.com\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n" }, { "commit": "6c37fe363429e7537a124b96f32fdb573bd3b896", "tree": "f5c67536fc123a1837e7dd453645aec70f2af0dc", "parents": [ "0c1fd8fc590f01cf5814c8f035aec638a2e6b092" ], "author": { "name": "Tian Gao", "email": "gaogaotiantian@hotmail.com", "time": "Wed Jul 29 17:49:52 2026 -0700" }, "committer": { "name": "Tian Gao", "email": "gaogaotiantian@hotmail.com", "time": "Wed Jul 29 17:49:52 2026 -0700" }, "message": "[SPARK-58159][PYTHON] Support `with` statement for connect session\n\n### What changes were proposed in this pull request?\n\nSupport `with` statement for connect session.\n\n### Why are the changes needed?\n\nIt is supported for classic spark session. We should support this for connect session too.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, user can use\n\n```python\nwith SparkSession.builder.remote(\"local[*]\").getOrCreate() as spark:\n ...\n```\n\nNow\n\n### How was this patch tested?\n\nA feature test is added.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nNo.\n\nCloses #57291 from gaogaotiantian/add-connect-session-context.\n\nAuthored-by: Tian Gao \u003cgaogaotiantian@hotmail.com\u003e\nSigned-off-by: Tian Gao \u003cgaogaotiantian@hotmail.com\u003e\n" }, { "commit": "0c1fd8fc590f01cf5814c8f035aec638a2e6b092", "tree": "6a0e397bd0736b8587932719c68602c9e8433d2c", "parents": [ "0c00d055fc49616f1cadaef0f548792fbfe77605" ], "author": { "name": "Mihailo Aleksic", "email": "mihailo.aleksic@databricks.com", "time": "Wed Jul 29 17:47:31 2026 -0700" }, "committer": { "name": "Tian Gao", "email": "gaogaotiantian@hotmail.com", "time": "Wed Jul 29 17:47:31 2026 -0700" }, "message": "[SPARK-58037][PYTHON][TEST][FOLLOWUP] Keep DataFrame golden tests internal\n\n### What changes were proposed in this pull request?\n This PR follows up on #57122 by keeping the PySpark DataFrame golden test helper internal to the test package:\n\n - Remove the `python/MANIFEST.in` rule that included `pyspark/sql/tests/df_golden` files in packaged Python artifacts.\n - Move `pyspark.testing.df_golden` to `pyspark.sql.tests.df_golden.df_golden`.\n - Update the DataFrame golden tests to import the helper from its new internal test location.\n\n### Why are the changes needed?\n`pyspark.testing` is user-accessible, so putting the golden-file helper there makes it look like a supported PySpark testing API. The helper is only intended for Spark\u0027s internal tests. The golden test files also should not be shipped to PySpark users.\n\n### Does this PR introduce _any_ user-facing change?\nNo.\n\n### How was this patch tested?\nTest only change.\n\n### Was this patch authored or co-authored using generative AI tooling?\nYes.\n\nCloses #57639 from mihailoale-db/spark-58037-python-tests-followup.\n\nAuthored-by: Mihailo Aleksic \u003cmihailo.aleksic@databricks.com\u003e\nSigned-off-by: Tian Gao \u003cgaogaotiantian@hotmail.com\u003e\n" }, { "commit": "0c00d055fc49616f1cadaef0f548792fbfe77605", "tree": "14c8457bd5c3bc64de2b6d22f18c2f122ce7b690", "parents": [ "a0229e7b5d5b605ca326acdcf2e7c9c063c106de" ], "author": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Wed Jul 29 21:45:55 2026 +0000" }, "committer": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Wed Jul 29 21:45:55 2026 +0000" }, "message": "[MINOR][BUILD] Fix duplicated word in spark-profiler POM description\n\n### What changes were proposed in this pull request?\nRemoves a duplicated word in the `spark-profiler` module\u0027s Maven `\u003cdescription\u003e`: \"based on the the async profiler\" becomes \"based on the async profiler\".\n\n### Why are the changes needed?\nA plain typo in the POM description string; the sibling README already uses the correct wording.\n\n### Does this PR introduce _any_ user-facing change?\nNo.\n\n### How was this patch tested?\nPOM description string only; no functional or build impact. No tests needed.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57623 from uros-b/pom-profiler-description-typo.\n\nAuthored-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\nSigned-off-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\n" }, { "commit": "a0229e7b5d5b605ca326acdcf2e7c9c063c106de", "tree": "60661a257ce8234b165adbe6c9706faa84c1dbbe", "parents": [ "7692b9fafa93243026a5c7a6b386189c78174f90" ], "author": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Wed Jul 29 21:34:59 2026 +0000" }, "committer": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Wed Jul 29 21:34:59 2026 +0000" }, "message": "[SPARK-58416][SQL][EXAMPLE] Fix wrong class name in SqlNetworkWordCount usage messages\n\n### What changes were proposed in this pull request?\nFixes the usage message in the `SqlNetworkWordCount` streaming examples, which printed the wrong class name (`JavaNetworkWordCount` / `NetworkWordCount`, copied from the sibling example). Corrected in both the Java and Scala versions.\n\n### Why are the changes needed?\nA user running the example with too few arguments saw a usage line naming a different class. Every other reference in each file already uses the correct class name.\n\n### Does this PR introduce _any_ user-facing change?\nYes (minor): the stderr usage message now prints the correct class name.\n\n### How was this patch tested?\nExample-only string change; verified the corrected name matches each file\u0027s class and header. No tests needed.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57622 from uros-b/example-javasqlnetworkwordcount-usage.\n\nAuthored-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\nSigned-off-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\n" }, { "commit": "7692b9fafa93243026a5c7a6b386189c78174f90", "tree": "ac25fc6d6c24eb15c0fac39ee12288f26cb71041", "parents": [ "8df89f20fc5c800ef4935c1b49a7525ec9df8921" ], "author": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Wed Jul 29 21:21:25 2026 +0000" }, "committer": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Wed Jul 29 21:21:25 2026 +0000" }, "message": "[SPARK-58417][PS] Remove stale Python 2 reference in DataFrame.to_latex docstring\n\n### What changes were proposed in this pull request?\nRemoves the stale Python 2 branch from the `encoding` parameter docstring of `DataFrame.to_latex`, collapsing it to a single accurate line stating the default is `utf-8`.\n\n### Why are the changes needed?\nSpark requires Python \u003e\u003d 3.11 (`python_requires` in setup.py), so the \"\u0027ascii\u0027 on Python 2\" note is dead and misleading.\n\n### Does this PR introduce _any_ user-facing change?\nNo.\n\n### How was this patch tested?\nDocstring-only change. No tests needed.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57621 from uros-b/deadcomment-tolatex-python2.\n\nAuthored-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\nSigned-off-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\n" }, { "commit": "8df89f20fc5c800ef4935c1b49a7525ec9df8921", "tree": "93a8dc6d9a8329bb99aa1b6beb94c3cae5f17480", "parents": [ "4d670c377ac7eded97012ac6fcf3ed63c8cf6421" ], "author": { "name": "Andreas Neumann", "email": "anew@apache.org", "time": "Wed Jul 29 11:44:46 2026 -0700" }, "committer": { "name": "Szehon Ho", "email": "szehon.apache@gmail.com", "time": "Wed Jul 29 11:44:46 2026 -0700" }, "message": "[SPARK-58347][SDP] Thread conf.resolver through ColumnSelection.applyToSchema\n\n### What changes were proposed in this pull request?\n\nThis is a follow-up refactor for SDP AutoCDC. It reworks `ColumnSelection.applyToSchema` (and its private helper `lookupFieldIndices`) to accept a `Resolver` instead of a `caseSensitive: Boolean`:\n - `ColumnSelection.applyToSchema` / `lookupFieldIndices` now take a `Resolver`. `lookupFieldIndices` resolves each requested column via `schema.fieldNames.indexWhere(resolver(_, name))` rather than switching between `StructType.getFieldIndex` / `getFieldIndexCaseInsensitive` on a boolean.\n - `Scd2BatchProcessor.computeTrackedHistoryColumns` now takes a `Resolver` directly instead of deriving `caseSensitiveResolution` / `caseInsensitiveResolution` from a boolean.\n - All callers pass `spark.sessionState.conf.resolver` instead of `conf.caseSensitiveAnalysis`: `Scd1BatchProcessor.projectTargetColumnsOntoMicrobatch` (2 sites), `Scd2BatchProcessor.projectTargetColumnsOntoMicrobatch` (2 sites) and its `computeTrackedHistoryColumns` instance method, and `AutoCdcMergeFlow` (the `userSelectedSchema` projection and the construction-time track-history validation).\n - The `caseSensitivity` message parameter of `AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA` is preserved via a new `CaseSensitivityLabels.of(resolver)` overload that classifies the resolver by probing it (`!resolver(\"a\", \"A\")`), so error messages are byte-for-byte unchanged.\n\n### Why are the changes needed?\n\nThe AutoCDC code already resolves identifiers everywhere else through `conf.resolver` (the canonical Spark abstraction for case-aware identifier comparison). `applyToSchema` was the odd one out, threading a raw `caseSensitive` boolean and re-deriving a resolver at each layer. Passing the `Resolver` directly makes column matching consistent with the rest of the pipeline, removes the boolean-to-resolver round-trips, and lets `computeTrackedHistoryColumns` take the resolver it actually needs rather than reconstructing one.\n\n### Does this PR introduce _any_ user-facing change?\nNo. This is a pure refactor with no behavior change; the `caseSensitivity` label in the `AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA` error is preserved.\n\n### How was this patch tested?\n\nExisting tests, updated to pass resolvers instead of booleans:\n - `ChangeArgsSuite` and `AutoCdcFlowSuite` (65 tests) — including the case-sensitive/insensitive selection and missing-column error-message cases.\n - `Scd2BatchProcessor*` and `Scd1BatchProcessorMergeSuite` (150 tests) — exercising the microbatch projection and history-tracking paths under both case-sensitivity settings.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57610 from anew/spark-58347-resolver-columnselection.\n\nAuthored-by: Andreas Neumann \u003canew@apache.org\u003e\nSigned-off-by: Szehon Ho \u003cszehon.apache@gmail.com\u003e\n" }, { "commit": "4d670c377ac7eded97012ac6fcf3ed63c8cf6421", "tree": "73ae31dca1abb3ebe87448e8a66d8ca7efdb2f3a", "parents": [ "8bee0297e94b1750c86aec76e69afd7145d6b279" ], "author": { "name": "Andreas Neumann", "email": "anew@apache.org", "time": "Wed Jul 29 11:42:35 2026 -0700" }, "committer": { "name": "Szehon Ho", "email": "szehon.apache@gmail.com", "time": "Wed Jul 29 11:42:35 2026 -0700" }, "message": "[SPARK-58271][SDP] Allow AUTO CDC clauses in any order\n\n### What changes were proposed in this pull request?\n\nThe `AUTO CDC` command previously required its clauses in a fixed order after `KEYS (...)`: `APPLY AS DELETE WHEN`, then `SEQUENCE BY`, then `COLUMNS`, then `STORED AS SCD TYPE`, then `TRACK HISTORY ON`. This relaxes the grammar so the optional clauses form an unordered set, accepted in any order.\n\n- **Grammar** (`SqlBaseParser.g4`): `autoCdcParameters` now matches the five clauses as a repeatable alternation `(autoCdcDeleteClause | autoCdcSequenceByClause | autoCdcColumnsClause | autoCdcStoredAsClause | autoCdcTrackHistoryClause)*` instead of a fixed positional sequence.\n- **`AstBuilder.parseAutoCdcParams`**: reads each clause from the resulting list; rejects a clause supplied more than once with `DUPLICATE_CLAUSES` via the shared `checkDuplicateClauses` helper; and enforces the still-mandatory `SEQUENCE BY` explicitly (the grammar no longer requires it positionally) with a targeted error.\n\n### Why are the changes needed?\n\nAs more AUTO CDC options are added, a fixed clause order is hard for users to remember. Allowing arbitrary ordering makes the syntax easier to use, consistent with how other Spark commands (e.g. `CREATE TABLE`) accept their optional clauses in any order.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, a SQL syntax relaxation.\n- Before: the AUTO CDC clauses had to be written in one fixed order; any other order failed with `PARSE_SYNTAX_ERROR`.\n- After: the optional clauses may be written in any order. Supplying the same clause twice now fails with `DUPLICATE_CLAUSES` (\"Found duplicate clauses: \u003cclauseName\u003e.\"), and omitting the required `SEQUENCE BY` fails with a clear \"AUTO CDC requires a SEQUENCE BY clause.\" message. Previously valid statements continue to parse unchanged.\n\nThis is a relaxation within the unreleased AUTO CDC feature on master; no released behavior changes.\n\n### How was this patch tested?\n\nUpdated `AutoCdcParserSuite` (67 tests, all passing):\n\n- The former \"wrong clause order\" negative tests are now positive tests asserting the clauses parse in the new orders, including one statement with a fully reversed clause order.\n- Added a duplicate-clause rejection test for each of the five clauses (asserting `DUPLICATE_CLAUSES`).\n- Updated the `SEQUENCE BY is required` tests to assert the new targeted error.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57612 from anew/spark-58271-flexible-autocdc-clause-order.\n\nAuthored-by: Andreas Neumann \u003canew@apache.org\u003e\nSigned-off-by: Szehon Ho \u003cszehon.apache@gmail.com\u003e\n" }, { "commit": "8bee0297e94b1750c86aec76e69afd7145d6b279", "tree": "d5610423edbdfff9e1eab18dec8bed967f08649d", "parents": [ "5db219c37591d79cbce26d1e1f8ad252020fd0cd" ], "author": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Wed Jul 29 11:40:41 2026 -0700" }, "committer": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Wed Jul 29 11:40:41 2026 -0700" }, "message": "[SPARK-55199][K8S][DOC] Improve K8s integration tests `README`\n\n### What changes were proposed in this pull request?\n\nThis PR improves the K8s integration tests `README.md` to match the current scripts and build:\n\n- Document the default mode which builds Docker images directly from the current source tree, and\n remove the stale `TODO` which was already implemented.\n- Add a new `Running the YuniKorn Integration Tests` section, symmetric to the Volcano section.\n- Document missing script flags (`--image-repo`, `--include-tags`, `--exclude-tags`,\n `--default-exclude-tags`, `--java-version`, `--hadoop-profile`, `--skip-building-dependencies`)\n and properties (`test.include.tags`, `test.exclude.tags`, `test.default.exclude.tags`).\n- Fix the default value of `spark.kubernetes.test.dockerFile` and remove an outdated note.\n\n### Why are the changes needed?\n\nThe README has drifted from the actual behavior. Accurate documentation helps developers run the\nK8s integration tests correctly.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is a documentation-only change.\n\n### How was this patch tested?\n\nManual review.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #57632 from dongjoon-hyun/SPARK-55199.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n" }, { "commit": "5db219c37591d79cbce26d1e1f8ad252020fd0cd", "tree": "46ff43f419d660e80c3f5b79e43d7967ed3784fb", "parents": [ "fd4c5869a0d5633eeb4853debe929282f268d573" ], "author": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Wed Jul 29 11:39:45 2026 -0700" }, "committer": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Wed Jul 29 11:39:45 2026 -0700" }, "message": "[SPARK-58422][K8S][TEST] Upgrade the minimum Minikube version to 1.38.0\n\n### What changes were proposed in this pull request?\n\nThis PR aims to update the Minikube minimum version check in `Minikube.getKubernetesClient` from `1.28.0` to `1.38.0`, along with the corresponding assertion message, for Apache Spark 5.0.0.\n\n### Why are the changes needed?\n\nApache Spark 5 dropped the support of K8s v1.34 and below and also upgraded the minimum Minikube version to `1.38.0` already. This PR only makes the code consistent with the documentation.\n- #57048\n- #56605\n\nMinikube v1.38.x has been used stably with K8s v1.35+ since 2026-01-28 in our CIs.\n- https://github.com/kubernetes/minikube/releases/tag/v1.38.1 (2026-02-19)\n - https://github.com/kubernetes/minikube/pull/22665\n- https://github.com/kubernetes/minikube/releases/tag/v1.38.0 (2026-01-28)\n - https://github.com/kubernetes/minikube/pull/22328\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is a test-only change.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #57633 from dongjoon-hyun/minikube-min-version-1.38.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n" }, { "commit": "fd4c5869a0d5633eeb4853debe929282f268d573", "tree": "c59bed8f377633073e2e7e8d800f675fa05db223", "parents": [ "3ff822a4d50ecd5cdc9e46c6157c6d9e33a07157" ], "author": { "name": "Kousuke Saruta", "email": "sarutak@apache.org", "time": "Wed Jul 29 09:02:58 2026 -0700" }, "committer": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Wed Jul 29 09:02:58 2026 -0700" }, "message": "[SPARK-58369][CONNECT][TEST] Introduce a way to `SparkConnectServiceKeepAliveSuite` to wait until unbinding a port is complete\n\n### What changes were proposed in this pull request?\nThis PR introduce `stopSparkConnectServiceAndUnbind()` to `SparkConnectServiceKeepAliveSuite` as a way to wait until unbinding a port is complete.\n\n### Why are the changes needed?\nSparkConnectServiceKeepAliveSuite sometimes flakily fails. #57342 tried to fix it but it but it still happens.\nhttps://github.com/apache/spark/actions/runs/30041141074/job/89325300649\n```\n[info] - SPARK-58094: disabling spark.connect.grpc.keepAlive.enabled reverts to the pre-fix hang *** FAILED *** (6 milliseconds)\n[info] java.net.BindException: Failed to bind to address 0.0.0.0/0.0.0.0:15788: Service \u0027org.apache.spark.sql.connect.service.SparkConnectService\u0027 failed after 0 retries (starting from 15788)! Consider explicitly setting the appropriate port for the service \u0027org.apache.spark.sql.connect.service.SparkConnectService\u0027 (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.\n[info] at io.grpc.netty.NettyServer.start(NettyServer.java:341)\n[info] at io.grpc.internal.ServerImpl.start(ServerImpl.java:185)\n[info] at io.grpc.internal.ServerImpl.start(ServerImpl.java:94)\n[info] at org.apache.spark.sql.connect.service.SparkConnectService$.$anonfun$startGRPCService$1(SparkConnectService.scala:451)\n[info] at org.apache.spark.sql.connect.service.SparkConnectService$.$anonfun$startGRPCService$1$adapted(SparkConnectService.scala:411)\n[info] at org.apache.spark.util.Utils$.$anonfun$startServiceOnPort$2(Utils.scala:2276)\n[info] at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:256)\n[info] at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2268)\n[info] at org.apache.spark.sql.connect.service.SparkConnectService$.startGRPCService(SparkConnectService.scala:471)\n[info] at org.apache.spark.sql.connect.service.SparkConnectService$.start(SparkConnectService.scala:483)\n[info] at org.apache.spark.sql.connect.service.SparkConnectServiceKeepAliveSuite.$anonfun$new$17(SparkConnectServiceKeepAliveSuite.scala:262)\n[info] at org.apache.spark.SparkTestSuite.withSparkEnvConfs(SparkTestSuite.scala:268)\n[info] at org.apache.spark.SparkTestSuite.withSparkEnvConfs$(SparkTestSuite.scala:257)\n[info] at org.apache.spark.SparkFunSuite.withSparkEnvConfs(SparkFunSuite.scala:33)\n[info] at org.apache.spark.sql.connect.service.SparkConnectServiceKeepAliveSuite.$anonfun$new$16(SparkConnectServiceKeepAliveSuite.scala:262)\n...\n```\n\nThe root cause of the issue is that `SparkConnectService.stop()` may return before unbinding the port is complete in Linux environment even though `io.grpc.Server#awaitTermination()` is called within `SparkConnectService.stop()`.\nWe can see this behavior with the following code. With this code, `SparkConnectService.start()` may throw `BindException`.\n\n```\nfor (_ \u003c- 1 to 3000) {\n SparkConnectService.stop(Some(30), Some(TimeUnit.SECONDS))\n withSparkEnvConfs(\n Connect.CONNECT_GRPC_BINDING_PORT.key -\u003e serverPort.toString,\n Connect.CONNECT_GRPC_KEEPALIVE_ENABLED.key -\u003e \"false\",\n Connect.CONNECT_GRPC_KEEPALIVE_TIME.key -\u003e \"1s\",\n Connect.CONNECT_GRPC_KEEPALIVE_TIMEOUT.key -\u003e \"1s\") {\n SparkConnectService.start(spark.sparkContext)\n }\n}\n```\n\nWe use Netty in `SparkConnectService` so the socket is closed in `AbstractChannel#doClose()`, more specifically in [AbstractEpollChannel#doClose()](https://github.com/netty/netty/blob/3703d79669ee024f6483c9d8697ac58ba546df33/transport-classes-epoll/src/main/java/io/netty/channel/epoll/AbstractEpollChannel.java#L214) in Linux.\n`doClose()` is also called from [AbstractChannel#doClose0()](https://github.com/netty/netty/blob/3703d79669ee024f6483c9d8697ac58ba546df33/transport/src/main/java/io/netty/channel/AbstractChannel.java#L617)\nIn Linux environment, `doClose0()` is called asynchronously [here](https://github.com/netty/netty/blob/3703d79669ee024f6483c9d8697ac58ba546df33/transport/src/main/java/io/netty/channel/AbstractChannel.java#L574) because `EpollSocketChannelUnsafe` overrides [prepareToClose()](https://github.com/netty/netty/blob/3703d79669ee024f6483c9d8697ac58ba546df33/transport-classes-epoll/src/main/java/io/netty/channel/epoll/EpollSocketChannel.java#L159) and [closeExecutor](https://github.com/netty/netty/blob/3703d79669ee024f6483c9d8697ac58ba546df33/transport/src/main/java/io/netty/channel/AbstractChannel.java#L567) will be non-null.\nOn the other hand, I don\u0027t see `prepareToClose()` overridden in any class under `io.netty.channel.kqueue`. So the flakiness should not affect in Mac environment.\n\n### Does this PR introduce _any_ user-facing change?\nNo.\n\n### How was this patch tested?\nConfirmed that the following code doesn\u0027t throw `BindException`.\n```\nfor (_ \u003c- 1 to 3000) {\n stopSparkConnectServiceAndUnbind()\n withSparkEnvConfs(\n Connect.CONNECT_GRPC_BINDING_PORT.key -\u003e serverPort.toString,\n Connect.CONNECT_GRPC_KEEPALIVE_ENABLED.key -\u003e \"false\",\n Connect.CONNECT_GRPC_KEEPALIVE_TIME.key -\u003e \"1s\",\n Connect.CONNECT_GRPC_KEEPALIVE_TIMEOUT.key -\u003e \"1s\") {\n SparkConnectService.start(spark.sparkContext)\n }\n}\n```\n\n### Was this patch authored or co-authored using generative AI tooling?\nNo.\n\nCloses #57560 from sarutak/fix-flaky-SparkConnectServiceKeepAliveSuite.\n\nAuthored-by: Kousuke Saruta \u003csarutak@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n" }, { "commit": "3ff822a4d50ecd5cdc9e46c6157c6d9e33a07157", "tree": "6b1eeead6153652a17092238bc506b96fcf79b28", "parents": [ "a06f5910cb25c0b83a68d9bf8ad239fe6b7bbde1" ], "author": { "name": "Cheng Pan", "email": "pan3793@gmail.com", "time": "Wed Jul 29 09:01:00 2026 -0700" }, "committer": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Wed Jul 29 09:01:00 2026 -0700" }, "message": "[SPARK-58400][SQL] Fix ANSI mode assumption for views created by Spark 4.0+\n\n### What changes were proposed in this pull request?\n\n1. `Analyzer.trySetAnsiValue`: replace `createSparkVersion.startsWith(\"4.\")` with `VersionUtils.majorMinorPatchVersion(createSparkVersion).exists { case (major, _, _) \u003d\u003e major \u003e\u003d 4 }`. The old check only matched the 4.x major; views created by Spark 5.x (ANSI still the default) wrongly fell to ANSI\u003dfalse.\n2. View resolution now forwards `createSparkVersion` to `View.effectiveSQLConf` on all paths that build the throwaway conf from persisted `viewSQLConfigs` -- `ViewResolver.resolve` (single-pass) and `SessionCatalog.parseMetricViewDefinition` -- matching `SessionCatalog.fromCatalogTable` / the fixed-point `ViewResolution.resolve`, which already forwarded it.\n\n### Why are the changes needed?\n\nANSI became the default in 4.0 (SPARK-44444) and remains so in 5.x. `trySetAnsiValue`\u0027s `startsWith(\"4.\")` only recognized 4.x, so a 5.x-created view with no persisted ANSI value was resolved as LEGACY instead of ANSI -- its schema-enforcement casts used the wrong eval mode.\n\nThe single-pass `ViewResolver` and `parseMetricViewDefinition` compounded this by dropping `createVersion` entirely. This surfaced as a dual-run `LOGICAL_PLAN_COMPARISON_MISMATCH`: the two analyzers\u0027 plans differed only in `Cast.evalMode`, which `NormalizePlan` does not strip. On master the failing test passes only by coincidence (`\"5.0.0\"` and `\"\"` both miss `startsWith(\"4.\")` and both resolve to LEGACY).\n\n### Does this PR introduce _any_ user-facing change?\n\nBug fix. Views created by Spark 4.0+ without a persisted ANSI value are now resolved with ANSI\u003dtrue (matching their creating version\u0027s default) in both analyzers. Empty/unparseable `createVersion` still maps to ANSI\u003dfalse, as documented.\n\n### How was this patch tested?\n\nAdded `AlwaysPersistedConfigsSuite` -\u003e \"ANSI value derived from createSparkVersion when not persisted for views\", exercising `View.effectiveSQLConf`/`trySetAnsiValue` for `createSparkVersion` in `{\"3.5.0\", \"4.0.0\", \"4.0.0-SNAPSHOT\", \"5.0.0\", \"\", \"bogus\"}` (ANSI true for major `\u003e\u003d 4`, false otherwise/unparseable). Existing suites: `HiveSQLViewSuite`, `SQLViewSuite`, `ViewResolverSuite`, `HybridAnalyzerSuite`, `MetricViewSuite`, `MetricViewV2CatalogSuite`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nYes. Generated-by: GLM 5.2\n\nCloses #57593 from pan3793/ansi-view-fix.\n\nAuthored-by: Cheng Pan \u003cpan3793@gmail.com\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n" }, { "commit": "a06f5910cb25c0b83a68d9bf8ad239fe6b7bbde1", "tree": "1f349608a1b1187c1d72d96b941ccec82ca9a1e4", "parents": [ "51ff60d9dfc95e65b8aaa40420a343ea2f4c0ba4" ], "author": { "name": "Liang-Chi Hsieh", "email": "viirya@gmail.com", "time": "Wed Jul 29 08:59:50 2026 -0700" }, "committer": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Wed Jul 29 08:59:50 2026 -0700" }, "message": "[SPARK-58414][SQL][TEST][FOLLOWUP] Use the imported LocalDateTime instead of the fully qualified name\n\n### What changes were proposed in this pull request?\n\nTwo cleanups to the test added by #57619, both from dongjoon-hyun\u0027s post-merge reviews:\n\n1. Replace the fully qualified `java.time.LocalDateTime` references with the bare `LocalDateTime`, which the suite already imports.\n2. Run the nested nanos round trip under both `spark.sql.inMemoryColumnarStorage.enableVectorizedReader` settings via the same `Seq(false, true)` loop the neighboring round-trip tests use, since the suite\u0027s default pins the conf to `false`.\n\n### Why are the changes needed?\n\nAddresses the post-merge reviews on #57619.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, test-only.\n\n### How was this patch tested?\n\nThe affected test passes under both conf values.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nYes, this pull request and its description were written by Claude Code.\n\nCloses #57631 from viirya/arrow-cache-nested-nanos-followup.\n\nAuthored-by: Liang-Chi Hsieh \u003cviirya@gmail.com\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n" }, { "commit": "51ff60d9dfc95e65b8aaa40420a343ea2f4c0ba4", "tree": "5ba75b58c815427b994e2e8d2482439234e7fc1a", "parents": [ "f27705db432c76119a09c7ef822927c2822eedc8" ], "author": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Wed Jul 29 08:53:28 2026 -0700" }, "committer": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Wed Jul 29 08:53:28 2026 -0700" }, "message": "[SPARK-58374][BUILD] Upgrade `joda-time` to 2.14.3\n\n### What changes were proposed in this pull request?\n\nThis PR upgrades `joda-time` to `2.14.3` for Apache Spark 5.\n\n### Why are the changes needed?\n\nTo bring in the latest time zone data and keep the dependency up to date.\n- https://www.joda.org/joda-time/changes-report.html#a2.14.3 (2026-07-26)\n- https://github.com/JodaOrg/joda-time/releases/tag/v2.14.3\n - https://github.com/JodaOrg/joda-time/pull/833\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #57569 from dongjoon-hyun/SPARK-58374.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n" }, { "commit": "f27705db432c76119a09c7ef822927c2822eedc8", "tree": "3c6ff385cc0238cf718e14d94b3bad3d533c07ba", "parents": [ "5ceafeda5c512457d012fef81cf5b5154c41b2c9" ], "author": { "name": "Anurag Mantripragada", "email": "amantripragada@apple.com", "time": "Wed Jul 29 08:47:58 2026 -0700" }, "committer": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Wed Jul 29 08:47:58 2026 -0700" }, "message": "[SPARK-58330][SQL] Prevent silent dropping of dynamic options on same table references.\n\n## What changes were proposed in this pull request?\n\nThis PR fixes dynamic table options being silently dropped when the same table is referenced more than once in a single statement. Options are now re-applied on a per-query relation cache hit, so each reference keeps its own options.\n\n## Why are the changes needed?\n\nThe per-query relation cache in RelationResolution is keyed by catalog, namespace, name, and time travel spec, but not by options. When the same table is referenced twice in one statement with different options, the second reference gets a cache hit and reuses the first reference\u0027s options, so the options the user wrote on the second reference are silently ignored. This affects all queries that self reference a table.\n\n## Does this PR introduce any user-facing change?\n\nYes. Options that were previously silently dropped on a second reference to the same table now take effect, and options that previously leaked from one reference onto another no longer do. This affects any statement with two or more references to one table.\n\nAdded a note to `docs/sql-migration-guide.md` scoped to SELECT/INSERT since those are the only two of the four statements that have shipped in a released version. UPDATE/MERGE self-reference support hasn\u0027t reached a release yet, so there\u0027s no upgrade path to document for those two.\n\n## How was this patch tested?\n\nAdded end-to-end tests, each asserting that both references keep their own options:\n\n- `DataSourceV2OptionSuite`: INSERT selecting from the same table, a self-join, and a streaming self-join and streaming CTE (both analysis-only, no writeStream.start())\n- `MergeIntoTableSuiteBase`: self-merge (target and source referencing the same table)\n- `UpdateTableSuiteBase`: a subquery and a CTE referencing the same table as the update target\n- `DDLSuite`: the same three shapes (self-join, INSERT self-select, CTE) for v1 (session-catalog/Hive) tables\n\n## Was this patch authored or co-authored using generative AI tooling?\nI used Claude Code (Claude Opus 4.8) to generate the code and tests and verified manually.\n\nCloses #57508 from anuragmantri/self-ref-options.\n\nLead-authored-by: Anurag Mantripragada \u003camantripragada@apple.com\u003e\nCo-authored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n" }, { "commit": "5ceafeda5c512457d012fef81cf5b5154c41b2c9", "tree": "400334017038a9fae2367da6af71a2e4b1d1624e", "parents": [ "584268cb19bf74604e20817e1f3741cd11aecac7" ], "author": { "name": "Liang-Chi Hsieh", "email": "viirya@gmail.com", "time": "Wed Jul 29 07:52:25 2026 -0700" }, "committer": { "name": "Liang-Chi Hsieh", "email": "viirya@gmail.com", "time": "Wed Jul 29 07:52:25 2026 -0700" }, "message": "[SPARK-58414][SQL][TEST] Add e2e coverage for nanosecond timestamps nested in complex types in the Arrow cache\n\n### What changes were proposed in this pull request?\n\nAdd a test to `ArrowCachedBatchSerializerSuite` that round-trips nanosecond timestamps nested inside an array, a struct field, and a map value through the Arrow cache, using a value outside the int64 epoch-nanos window (year 3000 at nanosecond precision) that the standard interchange encoding cannot represent, alongside an in-window value.\n\n### Why are the changes needed?\n\nThe suite covers top-level nanosecond timestamps and nested `CalendarInterval`, but had no round trip for nanosecond timestamps nested in complex types. The nested path relies on recursion in two places: `ArrowWriter`\u0027s field writers dispatch on `(type, vector)` recursively on the write side, and on the read side every container accessor in `ArrowColumnVector` (struct children, `ArrayAccessor`\u0027s data vector, `MapAccessor`\u0027s keys/values) wraps its element vector through the constructor that runs the lossless tagged-struct recognizers. A regression at any nesting level would silently decode wrong values, so the machinery deserves an end-to-end pin for the one lossless type family that has out-of-window domain values.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, test-only.\n\n### How was this patch tested?\n\nThe new test; full `ArrowCachedBatchSerializerSuite` and `ArrowCachedBatchKryoRegistrationSuite` pass (76 tests).\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nYes, this pull request and its description were written by Claude Code.\n\nCloses #57619 from viirya/arrow-cache-nested-nanos-test.\n\nAuthored-by: Liang-Chi Hsieh \u003cviirya@gmail.com\u003e\nSigned-off-by: Liang-Chi Hsieh \u003cviirya@gmail.com\u003e\n" }, { "commit": "584268cb19bf74604e20817e1f3741cd11aecac7", "tree": "0a72752af48fbc93dd0514f17c20a914ed50b10c", "parents": [ "2c8570a1d1080a03d828f8c1b7de025166d62d64" ], "author": { "name": "Chao Sun", "email": "chao@openai.com", "time": "Wed Jul 29 07:43:18 2026 -0700" }, "committer": { "name": "Chao Sun", "email": "chao@openai.com", "time": "Wed Jul 29 07:43:18 2026 -0700" }, "message": "[SPARK-58412][SQL] Prevent incomplete and stale cache materialization statistics\n\n### Why are the changes needed?\n\nSPARK-58412 tracks two lifecycle gaps in the cache-materialization bookkeeping introduced by SPARK-57547. That earlier change replaced raw cache-task completion counters with partition-keyed statistics so duplicate computations cannot make an `InMemoryRelation` appear materialized too early.\n\nFirst, `CachedRDDBuilder` currently publishes a task\u0027s partition statistics whenever the task completes successfully, even if a downstream consumer stops before exhausting the cache-building iterator. This can happen when a memory-only block cannot be stored and Spark returns the partially unrolled iterator to its consumer. Because the statistics accumulator is last-write-wins per partition, a partially consumed recomputation can replace that partition\u0027s complete row and byte counts while the cache still appears fully materialized. For example, a completed `(10 rows, N bytes)` entry can be replaced with `(0 rows, 0 bytes)` without removing the partition key, allowing AQE to treat a non-empty cache as empty.\n\nSecond, `clearCache` resets the existing accumulator in place. Tasks from the retired cache generation have already captured that same accumulator, so late completions can write stale partition keys and values into the rebuilt generation. If those stale keys cover every partition, the new cache can appear complete before its own partitions finish.\n\nThese are general `InMemoryRelation` correctness issues. They were identified while working on runtime-filter support in #57443, but they affect cache materialization independently of that optimizer feature.\n\n### What changes were proposed in this PR?\n\nThis change makes cache statistics represent only complete work from the current cache generation.\n\nA task now publishes its partition statistics only after the wrapped cache iterator has been exhausted and the task finishes without failure or interruption. A successful consumer that stops early therefore cannot overwrite a complete partition\u0027s statistics with partial values.\n\nClearing a cache now installs a newly registered `PartitionKeyedAccumulator` instead of resetting the previous accumulator. Each cache-building RDD captures the accumulator for its own generation, so tasks finishing after `clearCache` can update only the retired generation. The cache contents and storage semantics are unchanged; this change only tightens the lifecycle of materialization metadata.\n\n### How was this PR tested?\n\nThe existing `CachedTableSuite` clear-cache regression now injects late updates through the retired generation\u0027s accumulator before and after rebuilding the cache. It verifies that the new generation remains unloaded until its own partitions complete and that its row and byte statistics remain exact.\n\nA new `ConcurrentInMemoryRelationSuite` regression forces a cached partition to be recomputed, then runs a successful consumer that does not consume the returned cache iterator. It verifies that the partial attempt cannot replace the complete partition statistics.\n\nThe following validation passed:\n\n```\nbuild/sbt \u0027sql/testOnly org.apache.spark.sql.CachedTableSuite org.apache.spark.sql.execution.columnar.ConcurrentInMemoryRelationSuite\u0027\nbuild/sbt sql/scalastyle sql/test:scalastyle\ngit diff --check\n```\n\nThe two affected suites ran 108 tests successfully.\n\nCloses #57617 from sunchao/dev/chao/codex/cache-materialization-correctness.\n\nAuthored-by: Chao Sun \u003cchao@openai.com\u003e\nSigned-off-by: Chao Sun \u003cchao@openai.com\u003e\n" }, { "commit": "2c8570a1d1080a03d828f8c1b7de025166d62d64", "tree": "2807000641ec6c6d73ae9c8b05e0cba197521e4d", "parents": [ "87dbc2421b703d1df08ac39928151d96d49369ff" ], "author": { "name": "ChuckLin2025", "email": "lzequn@gmail.com", "time": "Wed Jul 29 20:56:51 2026 +0800" }, "committer": { "name": "Wenchen Fan", "email": "wenchen@databricks.com", "time": "Wed Jul 29 20:56:51 2026 +0800" }, "message": "[SPARK-58292][CORE] Recreate the netty worker EventLoopGroup when a worker event loop thread dies\n\n### What changes were proposed in this pull request?\n\nA netty worker event-loop thread that dies (a `Throwable` escaping `run()` at the `runIo()`/select level; per-task exceptions are swallowed by `safeExecute`) is driven to `ST_TERMINATED` by `SingleThreadEventExecutor.doStartThread()`\u0027s finally block. Once that happens the thread is:\n\n- **never replaced** in the fixed-size `MultithreadEventExecutorGroup` (`children` is final, there is no repopulation),\n- still **handed out** by the round-robin `EventExecutorChooser`, which has no liveness check, and\n- **not restartable** (`startThread()` only starts from `ST_NOT_STARTED`/`ST_SUSPENDED`, never from `ST_TERMINATED`).\n\nSo the dead loop permanently poisons any channel pinned to it, which surfaces in `TransportClientFactory` as two failure modes:\n\n1. **New connections (~1/N fail):** a fresh channel bound to the dead loop fails registration with `RejectedExecutionException(\"event executor terminated\")` (caught by `AbstractChannel.AbstractUnsafe.register`), so `createClient` throws `IOException`. Each connect round-robins across the N worker threads, so roughly 1 in N attempts binds to the dead loop and fails.\n2. **Reused cached client (worse — silent hang):** a pooled `TransportClient` pinned to the dead loop still has an open socket, so `isActive()` was true and `createClient` kept returning it. `writeAndFlush().addListener()` then submits to the dead loop; netty\u0027s `safeExecute` **swallows** the `RejectedExecutionException` (only logs \"Failed to submit a listener notification task. Event loop shut down?\"), the callback/listener is **orphaned**, and the fetch (broadcast/RDD/RPC) **hangs forever**.\n\nThis PR makes the client network stack self-heal in-process, all within `common/network-common`:\n\n- **`TransportClient.isActive()`** returns `false` when `channel.eventLoop().isShuttingDown()` is true, so a poisoned pooled client is no longer treated as active and is not reused — `createClient` creates a new one instead.\n- **`TransportClientFactory.createClient`**, when a connect fails and the cause chain contains a `RejectedExecutionException` whose message is exactly `\"event executor terminated\"` (the terminated-loop rejection only — the queue-full default handler throws with no message), replaces `workerGroup` with a fresh group and rethrows, so the existing `IOException` retry path (e.g. `RetryingBlockTransferor`) reconnects onto a fresh, all-live group.\n - `recreateWorkerGroup` is `synchronized` and **identity-guarded** (`workerGroup !\u003d connectGroup` → no-op), so N concurrent callers that all hit the same dead group swap it exactly once. `workerGroup` is `volatile`.\n - The superseded group is **not shut down eagerly** — its still-live threads may be serving already-open channels. It is retained via a `WeakReference` and shut down best-effort in `close()`; its threads are daemon, so a not-yet-collected group cannot block JVM shutdown.\n- Gated by a new config `spark.network.recreateWorkerGroupOnDeadEventLoop`, default `true`.\n\n### Why are the changes needed?\n\nWithout this, a single dead netty worker thread degrades the client network stack for the lifetime of the JVM: new connections fail ~1/N of the time, and — worse — a reused pooled client submits to the dead loop where the rejection is swallowed, orphaning the callback so the fetch hangs forever. Only a fresh JVM fully clears the poison. Recreating the worker group on the terminated-loop rejection lets the existing retry path recover in-process instead.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is an internal reliability fix. It adds an internal-style network config `spark.network.recreateWorkerGroupOnDeadEventLoop` (default `true`); when disabled, the previous behavior is preserved. When no event loop dies, behavior is unchanged.\n\n### How was this patch tested?\n\nNew unit tests in `common/network-common`:\n\n- `TransportClientSuite.isActiveFalseWhenEventLoopIsShuttingDown` — a client whose event loop reports `isShuttingDown()` is not active even though the channel still reports open/active.\n- `TransportClientFactorySuite.recreatesWorkerGroupWhenEventLoopIsDead` — shutting down the factory\u0027s worker group makes the next `createClient` fail with the terminated-loop rejection; the factory swaps in a fresh live group and a subsequent connection succeeds.\n- `TransportClientFactorySuite.doesNotRecreateWorkerGroupWhenDisabled` — negative control with the config off: the connect still fails and the worker group is left unchanged.\n\n`network-common/testOnly TransportClientSuite TransportClientFactorySuite` passes (12 tests). `core` compiles; `network-common` checkstyle (main + test) and `core` scalastyle report no issues.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Code (Anthropic)\n\nCloses #57462 from ChuckLin2025/SPARK-58292-recreate-eventloop.\n\nAuthored-by: ChuckLin2025 \u003clzequn@gmail.com\u003e\nSigned-off-by: Wenchen Fan \u003cwenchen@databricks.com\u003e\n" }, { "commit": "87dbc2421b703d1df08ac39928151d96d49369ff", "tree": "7170b02cac31625abe801ad34320de6d2ca1a867", "parents": [ "1fba3cc49139a5130febc2f408b378cc8b64d598" ], "author": { "name": "Linhong Liu", "email": "linhong.liu@databricks.com", "time": "Wed Jul 29 20:52:09 2026 +0800" }, "committer": { "name": "Wenchen Fan", "email": "wenchen@databricks.com", "time": "Wed Jul 29 20:52:09 2026 +0800" }, "message": "[SPARK-57271][PYTHON] Propagate traceback locals to Python planner runner\n\n### What changes were proposed in this pull request?\n\nThis PR propagates `SPARK_TRACEBACK_WITH_LOCALS` from `PythonPlannerRunner` when `spark.sql.execution.pyspark.udf.tracebackWithLocals.enabled` is enabled.\n\nIt also adds a UDTF `analyze` regression test that raises from the planner-side Python worker and verifies the surfaced traceback includes the local variable.\n\n### Why are the changes needed?\n\n`PythonPlannerRunner` already reads `spark.sql.execution.pyspark.udf.tracebackWithLocals.enabled`, but it did not add `SPARK_TRACEBACK_WITH_LOCALS` to the Python worker environment. As a result, planner-driven Python paths such as UDTF `analyze` did not honor the traceback-locals config, unlike regular Python UDF execution.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. When `spark.sql.execution.pyspark.udf.tracebackWithLocals.enabled` is enabled, Python exceptions raised through planner-driven Python paths can now include local variables in their tracebacks.\n\n### How was this patch tested?\n\nPassed:\n\n```\npython3 python/run-tests.py --testnames \"pyspark.sql.tests.test_udtf UDTFTests.test_udtf_analyze_traceback_with_locals\"\npython3 python/run-tests.py --testnames \"pyspark.sql.tests.test_udf UDFTests.test_udf_traceback_with_locals\"\n```\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: OpenAI Codex (GPT-5)\n\nCloses #57509 from linhongliu-db/task/oss-spark-spark-57271-pythonplannerrunner-which-doesn-t-extend-basepythonrunner-f822ad87ff/implementation.\n\nAuthored-by: Linhong Liu \u003clinhong.liu@databricks.com\u003e\nSigned-off-by: Wenchen Fan \u003cwenchen@databricks.com\u003e\n" }, { "commit": "1fba3cc49139a5130febc2f408b378cc8b64d598", "tree": "f2493ec2fe3a0db113a48d06b03cdd4107d50874", "parents": [ "6b793ce07d714207aa849a6bb64329cf22cc2750" ], "author": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Wed Jul 29 00:52:57 2026 -0700" }, "committer": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Wed Jul 29 00:52:57 2026 -0700" }, "message": "[SPARK-58413][SQL][TEST] Rename `sequential fetch` label to `pipelined fetch (1 client)` in `NettyTransportBenchmark`\n\n### What changes were proposed in this pull request?\n\nThis PR renames the benchmark case label `sequential fetch` to `pipelined fetch (1 client)` in the `File-Backed Shuffle Block Fetch` suite of `NettyTransportBenchmark`.\n\n### Why are the changes needed?\n\nThe label is misleading. The case fires all 100 `fetchChunk` requests at once on a single connection and then waits for all of them to complete via a semaphore (`fetchChunksSync`), so the client never waits for one chunk before requesting the next. This is a pipelined fetch over one connection, not a sequential request-response loop.\n\nThe mislabel skews the interpretation of the results. For example:\n\nhttps://github.com/apache/spark/blob/12785d56624a8b412e78559dd920b816d629704b/core/benchmarks/NettyTransportBenchmark-results.txt#L132-L133\n\nRead as \"sequential vs parallel\", the 1.8X looks like poor parallelization efficiency. The actual comparison is \"pipelined over 1 socket vs pipelined over 4 sockets\", where the single pipelined connection already runs at ~4.2 GB/s. The new label makes this clear and contrasts naturally with `parallel fetch (4 clients)`.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nLabel-only change; manually reviewed.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #57616 from dongjoon-hyun/SPARK-58413.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n" }, { "commit": "6b793ce07d714207aa849a6bb64329cf22cc2750", "tree": "f15715ef4d69e12e78a834b7679d76bb2fde3943", "parents": [ "24ad34b689be2afe543050ab2797918b836d7fe9" ], "author": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Wed Jul 29 00:51:50 2026 -0700" }, "committer": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Wed Jul 29 00:51:50 2026 -0700" }, "message": "[SPARK-58415][INFRA][TEST] Regenerate benchmark results\n\n### What changes were proposed in this pull request?\n\nThis PR aims to regenerate benchmark results to check the intermediate status as a part of Apache Spark 5.0.0 preparation\n\n### Why are the changes needed?\n\nTo make the benchmark up-to-date and fill the missing gaps in order to help the comparison with the upcoming Apache Spark 4.3.0.\n\n**1. Last Update (2026-02-13)**\n\n- https://github.com/apache/spark/pull/54313\n\n**2. Java Version Changes**\n\n```\n- OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.14.0-1017-azure\n+ OpenJDK 64-Bit Server VM 21.0.12+8-LTS on Linux 6.17.0-1020-azure\n```\n\n```\n- OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.14.0-1017-azure\n+ OpenJDK 64-Bit Server VM 21.0.12+8-LTS on Linux 6.17.0-1020-azure\n```\n\n```\n- OpenJDK 64-Bit Server VM 25.0.2+10-LTS on Linux 6.17.0-1008-azure\n+ OpenJDK 64-Bit Server VM 25.0.4+7-LTS on Linux 6.17.0-1020-azure\n```\n\n**3. MISSING BENCHMARK RESULT**\n\n- #55278 didn\u0027t generate the benchmark result at all.\n- #56291 didn\u0027t generate Java 21 and 25 result.\n- #56291 didn\u0027t generate Java 21 and 25 result.\n- #56485 didn\u0027t generate Java 21 and 25 result.\n- #57232 didn\u0027t generate Java 21 and 25 result.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nManual review.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nNo.\n\nCloses #57620 from dongjoon-hyun/SPARK-58415.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n" }, { "commit": "24ad34b689be2afe543050ab2797918b836d7fe9", "tree": "a8e7227f75017466d1307635ab738686fa24a9d4", "parents": [ "233f1a414c61b8494388106e4f4d7bea5d35b9ca" ], "author": { "name": "Andreas Neumann", "email": "anew@apache.org", "time": "Tue Jul 28 21:34:25 2026 -0700" }, "committer": { "name": "Szehon Ho", "email": "szehon.apache@gmail.com", "time": "Tue Jul 28 21:34:25 2026 -0700" }, "message": "[SPARK-58321][SDP] Wire SCD2 AutoCDC streaming write and enable SCD2 end to end\n\n### What changes were proposed in this pull request?\n\nAdds `Scd2MergeStreamingWrite` (mirroring `Scd1MergeStreamingWrite`): it resolves the auxiliary-table identifier, constructs an `Scd2ForeachBatchHandler` over an `Scd2BatchProcessor`, and drives it via Structured Streaming `foreachBatch`. `FlowPlanner` now routes an SCD2 `AutoCdcMergeFlow` to it, replacing the `AUTOCDC_SCD2_NOT_SUPPORTED` throw.\n\n### Why are the changes needed?\n\nThis removes the **last** remaining SCD2 gate. The flow-schema derivation (SPARK-58319), the auxiliary-table spec (SPARK-58320), the reserved-column / track-history validation (SPARK-57251, SPARK-58313), and the per-microbatch reconciliation handler (SPARK-57395) have all merged; this change makes SCD2 AutoCDC flows runnable end to end.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes: `AUTO CDC ... STORED AS SCD TYPE 2` pipelines are now supported and executable, where previously they failed with `AUTOCDC_SCD2_NOT_SUPPORTED`.\n\n### How was this patch tested?\n\n- New `AutoCdcScd2SinglePipelineSuite` runs SCD2 flows end to end through the pipeline and asserts SCD2 target semantics (an upsert opens a current record; an update closes the prior record and opens a new one; a delete closes the current record) plus auxiliary-table materialization. Also refreshed an `AutoCdcFlowSuite` test whose name/comment referenced the removed gate.\n\n- Generalized the randomized out-of-order convergence test (AutoCdcScd1OutOfOrderConvergenceSuite, renamed to AutoCdcOutOfOrderConvergenceSuite) to run under both SCD Type 1 and SCD Type 2. It generates a random CDC event stream (with deletes, duplicate events, and nulls), feeds it once as a single sorted micro-batch and once as several shuffled micro-batches, and asserts the two target tables converge — verifying SCD2 reconciliation is order-invariant end to end. Only the user-visible target is compared; the auxiliary tables legitimately differ by arrival order (deletedByBatchId stamps and cross-batch GC depend on batching) even when the target converges. Verified non-seed-fragile across several seeds.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Opus 4.8\n\nCloses #57584 from anew/spark-58321-scd2-streaming-write-v2.\n\nAuthored-by: Andreas Neumann \u003canew@apache.org\u003e\nSigned-off-by: Szehon Ho \u003cszehon.apache@gmail.com\u003e\n" }, { "commit": "233f1a414c61b8494388106e4f4d7bea5d35b9ca", "tree": "e135c71ae4fac3eef5f649916476bd55ad9b969c", "parents": [ "177ce56c0df55e5a21a44ff7c59105d075990a13" ], "author": { "name": "Ruifeng Zheng", "email": "ruifengz@apache.org", "time": "Wed Jul 29 08:40:19 2026 +0800" }, "committer": { "name": "Ruifeng Zheng", "email": "ruifengz@apache.org", "time": "Wed Jul 29 08:40:19 2026 +0800" }, "message": "[SPARK-58396][ML][CONNECT] Include FPGrowth metadata in size estimates\n\n### What changes were proposed in this pull request?\n\nAdds parameter metadata to `FPGrowthModel.estimatedSize` while retaining its existing frequent-itemset and item-support estimates. Adds regression coverage using the suite\u0027s small FP-growth dataset.\n\n### Why are the changes needed?\n\nThe specialized estimate counted learned data but omitted the model\u0027s parameter metadata.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nAdded `FPGrowthModel estimated size` coverage in `FPGrowthSuite`. The suite was not run locally.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Codex (GPT-5)\n\nCloses #57590 from zhengruifeng/fpgrowth-model-size-metadata-dev3.\n\nAuthored-by: Ruifeng Zheng \u003cruifengz@apache.org\u003e\nSigned-off-by: Ruifeng Zheng \u003cruifengz@apache.org\u003e\n" }, { "commit": "177ce56c0df55e5a21a44ff7c59105d075990a13", "tree": "af22e0212dc0f34cbc89eba28cfc933611264069", "parents": [ "ece8eacb8f44742395f39e67a0714eca2efc4a3e" ], "author": { "name": "Liang-Chi Hsieh", "email": "viirya@gmail.com", "time": "Tue Jul 28 16:17:08 2026 -0700" }, "committer": { "name": "Liang-Chi Hsieh", "email": "viirya@gmail.com", "time": "Tue Jul 28 16:17:08 2026 -0700" }, "message": "[SPARK-58408][CORE] Register TimestampNanosVal in KryoSerializer\n\n### What changes were proposed in this pull request?\n\nRegister `org.apache.spark.unsafe.types.TimestampNanosVal` in `KryoSerializer`, next to `UTF8String` which reached the registration list the same way (SPARK-51790).\n\n### Why are the changes needed?\n\nSince SPARK-57735, cached-batch statistics rows carry `TimestampNanosVal` min/max bounds for nanosecond-timestamp columns -- `TimestampNanosColumnStats` in the default cache serializer, and the Arrow cache serializer\u0027s vector-side statistics (SPARK-57268). The statistics `InternalRow` is a field of `DefaultCachedBatch` / `ArrowCachedBatch`, so it is serialized with the batch whenever the cache uses a serialized storage level.\n\nWith `spark.serializer\u003dKryoSerializer` and `spark.kryo.registrationRequired\u003dtrue`, materializing such a cache fails at the first block write:\n\n```\ncom.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException:\nClass is not registered: org.apache.spark.unsafe.types.TimestampNanosVal\n```\n\nThis is the same defect class as SPARK-51777 (`CachedBatch` classes) and SPARK-51790 (`UTF8String`): a new class reachable from the cached-batch object graph was introduced without a matching Kryo registration. Every other bound type in statistics rows (`UTF8String`, `Decimal`, primitives) is already registered; `TimestampNanosVal` was the only gap -- the remaining stats collectors for non-orderable types (Variant, CalendarInterval, Geometry) record only sizes, no value objects.\n\nUsers with the default `registrationRequired\u003dfalse` are unaffected (Kryo falls back to writing the class name). Nanosecond timestamp types are an unreleased 4.3.0 preview feature.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nNew test in `CacheTableInKryoSuite` (the suite from SPARK-51777/51790, which runs with `registrationRequired\u003dtrue`): persists a `TIMESTAMP_NTZ(9)` column with `DISK_ONLY`. It fails without the registration with the exception above and passes with it. `CacheTableInKryoSuite` (4 tests) and `core`\u0027s `*KryoSerializer*` suites (81 tests) pass.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nYes, this pull request and its description were written by Claude Code.\n\nCloses #57607 from viirya/kryo-register-timestamp-nanos.\n\nAuthored-by: Liang-Chi Hsieh \u003cviirya@gmail.com\u003e\nSigned-off-by: Liang-Chi Hsieh \u003cviirya@gmail.com\u003e\n" }, { "commit": "ece8eacb8f44742395f39e67a0714eca2efc4a3e", "tree": "2a845dbee509500d2eb93c59b74b246bcc95e24a", "parents": [ "86de29d22894b4ad131c6fbdedf953945e2dae24" ], "author": { "name": "Liang-Chi Hsieh", "email": "viirya@gmail.com", "time": "Tue Jul 28 16:15:36 2026 -0700" }, "committer": { "name": "Liang-Chi Hsieh", "email": "viirya@gmail.com", "time": "Tue Jul 28 16:15:36 2026 -0700" }, "message": "[SPARK-58381][SQL] Gate the Arrow cache zero-copy write path on physical congruence with the cache schema\n\n### What changes were proposed in this pull request?\n\nReplace the blocklist that decides zero-copy eligibility on the Arrow cache write path (`ColumnarBatchToArrowCachedBatchIterator`) with a per-column physical-congruence check against the cache schema, using `ArrowUtils.isCompatibleWithDeclaredField` (introduced by SPARK-58258 for the columnar Python UDF pass-through). Input vectors whose field trees are not physically congruent with the corresponding cache schema field now take the row-based conversion path, which rewrites the values through `ArrowWriter` under the cache schema. The `containsCacheSchemaMismatch` blocklist is removed.\n\n### Why are the changes needed?\n\nThe zero-copy path serializes the input vectors\u0027 buffers verbatim under the cache\u0027s own schema: `serializeBatch` writes only the record batch, and the read path reconstructs the schema from `cacheSchema` and loads the buffers positionally into it. The blocklist covered only large var-width vectors and interchange-shaped nanosecond timestamp / `CalendarInterval` vectors, so every other physical divergence was forwarded verbatim and silently reinterpreted under the canonical layout when the cached batch is read back -- the same defect class SPARK-58258 eliminated on the UDF read side, one lifecycle stage earlier. Shapes the blocklist missed include view and dictionary encodings, list offset widths (`LargeList`, `ListView`), tagged structs carrying extra children, map entry children in the wrong order, and timestamp units.\n\nThe corruption is concrete: an Arrow map vector whose entry-struct children sit in `[value, key]` order is tolerated by Arrow and read correctly by `ArrowColumnVector` (key/value are addressed by name), but its buffers are positional, so a cached `{123: 7}` comes back as `{7: 123}` -- silently, and for the lifetime of the cached relation. The new test demonstrates this: it fails on master with `Array((7, 123)) did not equal Array((123, 7))` and passes with the gate.\n\nIn-tree columnar producers happened to be covered by the blocklist (the cache scan itself produces canonical vectors; Python UDF output\u0027s large var-width and interchange nanos/interval shapes were the blocked entries), so this is reachable today only from external Arrow-backed DSv2 columnar sources -- `supportsColumnarInput` accepts any of them. The congruence check closes the whole class rather than the enumerated instances, and keeps the write gate symmetric with the read-side gate from SPARK-58258.\n\nA false reject is safe by construction: the row-based fallback re-encodes values under the cache schema (with its value-domain guards), so eligible batches lose only the zero-copy optimization, never correctness.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. The Arrow cache serializer (SPARK-57268) is unreleased.\n\n### How was this patch tested?\n\nNew test in `ArrowCachedBatchSerializerSuite` that builds a spec-tolerated but non-canonical map vector (entry children `[value, key]`), verifies the source reads correctly, round-trips it through the serializer\u0027s columnar write and read paths, and asserts the values survive. Without the fix it fails with the keys and values swapped. Full `ArrowCachedBatchSerializerSuite` passes (72 tests), confirming canonical shapes remain zero-copy eligible.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nYes, this pull request and its description were written by Claude Code.\n\nCloses #57577 from viirya/arrow-cache-write-zerocopy-gate.\n\nAuthored-by: Liang-Chi Hsieh \u003cviirya@gmail.com\u003e\nSigned-off-by: Liang-Chi Hsieh \u003cviirya@gmail.com\u003e\n" }, { "commit": "86de29d22894b4ad131c6fbdedf953945e2dae24", "tree": "3d9c75aeab70b68ec1e2595b66bcc6458324e4f8", "parents": [ "12785d56624a8b412e78559dd920b816d629704b" ], "author": { "name": "Kousuke Saruta", "email": "sarutak@amazon.co.jp", "time": "Wed Jul 29 07:37:47 2026 +0900" }, "committer": { "name": "Kousuke Saruta", "email": "sarutak@apache.org", "time": "Wed Jul 29 07:37:47 2026 +0900" }, "message": "[SPARK-57893][CORE] Implement `UserCredentialManager`\n\n### What changes were proposed in this pull request?\nThis is a part of the SPIP: OIDC Credential Propagation (SPARK-57703).\nThis PR adds `UserCredentialManager`, the driver-side credential renewal loop.\n\n`UserCredentialManager` is a sibling of `HadoopDelegationTokenManager` and handles the OIDC credential propagation path independently. Both managers run on independent threads and can both be active simultaneously.\n\n**Responsibilities:**\n1. Reads the current identity token via `TokenIngestor.load()`\n2. Calls `CredentialProvider.resolve()` for each configured scheme\n3. Serializes the resulting `UserCredentials` and invokes the propagation callback\n4. Schedules renewal based on `min(identity token expiry, service credential expiry) - safetyMargin`\n5. Retries with exponential backoff on failure\n\n**Key design decisions:**\n- **Fail-fast on startup:** If the initial credential acquisition fails, the application fails to start rather than running with null credentials.\n- **Per-provider error isolation:** One provider failure does not abort the entire resolution loop. Other providers continue to resolve.\n- **Separation of concerns:** Credential *resolution* failures trigger backoff; *propagation* failures are logged and retried on the next scheduled renewal.\n- **ObjectInputFilter:** Deserialization is restricted to only the classes needed for `UserCredentials`, preventing deserialization attacks.\n\n**New files:**\n- `core/.../deploy/security/UserCredentialManager.scala`: Main implementation\n- `core/.../deploy/security/UserCredentialManagerSuite.scala`: Unit tests\n\n**Modified files:**\n- `CredentialProviderLoader.java`: Added `discoverAllSchemes()` for auto-discovery of registered schemes\n- `config/package.scala`: Added `spark.security.oidc.*` configuration keys\n\n### Why are the changes needed?\nThere is no component to orchestrate credential acquisition, renewal scheduling, and propagation triggering on the driver for OIDC-based environments. This is the core driver-side engine that coordinates `TokenIngestor` and `CredentialProvider` to maintain fresh credentials.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. All new behavior is gated by `spark.security.oidc.enabled\u003dfalse` (default). No existing APIs are changed.\n\n### How was this patch tested?\n\nUnit tests in `UserCredentialManagerSuite` covering:\n- Initial credential acquisition and callback invocation\n- Fail-fast when TokenIngestor returns empty\n- Serialization/deserialization round-trip with ObjectInputFilter security\n- `computeRenewalDelay` edge cases (safety margin, min interval, no expiry)\n- Exponential backoff (growth, cap, zero-failure edge case)\n- Per-provider error isolation (partial failure, total failure)\n- Token rotation triggers new credential acquisition and propagation\n- Factory method (`create`) with various configurations\n- Lifecycle safety (double-start guard, stop after start)\n\n### Was this patch authored or co-authored using generative AI tooling?\nKiro CLI / Claude\n\nCloses #57387 from sarutak/oidc-propagation/subtask4-user-credential-manager.\n\nAuthored-by: Kousuke Saruta \u003csarutak@amazon.co.jp\u003e\nSigned-off-by: Kousuke Saruta \u003csarutak@apache.org\u003e\n" }, { "commit": "12785d56624a8b412e78559dd920b816d629704b", "tree": "94ccf6cc3105d6b016f621fcd767b069dd5f39e5", "parents": [ "b7d304f001690510691110e2acc43872b39f22f2" ], "author": { "name": "Szehon Ho", "email": "szehon.apache@gmail.com", "time": "Tue Jul 28 12:31:17 2026 -0700" }, "committer": { "name": "Szehon Ho", "email": "szehon.apache@gmail.com", "time": "Tue Jul 28 12:31:17 2026 -0700" }, "message": "[SPARK-57761][SQL] Add missing error class INVALID_XML_SCHEMA_MAP_TYPE\n\n### What changes were proposed in this pull request?\n`QueryCompilationErrors.invalidXmlSchema` throws the `INVALID_XML_SCHEMA_MAP_TYPE` error class (raised when reading XML with a user-specified schema that contains a MAP with a non-STRING key type), but the error class was never registered in `error-conditions.json`. This registers the error condition so it resolves correctly through the error framework, and adds a test.\n\n### Why are the changes needed?\nThe error class was used but not declared. Hitting this path raised `INTERNAL_ERROR` (\"Cannot find main error class \u0027INVALID_XML_SCHEMA_MAP_TYPE\u0027\") instead of the intended error.\n\n### Does this PR introduce _any_ user-facing change?\nNo. Previously this code path raised an `INTERNAL_ERROR` because the error class was unregistered; now the intended error message is produced.\n\n### How was this patch tested?\nAdded a unit test in `QueryCompilationErrorsSuite`. Also ran `SparkThrowableSuite`.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Cursor (Claude Opus 4.8)\n\nCloses #56871 from szehon-ho/SPARK-57761.\n\nAuthored-by: Szehon Ho \u003cszehon.apache@gmail.com\u003e\nSigned-off-by: Szehon Ho \u003cszehon.apache@gmail.com\u003e\n" }, { "commit": "b7d304f001690510691110e2acc43872b39f22f2", "tree": "1aec72db032f3e7805356692b66082fe23bdd0fe", "parents": [ "04d4c266592d74bea9c00cc4896cf3a6461725a9" ], "author": { "name": "Spenser Sun", "email": "haotian.sun@databricks.com", "time": "Tue Jul 28 11:41:31 2026 -0700" }, "committer": { "name": "Tian Gao", "email": "gaogaotiantian@hotmail.com", "time": "Tue Jul 28 11:41:31 2026 -0700" }, "message": "[SPARK-58312][PYTHON] Remove unnecessary type: ignore comments in pyspark.sql.group\n\n### What changes were proposed in this pull request?\nIn `python/pyspark/sql/group.py`, the `dfapi`/`df_varargs_api` decorated methods (`count`, `mean`, `avg`, `max`, `min`, `sum`) intentionally have docstring-only bodies, which trigger 6 `# type: ignore[empty-body]` comments. These per-method ignores are replaced by a single file-level `# mypy: disable-error-code\u003d\"empty-body\"` directive, matching the style already used in `dataframe.py`, `column.py`, `window.py`, and `table_arg.py`.\n\nNote: an earlier revision of this PR also touched `python/pyspark/sql/udf.py`. That part has been reverted per review feedback. This PR now only changes `group.py`.\n\n### Why are the changes needed?\nTo reduce type-checker noise and improve readability, and to make `group.py` consistent with the sibling files that already suppress `empty-body` at the file level.\n\n### Does this PR introduce _any_ user-facing change?\nNo\n\n### How was this patch tested?\nExisting tests. `mypy --namespace-packages --config-file python/mypy.ini python/pyspark` passes at full scope.\n\n### Was this patch authored or co-authored using generative AI tooling?\nNo\n\nCloses #57482 from Spenserrrr/haotian-sun_data/typeignore2.\n\nAuthored-by: Spenser Sun \u003chaotian.sun@databricks.com\u003e\nSigned-off-by: Tian Gao \u003cgaogaotiantian@hotmail.com\u003e\n" }, { "commit": "04d4c266592d74bea9c00cc4896cf3a6461725a9", "tree": "d5cbf087c58df80188b3153bc0a88e175f496e95", "parents": [ "37ca0a973f39d9c150afd4ddca635fa92faba11f" ], "author": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Tue Jul 28 09:42:05 2026 -0700" }, "committer": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Tue Jul 28 09:42:05 2026 -0700" }, "message": "[SPARK-58377][BUILD] Upgrade `netty-tcnative` to 2.0.81.Final\n\n### What changes were proposed in this pull request?\n\nThis PR aims to upgrade `netty-tcnative` to 2.0.81.Final.\n\n### Why are the changes needed?\n\nTo bring the latest bug fixes.\n\n- https://github.com/netty/netty-tcnative/releases/tag/netty-tcnative-parent-2.0.81.Final (2026-07-20)\n - https://github.com/netty/netty-tcnative/pull/991\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #57574 from dongjoon-hyun/SPARK-58377.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n" }, { "commit": "37ca0a973f39d9c150afd4ddca635fa92faba11f", "tree": "340aff805151c77c588996601cf57db62677199a", "parents": [ "7050d160dc22c5118cfe8c72bcfbae2df246339e" ], "author": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Tue Jul 28 08:15:40 2026 -0700" }, "committer": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Tue Jul 28 08:15:40 2026 -0700" }, "message": "[SPARK-58375][BUILD] Upgrade `ap-loader` to 4.5-13\n\n### What changes were proposed in this pull request?\n\nThis PR aims to upgrade `ap-loader` dependency to 4.5-13 for Apache Spark 5.\n\n### Why are the changes needed?\n\nTo bring the latest bug fixes and improvements from `ap-loader`.\n\n- https://github.com/jvm-profiling-tools/ap-loader/releases/tag/4.5-13 (2026-07-21)\n - https://github.com/async-profiler/async-profiler/releases/tag/v4.5\n - https://github.com/async-profiler/async-profiler/issues/1765: JDK 27 support\n - https://github.com/async-profiler/async-profiler/issues/1755: Span API for latency profiling\n - https://github.com/async-profiler/async-profiler/issues/1406: Long time-to-safepoint pause when attaching asprof\n - https://github.com/async-profiler/async-profiler/issues/1756: Runtime attach fails on JVMs with many native libraries\n - https://github.com/async-profiler/async-profiler/releases/tag/v4.4\n - https://github.com/async-profiler/async-profiler/issues/1553: Differential Flame Graphs\n - https://github.com/async-profiler/async-profiler/issues/1705: `memlimit` option to limit size of the call trace storage\n - https://github.com/async-profiler/async-profiler/issues/1715: Fix Zing crash when profiling cpu+wall together\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #57570 from dongjoon-hyun/SPARK-58375.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n" }, { "commit": "7050d160dc22c5118cfe8c72bcfbae2df246339e", "tree": "5eddf3df1ccd7f5afa82ec2ff929bef66a18fe2e", "parents": [ "f66e380ba181b9396a1d202341d662d787ffbc70" ], "author": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Tue Jul 28 08:14:18 2026 -0700" }, "committer": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Tue Jul 28 08:14:18 2026 -0700" }, "message": "[SPARK-58376][BUILD] Upgrade `byte-buddy` to 1.18.11\n\n### What changes were proposed in this pull request?\n\nThis PR aims to upgrade `byte-buddy` and `byte-buddy-agent` to 1.18.11.\n\n### Why are the changes needed?\n\nTo bring the latest bug fixes and official support for the newest JDKs. Note that version 1.18.6 was never released. The full release notes are available below:\n\n- https://github.com/raphw/byte-buddy/releases/tag/byte-buddy-1.18.11 (2026-07-02)\n - Add SBOM to published artifacts, and check for traversable paths injected into class files.\n- https://github.com/raphw/byte-buddy/releases/tag/byte-buddy-1.18.10 (2026-06-26)\n - Delay the change of default for unsafe use to Java 26 and improve the error message.\n- https://github.com/raphw/byte-buddy/releases/tag/byte-buddy-1.18.9 (2026-06-01)\n - Disable use of `Unsafe` by default on Java 25+, improve OpenJ9 attachment and diagnostics, avoid NPE on missing annotation types, and update ASM.\n- https://github.com/raphw/byte-buddy/releases/tag/byte-buddy-1.18.8 (2026-04-01)\n - Improve support for repeatable builds and fix reordering of the exception table in type initializers when instrumenting.\n- https://github.com/raphw/byte-buddy/releases/tag/byte-buddy-1.18.7 (2026-03-03)\n - Introduce a new versioning concept with the *-jdk5* suffix jar and a Java 8 baseline for the regular jar.\n- https://github.com/raphw/byte-buddy/releases/tag/byte-buddy-1.18.5 (2026-02-14)\n - Eagerly resolve canonical files during attach emulation and add missing super classes to hash code / equals computation in `Advice`.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is a test-only dependency.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #57572 from dongjoon-hyun/SPARK-58376.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n" }, { "commit": "f66e380ba181b9396a1d202341d662d787ffbc70", "tree": "05ef7c4d7ae79197dc24799288ef2c7b5c7c65ea", "parents": [ "fd5b0d48f7551fe38156be2069d06f27812ecf0e" ], "author": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Tue Jul 28 08:12:47 2026 -0700" }, "committer": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Tue Jul 28 08:12:47 2026 -0700" }, "message": "[SPARK-58379][BUILD] Upgrade `jnr-posix` to 3.2.1\n\n### What changes were proposed in this pull request?\n\nThis PR aims to upgrade `jnr-posix` test dependency to 3.2.1.\n\n### Why are the changes needed?\n\n- [Release 3.2.0](https://github.com/jnr/jnr-posix/releases/tag/3.2.0) (2026-06-26)\n- [Release 3.2.1](https://github.com/jnr/jnr-posix/releases/tag/3.2.1) (2026-06-27)\n\n`jnr-posix` is a test-only dependency used by `DiskBlockManagerSuite` to get/set\nthe process umask (SPARK-37618). The 3.2.x line brings:\n\n- `jnr-ffi` 2.2.11 -\u003e 2.3.0 (internal ASM 9.2 -\u003e 9.10.1) for better modern JDK\n class-file support\n- JPMS module visibility improvements on newer JDKs\n- Fallback to avoid `UnsatisfiedLinkError` on musl-based systems (Alpine)\n- RISC-V64 and LoongArch64 support\n- Temp-directory vulnerability fixes\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is a test-only dependency.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #57575 from dongjoon-hyun/SPARK-58379.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n" }, { "commit": "fd5b0d48f7551fe38156be2069d06f27812ecf0e", "tree": "8e4f91e808dcfb641049df8639c60093165e03f5", "parents": [ "d677bd365a88b5620005a63cb90f326211344f5c" ], "author": { "name": "YangJie", "email": "yangjie01@baidu.com", "time": "Tue Jul 28 22:52:05 2026 +0800" }, "committer": { "name": "yangjie01", "email": "yangjie01@baidu.com", "time": "Tue Jul 28 22:52:05 2026 +0800" }, "message": "[SPARK-57955][SQL] Raise a proper error for out-of-Int-range data type parameters\n\n### What changes were proposed in this pull request?\n\nThe data type parser converts integer type parameters (DECIMAL precision/scale, CHAR/VARCHAR length, TIME precision, GEOMETRY/GEOGRAPHY SRID) to `Int` with a raw `.toInt`. The grammar backs these with `INTEGER_VALUE : DIGIT+` (an unbounded digit run), so a value outside the 32-bit integer range overflows and throws a raw `java.lang.NumberFormatException` with no Spark error class.\n\nThis guards the conversion on all three data-type parse paths so an out-of-range parameter raises a proper Spark error:\n\n- `DataTypeAstBuilder` (the ANTLR parser, used by SQL/CAST and `DataTypeParser.parseDataType`)\n- `DataType.nameToType` (the JSON path, `DataType.fromJson`)\n- `LegacyTypeStringParser` (the case-class string parser kept for Spark 1.1 and earlier Parquet compatibility)\n\nRouting:\n\n- DECIMAL precision reuses `DECIMAL_PRECISION_EXCEEDS_MAX_PRECISION`\n- TIME precision reuses `UNSUPPORTED_TIME_PRECISION`\n- GEOMETRY/GEOGRAPHY SRID reuses `ST_INVALID_SRID_VALUE`\n- CHAR/VARCHAR length and DECIMAL scale use a new `DATATYPE_PARAMETER_VALUE_OUT_OF_RANGE` error condition\n\nThe `TIMESTAMP(p)` branch of the same parser already guarded this (raising `INVALID_TIMESTAMP_PRECISION`); the other type parameters are now brought in line. The unsupported-type branch additionally renders the raw token text instead of parsing it to `Int`, so an oversized parameter on an unsupported type (e.g. `FOO(9999999999)`) no longer leaks a `NumberFormatException` while building the `UNSUPPORTED_DATATYPE` message.\n\n### Why are the changes needed?\n\nAn out-of-`Int`-range type parameter surfaces a raw `java.lang.NumberFormatException` that is not a `SparkThrowable` -- it has no error condition and no SQLSTATE, so programmatic error handling (JDBC/Connect clients that read the condition/SQLSTATE) gets nothing, and callers that catch the parser\u0027s expected error surface may mistake it for an internal failure. For example:\n\n```sql\nSELECT CAST(1 AS DECIMAL(9999999999, 2));\n```\n\nthrows:\n\n```\njava.lang.NumberFormatException: For input string: \"9999999999\"\n```\n\nThe same happens via `StructType.fromDDL`, `DataType.fromJson`, `DataTypeParser.parseDataType`, and the legacy Parquet schema-string parser. This mirrors the fix in SPARK-56395 for the `NEAREST BY` num_results literal, which surfaced the same raw-`NumberFormatException` anti-pattern.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. An out-of-`Int`-range data type parameter now raises a Spark error with a proper error condition instead of a raw `NumberFormatException`. For example, `CAST(1 AS DECIMAL(9999999999, 2))` now raises `DECIMAL_PRECISION_EXCEEDS_MAX_PRECISION`, and `CAST(1 AS CHAR(9999999999))` raises the new `DATATYPE_PARAMETER_VALUE_OUT_OF_RANGE`. In-range values are unaffected.\n\n### How was this patch tested?\n\nA new test in `DataTypeParserSuite` covering DECIMAL precision (scale-present and scale-absent), DECIMAL scale, CHAR/VARCHAR length (bare and `COLLATE`), TIME precision, GEOMETRY/GEOGRAPHY SRID, and an unsupported parameterized type, across the ANTLR, JSON, and legacy parse paths, plus a valid `Int.MaxValue` boundary case (`CHAR(2147483647)`). `catalyst/testOnly *DataTypeParserSuite` and `core/testOnly org.apache.spark.SparkThrowableSuite` pass.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Code Opus 4.8\n\nCloses #57034 from LuciferYang/SPARK-datatype-param-overflow.\n\nAuthored-by: YangJie \u003cyangjie01@baidu.com\u003e\nSigned-off-by: yangjie01 \u003cyangjie01@baidu.com\u003e\n" }, { "commit": "d677bd365a88b5620005a63cb90f326211344f5c", "tree": "5159232c76f9f655b26609d8b9bbf6746da7ad8f", "parents": [ "2df516cffda1946dae90278deb561de33f7a10fb" ], "author": { "name": "akshatshenoi-db", "email": "akshat.shenoi@databricks.com", "time": "Tue Jul 28 22:20:11 2026 +0800" }, "committer": { "name": "Wenchen Fan", "email": "wenchen@databricks.com", "time": "Tue Jul 28 22:20:11 2026 +0800" }, "message": "[SPARK-58382][SQL] binaryFile archive support via wholeFile option\n\n### What changes were proposed in this pull request?\n\nThis PR lets `format(\"binaryFile\")` read tar/zip/7z archives, extending the archive-reader\nseries (SPARK-57135 CSV, SPARK-57419 JSON, SPARK-57478 text, SPARK-57479 XML, SPARK-57481 Avro).\nIt is gated by a new binaryFile option `wholeFile` (default `true`) and the existing\n`spark.sql.files.archive.reader.enabled` flag (default `false`).\n\n- `wholeFile\u003dtrue` (default): the archive is read as a single record (its raw bytes) -- unchanged\n from today\u0027s non-archive behavior.\n- `wholeFile\u003dfalse`: one row is emitted per inner entry. `content` holds the entry\u0027s unpacked\n bytes, `path` is `\u003carchive\u003e!/\u003centryName\u003e`, `length` is the entry\u0027s size, and `modificationTime`\n is the parent archive\u0027s (archive entry timestamps are optional, so the parent\u0027s is used).\n\n`BinaryFileFormat` mixes in `SupportsArchiveFormat`. On the `wholeFile\u003dfalse` archive path it\nstreams each entry via `SupportsArchiveFormat.readArchiveEntries` and builds one row per entry from\na synthetic per-entry `FileStatus`, reusing the same row-builder (`BinaryFileFormat.parse`) as the\nnon-archive path so length/modtime filter pushdown applies per entry.\n\nTo expose the entry\u0027s size to consumers, the `readArchiveEntries` `parseEntry` callback now\nreceives the `ArchiveEntry` instead of just the entry name; the existing CSV and `localizeEntries`\ncallers are updated to read `entry.getName`.\n\nKnown limitation (zip, until `ZipFile`): the streaming `ZipArchiveInputStream` reports an entry\u0027s\n`getSize` as `-1` when the size is only in a trailing data descriptor, so for such entries the\nper-entry `length` column is `-1` and the `spark.sql.sources.binaryFile.maxLength` guard\n(`status.getLen \u003e maxLength`) is skipped, i.e. the entry is read unbounded. tar and 7z report real\nsizes and are unaffected. This closes when zip reads move to `ZipFile`, which exposes entry sizes\nfrom the central directory up front.\n\n### Why are the changes needed?\n\nbinaryFile is the natural way to ingest opaque/blob files, and archives (tar/zip/7z) are a common\npackaging for large numbers of such files. This lets an archive be read as a directory of its\nentries without unpacking it to disk first, completing the archive-reader feature across the\nfile-based formats.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, gated behind `spark.sql.files.archive.reader.enabled` (default `false`). When enabled,\n`spark.read.format(\"binaryFile\").option(\"wholeFile\", \"false\").load(\u003carchive\u003e)` returns one row per\ninner entry. With the flag off, or with `wholeFile\u003dtrue` (the default), behavior is unchanged.\n\n### How was this patch tested?\n\nNew `BinaryFileArchiveReadBase` with `BinaryFileTarArchiveReadSuite`,\n`BinaryFileZipArchiveReadSuite`, and `BinaryFileSevenZArchiveReadSuite`, covering: whole-file vs\nper-entry reads, per-entry content/path/length, empty archives, hidden-entry skipping,\n`spark.sql.sources.binaryFile.maxLength` enforcement, length filter pushdown, single-partition\nscans, and corrupt-archive handling under `ignoreCorruptFiles`. The per-entry `length` assertions\nare skipped for zip because the streaming `ZipArchiveInputStream` reports a data-descriptor entry\u0027s\nsize as `-1`; they re-enable when zip reads move to `ZipFile`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Code\n\nCloses #57578 from akshatshenoi-db/archive-binaryfile.\n\nAuthored-by: akshatshenoi-db \u003cakshat.shenoi@databricks.com\u003e\nSigned-off-by: Wenchen Fan \u003cwenchen@databricks.com\u003e\n" }, { "commit": "2df516cffda1946dae90278deb561de33f7a10fb", "tree": "38fcd0be23358686c0a8cdbc5f4739f6c68a9a25", "parents": [ "073c8d8f4445ffa4ce279720fd28efaee053c234" ], "author": { "name": "Ruifeng Zheng", "email": "ruifengz@apache.org", "time": "Tue Jul 28 20:31:46 2026 +0800" }, "committer": { "name": "Ruifeng Zheng", "email": "ruifengz@apache.org", "time": "Tue Jul 28 20:31:46 2026 +0800" }, "message": "[SPARK-58402][PYTHON] Consolidate UNKNOWN_EXPLAIN_MODE into VALUE_NOT_ALLOWED\n\n### What changes were proposed in this pull request?\n\nConsolidate the specialized Spark Connect `UNKNOWN_EXPLAIN_MODE` error condition into `VALUE_NOT_ALLOWED`. Invalid explain modes now report the generic allowed-values condition, and the existing Spark Connect test asserts the new condition.\n\n### Why are the changes needed?\n\n`UNKNOWN_EXPLAIN_MODE` duplicates the generic allowed-values condition for a fixed argument allowlist. Removing it reduces narrowly scoped PySpark error conditions and aligns the validation with similar APIs.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. Invalid Spark Connect explain modes now use `VALUE_NOT_ALLOWED` and its generic message. The exception type remains `PySparkValueError`.\n\n### How was this patch tested?\n\nUpdated the existing Spark Connect explain-mode assertion. Static validation passed: `git diff --check`, JSON parsing, and a changed-Python-file line-length scan. The focused test suite was not run.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Codex (GPT-5)\n\nCloses #57594 from zhengruifeng/consolidate-unknown-explain-mode-dev0.\n\nAuthored-by: Ruifeng Zheng \u003cruifengz@apache.org\u003e\nSigned-off-by: Ruifeng Zheng \u003cruifengz@apache.org\u003e\n" }, { "commit": "073c8d8f4445ffa4ce279720fd28efaee053c234", "tree": "393855b3595535b2e904643e51aaa7fc116b7a8d", "parents": [ "b2918df83111fe7683c4b7e6c9aac3a2b347231d" ], "author": { "name": "Liang-Chi Hsieh", "email": "viirya@gmail.com", "time": "Tue Jul 28 10:18:09 2026 +0000" }, "committer": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Tue Jul 28 10:18:09 2026 +0000" }, "message": "[SPARK-58390][SQL] Emit row counts without deserializing Arrow payloads for empty-projection cache reads\n\n### What changes were proposed in this pull request?\n\nShort-circuit `ArrowCachedBatchSerializer.convertCachedBatchToInternalRow` when the projection is empty: emit `numRows` reused 0-field `UnsafeRow`s per cached batch, without deserializing or decompressing the batch\u0027s Arrow payload. The row count is already recorded on `ArrowCachedBatch`.\n\nAlso fixes the `ArrowCachedBatch` scaladoc, which listed the per-column statistics as `(upperBound, lowerBound, ...)` while both write paths produce the `ColumnStats.collectedStatistics` order `(lowerBound, upperBound, nullCount, count, sizeInBytes)`; the code was consistently lower-first everywhere, only the doc was wrong.\n\n### Why are the changes needed?\n\nAn empty projection (e.g. a count aggregate through the row-based reader, `spark.sql.inMemoryColumnarStorage.enableVectorizedReader\u003dfalse`) selects no columns, yet the reader still paid full IPC deserialization and decompression for every cached batch just to iterate its rows. That cost is pure waste: the answer is a stored integer.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. The Arrow cache serializer (SPARK-57268) is unreleased, and the change is performance-only; results are identical.\n\n### How was this patch tested?\n\nTwo new tests in `ArrowCachedBatchSerializerSuite`:\n\n- `empty projection emits row counts without deserializing the Arrow payload`: hands the reader a cached batch whose Arrow payload is garbage bytes. The empty projection returns the correct number of empty rows purely from `numRows` (fails before this change with `IllegalArgumentException: capacity \u003c 0` from the IPC reader, proving the payload used to be deserialized), while a projection that actually needs the payload still fails on the same batch, pinning that only the empty-projection case skips the read.\n- `count aggregate over the cached relation with the row-based reader`: end-to-end `count(*)` over a cached relation spanning many small Arrow batches with the vectorized reader disabled, plus a `sum` over the same cached data verifying projecting reads still decode the payload correctly.\n\nFull `ArrowCachedBatchSerializerSuite` (73 tests) and `ArrowCachedBatchKryoRegistrationSuite` pass.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nYes, this pull request and its description were written by Claude Code.\n\nCloses #57583 from viirya/arrow-cache-empty-projection.\n\nAuthored-by: Liang-Chi Hsieh \u003cviirya@gmail.com\u003e\nSigned-off-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\n" }, { "commit": "b2918df83111fe7683c4b7e6c9aac3a2b347231d", "tree": "5266e88c43a48e36d2f03763de174c98c60bc7b7", "parents": [ "5d440c7cc1935637d4885bef4281aaefe53fb7ef" ], "author": { "name": "Ruifeng Zheng", "email": "ruifengz@apache.org", "time": "Tue Jul 28 10:14:33 2026 +0000" }, "committer": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Tue Jul 28 10:14:33 2026 +0000" }, "message": "[SPARK-58397][ML][CONNECT] Add Bucketizer size estimate\n\n### What changes were proposed in this pull request?\n\nAdds a specialized `Bucketizer.estimatedSize` implementation that charges its parameter metadata. Adds a bounded-size regression test.\n\n### Why are the changes needed?\n\nBucketizer is a model whose state is entirely represented by its parameter maps, so its cache charge can use the precise metadata estimate instead of a generic object-graph walk.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nAdded `Bucketizer estimated size` coverage in `BucketizerSuite`. The suite was not run locally.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Codex (GPT-5)\n\nCloses #57591 from zhengruifeng/bucketizer-model-size-metadata-dev3.\n\nAuthored-by: Ruifeng Zheng \u003cruifengz@apache.org\u003e\nSigned-off-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\n" }, { "commit": "5d440c7cc1935637d4885bef4281aaefe53fb7ef", "tree": "db69a481e100d87e8098990c70910eab0fd150bd", "parents": [ "a6c3eb21763fb6196a150087672aa7437e8a8bda" ], "author": { "name": "Tian Gao", "email": "gaogaotiantian@hotmail.com", "time": "Tue Jul 28 10:12:37 2026 +0000" }, "committer": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Tue Jul 28 10:12:37 2026 +0000" }, "message": "[SPARK-58388][INFRA] Clean up the dependency list for lint/docs/gen-protos docker image\n\n### What changes were proposed in this pull request?\n\nMove the dependency list of 3 individual docker images to `pyproject.toml`.\n\nAlso cleaned up some unused packages in those images. The dependencies are now clearer - why those are needed. `docs` and `lint` images share some packages just because they were copied/pasted.\n\n`gen-proto` image now installs more packages than before, sharing the same dependency group with `lint`. This provides all necessary packages to `gen-proto`. This unification makes CI easier to understand. Notice that this docker image is *not* used by our CI. It\u0027s dev-only and I updated the README file about how to use it.\n\n### Why are the changes needed?\n\nWe want to manage our Python dependencies in a single file.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nCI.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nNo.\n\nCloses #57581 from gaogaotiantian/lint-doc-docker-image.\n\nAuthored-by: Tian Gao \u003cgaogaotiantian@hotmail.com\u003e\nSigned-off-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\n" }, { "commit": "a6c3eb21763fb6196a150087672aa7437e8a8bda", "tree": "41af683db0d00131e09db19ca18454214b53d833", "parents": [ "2ad6c5152c8663d3ba5f51a9c79a929425dfd003" ], "author": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Tue Jul 28 10:10:14 2026 +0000" }, "committer": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Tue Jul 28 10:10:14 2026 +0000" }, "message": "[MINOR][SQL] Simplify redundant boolean ternary in visitDeclareCursorStatement\n\n### What changes were proposed in this pull request?\nSimplifies a redundant boolean ternary in `visitDeclareCursorStatement`: `if (ctx.INSENSITIVE() !\u003d null) false else true` becomes `ctx.INSENSITIVE() \u003d\u003d null`.\n\n### Why are the changes needed?\nThe ternary maps a null check to a negated boolean; the direct comparison has the identical truth table and reads more clearly (asensitive is true exactly when the INSENSITIVE token is absent).\n\n### Does this PR introduce _any_ user-facing change?\nNo.\n\n### How was this patch tested?\nBehavior-preserving one-line simplification; existing parser tests cover it. No new tests needed.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57595 from uros-b/refactor-declarecursor-boolean.\n\nLead-authored-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\nCo-authored-by: Uros Bojanic \u003curos.bojanic@databricks.com\u003e\nSigned-off-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\n" }, { "commit": "2ad6c5152c8663d3ba5f51a9c79a929425dfd003", "tree": "f1db4342ebce18908d436ea68fb5d3c57e215aae", "parents": [ "e8cbd4840606db3f0ec6cca39f2e2d7dd0f43474" ], "author": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Tue Jul 28 10:09:26 2026 +0000" }, "committer": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Tue Jul 28 10:09:26 2026 +0000" }, "message": "[SPARK-58401][DOC] Update Parquet links to 1.17.1 in Parquet data source docs\n\n### What changes were proposed in this pull request?\nUpdates the two Parquet version-pinned links in the Parquet data source docs from 1.14.1 to 1.17.1 (the mock-KMS jar Maven directory and the `KmsClient` source link).\n\n### Why are the changes needed?\nThe build ships Parquet 1.17.1 (`pom.xml` and `dev/deps` both confirm `parquet-hadoop-1.17.1`), so the 1.14.1-pinned links are stale. This is the same maintenance previously done by SPARK-48177 (FOLLOWUP), which bumped the same two links.\n\n### Does this PR introduce _any_ user-facing change?\nNo.\n\n### How was this patch tested?\nDocs-only change. No tests needed.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57596 from uros-b/doc-parquet-version-1171.\n\nLead-authored-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\nCo-authored-by: Uros Bojanic \u003curos.bojanic@databricks.com\u003e\nSigned-off-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\n" }, { "commit": "e8cbd4840606db3f0ec6cca39f2e2d7dd0f43474", "tree": "93a311e07bcfe71a7550444e9aa72e59ee929c55", "parents": [ "29637348281f576ed4fe43024f35d244c75c840b" ], "author": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Tue Jul 28 10:08:48 2026 +0000" }, "committer": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Tue Jul 28 10:08:48 2026 +0000" }, "message": "[MINOR][INFRA] Fix doubled braces in structured logging style-check hint\n\n### What changes were proposed in this pull request?\nFixes the developer-facing hint printed by the structured-logging style check so its MDC example shows single braces `${MDC(TASK_ID, taskId)}` instead of the doubled `${{...}}`.\n\n### Why are the changes needed?\nThe example lives in a plain (non-f) string literal, so the doubled braces were rendered literally, teaching incorrect syntax to anyone who copied the hint. The real structured-logging API uses single braces.\n\n### Does this PR introduce _any_ user-facing change?\nNo (developer tooling message only).\n\n### How was this patch tested?\nManual inspection: confirmed the hint is emitted from a plain (non-f) string literal, so the doubled braces render literally, and that the real structured-logging API (`Logging.scala`) uses single braces. Reviewer can re-run `dev/structured_logging_style.py` to see the corrected hint.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57597 from uros-b/devexp-structured-logging-hint-braces.\n\nLead-authored-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\nCo-authored-by: Uros Bojanic \u003curos.bojanic@databricks.com\u003e\nSigned-off-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\n" }, { "commit": "29637348281f576ed4fe43024f35d244c75c840b", "tree": "1ab8e3e0d78ceff17893f4c2c4ce98a52ee685fe", "parents": [ "6f1257366a04a5fe54e866962dd2cb2851e5e42c" ], "author": { "name": "BRIJ RAJ KISHORE", "email": "22271048+brijrajk@users.noreply.github.com", "time": "Tue Jul 28 11:06:16 2026 +0200" }, "committer": { "name": "Peter Toth", "email": "peter.toth@gmail.com", "time": "Tue Jul 28 11:06:16 2026 +0200" }, "message": "[SPARK-58008][SQL] Support dynamic table options for DELETE\n\n### What changes were proposed in this pull request?\n\nExtends the per-statement `WITH (key \u003d value)` options clause to `DELETE` statements, consistent with the support already present for `SELECT` (SPARK-36680), `INSERT` (SPARK-49098), and `UPDATE` (SPARK-57681).\n\nExample syntax:\n```sql\nDELETE FROM catalog.db.table WITH (`write.split-size` \u003d 10) WHERE id \u003c 100\n\nDELETE FROM catalog.db.table AS t WITH (`k` \u003d \u0027v\u0027) WHERE t.id \u003c 100\n```\n\nThe options are forwarded through the `DataSourceV2Relation`, `RowLevelOperationInfo`, and `LogicalWriteInfo` so that DataSource V2 connectors receive them at every layer of the write path.\n\nOptions are also wired through the metadata-only DELETE paths:\n- `SupportsDeleteV2.deleteWhere(Predicate[], CaseInsensitiveStringMap)` — new back-compatible overload\n- `SupportsDeleteV2.truncateTable(CaseInsensitiveStringMap)` — new back-compatible overload\n- `TruncatableTable.truncateTable(CaseInsensitiveStringMap)` — new back-compatible overload\n- `DeleteFromTableWithFilters`, `DeleteFromTableExec`, `TruncateTableExec` — each carries `options`\n- `DataSourceV2Strategy` — threads `r.options` through all three non-row-level DELETE planning paths\n\n### Why are the changes needed?\n\n`DELETE` was the only DML statement missing this capability. Adding it completes the feature set across all row-level write operations and allows connectors to accept per-statement tuning options (e.g. split size, isolation level) on deletes without requiring separate configuration.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes — new SQL syntax. `DELETE FROM tbl WITH (k \u003d v) WHERE ...` is now valid.\n\n### How was this patch tested?\n\n- `DDLParserSuite`: two new parser tests covering `DELETE ... WITH (...)` without and with an alias.\n- `DeleteFromTableSuiteBase`: five new end-to-end tests verifying that options reach the `DataSourceV2Relation`, `RowLevelOperationInfo`, and `LogicalWriteInfo` layers across the row-level, subquery, CTE, metadata-only deleteWhere, and delete-all paths.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Sonnet 4.6\n\nCloses #57161 from brijrajk/SPARK-58008-delete-with-options.\n\nAuthored-by: BRIJ RAJ KISHORE \u003c22271048+brijrajk@users.noreply.github.com\u003e\nSigned-off-by: Peter Toth \u003cpeter.toth@gmail.com\u003e\n" }, { "commit": "6f1257366a04a5fe54e866962dd2cb2851e5e42c", "tree": "2ea49b570f644af1a95c1984bab3dcf2352d4f89", "parents": [ "a1c3f310a8c34a0c15661a41cb62cceae6032e1c" ], "author": { "name": "Ruifeng Zheng", "email": "ruifengz@apache.org", "time": "Tue Jul 28 16:19:45 2026 +0800" }, "committer": { "name": "Ruifeng Zheng", "email": "ruifengz@apache.org", "time": "Tue Jul 28 16:19:45 2026 +0800" }, "message": "[SPARK-58360][ML][CONNECT] Avoid nested parent overcount in RFormulaModel size estimate\n\n### What changes were proposed in this pull request?\n\nThis PR adds a specialized `RFormulaModel.estimatedSize` implementation. It counts the model metadata, the resolved formula, and the nested `PipelineModel` through its parent-safe estimate.\n\n### Why are the changes needed?\n\nThe default model-size estimate clears only the outer model parent. `RFormulaModel` retains its nested pipeline, whose stages can otherwise retain estimator-parent graphs and overcount Spark session or context state in ML Connect cache accounting.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nAdded `RFormulaSuite` coverage that fits a categorical R formula and verifies that the fitted model estimate stays below the expected upper bound.\n\nAlso ran `git diff --check`, ASCII, and source-line-length checks.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Codex (GPT-5)\n\nCloses #57552 from zhengruifeng/rformula-size-estimate-dev3.\n\nAuthored-by: Ruifeng Zheng \u003cruifengz@apache.org\u003e\nSigned-off-by: Ruifeng Zheng \u003cruifengz@apache.org\u003e\n" }, { "commit": "a1c3f310a8c34a0c15661a41cb62cceae6032e1c", "tree": "b3e5465020edddd480f9316a2a846c32220ea42b", "parents": [ "9ccb1ddab42aef78e6eb2d3b260fdedbb60e0002" ], "author": { "name": "Peter Toth", "email": "peter.toth@gmail.com", "time": "Tue Jul 28 09:43:06 2026 +0200" }, "committer": { "name": "Peter Toth", "email": "peter.toth@gmail.com", "time": "Tue Jul 28 09:43:06 2026 +0200" }, "message": "[SPARK-58370][SQL] Check table write privileges when the write target is also read in the same statement\n\n### What changes were proposed in this pull request?\n\n`RelationResolution.tryResolvePersistent` answers a hit in the per-query `AnalysisContext.relationCache` before reaching the branch that calls `CatalogV2Util.loadTable(catalog, ident, timeTravelSpec, Option(writePrivileges))`. This PR stops serving a reference that carries `REQUIRED_WRITE_PRIVILEGES` from that cache, so a write target always goes through `TableCatalog.loadTable(ident, writePrivileges)`:\n\n```scala\nval writePrivileges \u003d u.options.get(UnresolvedRelation.REQUIRED_WRITE_PRIVILEGES)\nval cached \u003d if (writePrivileges \u003d\u003d null) relationCache.get(key) else None\ncached.map(adaptCachedRelation(_, planId)).orElse { ... }\n```\n\nLoaded relations are still *stored* in the cache, so a self-MERGE\u0027s source keeps sharing the target\u0027s `Table` instance (one snapshot). `V2TableReference` already treats write targets this way (`V2TableReference.WriteTargetContext.cacheable \u003d false`).\n\n### Why are the changes needed?\n\n`TableCatalog.loadTable(Identifier, Set\u003cTableWritePrivilege\u003e)` (`since 3.5.3`) is where a catalog authorizes a write; the only other caller in the tree is `ResolveSchemaEvolution`. A write target hits the relation cache whenever another reference to the same table was resolved earlier in the same analyzer run, which is exactly the case for `InsertIntoStatement` and `V2WriteCommand` - their target is resolved at the command node *after* its query. So the catalog is never asked to authorize these statements:\n\n```sql\nINSERT INTO t SELECT * FROM t\nINSERT OVERWRITE t SELECT * FROM t WHERE 1 \u003d 0 -- empties the table\nINSERT INTO t REPLACE WHERE i \u003d 0 SELECT * FROM t\nINSERT INTO t SELECT i FROM (SELECT i FROM t) x\nWITH c AS (SELECT i FROM t) INSERT INTO t SELECT i FROM c\n```\n\n```scala\nspark.table(\"t\").writeTo(\"t\").append() // DataFrameWriterV2 passes the unanalyzed plan\n```\n\n`INSERT INTO t SELECT 1` on the same table is authorized normally, and `UPDATE` / `DELETE` / `MERGE` are unaffected (their target is a plain child, resolved before the source, so it is a cache miss), as is `DataFrameWriter.insertInto` (it passes the already-analyzed plan).\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. On a catalog that enforces privileges in `loadTable(ident, writePrivileges)`, the statements above are now authorized like every other write, so a user without the required privilege gets the catalog\u0027s error instead of silently writing. Catalogs that ignore the privileges argument (including Spark\u0027s built-in `V2SessionCatalog`) are unaffected. The write target of a self-referencing INSERT is now loaded once more, which is what a non-self-referencing INSERT already does.\n\n### How was this patch tested?\n\nNew test in `DataSourceV2SQLSuite` next to \"SPARK-49246: read-only catalog\" (whose write cases all use a constant source, which is why this gap was never covered), asserting the `ReadOnlyCatalog` privilege error for every statement above plus the UPDATE / DELETE / MERGE self-reference cases, for both a custom v2 catalog and a `ReadOnlyCatalog`-backed `spark_catalog`. It fails on master (\"Expected exception java.lang.RuntimeException to be thrown, but no exception was thrown\") and passes with the fix.\n\nAlso green: 17 suites / 1506 tests - `DataSourceV2SQLSuite` (V1+V2 filter), `DataSourceV2DataFrameSuite`, `DataSourceV2OptionSuite`, `PlanResolutionSuite`, `CachedTableSuite`, `*ViewTestSuite`, group/delta-based MERGE/UPDATE/DELETE suites, `InsertSuite`. `dev/lint-scala` clean.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Code (Claude Opus 5)\n\nCloses #57563 from peter-toth/SPARK-58370-write-privileges-self-reference.\n\nAuthored-by: Peter Toth \u003cpeter.toth@gmail.com\u003e\nSigned-off-by: Peter Toth \u003cpeter.toth@gmail.com\u003e\n" }, { "commit": "9ccb1ddab42aef78e6eb2d3b260fdedbb60e0002", "tree": "fe79338a496cfc1dc386712bcae1152e5e4d4398", "parents": [ "863694202aef47863bbdef35816d3c0e8f3066b9" ], "author": { "name": "Oleks V", "email": "10247224+comphead@users.noreply.github.com", "time": "Tue Jul 28 09:34:56 2026 +0200" }, "committer": { "name": "Peter Toth", "email": "peter.toth@gmail.com", "time": "Tue Jul 28 09:34:56 2026 +0200" }, "message": "[SPARK-55749][SQL][DOC] Clarify `array_contains` null-handling in the docs and example\n\n### What changes were proposed in this pull request?\n\nClarify the `array_contains` documentation:\n\n - `sql/api/.../functions.scala`: rewrote the Scaladoc summary to cover all three return cases (true / false / null).\n - `sql/catalyst/.../collectionOperations.scala`: added a SQL example showing the null result when the value is absent but the array contains a null element.\n\n### Why are the changes needed?\n\n The old docs said `array_contains` returns \"false otherwise\", omitting the null-propagation case:\n\n ```\n scala\u003e spark.sql(\"select array_contains(array(1, null, 3), 2)\").show(false)\n +------------------------------------+\n |array_contains(array(1, NULL, 3), 2)|\n +------------------------------------+\n |NULL |\n +------------------------------------+\n ```\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. Documentation-only.\n\n### How was this patch tested?\n\nN/A — docs only.\n\n### Was this patch authored or co-authored using generative AI tooling?\nNo\n\nCloses #57413 from comphead/arr_contains_doc.\n\nLead-authored-by: Oleks V \u003c10247224+comphead@users.noreply.github.com\u003e\nCo-authored-by: ovoievodin \u003co_voievodin@apple.com\u003e\nSigned-off-by: Peter Toth \u003cpeter.toth@gmail.com\u003e\n" }, { "commit": "863694202aef47863bbdef35816d3c0e8f3066b9", "tree": "2c290b6f6c64a8c6131bda676e0e297948179f5d", "parents": [ "35309c5336dbf08781212623d2ce05e7a1ba86f9" ], "author": { "name": "Spenser Sun", "email": "haotian.sun@databricks.com", "time": "Tue Jul 28 14:00:00 2026 +0800" }, "committer": { "name": "Ruifeng Zheng", "email": "ruifengz@apache.org", "time": "Tue Jul 28 14:00:00 2026 +0800" }, "message": "[SPARK-58332][PYTHON][TEST] Move compare_or_generate_golden_matrix into GoldenFileTestMixin\n\n### What changes were proposed in this pull request?\n\n`compare_or_generate_golden_matrix` was duplicated verbatim across three PyArrow golden-file test files:\n\n- `python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_cast.py`\n- `python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_default.py`\n- `python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py`\n\nThis PR moves it into `GoldenFileTestMixin` (`python/pyspark/testing/goldenutils.py`), which all three suites already inherit, and removes the local copies. Imports that became unused after the removal (`inspect`, `os`, `typing.Callable/List/Optional`) are dropped from the test files.\n\nThis is a follow-up to #57435, where reviewers asked to centralize the duplicated matrix driver into the mixin.\n\n### Why are the changes needed?\n\nRemoves duplicated test machinery so the golden-file matrix driver has a single implementation, making it easier to maintain and reuse for future golden-file suites.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. Test-only, behavior-preserving refactor.\n\n### How was this patch tested?\n\nExisting suites pass in compare mode (no golden files regenerated):\n\n```\npython -m pytest \\\n python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_cast.py \\\n python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_default.py \\\n python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py\n```\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57510 from Spenserrrr/centralize-golden-matrix-helper.\n\nAuthored-by: Spenser Sun \u003chaotian.sun@databricks.com\u003e\nSigned-off-by: Ruifeng Zheng \u003cruifengz@apache.org\u003e\n" }, { "commit": "35309c5336dbf08781212623d2ce05e7a1ba86f9", "tree": "4e7ae52181baff5a58103cb3dbae6e6ae6895cb7", "parents": [ "8ed480d6bd23c075c5e2ca34cb69d0aeb9692b8f" ], "author": { "name": "Boyang Jerry Peng", "email": "jerry.peng@databricks.com", "time": "Tue Jul 28 13:41:45 2026 +0800" }, "committer": { "name": "Wenchen Fan", "email": "wenchen@databricks.com", "time": "Tue Jul 28 13:41:45 2026 +0800" }, "message": "[SPARK-58263][CORE] Concurrently schedule pipelined-shuffle stage groups in the DAGScheduler\n\n### What changes were proposed in this pull request?\n\nThis PR adds native `DAGScheduler` support for **concurrently scheduling stages connected by a\n`PipelinedShuffleDependency`** (added earlier in the stack), together with the admission and\ncompletion semantics that make co-scheduling correct.\n\nA pipelined shuffle is incrementally readable: a consumer stage may begin reading the producer\u0027s\noutput while the producer is still running. The stock scheduler runs a consumer only after its\nproducer has fully materialized; this PR teaches the scheduler to co-schedule a producer and its\npipelined consumer as a **pipelined group** instead. Every new path is gated on a job actually\nusing a pipelined dependency, so behavior is unchanged for all existing jobs (the existing\n`DAGSchedulerSuite` is unaffected).\n\nThis PR supports a job that is **all-regular or all-pipelined** (a job mixing the two is rejected\nup front). So an all-pipelined job\u0027s whole stage graph is one pipelined group, which lets admission\nbe decided once, up front.\n\n- **Up-front gang admission (`handleJobSubmitted`).** All members of a pipelined group must run\n concurrently, so a group that cannot fit would deadlock (a consumer holds slots waiting for\n producer output while the producer cannot get slots to produce). Before any stage is created, the\n group\u0027s total task demand (computed from the RDD graph) is compared against the currently **free**\n slots of its resource profile -- total capacity (`maxNumConcurrentTasks`) minus the *outstanding*\n (running **plus enqueued**) task demand of other work in the same profile. Counting enqueued, not\n just running, demand prevents two groups from each passing the check yet failing to co-fit. If the\n group does not fit, the job fails fast with `CONCURRENT_SCHEDULER_INSUFFICIENT_SLOT` before any\n member runs -- true all-or-nothing admission that leaves no partial scheduler state (like the\n barrier slot check). There is no scheduler-side retry: a transient shortfall is the caller\u0027s to\n retry (a streaming query\u0027s batch loop reruns the batch). The check can be disabled with\n `spark.scheduler.pipelinedGroup.slotCheck.enabled\u003dfalse` for deployments that admit capacity\n out-of-band (e.g. a slot reservation).\n\n- **Co-scheduling (`submitStage`).** A stage\u0027s missing parents are classified by shuffle dependency\n type; a parent reached through a `PipelinedShuffleDependency` is a \"pipelined parent\". The stage is\n co-scheduled with its producers (tasks submitted immediately) only if every missing parent is\n pipelined **and** each is already running; otherwise it parks in `waitingStages` exactly as before.\n `submitWaitingPipelinedChildStages` is the \"producer started running\" analog of the existing\n \"producer completed\" hook: when a pipelined producer starts, its waiting consumers are reconsidered\n immediately, cascading transitively down a chain.\n\n- **Deferred completion for a co-scheduled consumer (`handleTaskCompletion` /\n `markStageAsFinished`).** A consumer co-scheduled with a still-running producer can finish first.\n Advancing its stage/job completion early would end the job and cancel the still-running producer,\n or make the consumer\u0027s output observable before the producer\u0027s. So the consumer\u0027s completion is\n deferred until the producer\u0027s outcome is final: its whole `CompletionEvent` is buffered and\n returns before any side effect runs (accumulator updates, `SparkListenerTaskEnd`, stage/job\n completion), which makes those side effects run exactly once, at replay. The buffered event is\n replayed on genuine producer success (applied normally), or dropped on producer failure -- on the\n drop the buffered tasks\u0027 `TaskEnd` events are still emitted (the tasks did finish, so active-task\n listeners must see them end) but no stage/job success is applied, since the group reruns. Deferrals\n are cleaned up on job end/abort so none outlive their job, flushing any still-buffered `TaskEnd`\n events on the way out.\n\n- **Other guards.** A job using a pipelined dependency is rejected up front when speculation is\n enabled (a speculative producer copy would race a consumer already reading partial output, with no\n commit barrier), when dynamic allocation is enabled (gang admission needs a stable slot set), or\n when a group member carries a **non-default resource profile** (gang admission measures capacity\n against the default profile, so the whole group must run on it -- per-profile admission is a\n follow-up); and a `PipelinedShuffleDependency` cannot be submitted as a map-stage job (no durable\n map output to compute statistics from).\n\nMain changes:\n\n- `DAGScheduler.scala` -- job classification, up-front gang admission, co-scheduling, and\n completion deferral. A one-pass RDD-graph walk (`rddGraphHasPipelinedDependency` /\n `classifyJobShuffleKinds`) keeps every new path inert for a job with no pipelined dependency.\n- `TaskSchedulerImpl.outstandingTasksForOtherWorkInProfile` -- a resource-profile-scoped\n outstanding-task (running + enqueued) count for the admission check (`private[scheduler]`).\n- `spark.scheduler.pipelinedGroup.slotCheck.enabled` -- a new `internal()` config (default `true`).\n- `CONCURRENT_SCHEDULER_INSUFFICIENT_SLOT` -- a new error condition.\n\nThis is a follow-up to `PipelinedShuffleDependency` and the dependency-typed shuffle-manager\nrouting (SPARK-58185, already in `master`); it is the first PR that makes the scheduler behave\ndifferently for a pipelined dependency. Group-atomic failure/rerun and additional fail-fast checks\nfor unsupported idioms follow in later PRs of the stack.\n\n### Why are the changes needed?\n\n`PipelinedShuffleDependency` and its incremental shuffle routing let a consumer read a producer\u0027s\noutput as it is produced, but nothing takes advantage of that until the scheduler co-schedules the\ntwo stages -- otherwise the consumer still waits for the producer to fully materialize and the\npipelining is never realized. Co-scheduling in turn requires admission control (a group that cannot\nco-fit must fail fast, not deadlock) and completion control (a fast-finishing consumer must not end\nthe job or cancel its producer). This PR provides both.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. All new behavior is gated on a job using a `PipelinedShuffleDependency`, which nothing constructs\nyet, so for every existing job the scheduler behaves exactly as before. The new\n`spark.scheduler.pipelinedGroup.slotCheck.enabled` config is `internal()` and defaults to `true`,\nand `CONCURRENT_SCHEDULER_INSUFFICIENT_SLOT` is a new error condition that can only surface for a job\nthat uses a pipelined dependency.\n\n### How was this patch tested?\n\nNew unit tests in `DAGSchedulerSuite` cover:\n\n- concurrent submission of a pipelined producer/consumer; inertness for a regular shuffle; a job\n mixing pipelined + regular shuffles rejected up front; a deep all-pipelined chain co-scheduled;\n transitive cascade when a producer starts; no double-submission on producer completion;\n- speculation, dynamic-allocation, and non-default-resource-profile rejection for a pipelined job\n (and that the corresponding regular jobs are not rejected -- including a regular job that merely\n attaches a non-default profile via `RDD.withResources`); a pipelined dependency submitted as a\n map-stage job rejected;\n- up-front admission: a group too large to co-fit failing fast with\n `CONCURRENT_SCHEDULER_INSUFFICIENT_SLOT`; a group that fits total capacity but not free slots\n failing fast; a group whose demand exactly equals free capacity admitted; other work\u0027s outstanding\n demand charged against admission; the slot check disabled admitting an over-capacity group;\n- deferred completion: an early-finishing consumer not ending the job or cancelling its running\n producer; its completion buffered until the producer finishes, then applied exactly once at replay\n (no buffer+replay `TaskEnd` duplication); normal producer-then-consumer ordering; the deferral\n dropped on producer failure; the deferral released only when the producer is genuinely available;\n an explicit job cancellation cleaning up the buffered deferral; and job teardown flushing a\n buffered consumer\u0027s `TaskEnd` events even when the release path never drained it (so active-task\n listeners do not leak the tasks as running).\n\n`TaskSchedulerImplSuite` covers `outstandingTasksForOtherWorkInProfile` counting running + enqueued\ntasks, excluding given stages, being resource-profile-scoped, and not double-counting a\nzombie + live attempt.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nCo-authored with: Claude Code (Opus 4.8)\n\nCloses #57341 from jerrypeng/stack/pipelined-shuffle-pr3-scheduling.\n\nAuthored-by: Boyang Jerry Peng \u003cjerry.peng@databricks.com\u003e\nSigned-off-by: Wenchen Fan \u003cwenchen@databricks.com\u003e\n" }, { "commit": "8ed480d6bd23c075c5e2ca34cb69d0aeb9692b8f", "tree": "d4ddd3f2b06ecf12ae68360706d4e8937f9f3ccf", "parents": [ "6f05a8309ab9d464ddf17d6ca34cc2e9a24c0090" ], "author": { "name": "Xiduo You", "email": "ulyssesyou18@gmail.com", "time": "Tue Jul 28 09:38:06 2026 +0800" }, "committer": { "name": "Xiduo You", "email": "ulyssesyou@apache.org", "time": "Tue Jul 28 09:38:06 2026 +0800" }, "message": "[SPARK-58317][SQL] Union output partitioning should support PartitioningCollection children\n\n### What changes were proposed in this pull request?\n\n`UnionExec.outputPartitioning` (SPARK-52921) passes a child partitioning through the union when every child is compatibly partitioned, letting a downstream operator (e.g. an aggregate) skip a shuffle. This PR extends that pass-through to children whose `outputPartitioning` is a `PartitioningCollection`.\n\n- `outputPartitioning` now treats each child\u0027s partitioning as a set of candidate partitionings (a `PartitioningCollection` flattens to its members; a single partitioning is a one-element set) and passes through the **intersection** across all children: empty -\u003e `UnknownPartitioning`; one member -\u003e that partitioning; many -\u003e a `PartitioningCollection`. Only index-co-locatable partitionings (`HashPartitioningLike` / `SinglePartition`) participate.\n- The existing `KeyedPartitioning` concatenation path is kept as a separate case that fires only when every child is a single `KeyedPartitioning`. It is a distinct physical strategy (`sparkContext.union`, `numPartitions \u003d sum`) and cannot be folded into the co-located intersection (`SQLPartitioningAwareUnionRDD`, `numPartitions \u003d N`), because a `PartitioningCollection` requires uniform `numPartitions` across its members.\n- `comparePartitioning` is documented as a leaf-only equivalence predicate; collections are flattened before it is called.\n\n### Why are the changes needed?\n\n`comparePartitioning` had no case for `PartitioningCollection`, so whenever a child reported one, the whole union collapsed to `UnknownPartitioning` and an extra shuffle was inserted. This is hit by a common query shape - a `UNION ALL` where one branch is an inner shuffled-hash join (whose `outputPartitioning` is `PartitioningCollection(Hash(k1), Hash(k2))`) and the other is a left join (single `HashPartitioning`), feeding a `GROUP BY`:\n\n```sql\nselect c1, c2, c3, count(*) from (\n select t1.c1, t1.c2, t1.c3 from t1 join t2 on t1.c1 \u003d t2.c1\n union all\n select t3.c1, t3.c2, t3.c3 from t3 left join t4 on t3.c1 \u003d t4.c1\n) group by c1, c2, c3\n```\n\nThe union failed to pass through and a redundant `ENSURE_REQUIREMENTS` shuffle appeared before the aggregate.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo change in query results. Under the default-on `spark.sql.unionOutputPartitioning`, affected plans drop a redundant shuffle.\n\n### How was this patch tested?\n\nAdded tests to `DataFrameSetOperationsSuite` covering: the mixed inner+left join union (intersection to a single `HashPartitioning`, shuffle eliminated); all children reporting collections (pass-through as a `PartitioningCollection`); empty intersection (fall back to `UnknownPartitioning`); and the collection pass-through under AQE. Existing `DataFrameSetOperationsSuite` and `KeyGroupedPartitioningSuite` (97 tests) regressions pass.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57491 from ulysses-you/SPARK-58317.\n\nAuthored-by: Xiduo You \u003culyssesyou18@gmail.com\u003e\nSigned-off-by: Xiduo You \u003culyssesyou@apache.org\u003e\n" }, { "commit": "6f05a8309ab9d464ddf17d6ca34cc2e9a24c0090", "tree": "8ba283fb8bb9203d052702adcf272a89aae49a98", "parents": [ "f711fdf2d5c587d3dea3b7ed564acef1e2deb3f7" ], "author": { "name": "Ruifeng Zheng", "email": "ruifengz@apache.org", "time": "Tue Jul 28 08:47:18 2026 +0800" }, "committer": { "name": "Ruifeng Zheng", "email": "ruifengz@apache.org", "time": "Tue Jul 28 08:47:18 2026 +0800" }, "message": "[SPARK-58367][ML][CONNECT] Include ALS metadata in size estimates\n\n### What changes were proposed in this pull request?\n\nThis PR updates ALS pre-fit and fitted-model size estimation to include parameter metadata as well as the distributed user and item factor data.\n\n### Why are the changes needed?\n\nALS has a specialized estimate for distributed factor data but previously omitted the model\u0027s parameter metadata. Including it makes ALS cache accounting consistent with other Spark ML implementations.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nUpdated `ALSSuite` to verify the metadata contribution in both estimation paths and assert that a fitted model remains below the expected upper bound.\n\nAlso ran `git diff --check`, ASCII, and source-line-length checks.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Codex (GPT-5)\n\nCloses #57554 from zhengruifeng/als-model-size-metadata-dev3.\n\nAuthored-by: Ruifeng Zheng \u003cruifengz@apache.org\u003e\nSigned-off-by: Ruifeng Zheng \u003cruifengz@apache.org\u003e\n" }, { "commit": "f711fdf2d5c587d3dea3b7ed564acef1e2deb3f7", "tree": "89d5b18579714aa0493d8981f8fcb22a1283f738", "parents": [ "f1a705837c69b63338ba82e1bbcc892aef66448e" ], "author": { "name": "Ruifeng Zheng", "email": "ruifengz@apache.org", "time": "Tue Jul 28 08:45:27 2026 +0800" }, "committer": { "name": "Ruifeng Zheng", "email": "ruifengz@apache.org", "time": "Tue Jul 28 08:45:27 2026 +0800" }, "message": "[SPARK-58361][ML][CONNECT] Include clustering model metadata in size estimates\n\n### What changes were proposed in this pull request?\n\nThis PR updates `KMeansModel`, `BisectingKMeansModel`, and `GaussianMixtureModel` size estimates to include their parameter metadata in addition to learned clustering state.\n\n### Why are the changes needed?\n\nThese models override the default estimator and previously omitted their parameter-map and UID metadata. Counting this metadata makes their cache accounting consistent with other Spark ML models.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nAdded one fitted-model upper-bound size-estimation test in each model\u0027s existing clustering suite.\n\nAlso ran `git diff --check`, ASCII, and source-line-length checks.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Codex (GPT-5)\n\nCloses #57553 from zhengruifeng/clustering-model-size-metadata-dev3.\n\nAuthored-by: Ruifeng Zheng \u003cruifengz@apache.org\u003e\nSigned-off-by: Ruifeng Zheng \u003cruifengz@apache.org\u003e\n" }, { "commit": "f1a705837c69b63338ba82e1bbcc892aef66448e", "tree": "ff99353e47b7b8ad62f07fb9c9656f3aee598c88", "parents": [ "d8d08b6c3170739ed22737af1e81bada846b9085" ], "author": { "name": "Ruifeng Zheng", "email": "ruifengz@apache.org", "time": "Tue Jul 28 08:33:49 2026 +0800" }, "committer": { "name": "Ruifeng Zheng", "email": "ruifengz@apache.org", "time": "Tue Jul 28 08:33:49 2026 +0800" }, "message": "[SPARK-58249][PS][FOLLOWUP] Use native function for NumPy invert\n\n### What changes were proposed in this pull request?\n\nReplace the scalar pandas UDF mapping for NumPy `invert` on pandas-on-Spark objects with the native Spark SQL `bitwise_not` function. Add the `int64` boundary-value coverage to the existing native NumPy ufunc parity test.\n\n### Why are the changes needed?\n\nSpark already provides a native, Spark Connect-compatible bitwise-not function. Using it removes the Python worker boundary and preserves NumPy integer results.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. `np.invert` now preserves its integral result type, matching NumPy, instead of using the scalar pandas UDF mapping declared with a double result type.\n\n### How was this patch tested?\n\n- Added pandas-on-Spark parity coverage for `np.invert` using `int64` boundary values.\n- Ran `build/sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 -Phive package`.\n- Ran `JAVA_HOME\u003d/usr/lib/jvm/java-17-openjdk-amd64 SPARK_TESTING\u003d1 SPARK_PREPEND_CLASSES\u003d1 PYSPARK_PYTHON\u003d.venv/bin/python PYSPARK_DRIVER_PYTHON\u003d.venv/bin/python python/run-tests --testnames pyspark.pandas.tests.test_numpy_compat`.\n- Ran `git diff --check`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Codex (GPT-5)\n\nCloses #57562 from zhengruifeng/pandas-native-invert-ufunc.\n\nAuthored-by: Ruifeng Zheng \u003cruifengz@apache.org\u003e\nSigned-off-by: Ruifeng Zheng \u003cruifengz@apache.org\u003e\n" }, { "commit": "d8d08b6c3170739ed22737af1e81bada846b9085", "tree": "a07bdeb232ca1a0acdf7d6603064662d3daea315", "parents": [ "a8c69339382fdfc88093f92f86fd860b02b20fe0" ], "author": { "name": "Venkata krishnan Sowrirajan", "email": "venkat.sowrirajan@gmail.com", "time": "Mon Jul 27 17:28:13 2026 -0700" }, "committer": { "name": "Chao Sun", "email": "chao@openai.com", "time": "Mon Jul 27 17:28:13 2026 -0700" }, "message": "[SPARK-58097][CONNECT] Preserve composite (userId, sessionId) session identity in Connect UI/status store\n\n### What changes were proposed in this pull request?\n\nSpark Connect identifies a session by `(userId, sessionId)`, and two users may share the same session UUID. The Connect UI listener/store keyed sessions by `sessionId` alone, so such sessions collapsed into one record (merged execution counts; one user\u0027s close finished/removed the other\u0027s row). History Server replay reproduced the same bug.\n\nThis keys `SessionInfo`\u0027s KVStore natural key and the listener\u0027s live-session map by a synthesized composite `uniqueId \u003d userId/sessionId`, and threads `userId` through the UI session page, its links, and the REST `sessions/{sessionId}` endpoint (added in SPARK-57941). Specifically:\n\n- **Composite identity:** `SessionInfo` is keyed on `(userId, sessionId)`; the session page filters operations on both fields so one user\u0027s operations don\u0027t leak into another\u0027s page.\n- **Opaque user ids through the UI:** `userId` is arbitrary and part of the key, but the UI wraps requests in `XssSafeRequest` (strips apostrophes, HTML-escapes) and `PagedTable` re-echoes decoded params. So `userId` is carried as an unpadded base64url token, whose alphabet (`A-Za-z0-9-_`) survives both untouched. The REST endpoint takes the same token; an empty token is a valid empty user id (the protobuf default), so only an absent parameter is rejected.\n- **Legacy History Server stores:** `getSession` falls back to a field scan on a composite-key miss, so disk stores cached by an older Spark (keyed on `sessionId` alone) still resolve after an upgrade with no replay.\n\n### Why are the changes needed?\n\nTwo users sharing a session UUID are distinct sessions in Spark Connect (see `SparkConnectServiceE2ESuite`), but the UI/status store merged them, showing incorrect counts and lifecycle state. Surfaced during review of SPARK-57941.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, within the unreleased master/branch line (the REST API and this fix are unreleased):\n\n- The Connect session detail page and the REST `sessions/{sessionId}` endpoint now require a `userId` parameter (a base64url token) alongside the session id.\n- **Trade-off on legacy persistent stores:** the natural key changes without bumping `AppStatusStore.CURRENT_VERSION`. A version bump is the only replay knob and is global, forcing every application\u0027s disk store (core, SQL, Hive Thrift, Connect) to rebuild, which is disproportionate for a Connect-only key change. Instead the `getSession` fallback resolves legacy rows in place. Its one limitation: rows where two users genuinely shared a UUID *before* the upgrade were already merged under the old key and cannot be reconstructed without replay; those recover on cache eviction. Discussed and agreed with the reviewer on the PR.\n\n### How was this patch tested?\n\nAdded to `SparkConnectServerListenerSuite`: same-UUID/different-user coverage (live + History Server replay) and a legacy `sessionId`-keyed row that the fallback resolves. Added to `SparkConnectServerPageSuite`: per-user operation isolation and a base64url round-trip through the real `XssSafeRequest.stripXSS` transform for values containing `\u0027`, `+`, `\u003d`. Added to `ConnectResourceWithActualDataSuite`: REST lookup by the `userId` token over the real HTTP filter, plus an empty-user session addressable via an empty token. `connect/testOnly org.apache.spark.sql.connect.ui.* org.apache.spark.status.api.v1.connect.*` passes.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Code (Claude Opus 4.8)\n\nCloses #57219 from venkata91/SPARK-58097-session-identity.\n\nAuthored-by: Venkata krishnan Sowrirajan \u003cvenkat.sowrirajan@gmail.com\u003e\nSigned-off-by: Chao Sun \u003cchao@openai.com\u003e\n" }, { "commit": "a8c69339382fdfc88093f92f86fd860b02b20fe0", "tree": "b2077ad12cbe5e7eeea06bed5af821a61a6d0e68", "parents": [ "b66553d39c206a14c87f638953f64a3432b24d0e" ], "author": { "name": "Kousuke Saruta", "email": "sarutak@apache.org", "time": "Tue Jul 28 08:24:11 2026 +0900" }, "committer": { "name": "Kousuke Saruta", "email": "sarutak@apache.org", "time": "Tue Jul 28 08:24:11 2026 +0900" }, "message": "[SPARK-58364][CORE] Rename the configuration namespace from `spark.security.credentials` to `spark.security.oidc`\n\n### What changes were proposed in this pull request?\nThis PR renames the configuration namespace for the OIDC credential propagation feature from `spark.security.credentials` to `spark.security.oidc`.\nThis PR also renames relevant words in comments, strings and identifiers.\n\n### Why are the changes needed?\nDuring the discussion on another [PR](https://github.com/apache/spark/pull/57387#issuecomment-5084689771), reusing the spark.security.credentials.* for OIDC creates confusion\n\n### Does this PR introduce _any_ user-facing change?\nNo. This feature is under development.\n\n### How was this patch tested?\nGA.\n\n### Was this patch authored or co-authored using generative AI tooling?\nKiro CLI / Claude\n\nCloses #57557 from sarutak/oidc-propagation/rename-config-namespace.\n\nAuthored-by: Kousuke Saruta \u003csarutak@apache.org\u003e\nSigned-off-by: Kousuke Saruta \u003csarutak@apache.org\u003e\n" }, { "commit": "b66553d39c206a14c87f638953f64a3432b24d0e", "tree": "c6e8451ebc602d145b52a7d4141d015d02cca8ba", "parents": [ "3b28be3c85f2c577076a9b821791dfd1bf4e65e1" ], "author": { "name": "Andreas Neumann", "email": "anew@apache.org", "time": "Mon Jul 27 14:25:04 2026 -0700" }, "committer": { "name": "Szehon Ho", "email": "szehon.apache@gmail.com", "time": "Mon Jul 27 14:25:04 2026 -0700" }, "message": "[SPARK-57395][SDP] Implement SCD2 Batch Processor; foreachBatch Callback\n\n### What changes were proposed in this pull request?\n\nThis PR adds Scd2ForeachBatchHandler, the entry point that drives one SCD Type 2 AutoCDC microbatch reconciliation from a Structured Streaming foreachBatch callback. It is the SCD2 analog of the existing Scd1ForeachBatchHandler, and it composes the previously-landed Scd2BatchProcessor transforms into the single end-to-end per-batch pipeline:\n\n 1. Validate the incoming microbatch (ScdBatchValidator — null keys / null sequence / orderable sequence).\n 2. Preprocess the microbatch and compute the per-key minimum sequence.\n 3. Pull in the affected rows from both the auxiliary and target tables for the keys in this batch.\n 4. Union the microbatch with the affected rows, then run decomposition (decomposeOutOfOrderRows, dropRedundantRowsPostDecomposition, assertWellFormedRowsPostDecomposition).\n 5. Reconcile and route (reconcileStartAndEndAt, dropLeftoverDeletesPostReconciliation, promoteDecompositionTailsToTombstones, identifyAndTagAuxRows).\n 6. Merge the reconciled rows into the auxiliary table and the target table.\n\nThe handler is idempotent under same-batchId replay, relying on the logical-delete / garbage-collection scheme already implemented in Scd2BatchProcessor.\n\nFiles:\n - Scd2ForeachBatchHandler.scala — the new handler (~94 lines).\n - Scd2ForeachBatchHandlerSuite.scala — an end-to-end behavioral suite.\n\n### Why are the changes needed?\nThe SCD2 batch processor and its individual reconciliation transforms landed in prior PRs (e.g. SPARK-57378), but there was no component wiring them into a runnable per-microbatch callback. This handler is the missing piece that lets an SCD2 AutoCDC flow actually execute against the auxiliary and target tables on each streaming batch, and it provides a single place to assert the composed reconciliation behaves correctly end-to-end.\n\n### Does this PR introduce any user-facing change?\nNo. This adds internal SDP/AutoCDC machinery only. SCD2 AutoCDC flows are still gated (AUTOCDC_SCD2_NOT_SUPPORTED) and not yet reachable by users, so there is no change to released or user-facing behavior.\n\n### How was this patch tested?\nA new end-to-end suite Scd2ForeachBatchHandlerSuite (35 tests) exercises the handler against real auxiliary and target tables, covering:\n - input validation (null key / null sequence fail the batch without applying changes);\n - basic SCD2 semantics (insert opens a current record; update closes the old and opens a new; delete closes the current record);\n - in-batch sequences (multiple updates, insert+delete, insert/update/delete/re-insert);\n - out-of-order / late-arriving events (late insert, late update bisecting a record, late delete shortening a record, multiple bisections in one batch);\n - idempotency under same-batch replay (updates, deletes, repeated values, redelivered events) and cross-batch garbage collection of logically-deleted tombstones;\n - tracked vs. untracked column changes (untracked change updates in place; tracked change opens history);\n - multi-key / composite-key independence and case-sensitive/insensitive key resolution.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-By: Opus 4.8\n\nCloses #57495 from anew/SPARK-57395-scd2-foreachBatch-callback.\n\nLead-authored-by: Andreas Neumann \u003canew@apache.org\u003e\nCo-authored-by: Anish Mahto \u003canish.mahto99@gmail.com\u003e\nSigned-off-by: Szehon Ho \u003cszehon.apache@gmail.com\u003e\n" }, { "commit": "3b28be3c85f2c577076a9b821791dfd1bf4e65e1", "tree": "f14a3d9cec4126a85bf6ce993da89f0d1b05dd1d", "parents": [ "5ce51a624421acceec378e0170da24a8fc2a96eb" ], "author": { "name": "Tian Gao", "email": "gaogaotiantian@hotmail.com", "time": "Mon Jul 27 13:12:48 2026 -0700" }, "committer": { "name": "Tian Gao", "email": "gaogaotiantian@hotmail.com", "time": "Mon Jul 27 13:12:48 2026 -0700" }, "message": "[SPARK-58308][INFRA] Remove usage of ipython_genutils\n\n### What changes were proposed in this pull request?\n\nRemoved all usage of `ipython_genutils`.\n\n### Why are the changes needed?\n\nThis package was originally introduced by [SPARK-38517](https://issues.apache.org/jira/browse/SPARK-38517) because of https://github.com/jupyter/nbconvert/issues/1725 . However, `ipython_genutils` has not been maintained for 10 years and `nbconvert` quickly removed the usage and dependency of it. So now our doc gen CI should not rely on it either.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nCI should confirm that we don\u0027t need it anymore.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nNo.\n\nCloses #57478 from gaogaotiantian/remove-ipython_genutils.\n\nAuthored-by: Tian Gao \u003cgaogaotiantian@hotmail.com\u003e\nSigned-off-by: Tian Gao \u003cgaogaotiantian@hotmail.com\u003e\n" }, { "commit": "5ce51a624421acceec378e0170da24a8fc2a96eb", "tree": "09a2540d76671a8cab21e43e1b65545184dc7c50", "parents": [ "149ba07b5606a1f9443b313d74bbee8baa717ff4" ], "author": { "name": "Tian Gao", "email": "gaogaotiantian@hotmail.com", "time": "Mon Jul 27 11:53:19 2026 -0700" }, "committer": { "name": "Tian Gao", "email": "gaogaotiantian@hotmail.com", "time": "Mon Jul 27 11:53:19 2026 -0700" }, "message": "[SPARK-58340][PYTHON] Add missing package to pyproject.toml for lint\n\n### What changes were proposed in this pull request?\n\nAdd `pytest` and `types-protobuf` to `pyproject.toml`.\n\n### Why are the changes needed?\n\nThese two packages are used in lint CI. They are also needed if you want to run the full `lint-python` script. We should just put it into `pyproject.toml` to keep consistency.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nCI.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nNo.\n\nCloses #57515 from gaogaotiantian/add-required-package-to-lint.\n\nAuthored-by: Tian Gao \u003cgaogaotiantian@hotmail.com\u003e\nSigned-off-by: Tian Gao \u003cgaogaotiantian@hotmail.com\u003e\n" }, { "commit": "149ba07b5606a1f9443b313d74bbee8baa717ff4", "tree": "791d1fe34e61191492de3b73955d9b5053228c48", "parents": [ "a646dc80468ce2b9e3e1ca3ff3191b30d071a14f" ], "author": { "name": "Spenser Sun", "email": "haotian.sun@databricks.com", "time": "Mon Jul 27 11:47:28 2026 -0700" }, "committer": { "name": "Tian Gao", "email": "gaogaotiantian@hotmail.com", "time": "Mon Jul 27 11:47:28 2026 -0700" }, "message": "[SPARK-58346][PYTHON] Remove unnecessary type: ignore[union-attr] comments in PySpark\n\n### What changes were proposed in this pull request?\n\nRemoves 16 `# type: ignore[union-attr]` comments across 4 files in `python/pyspark` by making the implicit non-None assumption explicit at each access site:\n\n- **`sql/types.py` (9):** In `_parse_datatype_json_value`, each atomic-type branch matched its regex twice - once in the `elif` test and again to bind `m` - then suppressed the `Optional[re.Match]` on `m.group(...)`. Converting these to the walrus form (`elif m :\u003d PATTERN.match(json_value):`) matches once, narrows `m` for the branch body, and drops the ignore. This also removes the redundant second match and makes these branches consistent with the geometry/geography branches in the same function, which already use single-match-plus-guard.\n- **`errors/exceptions/captured.py` (5):** The `CapturedException` accessors access `SparkContext._jvm.PythonErrorUtils`, where `_jvm` is `Optional[JVMView]`. Each method already asserts `SparkContext._gateway is not None`; adding the matching `assert SparkContext._jvm is not None` (the same pattern already used elsewhere in the file) narrows `_jvm` and removes the ignore.\n- **`sql/connect/client/core.py` (1):** The code narrowed a local `session` via `if session is not None:` but then re-read the Optional class attribute `PySparkSession._instantiatedSession._jvm`. Using the narrowed local (`session._jvm`) removes the ignore and the redundant re-read.\n- **`sql/catalog.py` (1):** The code asserted `sc is not None` but the ignore was about `sc._gateway` being Optional. Adding `assert sc._gateway is not None` completes the guard the code had already started.\n\n### Why are the changes needed?\n\nThese ignores suppressed mypy `[union-attr]` errors that arise only because a value is typed Optional at the access site, even though the surrounding code guarantees it is not None. Making the assumption explicit is clearer and matches idioms the codebase already uses nearby. The `assert` additions do not introduce a new failure mode - the affected code already required a live JVM/gateway on these paths.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\n`mypy` passes at full scope over `python/pyspark` (1287 source files). No behavior change; existing tests cover the affected code paths.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57521 from Spenserrrr/nonetype-guards.\n\nAuthored-by: Spenser Sun \u003chaotian.sun@databricks.com\u003e\nSigned-off-by: Tian Gao \u003cgaogaotiantian@hotmail.com\u003e\n" }, { "commit": "a646dc80468ce2b9e3e1ca3ff3191b30d071a14f", "tree": "56454eea2857eadce7d4daf3100b2815e6a1322c", "parents": [ "e8d5e4a962b434821ede3cf9cc3a046a8eead8a8" ], "author": { "name": "Fabian Paul", "email": "fpaul@apache.org", "time": "Mon Jul 27 11:43:28 2026 -0700" }, "committer": { "name": "Tian Gao", "email": "gaogaotiantian@hotmail.com", "time": "Mon Jul 27 11:43:28 2026 -0700" }, "message": "[SPARK-58119][CORE][PYTHON] Introduce PythonWorkerHandle abstraction for the Python worker path\n\n### What changes were proposed in this pull request?\n\nThis PR introduces a small `PythonWorkerHandle` trait in `org.apache.spark.api.python` that\nabstracts the concrete Python-worker process backend, exposing only what the runner needs:\n\n- `isAlive(): Boolean`\n- `destroy(): Boolean`\n- `terminationDiagnostics(): Option[String]`\n\n`LocalPythonWorkerHandle` is the implementation backed by a local OS process\n(`java.lang.ProcessHandle`). Its\n`terminationDiagnostics` reads the worker\u0027s Python faulthandler log at `\u003cfaultHandlerLogDir\u003e/\u003cpid\u003e`\n(read-once, deleting the file after reading), where `faultHandlerLogDir: Option[File]` is bound when\nthe handle is created (`None` means faulthandler output is disabled for that worker).\n`PythonWorkerHandle.of(pid, faultHandlerLogDir)` constructs one.\n\n`PythonWorkerFactory` derives `faultHandlerLogDir` from `envVars` (the `PYTHON_FAULTHANDLER_DIR`\nentry it already passes to the worker process) and builds the handle. As a result the reader path no\nlonger needs a separate `faultHandlerEnabled` flag or a pre-projected pid threaded through it:\n`SparkEnv.createPythonWorker`, `BasePythonRunner` (`ReaderIterator`, `ReaderInputStream`,\n`createPipelinedDataIn`, `newReaderIterator`, `pythonWorkerStatusMessageWithContext`),\n`PythonUDFRunner`, `PythonArrowOutput`, and `PythonPlannerRunner` now carry\n`Option[PythonWorkerHandle]`, and the crash / idle-timeout messages call\n`handle.flatMap(_.terminationDiagnostics())`. The now-unused\n`BasePythonRunner.tryReadFaultHandlerLog` / `faultHandlerLogPath` helpers are removed (their logic\nnow lives in `LocalPythonWorkerHandle`).\n\n### Why are the changes needed?\n\nCurrently the runner depends directly on `java.lang.ProcessHandle` and spreads two separate concerns\nacross the reader path: how to talk to the worker process (liveness / termination) and how to obtain\nits post-mortem diagnostics (the faulthandler log, gated by a separate boolean and a pre-projected\npid). Abstracting the worker behind a handle decouples the runner from the concrete OS-process type,\ncentralizes diagnostics ownership on the handle, and makes the worker backend unit-testable.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is an internal refactor; worker lifecycle and the existing crash / idle-timeout error\nmessages are unchanged.\n\n### How was this patch tested?\n\nAdded `PythonWorkerHandleSuite`, covering `isAlive`/`destroy`, `PythonWorkerHandle.of`, and\n`terminationDiagnostics` (log present, log absent, and disabled when no log directory is bound).\nBuilt and test-compiled `core` and `sql/core` locally; the existing Python worker test suites\nexercise the reader path end to end.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: generative AI tooling (assisted authoring and refactoring).\n\nCloses #57247 from fapaul/processhandle-oss-demo.\n\nAuthored-by: Fabian Paul \u003cfpaul@apache.org\u003e\nSigned-off-by: Tian Gao \u003cgaogaotiantian@hotmail.com\u003e\n" }, { "commit": "e8d5e4a962b434821ede3cf9cc3a046a8eead8a8", "tree": "8e6af571705c1f8031dbc75328b8bae50dcefd9a", "parents": [ "18ad81147cbd86373ece320878b8469acd7f2e5f" ], "author": { "name": "Yicong Huang", "email": "17627829+Yicong-Huang@users.noreply.github.com", "time": "Mon Jul 27 09:30:50 2026 -0700" }, "committer": { "name": "Yicong Huang", "email": "17627829+Yicong-Huang@users.noreply.github.com", "time": "Mon Jul 27 09:30:50 2026 -0700" }, "message": "[SPARK-58354][PYTHON] Remove unused ArrowStreamPandasSerializer base class\n\n### What changes were proposed in this pull request?\n\nRemove the unused `ArrowStreamPandasSerializer` base class. After all Pandas UDF serializer subclasses were removed (`ArrowStreamPandasUDFSerializer`, `GroupPandasUDFSerializer`, `ArrowStreamAggPandasUDFSerializer`, `ApplyInPandasWithStateSerializer`, `CogroupPandasUDFSerializer`, `TransformWithStateInPandasSerializer`), the base class became an orphan with no subclasses, no instantiations, and no other references anywhere in the Python tree. This also drops the now-orphaned `_normalize_packed` helper (only used by that class) and the imports that are no longer needed.\n\n### Why are the changes needed?\n\nDead code cleanup. The class and its sole helper are unreachable.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nExisting tests. No behavior change.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nNo.\n\nCloses #57544 from Yicong-Huang/SPARK-58354.\n\nAuthored-by: Yicong Huang \u003c17627829+Yicong-Huang@users.noreply.github.com\u003e\nSigned-off-by: Yicong Huang \u003c17627829+Yicong-Huang@users.noreply.github.com\u003e\n" }, { "commit": "18ad81147cbd86373ece320878b8469acd7f2e5f", "tree": "72eaa55542644384d97dfd92414e182d7301286b", "parents": [ "adad83831f22076a855456aeae7ea62b3895e851" ], "author": { "name": "Andreas Neumann", "email": "andreas.neumann@databricks.com", "time": "Mon Jul 27 09:05:47 2026 -0700" }, "committer": { "name": "Jose Torres", "email": "jtorres@apache.org", "time": "Mon Jul 27 09:05:47 2026 -0700" }, "message": "[SPARK-57251][SDP] Validate SCD2 reserved framework columns at AutoCDC flow construction\n\n### What changes were proposed in this pull request?\nWhat changes were proposed in this pull request?\n\nThis PR replaces https://github.com/apache/spark/pull/57488, which had to be abandoned due to severe semantic merge conflicts. It is an exact cherry-pick of the previous commits, plus a fix for the test cases that failed due to the merge conflicts.\n\nWhat were the merge conflicts? Two PRs disagreeing on the semantics when a source contains a reserved column but does not include that column in the column selection. This PR rejects that because it validates pre-column selection. The other PR (SPARK-58313) allowed that because it was doing the validation port column selection. I decided to fall back to the original behavior that already existed for SCD type 1 validate pre column selection. And we have SPARK-58325 to reconsider whether we want to do validation post column selection.\n\nAutoCdcMergeFlow validates at construction time that a flow\u0027s source change-data feed does not carry columns that collide with AutoCDC\u0027s internal columns. Until now, that validation (requireReservedPrefixAbsentInSourceColumns) only rejected column names starting with the reserved prefix __spark_autocdc_.\n\n SCD2, however, persists two framework columns to the target that do not carry that prefix — __START_AT and __END_AT (see Scd2BatchProcessor.reservedFrameworkColNames). A source column named __START_AT or __END_AT therefore slipped past the guard and would be silently overwritten during microbatch preprocessing.\n\n This PR closes that gap (the TODO(SPARK-57251) in Scd2BatchProcessor):\n\n - Adds requireReservedFrameworkColumnsAbsentInSourceColumns() to the AutoCdcMergeFlow constructor. For SCD2 flows it rejects any source column whose name collides (by exact name, resolver-aware so it respects spark.sql.caseSensitive) with a non-prefixed reserved framework column. SCD1 targets carry no such columns, so the check is a no-op for SCD1.\n - Runs the check before the flow\u0027s schema val is forced, so the actionable reserved-name error surfaces ahead of the temporary AUTOCDC_SCD2_NOT_SUPPORTED gate, and the check remains correct once SCD2 support lands.\n - Adds a new error condition AUTOCDC_RESERVED_COLUMN_NAME_CONFLICT (SQLSTATE 42710), distinct from the existing prefix-based AUTOCDC_RESERVED_COLUMN_NAME_PREFIX_CONFLICT, since this collision is by exact name rather than by prefix.\n - Widens the visibility of Scd2BatchProcessor.reservedFrameworkColNames to private[pipelines] so the flow layer can validate against the single source of truth.\n\n### Why are the changes needed?\n Without this check, a user whose CDC source happens to contain a __START_AT or __END_AT column would have that data silently overwritten by AutoCDC\u0027s SCD2 framework columns, with no error and no diagnostic — a data-correctness footgun. Failing fast at flow construction with a user-actionable error (\"rename or remove the column\") is the intended UX, consistent with the existing reserved-prefix guard.\n\n### Does this PR introduce _any_ user-facing change?\nYes. An AutoCDC SCD2 flow whose source change-data feed contains a column named __START_AT or __END_AT (subject to case-sensitivity settings) now fails at flow construction with AUTOCDC_RESERVED_COLUMN_NAME_CONFLICT instead of silently overwriting the column. There is no change for SCD1 flows, and no change for SCD2 flows whose sources do not use these names. (Note: SCD2 AutoCDC flows are not yet generally supported on master — they are still gated by AUTOCDC_SCD2_NOT_SUPPORTED — so no released behavior changes.)\n\n### How was this patch tested?\nNew unit tests in AutoCdcFlowSuite covering:\n - an SCD2 flow with a __START_AT/__END_AT source column is rejected with AUTOCDC_RESERVED_COLUMN_NAME_CONFLICT;\n - the reserved-name check fires before the AUTOCDC_SCD2_NOT_SUPPORTED gate;\n - an SCD1 flow with the same column name is allowed and the column survives into the flow schema;\n - case-sensitivity behavior (spark.sql.caseSensitive true/false);\n - a guard test asserting the set of non-prefixed reserved names is exactly {__START_AT, __END_AT}, so a future rename can\u0027t silently un-cover the validation.\n\n### Was this patch authored or co-authored using generative AI tooling?\n Generated-by: Claude Code (Opus 4.8)\n\nCloses #57527 from anew/spark-57251-reserved-columns-v2.\n\nAuthored-by: Andreas Neumann \u003candreas.neumann@databricks.com\u003e\nSigned-off-by: Jose Torres \u003cjtorres@apache.org\u003e\n" }, { "commit": "adad83831f22076a855456aeae7ea62b3895e851", "tree": "1d547fd9803f0239a7826d315e9a6508ab7c7c9f", "parents": [ "90b1f8984ea8868bb1ea3edb9c66382de2930495" ], "author": { "name": "Qiegang Long", "email": "qlong@users.noreply.github.com", "time": "Tue Jul 28 00:04:08 2026 +0800" }, "committer": { "name": "Wenchen Fan", "email": "wenchen@databricks.com", "time": "Tue Jul 28 00:04:08 2026 +0800" }, "message": "[SPARK-58089][SQL] Push variant extractions through Aggregate/Sort/Join\n\n### What changes were proposed in this pull request?\nIntroduce a new optimizer rule `PullOutVariantExtractions` that hoists\n`variant_get` / `Cast(variant)` extractions out of three operator types\nthat the existing `PushVariantIntoScan` / `V2ScanRelationPushDown` rules\ncannot see through:\n\n- **Aggregate function arguments** – e.g. `max(variant_get(v, \u0027$.price\u0027, \u0027int\u0027))`:\n the extraction is moved into a `Project` directly below the `Aggregate` and the\n aggregate references the resulting alias. The bare variant column is suppressed\n unless it is also needed raw (e.g. as a `GROUP BY` key), so no redundant\n full-variant slot is generated.\n\n- **Sort order keys** – e.g. `ORDER BY variant_get(v, \u0027$.price\u0027, \u0027int\u0027)`:\n matched as `Project → Sort`; the extraction is hoisted below the `Sort` and\n the original `Project` is reproduced to prevent the alias from leaking into\n the output.\n\n- **Join conditions and projections above joins** – matched as `Project → Join`;\n extractions in both the join condition and the outer `Project` are routed to\n the owning join side. A `pushSideAliases` helper then pushes the aliases\n *through* any depth of chained joins so they land in a `Project` directly\n above the scan (where `PhysicalOperation` collapses them with the scan, making\n them visible to the pushdown). This is necessary because `PhysicalOperation`\n stops at a `Join` node.\n\nA `Sort` sitting over a `Join` is handled by fusing the two cases: the\norder-key aliases are pushed through the join tree, not left in a `Project`\nabove it.\n\nThe rule is gated by a new internal config\n`spark.sql.variant.pushVariantIntoScan.pullOutExtractions` (default `true`)\nand is a no-op unless `spark.sql.variant.pushVariantIntoScan` is also enabled.\nNon-variant plans are untouched.\n\nThe rule is registered as the first rule in `SparkOptimizer.earlyScanPushDownRules`,\nbefore `SchemaPruning` and the V2 scan pushdown rules.\n\n### Why are the changes needed?\nBefore this change, a `variant_get` inside an aggregate function argument, sort key,\nor join condition caused the whole variant column to be read raw (or shredded with a\nredundant full-variant slot). For example:\n\n```sql\nSELECT name, max(variant_get(v, \u0027$.price\u0027, \u0027int\u0027)) FROM T GROUP BY name\n```\nread the entire v column even though only the price field was needed.\nAfter this change, Spark shreds only the requested typed fields, avoiding the\nfull-variant I/O.\n\nThe change improves query performance for variant referenced in aggregrate, join, sort.\n\n### Does this PR introduce _any_ user-facing change?\nNo\n\n### How was this patch tested?\n\nAdded new units. Also run correctness tests against some known workload.\n\n#### Performance result\n\nTest framework: https://github.com/cloudera-labs/variant-conformance-benchmark\nDataset: tpc-ds dataset (SF\u003d5), spark native parquet table, payload in variant or json.\nRun setup: pre-warm jvm, three runs, median query timing reported by spark.\n\nRun A with pullout rule enabled vs B with the rule disabled.\n```\n\nCompare: tpcds-flat-pullout-20260714 vs tpcds-flat-no-pullout-20260714 (metric: query_median)\n Run A: /Users/qlong/opensources/variant-conformance-benchmark/results/tpcds-flat-pullout-20260714/tpcds-flat/timings-variant.csv\n Run B: /Users/qlong/opensources/variant-conformance-benchmark/results/tpcds-flat-no-pullout-20260714/tpcds-flat/timings-variant.csv\n\nquery median_A median_B delta_s delta_%\n------ -------- -------- ------- -------\nq07 1.40 3.28 -1.88 -57.2%\nq12 0.07 0.07 -0.00 -2.8%\nq19 0.06 0.07 -0.01 -9.9%\nq26 1.14 3.40 -2.26 -66.4%\nq42 0.56 2.59 -2.03 -78.2%\nq52 0.61 3.04 -2.43 -80.0%\nq55 0.54 2.68 -2.14 -79.9%\nq63 0.58 2.95 -2.37 -80.4%\nq68 1.00 3.42 -2.42 -70.8%\nq73 0.57 2.88 -2.31 -80.1%\nq79 0.85 3.26 -2.41 -73.9%\nq98 0.65 3.32 -2.67 -80.5%\n\nSummary: 12 queries, 12 comparable\n Geo-mean delta: -69.5% (Run A faster)\n Total (query_median): 8.0s vs 31.0s\n\n ```\n\nRun A with pullout rule enabled vs B using json for payload, this test shows the **performance advantage of variant over json**.\n\n```\nCompare: tpcds-flat-pullout-20260714 vs tpcds-flat-pullout-20260714 (metric: query_median)\n Run A: /Users/qlong/opensources/variant-conformance-benchmark/results/tpcds-flat-pullout-20260714/tpcds-flat/timings-variant.csv\n Run B: /Users/qlong/opensources/variant-conformance-benchmark/results/tpcds-flat-pullout-20260714/tpcds-flat/timings-string_json.csv\n\nquery median_A median_B delta_s delta_%\n------ -------- -------- ------- -------\nq07 1.40 3.26 -1.85 -56.9%\nq12 0.07 0.13 -0.06 -47.0%\nq19 0.06 0.23 -0.17 -72.2%\nq26 1.14 2.17 -1.03 -47.5%\nq42 0.56 0.56 +0.01 +1.1%\nq52 0.61 1.09 -0.48 -44.1%\nq55 0.54 0.47 +0.07 +15.2%\nq63 0.58 0.88 -0.30 -34.2%\nq68 1.00 1.34 -0.34 -25.1%\nq73 0.57 0.70 -0.13 -18.0%\nq79 0.85 1.62 -0.77 -47.4%\nq98 0.65 1.19 -0.54 -45.4%\n\nSummary: 12 queries, 12 comparable\n Geo-mean delta: -39.3% (Run A faster)\n Total (query_median): 8.0s vs 13.6s\n```\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nCo-authored with Claude Code\n\nCloses #57190 from qlong/variant-pushout-aggregate.\n\nAuthored-by: Qiegang Long \u003cqlong@users.noreply.github.com\u003e\nSigned-off-by: Wenchen Fan \u003cwenchen@databricks.com\u003e\n" }, { "commit": "90b1f8984ea8868bb1ea3edb9c66382de2930495", "tree": "371d48938dc0d539fe96149a30c32c779d89d9e6", "parents": [ "2d5e1b65350d0ae3d61c94c034a5207221a88950" ], "author": { "name": "Andreas Neumann", "email": "andreas.neumann@databricks.com", "time": "Mon Jul 27 09:03:05 2026 -0700" }, "committer": { "name": "Jose Torres", "email": "jtorres@apache.org", "time": "Mon Jul 27 09:03:05 2026 -0700" }, "message": "[SPARK-58313][SDP] Validate SCD2 track-history columns at AutoCDC flow construction\n\n### What changes were proposed in this pull request?\nThis PR replaces https://github.com/apache/spark/pull/57490, which had to be abandoned due to severe merge conflicts. It is an exact cherry-pick of the previous commits, plus a fix for the test cases that failed due to the merge conflicts.\n\nAn SCD2 AutoCDC flow can restrict which columns define a \"run\" via TRACK HISTORY ON (...), which populate ChangeArgs.trackHistorySelection. Until now, that selection was only resolved when the first microbatch ran, inside Scd2BatchProcessor.computeTrackedHistoryColumns during reconciliation. An unresolvable or ineligible tracking column — one that is absent from the source, is a key, is a reserved framework column, or was dropped by the flow\u0027s column_list — therefore surfaced mid-stream rather than at flow construction, unlike every other AutoCDC misconfiguration (keys, column selection, reserved names), which fail eagerly.\n\nThis PR validates trackHistorySelection at AutoCdcMergeFlow construction time, mirroring the existing requireKeysPresentInSelectedSchema check:\n\n - The eligibility + resolution logic is extracted from Scd2BatchProcessor.computeTrackedHistoryColumns into a schema-based companion helper Scd2BatchProcessor.computeTrackedHistoryColumns(schema, changeArgs, caseSensitive). Both the per-microbatch runtime path and the new construction-time validator call it, so the two can never diverge. The refactor is behavior-preserving.\n - AutoCdcMergeFlow gains requireTrackHistoryColumnsResolvableInSelectedSchema, invoked when deriving the user-selected schema (right after the key-presence check). It runs before the flow\u0027s schema is forced, so the actionable error surfaces ahead of the temporary AUTOCDC_SCD2_NOT_SUPPORTED gate and remains correct once SCD2 support lands.\n - No new error condition: an unresolvable selection reuses the existing AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA (schema name trackHistorySelection).\n - The check is a no-op when trackHistorySelection is None, which covers all SCD1 flows (enforced by ChangeArgs) and SCD2 flows that do not restrict tracking.\n\n### Why are the changes needed?\nDeferring this validation to reconciliation means a simple typo or misconfiguration (TRACK HISTORY ON (typo), or tracking a key/excluded column) is not reported at graph analysis time; it only fails once data flows, with an error raised deep in the SCD2 batch processor. Validating at flow construction gives a fail-fast, user-actionable error consistent with the rest of the AutoCDC configuration surface (keys, column_list, reserved names).\n\n### Does this PR introduce any user-facing change?\nYes. An AutoCDC SCD2 flow whose TRACK HISTORY ON (...) references a column that is not an eligible history-tracking column (absent, a key, a framework column, or excluded by column_list) now fails at flow construction with AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA instead of failing when the first microbatch runs. Valid selections are unaffected, and there is no change for SCD1 flows. (Note: SCD2 AutoCDC flows are not yet generally supported on master — still gated by AUTOCDC_SCD2_NOT_SUPPORTED — so no released behavior changes.)\n\n### How was this patch tested?\nNew unit tests in AutoCdcFlowSuite covering: an SCD2 flow tracking a non-existent column, a key column (ineligible), and a column dropped by columnSelection are each rejected at construction with AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA; a resolvable selection passes the check (falling through to the SCD2-not-supported gate); and case-sensitive/insensitive resolution behavior.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57524 from anew/spark-58313-track-history-v2.\n\nAuthored-by: Andreas Neumann \u003candreas.neumann@databricks.com\u003e\nSigned-off-by: Jose Torres \u003cjtorres@apache.org\u003e\n" }, { "commit": "2d5e1b65350d0ae3d61c94c034a5207221a88950", "tree": "2f81af9b1ad1c18114d2420924b62c84308745f3", "parents": [ "014c53b454c416d93993497b20a35ce4602068ee" ], "author": { "name": "Wenchen Fan", "email": "wenchen@databricks.com", "time": "Mon Jul 27 23:45:45 2026 +0800" }, "committer": { "name": "Wenchen Fan", "email": "wenchen@databricks.com", "time": "Mon Jul 27 23:45:45 2026 +0800" }, "message": "[SPARK-56460][CORE] Define configs in text files\n\n### What changes were proposed in this pull request?\nThis PR adds a new module `common/config` which introduces a framework to define configs in prototext files. When the Spark application starts, we load config definitions from all prototext files and put them in a map with config name as the key, and the proto binary of the config definition as the value. These proto-backed configs are also registered in `ConfigEntry.knownConfigs` via a wrapper `ProtoBackedConfigEntry`, together with existing Spark configs.\n\nThe proto schema defines each config entry with: key, value type, default value, scope (cluster vs session), mutability (static vs dynamic, i.e. whether the config can be changed after system initialization), visibility (public vs internal), binding policy (session/persisted/not_applicable for SQL views, UDFs, and procedures), documentation, and version. Enum values follow proto3 naming conventions with type-name prefixes (e.g. `SCOPE_CLUSTER`, `VALUE_TYPE_BOOL`, `VISIBILITY_PUBLIC`, `BINDING_POLICY_SESSION`, `MUTABILITY_STATIC`) to avoid namespace collisions. The `BindingPolicy` field maps to the existing Scala `ConfigBindingPolicy` enum. The `Mutability` field is used by `SQLConf.isStaticConfigKey` to determine whether a config can be changed at runtime.\n\nThe new framework will co-exist with the existing config framework, until we migrate all existing configs to this new framework.\n\nFor simple configs that are only accessed in one place, we can hardcode the config name in the place that accesses it, with a new API `SQLConf#getConfByKeyStrict` to avoid typo. Example in this PR: `spark.sql.optimizer.datasourceV2ExprFolding`\n\nFor configs that are accessed in multiple places and we want to avoid hardcoding config name, or configs that need custom validation, we can still have an entry in `object SQLConf` to reference the config definition. Examples in this PR: `spark.sql.optimizer.maxIterations` and `spark.sql.shuffledHashJoinFactor`.\n\nAs part of the migration, the 4 example configs are removed from the binding-policy exceptions allowlist (`configs-without-binding-policy-exceptions`) and assigned `NOT_APPLICABLE`, aligning them with standard binding-policy handling. Since `NOT_APPLICABLE` configs do not affect view/UDF/procedure resolution results, no resolved plans change.\n\n### Why are the changes needed?\nDefining configs in various Scala objects is a bad practice:\n- Configs are registered when the Scala objects are loaded. To list all configs we must know all these Scala objects and load them.\n- It\u0027s hard to audit configs as they spread all over the codebase.\n- We will hit JVM limitation one day sooner or later, as defining configs in a Scala object is basically doing heavy work in the constructor, which has limitation of 64 kb bytecode size.\n\n### Does this PR introduce _any_ user-facing change?\nNo, it\u0027s developer facing\n\n### How was this patch tested?\nnew tests\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Claude Code\n\nCloses #53488 from cloud-fan/conf.\n\nAuthored-by: Wenchen Fan \u003cwenchen@databricks.com\u003e\nSigned-off-by: Wenchen Fan \u003cwenchen@databricks.com\u003e\n" }, { "commit": "014c53b454c416d93993497b20a35ce4602068ee", "tree": "8422f2bd586290b9b6951659facef4346394675e", "parents": [ "54d5f5125972fc0f358a9716c30ab8e82bd53b40" ], "author": { "name": "YangJie", "email": "yangjie01@baidu.com", "time": "Mon Jul 27 23:40:33 2026 +0800" }, "committer": { "name": "yangjie01", "email": "yangjie01@baidu.com", "time": "Mon Jul 27 23:40:33 2026 +0800" }, "message": "[SPARK-58371][DOC] Update `json` gem version to 2.21.1\n\n### What changes were proposed in this pull request?\n\nThis PR upgrades the `json` gem from 2.12.2 to 2.21.1 in `docs/Gemfile.lock`. It is a transitive dependency (pulled in by `jekyll`, which requires `json (~\u003e 2.6)`), and 2.21.1 satisfies that constraint, so only the locked spec version changes.\n\n`json` has no runtime dependencies, so no other lock entries change and `docs/Gemfile` does not need to be touched.\n\n### Why are the changes needed?\n\n2.21.1 includes the fix for a security advisory that affects all versions in `\u003e\u003d 2.9.0, \u003c 2.19.9`:\n\n- [GHSA-x2f5-4prf-w687](https://github.com/advisories/GHSA-x2f5-4prf-w687) / CVE-2026-54696 (low): heap out-of-bounds write in the JSON generator when streaming to an IO. On the IO path, `fbuffer_do_inc_capa()` compared the requested size against the buffer\u0027s total capacity instead of its remaining capacity, so `JSON.dump(obj, io)` or `JSON::State#generate(obj, io)` could write past the buffer when serializing a string near 16 KB. Fixed in 2.19.9.\n\nFollowing the same pattern as SPARK-57633 (`concurrent-ruby` 1.3.7), this picks up the latest release rather than the minimum patched version.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This only affects the documentation build toolchain.\n\n### How was this patch tested?\n\nManually verified in a scratch copy of `docs/Gemfile` and `docs/Gemfile.lock`:\n\n1. Lock resolution matches. Running `bundle lock --update\u003djson` produces a lock file byte-identical to the hand-edited one, confirming 2.21.1 satisfies jekyll\u0027s `json (~\u003e 2.6)` and that no other locked spec is affected.\n\n2. Frozen install succeeds. `BUNDLE_FROZEN\u003dtrue bundle install` installs all 36 gems without modifying the lock file, and `bundle list` reports `json (2.21.1)`.\n\n3. The advisory\u0027s PoC no longer crashes:\n\n```ruby\nrequire \"json\"\nrequire \"stringio\"\n\nio \u003d StringIO.new\nbig \u003d \"a\" * 16385\nbig[16382] \u003d \u0027\"\u0027 # escapable byte near the buffer boundary\n\nJSON.dump([big], io)\n```\n\n```text\njson version: 2.21.1\nPoC (JSON.dump with IO): no crash, output bytes \u003d 16390\nroundtrip ok: true\n```\n\n4. The docs site builds:\n\n```\n$ cd docs \u0026\u0026 SKIP_API\u003d1 bundle exec jekyll build\nConfiguration file: .../docs/_config.yml\n\n************************\n* Building error docs. *\n************************\nGenerated: docs/_generated/error-conditions.html\n Source: .../docs\n Destination: .../docs/_site\n Incremental build: disabled. Enable with --incremental\n Generating...\nWarning: Tolerating missing API files because the following skip flags are set: SKIP_API\n done in 3.125 seconds.\n Auto-regeneration: disabled. Use --watch to enable.\n```\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Code (Opus 5)\n\nCloses #57565 from LuciferYang/docs-json-gem-2.21.1.\n\nAuthored-by: YangJie \u003cyangjie01@baidu.com\u003e\nSigned-off-by: yangjie01 \u003cyangjie01@baidu.com\u003e\n" }, { "commit": "54d5f5125972fc0f358a9716c30ab8e82bd53b40", "tree": "cf3639cffae2cdd9845fb66b93546975f88d9f61", "parents": [ "0747e287579272958600e3db3ccf03044378576f" ], "author": { "name": "YangJie", "email": "yangjie01@baidu.com", "time": "Mon Jul 27 23:36:57 2026 +0800" }, "committer": { "name": "yangjie01", "email": "yangjie01@baidu.com", "time": "Mon Jul 27 23:36:57 2026 +0800" }, "message": "[SPARK-58363][SQL] Assign a name to the error condition _LEGACY_ERROR_TEMP_2214-2219\n\n### What changes were proposed in this pull request?\n\nThe catalog-plugin loading failures in `Catalogs.load` were reported with six placeholder error conditions `_LEGACY_ERROR_TEMP_2214` through `_LEGACY_ERROR_TEMP_2219`, none of which carried a SQLSTATE.\n\nGroup the six into a single umbrella error condition `CANNOT_LOAD_CATALOG` (SQLSTATE `46103`), with one subclass per failure mode in the `Catalogs.load` try/catch:\n\n- `NOT_A_CATALOG_PLUGIN` — the class does not implement `CatalogPlugin`\n- `PLUGIN_CLASS_NOT_FOUND` — `ClassNotFoundException`\n- `CONSTRUCTOR_NOT_FOUND` — `NoSuchMethodException`\n- `CONSTRUCTOR_NOT_ACCESSIBLE` — `IllegalAccessException`\n- `ABSTRACT_CLASS` — `InstantiationException`\n- `CONSTRUCTOR_FAILURE` — `InvocationTargetException`\n\nThis also drops a stray trailing `)` in the legacy `_2216`/`_2217` messages.\n\n### Why are the changes needed?\n\nThe error-conditions README disallows new `_LEGACY_ERROR_TEMP_*` entries and asks existing ones to be resolved. This resolves six of them. SQLSTATE `46103` (\"Java Error\") is consistent with the sibling `CANNOT_LOAD_FUNCTION_CLASS`; the failures are triggered by a user-supplied `spark.sql.catalog.\u003cname\u003e` plugin class, so they are user-actionable rather than system errors.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. The `_LEGACY_ERROR_TEMP_*` names are not part of the public API. Message text is preserved (except the stray `)` typo removal), now rendered under the umbrella prefix.\n\n### How was this patch tested?\n\nUpdated the existing `checkError`/condition assertions in `CatalogLoadingSuite` and `SupportsCatalogOptionsSuite`, and added tests for the previously-uncovered `CONSTRUCTOR_NOT_FOUND` and `ABSTRACT_CLASS` subclasses. `build/sbt \"catalyst/testOnly *CatalogLoadingSuite\" \"core/testOnly org.apache.spark.SparkThrowableSuite\"` passes.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57556 from LuciferYang/assign-name-legacy-2214-2219.\n\nAuthored-by: YangJie \u003cyangjie01@baidu.com\u003e\nSigned-off-by: yangjie01 \u003cyangjie01@baidu.com\u003e\n" }, { "commit": "0747e287579272958600e3db3ccf03044378576f", "tree": "872ca4ffee9b73fbcca1dbe2b45aaf534e8d08fa", "parents": [ "effd582afa3980a6b674d212a705c0d5212f2948" ], "author": { "name": "YangJie", "email": "yangjie01@baidu.com", "time": "Mon Jul 27 23:31:18 2026 +0800" }, "committer": { "name": "yangjie01", "email": "yangjie01@baidu.com", "time": "Mon Jul 27 23:31:18 2026 +0800" }, "message": "[SPARK-57638][SQL] Avoid busy-waiting in Declarative Pipelines flow resolution\n\n### What changes were proposed in this pull request?\n`DataflowGraphTransformer.transformDownNodes` resolves flows on a bounded thread pool and drives them from a `while` loop that, each pass, partitioned the in-flight futures with the non-blocking `future.isDone`, reaped the completed ones, and scheduled a new flow if a slot was free. When all slots were in flight (or the queue was drained and only the last futures remained) and none had completed, the pass reaped nothing and scheduled nothing, then looped again immediately - busy-spinning on `isDone` and pinning a core for the duration of resolution.\n\nThis drives the loop with an `ExecutorCompletionService` instead: completed tasks are drained with the non-blocking `poll()`, and when nothing can be scheduled but tasks are still running, the loop blocks on `take()` until the next one finishes rather than spinning. Behavior is otherwise unchanged - the same flows are scheduled in the same order, exceptions are still propagated via `Future.get()`, and an `outstanding` counter replaces the `ArrayBuffer[Future]` for slot bookkeeping.\n\n### Why are the changes needed?\nResolving a graph with more flows than the parallelism (10) kept one CPU core busy at 100% doing no useful work for the whole resolution, which is wasteful and shows up as unexplained driver CPU.\n\n### Does this PR introduce _any_ user-facing change?\nNo.\n\n### How was this patch tested?\nTwo new cases in `ConnectValidPipelineSuite` cover the regime this PR changes - more flows than `parallelism` (10), so the slots fill and the loop reaches the blocking `take()` branch that replaces the busy-wait. The small graphs in the existing suites never get there.\n\n- `resolution terminates and resolves all flows when flow count exceeds parallelism` - 25 independent flows.\n- `resolution re-queues retryable flows under load when consumers exceed parallelism` - 20 consumers registered before their source `src`, so the first batch throws `TransformNodeRetryableException`, parks as dependents of `src`, and is re-queued once `src` resolves; this exercises the retryable re-queue path together with the blocking branch.\n\nBoth assert only the outcome (every flow resolves and the call returns), so they are deterministic and have no timing dependence - a regression that deadlocked would hang until the suite times out. Asserting the absence of a busy-wait directly is not included, since that requires CPU-time or timing measurements that are flaky in CI.\n\nExisting graph-resolution suites (`ConnectValidPipelineSuite`, `ConnectInvalidPipelineSuite`, `SqlPipelineSuite`, `TriggeredGraphExecutionSuite`, `MaterializeTablesSuite`) still pass; the change only affects how the loop waits, not what it resolves.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Claude Code (Claude Opus 4.8)\n\nCloses #56700 from LuciferYang/sdp-resolution-busy-wait.\n\nAuthored-by: YangJie \u003cyangjie01@baidu.com\u003e\nSigned-off-by: yangjie01 \u003cyangjie01@baidu.com\u003e\n" }, { "commit": "effd582afa3980a6b674d212a705c0d5212f2948", "tree": "ed7a7275c7633b59423bb1cd4258c5ade7c69829", "parents": [ "a663376d954fc950cc949d5f0a9d385cb262edfb" ], "author": { "name": "YangJie", "email": "yangjie01@baidu.com", "time": "Mon Jul 27 23:22:32 2026 +0800" }, "committer": { "name": "yangjie01", "email": "yangjie01@baidu.com", "time": "Mon Jul 27 23:22:32 2026 +0800" }, "message": "[SPARK-57635][SQL] Make Declarative Pipelines run-termination reason deterministic\n\n### What changes were proposed in this pull request?\n`TriggeredGraphExecution.getRunTerminationReason` decided which failed flow\u0027s reason to report by calling `collectFirst` over `failureTracker`, a `ConcurrentHashMap` whose iteration order is unspecified. When more than one flow exhausts its retries (a non-retryable `StopFlowExecution`), the flow whose reason gets surfaced therefore varied from run to run.\n\nThis extracts a small pure helper, `chooseRunTerminationReason`, that considers only the stopped flows and picks the earliest one by `(lastFailTimestamp, flowName)`, so the reported reason is stable across otherwise-identical runs. `getRunTerminationReason` now calls it and falls back to `UnexpectedRunFailure()`. The previous code also computed `graphForExecution.flow(...)` and `lastException` only to discard them; those are dropped.\n\n### Why are the changes needed?\nTwo runs that fail the same way could report different termination reasons (different flow name and cause), which is confusing in logs and events and makes the outcome non-reproducible.\n\n### Does this PR introduce _any_ user-facing change?\nNo. The reported reason was already one of the failing flows; it is now chosen deterministically.\n\n### How was this patch tested?\nAdded unit tests for `chooseRunTerminationReason` in `TriggeredGraphExecutionSuite`: the earliest failure wins regardless of iteration order, ties are broken by flow name, and retryable failures are ignored. They fail against the previous order-dependent selection and pass with this change.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Claude Code (Claude Opus 4.8)\n\nCloses #56698 from LuciferYang/sdp-deterministic-termination-reason.\n\nAuthored-by: YangJie \u003cyangjie01@baidu.com\u003e\nSigned-off-by: yangjie01 \u003cyangjie01@baidu.com\u003e\n" }, { "commit": "a663376d954fc950cc949d5f0a9d385cb262edfb", "tree": "b6bc6b844e8ead4c52c67de7e75b4c97f567e98f", "parents": [ "18ca27f34cc1fe72acbc6971e0cb89ea823b7bac" ], "author": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Mon Jul 27 11:32:57 2026 +0000" }, "committer": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Mon Jul 27 11:32:57 2026 +0000" }, "message": "[SPARK-58355][SS] Fix grammar in metadata log null-check message\n\n### What changes were proposed in this pull request?\nFixes the grammar of the null-metadata `require(...)` message in `HDFSMetadataLog` and `AsyncOffsetSeqLog`: \"\u0027null\u0027 metadata cannot written to a metadata log\" becomes \"... cannot be written ...\".\n\n### Why are the changes needed?\nThe message was missing the word \"be\". The sibling `AsyncCommitLog` already uses the correct phrasing, so this brings the two outliers in line. The message is a plain `require()` string, not part of the error-condition framework, so it is not golden-tested.\n\n### Does this PR introduce _any_ user-facing change?\nNo.\n\n### How was this patch tested?\nMessage-text-only change; no behavior change. No tests needed.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57546 from uros-b/error-metadatalog-cannot-be-written.\n\nLead-authored-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\nCo-authored-by: Uros Bojanic \u003curos.bojanic@databricks.com\u003e\nSigned-off-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\n" }, { "commit": "18ca27f34cc1fe72acbc6971e0cb89ea823b7bac", "tree": "99da0fac37b32b743d45512b657d5e36f40de687", "parents": [ "781ca6aeecf071c47b3b643dafcad39f10a1c6a8" ], "author": { "name": "Wenchen Fan", "email": "wenchen@databricks.com", "time": "Mon Jul 27 13:38:48 2026 +0800" }, "committer": { "name": "Wenchen Fan", "email": "wenchen@databricks.com", "time": "Mon Jul 27 13:38:48 2026 +0800" }, "message": "[SPARK-58069][SQL][FOLLOWUP] Handle empty approx_top_k combine buffers\n\n### What changes were proposed in this pull request?\n\nFollowup to https://github.com/apache/spark/pull/57177.\n\nThis change makes `CombineInternal` serialize and deserialize the placeholder buffer created for\nan empty partition. It encodes a missing item type as a zero-length type section and uses a default\nstring serde for the necessarily empty sketch.\n\n### Why are the changes needed?\n\nAn untouched `approx_top_k_combine` buffer has no item type. It can nevertheless be serialized\nbetween aggregation stages, where selecting a type-specific serde or serializing the type currently\nthrows. Preserving the empty placeholder allows the later merge to initialize it from real input.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. `approx_top_k_combine` no longer fails when an empty partition contributes an untouched\nplaceholder buffer between aggregation stages.\n\n### How was this patch tested?\n\nAdded a regression test to `ApproxTopKSuite` that round-trips an empty combine buffer and verifies\nthat its missing item type, sketch, and configured size are preserved. The staged workflow will run\nthe required fail-before and pass-after suite cycle before opening the pull request.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Codex (GPT-5)\n\nCloses #57375 from cloud-fan/SPARK-58069-followup-238032.\n\nAuthored-by: Wenchen Fan \u003cwenchen@databricks.com\u003e\nSigned-off-by: Wenchen Fan \u003cwenchen@databricks.com\u003e\n" }, { "commit": "781ca6aeecf071c47b3b643dafcad39f10a1c6a8", "tree": "3dbf0ba85a2534580fae8262738fee74bff1ff8a", "parents": [ "59734428bf762dc544c2d46e53a1bd55d73a40ef" ], "author": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Mon Jul 27 04:11:39 2026 +0000" }, "committer": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Mon Jul 27 04:11:39 2026 +0000" }, "message": "[SPARK-58357][SQL] Remove unused withCatalogIdentClause in SparkSqlAstBuilder\n\n### What changes were proposed in this pull request?\nRemoves the unused private method `withCatalogIdentClause` in `SparkSqlAstBuilder`, along with the `SparkException` and `PlanWithUnresolvedIdentifier` imports that it was the sole remaining user of.\n\n### Why are the changes needed?\nIts only caller (`visitSetCatalog`) was rewritten to call `expression(...)` directly, leaving the method with zero references. Removing it and its now-orphaned imports is dead-code cleanup with no behavior change.\n\n### Does this PR introduce _any_ user-facing change?\nNo.\n\n### How was this patch tested?\nPure removal of unreferenced code; existing parser tests unaffected. No new tests needed.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57548 from uros-b/deadcode-sparksqlparser-withcatalogidentclause.\n\nLead-authored-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\nCo-authored-by: Uros Bojanic \u003curos.bojanic@databricks.com\u003e\nSigned-off-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\n" }, { "commit": "59734428bf762dc544c2d46e53a1bd55d73a40ef", "tree": "ddab587cdb9960dd5873e84880b1ef7e99c1438b", "parents": [ "44d1160ae32d1bba04b2da0c36524309a263573b" ], "author": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Mon Jul 27 04:10:54 2026 +0000" }, "committer": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Mon Jul 27 04:10:54 2026 +0000" }, "message": "[SPARK-58356][CORE] Use exists instead of filter(...).nonEmpty in MasterPage\n\n### What changes were proposed in this pull request?\nReplaces `filter(_.resourcesInfoUsed.nonEmpty).nonEmpty` with `exists(_.resourcesInfoUsed.nonEmpty)` in `MasterPage` when deciding whether to show the resources column.\n\n### Why are the changes needed?\n`state.workers` is an `Array[WorkerInfo]`; the `filter(...).nonEmpty` form allocates a throwaway filtered array and scans every worker, while `exists` allocates nothing and short-circuits on the first match. The resulting Boolean is identical.\n\n### Does this PR introduce _any_ user-facing change?\nNo.\n\n### How was this patch tested?\nEquivalent, behavior-preserving change on a UI code path. No new tests needed.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57547 from uros-b/perf-masterpage-exists.\n\nLead-authored-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\nCo-authored-by: Uros Bojanic \u003curos.bojanic@databricks.com\u003e\nSigned-off-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\n" }, { "commit": "44d1160ae32d1bba04b2da0c36524309a263573b", "tree": "0e6fa718141ced38f1d5943e7ef743975f4e0fcd", "parents": [ "f6975c913fd1107143d0dd0a42b265496a99149a" ], "author": { "name": "YangJie", "email": "yangjie01@baidu.com", "time": "Mon Jul 27 11:36:46 2026 +0800" }, "committer": { "name": "yangjie01", "email": "yangjie01@baidu.com", "time": "Mon Jul 27 11:36:46 2026 +0800" }, "message": "[SPARK-58296][SQL] Fix to_time returning STRING type when the format is a foldable NULL\n\n### What changes were proposed in this pull request?\n\n`ToTime` (the `to_time` function) is a `RuntimeReplaceable`, so its `dataType` comes from `replacement.dataType`. When the format argument is foldable and evaluates to NULL, the replacement fell back to `Literal(null, expr.dataType)`, where `expr` is the format argument, so the whole expression reported the format\u0027s type (STRING) instead of TIME. This changes the fallback to `Literal(null, TimeType())`, so `to_time` reports `TimeType()` like all of its other branches.\n\n### Why are the changes needed?\n\n`to_time(str, fmt)` with a foldable NULL `fmt` reports the wrong type. `SELECT typeof(to_time(\u002700:12:00\u0027, null))` returns `string` instead of `time(6)`, and the analyzed plan carries STRING for that column, so schema inference and UNION type coercion (against a real TIME column) use the wrong type. The value is NULL either way, but the type is user-visible and wrong. The inconsistency is also internal: a non-foldable NULL format takes the `invokeParser()` branch and correctly yields `time(6)`, so the result type depended on whether the format happened to be foldable.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. `to_time(str, fmt)` with a foldable NULL `fmt` now has type `time(6)` instead of STRING, consistent with all other forms of `to_time`. The returned value (NULL) is unchanged. This is a bug fix: the wrong type has been present since `to_time` was introduced in 4.1.0.\n\n### How was this patch tested?\n\nAdded a `TimeExpressionsSuite` case asserting `ToTime(...).dataType \u003d\u003d\u003d TimeType()` for both an untyped and a STRING-typed foldable NULL format; it fails on the unfixed tree with `StringType did not equal TimeType(6)`. The same case also asserts the value is still NULL via `checkEvaluation`. Also added two golden queries in `time.sql` whose schema now locks in `time(6)`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57486 from LuciferYang/SPARK-58296-totime-null-format.\n\nAuthored-by: YangJie \u003cyangjie01@baidu.com\u003e\nSigned-off-by: yangjie01 \u003cyangjie01@baidu.com\u003e\n" }, { "commit": "f6975c913fd1107143d0dd0a42b265496a99149a", "tree": "7314c288465b46d235f8e76f5eeaeea33c089664", "parents": [ "160b6193ee35b89bb5825912b1596e6d8c906c1f" ], "author": { "name": "Wenchen Fan", "email": "wenchen@databricks.com", "time": "Mon Jul 27 10:15:22 2026 +0800" }, "committer": { "name": "Wenchen Fan", "email": "wenchen@databricks.com", "time": "Mon Jul 27 10:15:22 2026 +0800" }, "message": "[SPARK-52825][SQL] Register existing dialects for additional URL prefixes\n\n### What changes were proposed in this pull request?\n\nThis PR lets users register an existing `JdbcDialect` for an additional JDBC URL prefix. The\nregistration affects dialect lookup only; Spark still passes the original URL to the JDBC driver.\n\nIt adds `JdbcDialects.registerDialectForUrlPrefix` and\n`unregisterDialectForUrlPrefix`. It also adds `getBuiltInDialect`, a case-insensitive lookup for\nthe built-in `mysql`, `postgresql`, `db2`, `sqlserver`, `derby`, `oracle`, `teradata`, `h2`,\n`snowflake`, and `databricks` dialects. For example, a user can reuse the built-in MySQL dialect\nfor a known-compatible wrapper:\n\n```scala\nJdbcDialects.registerDialectForUrlPrefix(\n \"jdbc:aws-wrapper:mysql:\",\n JdbcDialects.getBuiltInDialect(\"mysql\"))\n```\n\nAn unsupported built-in dialect name raises a `SparkIllegalArgumentException` with the\n`UNSUPPORTED_BUILT_IN_JDBC_DIALECT` error condition.\n\nPrefixes are case-insensitive, validated to start with `jdbc:` and end with `:`, and consulted only\nwhen no registered dialect handles the original URL through `canHandle`. This preserves the\nprecedence of user-provided dialects. When prefixes overlap, the longest matching prefix wins.\n\n### Why are the changes needed?\n\nWrapper JDBC drivers can use URLs such as `jdbc:aws-wrapper:mysql://...` while speaking the same\ndatabase dialect as the underlying MySQL or PostgreSQL driver.\n\nMatching arbitrary nested protocols with a broad regular expression assumes that every URL of the\nform `jdbc:\u003canything\u003e:mysql` or `jdbc:\u003canything\u003e:postgresql` is compatible with the corresponding\nbuilt-in dialect. It can also inspect unrelated portions of the URL. Registering a specific prefix\ndirectly to an existing dialect makes that relationship deliberate and bounded without requiring\nusers to inherit from or copy a built-in dialect. The named lookup makes all built-in dialects\ndiscoverable without exposing a public collection or requiring knowledge of canonical JDBC URLs.\nSpark registers no additional prefixes by default, so compatibility remains an explicit user\ndecision.\n\nThis follows up on the discussion in #53902.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. Users can look up a built-in dialect by its documented name, and register or unregister an\nexisting dialect for an additional JDBC URL prefix through `JdbcDialects`. Wrapper URLs remain\nunmatched until a user explicitly registers a dialect for their prefix.\n\n### How was this patch tested?\n\nAdded tests covering all ten built-in dialect names, case-insensitive named lookup, structured\nerrors for unsupported names, explicit prefix registration and replacement, case-insensitive\nprefix matching, unregistration, prefix validation, longest-prefix matching, user-registered\ndialect precedence, and negative cases where unregistered wrappers or database names in the URL\nauthority must not select a dialect.\n\nThe following commands were run:\n\n```\nbuild/sbt \u0027sql/testOnly *JDBCSuite -- -z \"JDBC dialect\"\u0027\nbuild/sbt sql/scalastyle sql/Test/scalastyle\n```\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: OpenAI Codex (GPT-5)\n\nCloses #57455 from cloud-fan/jdbc-dialect-url-aliases.\n\nAuthored-by: Wenchen Fan \u003cwenchen@databricks.com\u003e\nSigned-off-by: Wenchen Fan \u003cwenchen@databricks.com\u003e\n" }, { "commit": "160b6193ee35b89bb5825912b1596e6d8c906c1f", "tree": "3f03b8b0ae4d98bdb6b7af80a8518bd803cc34e5", "parents": [ "ef9cc16ed52d141dca580c0a9d8ed14e35e95ead" ], "author": { "name": "Wenchen Fan", "email": "wenchen@databricks.com", "time": "Mon Jul 27 10:12:06 2026 +0800" }, "committer": { "name": "Wenchen Fan", "email": "wenchen@databricks.com", "time": "Mon Jul 27 10:12:06 2026 +0800" }, "message": "[MINOR][DOC] Use AGENTS.md for nested project instructions\n\n### What changes were proposed in this pull request?\n\nAdd project-instruction discovery guidance to the root `AGENTS.md`. Agents should discover nested\n`AGENTS.md` files in the current directory and its ancestors, with more specific instructions taking\nprecedence for their directory scope.\n\n### Why are the changes needed?\n\nNested `AGENTS.md` files are a common pattern for providing subsystem-specific guidance. Spark is\nexpected to adopt this pattern soon, so the root instructions should define how agents discover and\nprioritize those files consistently.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nDocumentation-only change. Verified with `git diff --check`; no runtime tests were needed.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: OpenAI Codex (GPT-5)\n\nCloses #57501 from cloud-fan/agents-md-nested-instructions.\n\nAuthored-by: Wenchen Fan \u003cwenchen@databricks.com\u003e\nSigned-off-by: Wenchen Fan \u003cwenchen@databricks.com\u003e\n" }, { "commit": "ef9cc16ed52d141dca580c0a9d8ed14e35e95ead", "tree": "15c501ca4b1e4f49bdf9e72126bbfe6e4498ec9f", "parents": [ "714cb540ebda17e21ba2c53959628087a6412427" ], "author": { "name": "Xiduo You", "email": "ulyssesyou18@gmail.com", "time": "Mon Jul 27 09:31:14 2026 +0800" }, "committer": { "name": "Xiduo You", "email": "ulyssesyou@apache.org", "time": "Mon Jul 27 09:31:14 2026 +0800" }, "message": "[SPARK-58210][SQL] Enable ReplaceHashWithSortAgg and CombineAdjacentAggregation by default\n\n### What changes were proposed in this pull request?\n\nThis PR enables two physical aggregate rules by default and decouples their configurations:\n\n- `spark.sql.execution.replaceHashWithSortAgg` now defaults to `true`. It replaces a hash-based aggregate with a sort aggregate when the aggregate\u0027s child already satisfies the grouping-key sort order.\n- `spark.sql.execution.combineAdjacentAggregation` now defaults to `true` and is no longer a `fallbackConf` of `replaceHashWithSortAgg`; it is an independent config. It merges an adjacent partial/final aggregate pair (with no shuffle between them) into a single complete-mode aggregate.\n- `ReplaceHashWithSortAgg` switches its plan traversal from `transformDown` to `transformUp`. `HashAggregateExec` does not expose a grouping-key output ordering (`outputOrdering \u003d Nil`), while `SortAggregateExec` does (it is order-preserving on the grouping keys). With `transformDown`, an outer (final) aggregate inspected before its inner (partial) aggregate sees the partial\u0027s `Nil` ordering and is not replaced; with `transformUp`, the partial is replaced first, so its sort-aggregate output ordering satisfies the final aggregate, which is then also replaced. This lets nested partial/final pairs both be converted when the leaf child is already sorted.\n\nAffected tests and golden files are updated, and migration-guide entries are added under \"Upgrading from Spark SQL 4.2 to 4.3\".\n\nTwo correctness fixes that enabling combining by default depends on were split out and merged separately as prerequisites:\n\n- #57464 (SPARK-58294) makes `HiveUDAFFunction` Complete-safe, so a Complete-mode `ObjectHashAggregateExec` (produced by combining) does not crash mode-aware Hive UDAFs.\n- #57460 (SPARK-58291) fixes the empty-buffer merge overflow in the statistical aggregates (`CentralMomentAgg` / `Covariance` / `Corr`), so the partial/final merge path returns the same finite result as the combined Complete-mode path.\n\nThis PR is rebased on top of both, so its diff no longer contains those source changes.\n\n### Why are the changes needed?\n\nBoth rules improve aggregate execution and have been available but off by default. `CombineAdjacentAggregation` was introduced (SPARK-43317) as a `fallbackConf` of `replaceHashWithSortAgg`, so enabling one enabled both. They are logically independent (one reuses existing ordering, the other merges an adjacent pair), so this PR turns each on by default and gives each its own config, letting users enable/disable them separately.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. On default config, Spark may now plan a sort aggregate instead of a hash aggregate (when the child is already sorted on the grouping keys) and may merge an adjacent partial/final aggregate pair into a single complete-mode aggregate. The physical plan of some queries changes as a result. To restore the previous behavior, set `spark.sql.execution.replaceHashWithSortAgg` and/or `spark.sql.execution.combineAdjacentAggregation` to `false` (both must be `false` to fully restore the previous partial/final staging, since the settings are now independent).\n\n### How was this patch tested?\n\nUpdated existing tests:\n- `ReplaceHashWithSortAggSuite` / `CombineAdjacentAggregationSuite`: the `checkAggs` helper now toggles both configs together; the former \"falls back to replaceHashWithSortAgg\" test is replaced by an independence test. A new `SPARK-58210: bottom-up traversal replaces both partial and final hash aggregate` case exercises the `transformDown` -\u003e `transformUp` change directly (SMJ -\u003e partial -\u003e final, combining disabled), expecting both aggregates replaced (0 hash, 2 sort) where top-down would leave 1 hash.\n- `SQLMetricsSuite` (SPARK-25497): pins `combineAdjacentAggregation\u003dfalse` so the single-partition query keeps its partial/final structure for the limit/codegen metric assertions.\n- `PlannerSuite` (SPARK-40086): accepts a sort aggregate for the single-partition sorted-input queries while still asserting no extra shuffle.\n- `HiveUDAFSuite` (SPARK-24935): pins `combineAdjacentAggregation\u003dfalse` to keep the partial/final staging this test targets (the combined Complete-mode path is covered separately by the SPARK-58294 test added in #57464).\n\nRegenerated golden files via `SPARK_GENERATE_GOLDEN_FILES\u003d1`:\n- `sql-tests/results/explain-cbo.sql.out`\n- `tpcds-plan-stability` approved plans for the affected TPC-DS queries.\n\nA total of **34** golden plans changed, all caused by the two rules enabled in this PR, with **no regressions**.\n\n| Type | Queries |\n|---|---|\n| Pure combine | q14a, q14a.sf100, q22a, q22a.sf100, q33, q33.sf100, q46.sf100 (modified \u0026 v1_4), q49, q49.sf100, q49.sf100 (v2_7), q51a, q51a.sf100, q56, q56.sf100, q60, q60.sf100, q66, q66.sf100, q68.sf100, q77a.sf100 |\n| Pure replace | q16, q16.sf100, q64.sf100, q94, q94.sf100, q95, q95.sf100, q64.sf100 (v2_7) |\n| Both | q23a.sf100, q23b.sf100, q64, q64 (v2_7) |\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Code\n\nCloses #57363 from ulysses-you/enable-hashtosort.\n\nAuthored-by: Xiduo You \u003culyssesyou18@gmail.com\u003e\nSigned-off-by: Xiduo You \u003culyssesyou@apache.org\u003e\n" }, { "commit": "714cb540ebda17e21ba2c53959628087a6412427", "tree": "e4def0d32f16de9fe412fc0469861c80fdf2f443", "parents": [ "5ebf3b2bd7128da15fc43b49d7dec7c53c8e2e2f" ], "author": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Sun Jul 26 22:23:12 2026 +0000" }, "committer": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Sun Jul 26 22:23:12 2026 +0000" }, "message": "[MINOR][ML] Fix doubled word in HasTrainingSummary scaladoc\n\n### What changes were proposed in this pull request?\nRemoves a duplicated word in the `summary` scaladoc of `HasTrainingSummary`: \"thrown if if `hasSummary` is false\" becomes \"thrown if `hasSummary` is false\".\n\n### Why are the changes needed?\nThe doubled \"if\" is a plain typo in a public API doc comment.\n\n### Does this PR introduce _any_ user-facing change?\nNo.\n\n### How was this patch tested?\nComment-only change. No tests needed.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57541 from uros-b/typo-hastrainingsummary-if-if.\n\nLead-authored-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\nCo-authored-by: Uros Bojanic \u003curos.bojanic@databricks.com\u003e\nSigned-off-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\n" }, { "commit": "5ebf3b2bd7128da15fc43b49d7dec7c53c8e2e2f", "tree": "ec3d48bc1a9603bc308793ba9c5af9e0a2bf4082", "parents": [ "ccf0eb5162e0cd3aee480036969361ac3be55b21" ], "author": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Sun Jul 26 22:22:25 2026 +0000" }, "committer": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Sun Jul 26 22:22:25 2026 +0000" }, "message": "[SPARK-58353][CONNECT] Extract parseExplainMode helper in Connect Dataset\n\n### What changes were proposed in this pull request?\nExtracts the two byte-identical explain-mode parsing blocks in Connect\u0027s `Dataset` (`explain(String)` and `explainString(String)`) into a single private `parseExplainMode(mode: String)` helper, and calls it from both sites.\n\n### Why are the changes needed?\nThe seven-line string-to-`ExplainMode` match was duplicated verbatim. Factoring it out removes the duplication with no behavior change (same case arms, same `\"Unsupported explain mode: \"` error on the untrimmed input).\n\n### Does this PR introduce _any_ user-facing change?\nNo.\n\n### How was this patch tested?\nBehavior-preserving refactor; the private helper is exercised by existing explain-mode coverage. No new tests needed.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57542 from uros-b/helper-connect-parse-explain-mode.\n\nLead-authored-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\nCo-authored-by: Uros Bojanic \u003curos.bojanic@databricks.com\u003e\nSigned-off-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\n" }, { "commit": "ccf0eb5162e0cd3aee480036969361ac3be55b21", "tree": "e7f1cb4733eca4d0bff704052a80da632afe46a6", "parents": [ "12f4f9bd8e38cc8048d7b747ffe38af9462ddc70" ], "author": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Sun Jul 26 22:21:45 2026 +0000" }, "committer": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Sun Jul 26 22:21:45 2026 +0000" }, "message": "[MINOR][DOC] Fix grammar in spark.speculation.efficiency.processRateMultiplier doc\n\n### What changes were proposed in this pull request?\nFixes the grammar of the `spark.speculation.efficiency.processRateMultiplier` description: \"A multiplier that used when evaluating inefficient tasks\" becomes \"A multiplier that is used ...\". The fix is applied in both the config\u0027s `.doc()` string (`core/.../config/package.scala`) and the mirrored `docs/configuration.md` entry so they stay in sync.\n\n### Why are the changes needed?\nThe relative clause was missing its verb. Both the source `.doc()` and the generated docs table had the same defect.\n\n### Does this PR introduce _any_ user-facing change?\nNo.\n\n### How was this patch tested?\nDocs/description-string change; no behavior change. No tests needed.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57543 from uros-b/doc-config-processratemultiplier-grammar.\n\nLead-authored-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\nCo-authored-by: Uros Bojanic \u003curos.bojanic@databricks.com\u003e\nSigned-off-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\n" }, { "commit": "12f4f9bd8e38cc8048d7b747ffe38af9462ddc70", "tree": "fd5f9c2edbd6afa798b68ead75400edbc2357827", "parents": [ "c7b2f1a865cce621d2c702bdfdff5132e251ca74" ], "author": { "name": "Yash Bapat", "email": "ybapat@purdue.edu", "time": "Sun Jul 26 13:50:25 2026 -0700" }, "committer": { "name": "Yicong Huang", "email": "17627829+Yicong-Huang@users.noreply.github.com", "time": "Sun Jul 26 13:50:25 2026 -0700" }, "message": "[MINOR][CONNECT] Remove redundant user_context.user_id assignment in execute_command methods\n\n### What changes were proposed in this pull request?\n\nRemove two redundant assignments of `req.user_context.user_id \u003d self._user_id` from `SparkConnectClient.execute_command` and `SparkConnectClient.execute_command_as_iterator`.\n\nBoth methods call `_execute_plan_request_with_metadata()` to build the request, and that helper already sets `req.user_context.user_id` under the identical `if self._user_id:` guard. The subsequent reassignments in the two callers are dead code.\n\n```python\n# _execute_plan_request_with_metadata() already does this:\nif self._user_id:\n req.user_context.user_id \u003d self._user_id\n\n# These lines in execute_command / execute_command_as_iterator were redundant:\n- if self._user_id:\n- req.user_context.user_id \u003d self._user_id\n```\n\nRelates to GitHub issue #56408.\n\n### Why are the changes needed?\n\nDead code removal / readability improvement. Setting the same protobuf field to the same value twice in sequence is misleading — it implies the first assignment might not be sufficient when it is.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. The resulting `req.user_context.user_id` value is identical before and after this change.\n\n### How was this patch tested?\n\nNo behavior change; existing Spark Connect unit tests cover `execute_command` / `execute_command_as_iterator` request construction.\n\n\u003e This PR was created with the assistance of Claude (AI). Disclosed per Apache Spark contribution guidelines.\n\nCloses #57493 from ybapat/issue-56408-dup-user-context.\n\nAuthored-by: Yash Bapat \u003cybapat@purdue.edu\u003e\nSigned-off-by: Yicong Huang \u003c17627829+Yicong-Huang@users.noreply.github.com\u003e\n" }, { "commit": "c7b2f1a865cce621d2c702bdfdff5132e251ca74", "tree": "6631a9319fa0839613819d84882f6a6f673e9490", "parents": [ "dee0c17c5c1ddff2f896d5e88d33c5d23247340b" ], "author": { "name": "Yicong Huang", "email": "17627829+Yicong-Huang@users.noreply.github.com", "time": "Sun Jul 26 14:36:05 2026 +0000" }, "committer": { "name": "Yicong-Huang", "email": "17627829+Yicong-Huang@users.noreply.github.com", "time": "Sun Jul 26 14:36:05 2026 +0000" }, "message": "[SPARK-58314][PYTHON] Remove ArrowStreamUDFSerializer\n\n### What changes were proposed in this pull request?\n\nRemoves the now-unused `ArrowStreamUDFSerializer` class from `python/pyspark/sql/pandas/serializers.py`, and updates two stale docstring references to it in `conversion.py` (`ArrowBatchTransformer.flatten_struct` / `wrap_struct`).\n\nAfter the PythonEvalType refactor moved serializer construction into `worker.py` and its last subclass `TransformWithStateInPySparkRowSerializer` was removed (SPARK-58297), `ArrowStreamUDFSerializer` has no remaining runtime users. The `flatten_struct` / `wrap_struct` transformer methods it used to call are retained -- they are still used directly by the Arrow UDF mappers in `worker.py`.\n\n### Why are the changes needed?\n\nDead-code cleanup. Part of SPARK-55388.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nExisting tests. No behavior change.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nNo.\n\nCloses #57507 from Yicong-Huang/SPARK-58314.\n\nAuthored-by: Yicong Huang \u003c17627829+Yicong-Huang@users.noreply.github.com\u003e\nSigned-off-by: Yicong-Huang \u003c17627829+Yicong-Huang@users.noreply.github.com\u003e\n" }, { "commit": "dee0c17c5c1ddff2f896d5e88d33c5d23247340b", "tree": "fc5d9a62e0ed0abce72f04e657fee3ab45b1710f", "parents": [ "f8c6ded7b58d681b91a32633b07e5f7ffcc2fb50" ], "author": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Sun Jul 26 14:32:51 2026 +0000" }, "committer": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Sun Jul 26 14:32:51 2026 +0000" }, "message": "[MINOR][DOC] Fix typo \u0027commited\u0027 in state data source docs\n\n### What changes were proposed in this pull request?\nCorrects the spelling \"commited\" to \"committed\" in the `changeEndBatchId` default-value description in the Structured Streaming state data source docs.\n\n### Why are the changes needed?\n`committed` doubles both the m and the t. The description is also semantically accurate: the default resolves to the latest committed batchId.\n\n### Does this PR introduce _any_ user-facing change?\nNo.\n\n### How was this patch tested?\nDocs-only change. No tests needed.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57520 from uros-b/doc-typo-committed-state-source.\n\nLead-authored-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\nCo-authored-by: Uros Bojanic \u003curos.bojanic@databricks.com\u003e\nSigned-off-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\n" }, { "commit": "f8c6ded7b58d681b91a32633b07e5f7ffcc2fb50", "tree": "3b074f1ab9d98a32e9976edfffc4e8007c7b4ef2", "parents": [ "fd3169438dbdada4623d310380b7c300543833af" ], "author": { "name": "Cheng Pan", "email": "pan3793@gmail.com", "time": "Sun Jul 26 18:59:20 2026 +0800" }, "committer": { "name": "Cheng Pan", "email": "chengpan@apache.org", "time": "Sun Jul 26 18:59:20 2026 +0800" }, "message": "[SPARK-58192][CORE] Support fractional spark.task.cpus\n\n### What changes were proposed in this pull request?\n\nThis PR adds support for fractional values of `spark.task.cpus` (e.g. `0.5`, `1.5`) and for\nfractional stage-level task cpus requests, so that the number of concurrent tasks on an\nexecutor is `floor(executor cores / task cpus)` computed exactly.\n\n**Exact decimal accounting.**\n\n- CPU bookkeeping moves from `Int` to `scala.math.BigDecimal`, normalized to a fixed scale of\n 9 by a new internal `CpuAmount` object (compact, long-backed, drift-free).\n- The exact value flows through `WorkerOffer`, `ExecutorData`, `TaskDescription`,\n `StatusUpdate`, and `TaskContext`.\n- Slot math uses exact `FLOOR` division (`1.0 / 0.1` is 10, not 9), clamps to\n `[0, Int.MaxValue]`, and slot sums saturate in `Long`.\n\n**Configuration and API.**\n\n- A new `ConfigBuilder.decimalConf` parses the exact decimal string; `spark.task.cpus` must be\n in `[1e-9, Int.MaxValue]` with at most 9 decimal places, validated before normalization so\n extreme exponents fail fast.\n- `TaskContext.cpuAmount()` (new, Scala and Python) returns the exact amount;\n `TaskContext.cpus()` is deprecated and returns its ceiling.\n- `TaskResourceRequests.cpus` accepts `Double`/`float`; amounts are rounded to the nearest\n `1e-9` and must be in the same range as the config. The addressable-resource assertions\n (`\u003c\u003d 1.0 || whole`, `\u003c\u003d 0.5 || whole`) no longer apply to cpus, which is a plain quantity\n from the core pool.\n- Validation happens at the request entry points and again at profile registration:\n `ResourceProfileManager.addResourceProfile` checks the cpus amount under the resource map\n key and forces the limiting-resource computation *before* inserting, so a malformed profile\n from any live path (raw construction, mutated builder map, Spark Connect) never enters the\n registry.\n- The `TaskResourceRequest` constructor stays lenient because it deserializes persisted data,\n which can carry anything an earlier release accepted (including `0` and `-Infinity`). The\n history server additionally parses replayed cpus values defensively: string length and\n range checked before parse/normalize, invalid values fall back to the default instead of\n failing replay.\n\n**PySpark executor memory** is split by the executor\u0027s concurrent task capacity (passed as a\ntask local property) rather than the raw core count, since fractional cpus can admit more\nworkers than cores.\n\n- A profile that does not override `pysparkMemory` inherits the default profile\u0027s allocation,\n matching how its executors are sized.\n- With dynamic allocation off -- where default-profile and task-only-profile tasks can share\n an executor -- both propagate `ceil(cores / taskCpus)`, so any core-feasible mix of workers\n stays within the budget.\n- Otherwise the profile\u0027s real limiting resource is used, with custom-resource bounds capped\n by explicit profile cores when the cores limit is unknown (SPARK-30299).\n- A *positive* allocation whose per-worker share rounds to 0 MiB fails fast; `0` still means\n \"no limit\".\n\n**Scheduler.**\n\n- Each stage snapshots the job `Properties` before profile-specific values are written, so\n sibling stages with different profiles cannot clobber each other.\n- Null job properties are materialized as an empty map.\n\n**Executor / runtime.**\n\n- Python workers derive `OMP_NUM_THREADS` and the integer worker `cpus` from the task\u0027s own\n cpu amount (stage-level profiles honored) unless `spark.executorEnv.OMP_NUM_THREADS` is set.\n- `executorRunTime` subtracts in-run deserialization before weighting by the cpus amount, with\n the weight floored at 1 so sub-core tasks do not inflate UI/REST scheduler delay.\n- Script-transform child processes get `OMP_NUM_THREADS \u003d ceil(spark.task.cpus)`.\n- The pipelined Python UDF writer pool is unbounded: task concurrency is its natural cap, and\n a processor-count cap can deadlock barrier stages once tasks outnumber cores.\n\n**Kubernetes.**\n\n- Allocation recovery mode announces `ceil(spark.task.cpus)` cores (global setting only).\n- With `spark.task.cpus \u003c\u003d 0.5` a recovery executor accepts `floor(1 / spark.task.cpus)`\n tasks instead of one; a one-time warning and docs cover this.\n\n**Docs / misc.**\n\n- `core-migration-guide.md` entries for the PySpark memory behavior changes;\n `configuration.md` and `running-on-kubernetes.md` updated.\n- `cpuAmount` added to the PySpark API reference.\n- MiMa excludes in `v43excludes` to match `Since(\"4.3.0\")`.\n\n**Out of scope** (pre-existing, left to follow-ups):\n\n- K8s recovery-mode cores from stage-level profiles.\n- Stage-profile-aware script-transform `OMP_NUM_THREADS`.\n- Using the actual registered executor cores for the memory split in standalone (SPARK-30299).\n- `shouldCheckExecutorCores` ignoring profile-scoped cores.\n- Budgeting multiple concurrently active Python workers within one task (e.g. chained\n `mapInPandas`).\n\n### Why are the changes needed?\n\n`spark.task.cpus` only accepted whole numbers, so the finest scheduling granularity was one\ntask per core. Lightweight tasks (I/O-bound, or PySpark tasks whose heavy lifting happens in\na shared native library or external service) cannot use an executor efficiently, and tasks\nneeding e.g. `1.5` cores had to round up to 2 and waste capacity:\n\n```bash\n# 8 concurrent tasks on a 4-core executor\nspark-submit --conf spark.executor.cores\u003d4 --conf spark.task.cpus\u003d0.5 ...\n\n# 2 concurrent tasks on a 4-core executor: floor(4 / 1.5) \u003d 2\nspark-submit --conf spark.executor.cores\u003d4 --conf spark.task.cpus\u003d1.5 ...\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes.\n\n- `spark.task.cpus` and `TaskResourceRequests.cpus` accept fractional values; whole-number\n configurations schedule exactly as before. Out-of-range or over-precise values are rejected\n with a clear error.\n- New API `TaskContext.cpuAmount()` (Scala and Python); `TaskContext.cpus()` is deprecated.\n- In K8s allocation recovery mode with `spark.task.cpus \u003c\u003d 0.5`, recovery executors accept\n more than one task (warned and documented).\n\nFixing places that equated \"executor cores\" with \"concurrent tasks\" also changes behavior for\nsome existing whole-number workloads (covered by `core-migration-guide.md` entries):\n\n- Per-worker PySpark memory follows real concurrency: larger shares for `spark.task.cpus \u003e 1`\n and (with dynamic allocation on) for custom-resource-limited profiles; cpus-proportional\n shares on shareable executors. Aggregate limits stay within the executor-wide allocation.\n- Profiles without an explicit `pysparkMemory` now always carry the default limit (previously\n order-dependent).\n- An unsatisfiable positive allocation now fails fast instead of silently dropping the limit\n (`0` still disables it).\n- `executorRunTime` no longer scales the deserialization interval for `cpus \u003e 1`.\n- `OMP_NUM_THREADS` honors stage-level profiles.\n- Stages carry per-stage snapshots of the job `Properties` instead of sharing one mutable\n instance.\n\n### How was this patch tested?\n\nNew and updated unit tests:\n\n- `SparkContextSuite`: config bounds/precision matrix.\n- `ResourceProfileSuite` / `ResourceProfileManagerSuite`: fractional profiles, slot math,\n entry-point and registration validation with registry-unchanged assertions.\n- `TaskSchedulerImplSuite`: fractional scheduling end-to-end, slot saturation.\n- `TaskDescriptionSuite`: exact fractional encode/decode round-trip.\n- `DAGSchedulerSuite`: per-stage property isolation, null job properties, memory-split\n propagation across profile shapes (GPU-limited, task-only, ceiling division, capped custom\n bounds).\n- `ExecutorSuite`: cpu-weighted runtime flooring.\n- `AppStatusListenerSuite` / `JsonProtocolSuite` / `KVStoreProtobufSerializerSuite`:\n fractional parsing, hostile replayed values, legacy-profile fallbacks.\n- `BasePythonRunnerSuite`: worker memory split, fail-fast boundaries, writer-pool concurrency.\n- `BasicExecutorFeatureStepSuite` / `ExecutorPodsAllocatorSuite`: K8s recovery-mode ceiling\n and one-time warning.\n- PySpark `test_taskcontext.py` / `test_worker.py` / `test_resources.py`: `cpuAmount`, worker\n env, request round-trips.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Code (Claude Fable 5)\n\nCloses #57332 from pan3793/SPARK-58192.\n\nAuthored-by: Cheng Pan \u003cpan3793@gmail.com\u003e\nSigned-off-by: Cheng Pan \u003cchengpan@apache.org\u003e\n" }, { "commit": "fd3169438dbdada4623d310380b7c300543833af", "tree": "596f4798e4dedf871478b990e097c274c1702cf0", "parents": [ "ced139fdfcd49331ad04b2bc6bb89352118bc67c" ], "author": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Sun Jul 26 10:44:25 2026 +0000" }, "committer": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Sun Jul 26 10:44:25 2026 +0000" }, "message": "[MINOR][SS] Use filterNot in HDFSBackedStateStoreProvider snapshot filtering\n\n### What changes were proposed in this pull request?\nReplaces `filter(_.isSnapshot \u003d\u003d false)` with the equivalent, idiomatic `filterNot(_.isSnapshot)` in `HDFSBackedStateStoreProvider.doSnapshot`.\n\n### Why are the changes needed?\n`StoreFile.isSnapshot` is a plain `Boolean` field, so comparing it to `false` is redundant. `filterNot` is the idiomatic form used throughout Spark and reads more clearly.\n\n### Does this PR introduce _any_ user-facing change?\nNo.\n\n### How was this patch tested?\nBehavior-preserving one-line change; existing state-store tests cover the path. No new tests needed.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57519 from uros-b/cleanup-hdfs-state-filternot.\n\nLead-authored-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\nCo-authored-by: Uros Bojanic \u003curos.bojanic@databricks.com\u003e\nSigned-off-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\n" }, { "commit": "ced139fdfcd49331ad04b2bc6bb89352118bc67c", "tree": "6759306eca0770cc9addaebb92d223a97134514f", "parents": [ "d2693b5baf5cd7c449a521e93564fac4b296a167" ], "author": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Sun Jul 26 10:43:06 2026 +0000" }, "committer": { "name": "Uros Bojanic", "email": "221401595+uros-b@users.noreply.github.com", "time": "Sun Jul 26 10:43:06 2026 +0000" }, "message": "[SPARK-58343][SQL][TEST] Add unit test for TypeUtils.typeWithProperEquals\n\n### What changes were proposed in this pull request?\nAdds a focused unit test for `TypeUtils.typeWithProperEquals` in `TypeUtilsSuite`. The predicate was previously only referenced in a code comment, never directly asserted. The test covers atomic types (true), UTF8_BINARY string (true), a collated string (false), binary type (false), and complex types (false).\n\n### Why are the changes needed?\n`typeWithProperEquals` gates several expressions (higher-order functions, complex-type extractors, collection ops, PivotFirst) but had no direct coverage. The added assertions pin its contract, including the special-cased BinaryType and collation branches.\n\n### Does this PR introduce _any_ user-facing change?\nNo.\n\n### How was this patch tested?\nThis PR is the test. `build/sbt \"catalyst/testOnly *TypeUtilsSuite\"`.\n\n### Was this patch authored or co-authored using generative AI tooling?\nGenerated-by: Claude Code (Opus 4.8)\n\nCloses #57518 from uros-b/test-typeutils-properequals.\n\nLead-authored-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\nCo-authored-by: Uros Bojanic \u003curos.bojanic@databricks.com\u003e\nSigned-off-by: Uros Bojanic \u003c221401595+uros-b@users.noreply.github.com\u003e\n" }, { "commit": "d2693b5baf5cd7c449a521e93564fac4b296a167", "tree": "b9b28fed991e327a7da788e56d0c8ab22c837b6b", "parents": [ "b910d6f4101c53dd5f9b866122d92e4c41c768f3" ], "author": { "name": "naveenp2708", "email": "naveenp2708@gmail.com", "time": "Sun Jul 26 10:16:06 2026 +0200" }, "committer": { "name": "Peter Toth", "email": "peter.toth@gmail.com", "time": "Sun Jul 26 10:16:06 2026 +0200" }, "message": "[SPARK-52246][SQL][TEST] Add bucket transform regression test for one-side shuffle with join key tail of partition keys\n\n### What changes were proposed in this pull request?\n\nAdd a regression test to `KeyGroupedPartitioningSuite` for the scenario reported in [SPARK-52246](https://issues.apache.org/jira/browse/SPARK-52246) (test adapted from the reproduction in the JIRA description): a storage-partitioned join where one side is auto-shuffled, the partitioned side uses a bucket transform, and the partition key is the tail of the join keys.\n\n### Why are the changes needed?\n\nSPARK-52246 reported silent incorrect results (empty join output) for this scenario on 4.0.0/4.0.1. The defect was fixed by SPARK-54439, but the tests added there cover identity and years transforms only, not bucket. This locks in coverage for the bucket case.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nNew test passes on master; verified it fails on v4.0.1 (pre-SPARK-54439), reproducing the original wrong results.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nNo.\n\nCloses #57442 from naveenp2708/repro-SPARK-52246.\n\nAuthored-by: naveenp2708 \u003cnaveenp2708@gmail.com\u003e\nSigned-off-by: Peter Toth \u003cpeter.toth@gmail.com\u003e\n" }, { "commit": "b910d6f4101c53dd5f9b866122d92e4c41c768f3", "tree": "a4dec7c6510db4b14595ce81b672825b6c6e239a", "parents": [ "cce435599ed332cd5b785d49b26d4e7dde631d8c" ], "author": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Sat Jul 25 12:39:30 2026 -0700" }, "committer": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Sat Jul 25 12:39:30 2026 -0700" }, "message": "[SPARK-58341][SQL] Fix wrong results with 5+ positional parameters in `sql`\n\n### What changes were proposed in this pull request?\n\nThis PR fixes a correctness issue where `spark.sql` binds 5 or more positional parameters in the wrong order. The resolved parameters were collected via `resolvedParams.values.toSeq` from a `Map` keyed by `_pos_\u003cidx\u003e`, but Scala\u0027s immutable `Map` does not preserve insertion order for 5+ entries. This PR looks them up by the positional key instead, in both affected places:\n\n- `classic.SparkSession.sql(sqlText, args: Array[_], tracker)`\n- `SparkConnectPlanner.buildParameterContext` (`pos_arguments`)\n\n### Why are the changes needed?\n\nSince Apache Spark 4.1.0 (SPARK-53573), queries with 5+ positional parameters\nsilently return wrong results in both the classic API and Spark Connect.\n\n- https://github.com/apache/spark/pull/52334\n\n**BEFORE (Spark 4.2.0)**\n```\n$ bin/spark-shell\nWARNING: Using incubator modules: jdk.incubator.vector\nUsing Spark\u0027s default log4j profile: org/apache/spark/log4j2-defaults.properties\nSetting default log level to \"WARN\".\nTo adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\nWelcome to\n ____ __\n / __/__ ___ _____/ /__\n _\\ \\/ _ \\/ _ `/ __/ \u0027_/\n /___/ .__/\\_,_/_/ /_/\\_\\ version 4.2.0\n /_/\n\nUsing Scala version 2.13.18 (OpenJDK 64-Bit Server VM, Java 21.0.12)\nType in expressions to have them evaluated.\nType :help for more information.\n26/07/24 15:46:06 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\nSpark context Web UI available at http://localhost:4040\nSpark context available as \u0027sc\u0027 (master \u003d local[*], app id \u003d local-1784933166639).\nSpark session available as \u0027spark\u0027.\n\nscala\u003e spark.sql(\"SELECT ?, ?, ?, ?, ?, ?\", Array(1, 2, 3, 4, 5, 6)).collect()\nval res0: Array[org.apache.spark.sql.Row] \u003d Array([6,2,5,3,4,1])\n```\n\n**AFTER (this PR)**\n```\n$ bin/spark-shell\nWARNING: Using incubator modules: jdk.incubator.vector\nUsing Spark\u0027s default log4j profile: org/apache/spark/log4j2-defaults.properties\nSetting default log level to \"WARN\".\nTo adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\nWelcome to\n ____ __\n / __/__ ___ _____/ /__\n _\\ \\/ _ \\/ _ `/ __/ \u0027_/\n /___/ .__/\\_,_/_/ /_/\\_\\ version 5.0.0-SNAPSHOT\n /_/\n\nUsing Scala version 2.13.18 (OpenJDK 64-Bit Server VM, Java 21.0.12)\nType in expressions to have them evaluated.\nType :help for more information.\n26/07/24 15:48:50 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\nSpark context Web UI available at http://localhost:4040\nSpark context available as \u0027sc\u0027 (master \u003d local[*], app id \u003d local-1784933331484).\nSpark session available as \u0027spark\u0027.\n\nscala\u003e spark.sql(\"SELECT ?, ?, ?, ?, ?, ?\", Array(1, 2, 3, 4, 5, 6)).collect()\nval res0: Array[org.apache.spark.sql.Row] \u003d Array([1,2,3,4,5,6])\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, this is a correctness bug fix of the released versions 4.1.0, 4.1.1, 4.1.2, 4.1.3 and 4.2.0.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test case.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #57516 from dongjoon-hyun/SPARK-58341.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n" }, { "commit": "cce435599ed332cd5b785d49b26d4e7dde631d8c", "tree": "5ec08dc6d8599189df15a0321d644df24c96d451", "parents": [ "faa1d1ecad4c64dc528ee83be6eb19bb5291c439" ], "author": { "name": "Peter Toth", "email": "peter.toth@gmail.com", "time": "Sat Jul 25 11:49:53 2026 +0200" }, "committer": { "name": "Peter Toth", "email": "peter.toth@gmail.com", "time": "Sat Jul 25 11:49:53 2026 +0200" }, "message": "[SPARK-58324][SQL] Drop unused sameOrderExpressions from GroupPartitionsExec k-way merge ordering\n\n### What changes were proposed in this pull request?\n\n`GroupPartitionsExec` builds a `SortedMergeCoalescedRDD` for the k-way merge and hands it a `LazyCodeGenOrdering` built from `child.outputOrdering`. The generated comparator (`GenerateOrdering`) only needs each `SortOrder`\u0027s sort key (child, direction, null ordering), so this drops `sameOrderExpressions` -- planner-only metadata -- via a small `kWayMergeOrdering` helper before constructing the ordering, so it is not serialized with the RDD in every task.\n\n### Why are the changes needed?\n\n`sameOrderExpressions` is unused by the merge comparator and is unnecessary payload serialized with every task. It was also the vector for the `StackOverflowError` fixed in SPARK-58323 (an unforced, deeply-nested `LazyList`); not carrying it here removes this operator\u0027s exposure to any such ordering entirely (defense-in-depth), independent of that fix.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nNew unit test in `GroupPartitionsExecSuite` asserting `kWayMergeOrdering` keeps the sort key but drops `sameOrderExpressions`. Existing SPARK-55715 sorted-merge tests cover comparator correctness.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 4.8\n\nCloses #57503 from peter-toth/SPARK-58324-drop-sameorderexpressions-kway-merge.\n\nAuthored-by: Peter Toth \u003cpeter.toth@gmail.com\u003e\nSigned-off-by: Peter Toth \u003cpeter.toth@gmail.com\u003e\n" }, { "commit": "faa1d1ecad4c64dc528ee83be6eb19bb5291c439", "tree": "be816c349ea339049a2114d2cb24a3593b5c04fc", "parents": [ "c784ac6d6c362a937fc4313b1e39aaea22b7dee4" ], "author": { "name": "Peter Toth", "email": "peter.toth@gmail.com", "time": "Sat Jul 25 11:22:36 2026 +0200" }, "committer": { "name": "Peter Toth", "email": "peter.toth@gmail.com", "time": "Sat Jul 25 11:22:36 2026 +0200" }, "message": "[SPARK-58323][SQL] Materialize AliasAware output ordering and partitioning to avoid StackOverflowError\n\n### What changes were proposed in this pull request?\n\n`AliasAwareQueryOutputOrdering.outputOrdering` and `PartitioningPreservingUnaryExecNode.outputPartitioning` (and `BroadcastHashJoinExec`\u0027s output-partitioning expansion) build their results with `multiTransform`, which returns a `LazyList`, and store the bounded result unforced. This forces them into strict collections with `.toList`, right after the existing `.take(...)` so the lazy short-circuit during candidate generation is preserved.\n\nNote: for `BroadcastHashJoinExec`, using a strict `List` also makes the existing `case p :: Nil \u003d\u003e p` unwrap in `outputPartitioning` reachable — so a single-element expansion now returns the `HashPartitioning` directly instead of a one-element `PartitioningCollection` (that branch was previously dead because the expansion was an unforced `LazyList`, which a `List`-cons pattern never matches). This is the cleaner, intended form; the corresponding `BroadcastJoinSuite` assertion is updated accordingly.\n\n### Why are the changes needed?\n\nEach plan node re-wraps the child ordering/partitioning\u0027s unforced `LazyList`, so across a deep projection chain the deferred nesting grows unbounded and overflows the stack when the ordering/partitioning is later serialized or deeply traversed. Concretely, task serialization of a `SortedMergeCoalescedRDD` (SPARK-55715) that captures the child `outputOrdering` was observed to fail on the driver with a `StackOverflowError` at `DAGScheduler.submitMissingTasks` on a large bucketed `MERGE`. The lazy `LazyList` in `SortOrder.sameOrderExpressions` (and in the projected `PartitioningCollection`) is the root cause; this has existed since the `multiTransform` path was introduced in SPARK-42049.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nNew test in `ProjectedOrderingAndPartitioningSuite` asserting `sameOrderExpressions` and the projected `PartitioningCollection.partitionings` are strict (not `LazyList`); it fails before this change (they were `LazyList`) and passes after. `BroadcastJoinSuite` / `BroadcastJoinSuiteAE` updated for the single-element output-partitioning form and pass. Existing `ProjectedOrderingAndPartitioningSuite` / `PlannerSuite` / `EnsureRequirementsSuite` pass.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 4.8\n\nCloses #57502 from peter-toth/SPARK-58323-aliasaware-strict-ordering-partitioning.\n\nAuthored-by: Peter Toth \u003cpeter.toth@gmail.com\u003e\nSigned-off-by: Peter Toth \u003cpeter.toth@gmail.com\u003e\n" }, { "commit": "c784ac6d6c362a937fc4313b1e39aaea22b7dee4", "tree": "8913108d5205ea78b7ad2aa3f3f49102b2720335", "parents": [ "6f1c904686cb54e52c2282882fab898a125a7eac" ], "author": { "name": "Andreas Neumann", "email": "andreas.neumann@databricks.com", "time": "Fri Jul 24 21:03:02 2026 -0700" }, "committer": { "name": "Jose Torres", "email": "jtorres@apache.org", "time": "Fri Jul 24 21:03:02 2026 -0700" }, "message": "[SPARK-58342][SDP] Add unit tests for the SCD1 AutoCDC auxiliary table spec\n\n### What changes were proposed in this pull request?\n\n`AutoCdcAuxiliaryTable.buildScd1AuxiliaryTableSpecFor` had no focused unit test; its behavior was only exercised indirectly via higher-level pipeline suites. [SPARK-58320](https://issues.apache.org/jira/browse/SPARK-58320) added `AutoCdcScd2AuxiliaryTableSpecSuite` for the SCD2 spec builder; this adds the parallel SCD1 coverage.\n\n`AutoCdcScd1AuxiliaryTableSpecSuite` resolves a real SCD1 AutoCDC graph and asserts the derived auxiliary-table spec:\n - the schema is exactly the key columns plus the CDC metadata column -- no user data columns and no SCD2 framework columns, since the SCD1 auxiliary table stores per-key tombstones rather than full rows;\n - the CDC metadata column uses the SCD1 metadata struct schema and is non-null;\n - the spec records `ScdType.Type1`, the key column names (property + `expectedKeyFields`), and the correct identifiers;\n - composite keys are all carried through.\n\n### Why are the changes needed?\n\nFills a test-coverage gap for the SCD1 auxiliary-table spec derivation, matching the coverage that now exists for SCD2, so future changes to `buildScd1AuxiliaryTableSpecFor` (schema shape, recorded drift metadata) are guarded by a focused unit test.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. Test-only change; no production code is modified.\n\n### How was this patch tested?\n\n- `build/sbt \u0027pipelines/testOnly *AutoCdcScd1AuxiliaryTableSpecSuite\u0027` -- 5/5 pass\n- `dev/lint-scala` -- Scalastyle and Scalafmt clean\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Opus 4.8\n\nCloses #57517 from anew/spark-scd1-aux-spec-tests.\n\nAuthored-by: Andreas Neumann \u003candreas.neumann@databricks.com\u003e\nSigned-off-by: Jose Torres \u003cjtorres@apache.org\u003e\n" }, { "commit": "6f1c904686cb54e52c2282882fab898a125a7eac", "tree": "f6f7f6fee05d067fa38a8b9015439fe82874c863", "parents": [ "7865fb840daf027ca49fed81d894e6ebbb97b18a" ], "author": { "name": "Chao Sun", "email": "sunchao@apache.org", "time": "Fri Jul 24 19:57:57 2026 -0700" }, "committer": { "name": "Chao Sun", "email": "chao@openai.com", "time": "Fri Jul 24 19:57:57 2026 -0700" }, "message": "[SPARK-58310][SQL] Avoid runtime Bloom-filter subqueries containing Python UDFs\n\n### Why are the changes needed?\n\nWith adaptive execution and runtime Bloom filters enabled, a nested broadcast join can cause `InjectRuntimeFilter` to copy a creation-side plan containing a Python UDF into a newly constructed Bloom-filter scalar subquery. Runtime filters are introduced after subquery optimization, so the copied Python UDF never passes through Python-UDF extraction. The physical plan subsequently attempts to generate JVM code for the unevaluable Python expression while executing `BloomFilterAggregate`, causing an otherwise valid query to fail through `SubqueryExec` and `ObjectHashAggregateExec`.\n\nThe problem affects ordinary Apache Spark and does not depend on an external table provider. A self-contained PySpark setup is:\n\n```python\nfrom pyspark.sql.functions import pandas_udf, udf\n\nudf(\"string\")\ndef normalize_runtime_bloom_url(value):\n return value\n\npandas_udf(\"string\")\ndef normalize_runtime_bloom_pandas_url(values):\n return values\n\nspark.udf.register(\"normalize_runtime_bloom_url\", normalize_runtime_bloom_url)\nspark.udf.register(\n \"normalize_runtime_bloom_pandas_url\", normalize_runtime_bloom_pandas_url\n)\n\nspark.range(20000).selectExpr(\n \"cast(id as int) as id\",\n \"concat(\u0027url-\u0027, cast(id % 20 as string)) as url\",\n).write.mode(\"overwrite\").parquet(\"/tmp/spark-58310-fact\")\n\nspark.range(20).selectExpr(\n \"concat(\u0027url-\u0027, cast(id as string)) as url\",\n \"case when id \u003c 10 then \u0027NL\u0027 else \u0027US\u0027 end as country\",\n).write.mode(\"overwrite\").parquet(\"/tmp/spark-58310-dimension\")\n\nspark.read.parquet(\"/tmp/spark-58310-fact\").createOrReplaceTempView(\n \"runtime_bloom_fact\"\n)\nspark.read.parquet(\"/tmp/spark-58310-dimension\").createOrReplaceTempView(\n \"runtime_bloom_dimension\"\n)\n\nspark.conf.set(\"spark.sql.adaptive.enabled\", \"true\")\nspark.conf.set(\"spark.sql.optimizer.dynamicPartitionPruning.enabled\", \"false\")\nspark.conf.set(\"spark.sql.optimizer.runtime.bloomFilter.enabled\", \"true\")\nspark.conf.set(\n \"spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold\", \"0\"\n)\nspark.conf.set(\"spark.sql.optimizer.runtime.bloomFilter.creationSideThreshold\", \"100MB\")\nspark.conf.set(\"spark.sql.autoBroadcastJoinThreshold\", \"-1\")\n```\n\nThe following query reproduces the failure; substituting `normalize_runtime_bloom_pandas_url` reproduces the Arrow/Pandas variant:\n\n```sql\nWITH coverage AS (\n SELECT url, normalize_runtime_bloom_url(url) AS normalized_url\n FROM runtime_bloom_dimension\n WHERE country \u003d \u0027NL\u0027\n),\nnormalized_keys AS (\n SELECT DISTINCT normalized_url\n FROM coverage\n WHERE normalized_url IS NOT NULL\n),\nmatched_urls AS (\n SELECT /*+ BROADCAST(k) */ fact.url\n FROM runtime_bloom_fact fact\n JOIN normalized_keys k ON fact.url \u003d k.normalized_url\n),\nsnapshot AS (\n SELECT max(url) AS snapshot_url\n FROM runtime_bloom_fact\n)\nSELECT /*+ BROADCAST(m), BROADCAST(snapshot) */\n coverage.url,\n coverage.normalized_url,\n matched_urls.url AS matched_url,\n snapshot.snapshot_url\nFROM coverage\nLEFT JOIN matched_urls ON coverage.normalized_url \u003d matched_urls.url\nCROSS JOIN snapshot\n```\n\n### What changes were proposed in this PR?\n\nAfter constructing and column-pruning the runtime Bloom-filter aggregate, return the unchanged application-side plan when the pruned aggregate still contains the `PYTHON_UDF` tree pattern. Checking after column pruning preserves safe Bloom filters when a Python UDF is needed only by the outer query output and is absent from the actual Bloom-filter scalar subquery. This guard is deliberately limited to unsafe Bloom-filter creation. It does not disable runtime filtering, does not reject a Python UDF on the application side, and does not change Python-UDF extraction or adaptive execution.\n\nAdd `SPARK-58310` coverage to `InjectRuntimeFilterSuite` that:\n\n- Runs both standard scalar Python and Arrow/Pandas scalar UDFs when available.\n- Reproduces nested CTEs, explicit broadcast joins, scalar aggregates, and adaptive execution with dynamic partition pruning disabled.\n- Compares the complete result with the Bloom-disabled baseline and verifies a real Parquet write/read round trip.\n- Proves that safe application-side and pruned creation-side runtime Bloom filters are still injected and produce the same results.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. Queries that previously failed because a runtime Bloom-filter subquery attempted to code-generate a Python UDF now complete successfully. Safe runtime Bloom-filter optimizations remain enabled.\n\n### How was this PR tested?\n\nFirst, the new `SPARK-58310` regression was run against unmodified Apache Spark `master` and failed with the expected Python-UDF scalar-subquery/`ObjectHashAggregateExec` error. The same regression then passed after the creation-side guard was applied, with both `pandas` and `pyarrow` installed.\n\nThe following focused upstream regression suites and Scala-style checks were also run on the public Apache Spark branch:\n\n```bash\nbuild/sbt \\\n \u0027sql/testOnly org.apache.spark.sql.InjectRuntimeFilterSuite org.apache.spark.sql.DynamicPartitionPruningV1SuiteAEOn org.apache.spark.sql.SubquerySuite org.apache.spark.sql.execution.adaptive.AdaptiveQueryExecSuite org.apache.spark.sql.execution.python.ExtractPythonUDFsSuite\u0027 \\\n \u0027catalyst/scalastyle\u0027 \\\n \u0027sql/scalastyle\u0027 \\\n \u0027sql/Test/scalastyle\u0027\n```\n\n### How was this patch tested?\n\nThe upstream before/after reproduction, Python and Arrow regression, adjacent SQL suites, and Scala-style checks are described above.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nYes. Generated-by: OpenAI Codex (GPT-5).\n\nCloses #57485 from sunchao/dev/chao/codex/python-udf-runtime-bloom-filter.\n\nAuthored-by: Chao Sun \u003csunchao@apache.org\u003e\nSigned-off-by: Chao Sun \u003cchao@openai.com\u003e\n" }, { "commit": "7865fb840daf027ca49fed81d894e6ebbb97b18a", "tree": "6bddb56d4dec2fee87b6ddee376e0486a6feb988", "parents": [ "0700340fa6fc54f1f625b49503c205eab44394eb" ], "author": { "name": "Szehon Ho", "email": "szehon.apache@gmail.com", "time": "Fri Jul 24 18:48:33 2026 -0700" }, "committer": { "name": "Szehon Ho", "email": "szehon.apache@gmail.com", "time": "Fri Jul 24 18:48:33 2026 -0700" }, "message": "[SPARK-58311][SQL] Gate generated column values on write with a table capability\n\n### What changes were proposed in this pull request?\n\nThis PR moves the gate that decides whether Spark auto-fills and enforces generated column values on a DSv2 write from a **catalog-level** capability to a **table-level** capability.\n\nSpecifically:\n- Adds `TableCapability.GENERATE_COLUMN_VALUES_ON_WRITE`, checked via `Table.capabilities()`.\n- Removes `TableCatalogCapability.SUPPORT_GENERATED_COLUMN_ON_WRITE` (added by SPARK-57644 and still unreleased).\n- `GeneratedColumn.supportsGeneratedColumnsOnWrite` now takes a `Table` and checks the table\u0027s capabilities instead of the `TableCatalog`\u0027s.\n- Updates the four call sites to consult the table:\n - `Analyzer` (exposing generation-expression metadata to `TableOutputResolver` so missing values are auto-filled),\n - `ResolveTableConstraints` (adding the generated-column `CheckInvariant` constraints),\n - `RewriteRowLevelCommand` (blocking MERGE/UPDATE),\n - `ResolveWriteToStream` (blocking streaming writes).\n- Test infra: the in-memory test table advertises the new capability, gated by a `generate-column-values-on-write` table property (default `true`), mirroring the existing `accept-any-schema` / `auto-schema-evolution` toggles.\n\n### Why are the changes needed?\n\nWhether Spark should generate/enforce generated column values is a property of an individual **table**, not of the whole **catalog**. A single catalog can expose tables from formats or protocol versions with differing generated-column support, so a catalog-wide flag is too coarse.\n\nThis also aligns with how Delta Lake gates the feature: it is gated per-table on the table\u0027s protocol. `GeneratedColumn.satisfyGeneratedColumnProtocol(protocol)` returns `protocol.isFeatureSupported(GeneratedColumnsTableFeature)` -- i.e. it checks whether the `GeneratedColumnsTableFeature` is enabled in that specific table\u0027s protocol. A per-table `TableCapability` is the natural DSv2 analogue and matches the existing pattern for other per-table write behaviors (`BATCH_WRITE`, `OVERWRITE_DYNAMIC`, `TRUNCATE`, ...).\n\n### Does this PR introduce _any_ user-facing change?\n\nNo change relative to a released version. The catalog capability being removed (`SUPPORT_GENERATED_COLUMN_ON_WRITE`) was introduced by SPARK-57644 and has not shipped in any release, so this only reshapes an unreleased API within the development branch. Connectors opt in by having their `Table` advertise `TableCapability.GENERATE_COLUMN_VALUES_ON_WRITE`. Auto-filling missing generated column values applies to by-name writes; ordinary by-position writes must still provide a value for every table column.\n\n### How was this patch tested?\n\n`GeneratedColumnWriteSuite` (69 tests) passes. The existing \"connector without the write capability does not auto-fill or enforce generated columns\" coverage is retained (renamed to \"table without write capability ...\"), now driven by creating a table with `TBLPROPERTIES (\u0027generate-column-values-on-write\u0027 \u003d \u0027false\u0027)` instead of a dedicated no-capability catalog.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Cursor (Opus 4.8)\n\nCloses #57481 from szehon-ho/dsv2-generated-column-gate.\n\nAuthored-by: Szehon Ho \u003cszehon.apache@gmail.com\u003e\nSigned-off-by: Szehon Ho \u003cszehon.apache@gmail.com\u003e\n" }, { "commit": "0700340fa6fc54f1f625b49503c205eab44394eb", "tree": "ea224097a264c065d5a344f091e8db3f93c648b2", "parents": [ "4a5bb9502acc0636e1748fa7e1b8758c851c5e73" ], "author": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Fri Jul 24 17:30:46 2026 -0700" }, "committer": { "name": "Dongjoon Hyun", "email": "dongjoon@apache.org", "time": "Fri Jul 24 17:30:46 2026 -0700" }, "message": "[SPARK-58334][K8S][DOC] Introduce Apache Spark K8s Operator in `running-on-kubernetes.md`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to introduce `Apache Spark Kubernetes Operator` in `running-on-kubernetes.md` by adding a short section after `How it works` section.\n\n- What the operator is: a separate Apache Spark project that deploys and manages Spark\n workloads declaratively via `SparkApp` and `SparkCluster` custom resources.\n- Its relationship with the `spark-submit` based submission covered by this document:\n the two approaches are complementary and the operator runs on top of the same native\n Kubernetes scheduler backend.\n- Links to the operator repository and examples, consistent with the existing links in\n `docs/index.md`.\n\n### Why are the changes needed?\n\n`Apache Spark Kubernetes Operator` v1.0.0 is going to be released Tomorrow. Currently, it is only mentioned in `docs/index.md`. Users who land directly on `running-on-kubernetes.md`, the main documentation page for running Spark on Kubernetes, have no way to discover the operator. This closes that discoverability gap while keeping the detailed operator usage out of scope (deferred to the operator repository).\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is a documentation-only change.\n\n### How was this patch tested?\n\nManually check.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #57512 from dongjoon-hyun/SPARK-58334.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n" } ], "next": "4a5bb9502acc0636e1748fa7e1b8758c851c5e73" }