)]}'
{
  "log": [
    {
      "commit": "8f5014348186d37529f1e577b71a2a313f051edc",
      "tree": "44b13f75fce4f391873332225444eb0968c6aeab",
      "parents": [
        "ada42aea8b97ed9152fa212f911665c37410dbe2"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Nov 09 22:43:32 2026 -0800"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Nov 09 22:43:32 2026 -0800"
      },
      "message": "[SPARK-61339] Upgrade gRPC Swift NIO Transport to 2.12.0\n\n### What changes were proposed in this pull request?\n\nThis PR upgrades `grpc-swift-nio-transport` from 2.11.0 to 2.12.0.\n\n- `Package.swift`: bump the `exact` pin of `grpc-swift-nio-transport` to `2.12.0`.\n- `README.md`: update the listed dependency version to [2.12.0 (November 2026)](https://github.com/grpc/grpc-swift-nio-transport/releases/tag/2.12.0).\n\n### Why are the changes needed?\n\nTo use the latest release of the dependency. 2.12.0 was released on 2026-11-02.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs. `swift build -c release` was also verified locally.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Sonnet 5.5\n\nCloses #626 from dongjoon-hyun/SPARK-61339.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "ada42aea8b97ed9152fa212f911665c37410dbe2",
      "tree": "e7512e27afee759480c3e939c35da131b1730df2",
      "parents": [
        "f7557d74b6be547a414c2c82d0178b5d044189d5"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Nov 09 22:42:46 2026 -0800"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Nov 09 22:42:46 2026 -0800"
      },
      "message": "[SPARK-61338] Use Apache Spark 4.0.5, 4.1.4, and 4.2.1 Docker images in CI\n\n### What changes were proposed in this pull request?\n\nThis PR aims to use the latest maintenance versions of the Apache Spark Docker images in CI.\n\n- `apache/spark:4.0.4-java21-scala` -\u003e `apache/spark:4.0.5-java21-scala`\n- `apache/spark:4.1.3-scala` -\u003e `apache/spark:4.1.4-scala`\n- `apache/spark:4.2.0-scala-java25` -\u003e `apache/spark:4.2.1-scala-java25`\n\n### Why are the changes needed?\n\nTo test against the latest maintenance releases of each Apache Spark line, consistent with the binary distributions (4.0.5, 4.1.4, 4.2.1) already used elsewhere in CI.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs. Verified that all three image tags exist on Docker Hub.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Sonnet 5.5\n\nCloses #625 from dongjoon-hyun/SPARK-61338.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "f7557d74b6be547a414c2c82d0178b5d044189d5",
      "tree": "0a83bff0d0485605afd7efd56f0b419cf7eab765",
      "parents": [
        "7ac060eb652c47aff9e14e2567c73e01f5ba72bb"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Nov 09 21:06:40 2026 -0800"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Nov 09 21:06:40 2026 -0800"
      },
      "message": "[SPARK-61330] Use Apache Spark 4.3.0 official release and Docker image in CI and docs\n\n### What changes were proposed in this pull request?\n\nThis PR updates CI and documentation to use the official Apache Spark 4.3.0 release.\n\n- `integration-test-mac-spark43`: remove the RC1 `url:` input so that `setup-spark` falls back to the default ASF mirror, and update `sha512:` to the checksum of the official tarball (verified against both `downloads.apache.org` and `dlcdn.apache.org`).\n- Add `integration-test-ubuntu-spark43`, which runs the integration tests against the `apache/spark:4.3.0-scala-java25` Docker image.\n- Update `README.md`, `Examples/*/README.md`, `Documentation.docc/Examples.md`, and `AGENTS.md` to use `apache/spark:4.3.0` and list 4.3.0 as a tested version.\n\n### Why are the changes needed?\n\nApache Spark 4.3.0 has been released, so CI should test against the official release and Docker image instead of RC1, and the documentation should reference the latest release.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. Only CI and documentation are changed.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Sonnet 5.5\n\nCloses #624 from dongjoon-hyun/SPARK-61330.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "7ac060eb652c47aff9e14e2567c73e01f5ba72bb",
      "tree": "6543f4127d43ea07ddbd7e14a4083e575d456a20",
      "parents": [
        "a8d5ff58d657dd08621801a44acadac247e18143"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 28 10:27:22 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 28 10:27:22 2026 -0700"
      },
      "message": "[SPARK-60619] Use Apache Spark 4.0.5, 4.1.4, and 4.2.1 instead of RCs\n\n### What changes were proposed in this pull request?\n\nThis PR updates the `integration-test-mac`, `integration-test-mac-spark41`, and `integration-test-mac-spark42` CI jobs to use the official Apache Spark releases instead of RC1, with the updated SHA-512 checksums.\n\n| Job | Version | Before | After |\n|---|---|---|---|\n| `integration-test-mac` | 4.0.5 | RC1 (`dist.apache.org/repos/dist/dev`) | official release |\n| `integration-test-mac-spark41` | 4.1.4 | RC1 (`dist.apache.org/repos/dist/dev`) | official release |\n| `integration-test-mac-spark42` | 4.2.1 | RC1 (`dist.apache.org/repos/dist/dev`) | official release |\n\n- The RC1 `url:` input is removed so that the `setup-spark` action falls back to its default ASF mirror redirector (`closer.lua`).\n- The `sha512:` input is updated to the checksum of the official tarball (verified against both `downloads.apache.org` and `dlcdn.apache.org`).\n- `AGENTS.md` is updated to list `4.0.5 / 4.1.4 / 4.2.1` as the tested Spark versions.\n\nThe 4.3.0 job still uses RC1 because it is not released yet.\n\n### Why are the changes needed?\n\nTo test against the official Apache Spark releases.\n- https://dlcdn.apache.org/spark/spark-4.0.5/\n- https://dlcdn.apache.org/spark/spark-4.1.4/\n- https://dlcdn.apache.org/spark/spark-4.2.1/\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Sonnet 5.5\n\nCloses #623 from dongjoon-hyun/SPARK-60619.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "a8d5ff58d657dd08621801a44acadac247e18143",
      "tree": "767124b1332f26101ad4b67c5f3b393c68be4e27",
      "parents": [
        "7dc5f0b0d13bfb280dae517812a319123d68970e"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 23 10:05:17 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 23 10:05:17 2026 -0700"
      },
      "message": "[SPARK-60516] Fix `DataFrame.toDF` to rename columns instead of selecting them\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix `DataFrame.toDF(_:)` to rename the columns by position with the `ToDF` relation, like the Scala and PySpark Spark Connect clients.\n\n```swift\n-        DataFrame(spark: self.spark, plan: SparkConnectClient.getProject(self.plan.root, cols))\n+        DataFrame(spark: self.spark, plan: SparkConnectClient.getToDF(self.plan.root, cols))\n```\n\n`toDF()` without column names still returns the `DataFrame` as is, like `Dataset.toDF()` in Scala.\n\n### Why are the changes needed?\n\n`toDF` was added for feature parity in SPARK-51995 ([#111](https://github.com/apache/spark-connect-swift/pull/111)), but it sends a `Project` relation with the given names, which is the same as `select(_:)`. Its documentation was also copied from `select(_:)`.\n\nIn Scala and PySpark, `toDF` renames all columns by position, and both Spark Connect clients send the `ToDF` relation.\n\n- [Apache Spark 4.2.0 (2026-07-11) `Dataset.toDF`](https://github.com/apache/spark/blob/v4.2.0/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala#L186-L190)\n\n```scala\ndef toDF(colNames: String*): DataFrame \u003d sparkSession.newDataFrame { builder \u003d\u003e\n  builder.getToDfBuilder\n    .setInput(plan.getRoot)\n    .addAllColumnNames(colNames.asJava)\n}\n```\n\n- [Apache Spark 4.2.0 (2026-07-11) PySpark `DataFrame.toDF`](https://github.com/apache/spark/blob/v4.2.0/python/pyspark/sql/connect/dataframe.py#L2080-L2091)\n\n```python\nreturn DataFrame(plan.ToDF(self._plan, list(cols)), self._session)\n```\n\nAs a result, the same code behaves differently in the Swift client. `spark.range(1).toDF(\"x\")` fails only in Swift, and `toDF(\"b\", \"a\")` on the columns `a` and `b` swaps the values without an error.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. `toDF` renames the columns by position like Scala and PySpark. This is a user-facing change compared to all released versions from [v0.1.0 (2025-05-04)](https://github.com/apache/spark-connect-swift/releases/tag/v0.1.0) to [0.7.0 (2026-07-22)](https://github.com/apache/spark-connect-swift/releases/tag/0.7.0).\n\nThe number of column names must be the same as the number of columns. To select or reorder columns by name, use `select(_:)` instead. `toDF()` and `toDF` with all column names in the same order return the same results as before.\n\n**BEFORE**\n\n```swift\nlet df \u003d try await spark.sql(\"SELECT 1 a, 2 b\")\ntry await spark.range(1).toDF(\"x\").columns  // throws [UNRESOLVED_COLUMN.WITH_SUGGESTION]\ntry await df.toDF(\"b\", \"a\").collect()       // [Row(2, 1)]\ntry await df.toDF(\"a\").columns              // [\"a\"]\n```\n\n**AFTER**\n\n```swift\nlet df \u003d try await spark.sql(\"SELECT 1 a, 2 b\")\ntry await spark.range(1).toDF(\"x\").columns  // [\"x\"]\ntry await df.toDF(\"b\", \"a\").collect()       // [Row(1, 2)]\ntry await df.toDF(\"a\").columns              // throws [ASSIGNMENT_ARITY_MISMATCH]\n```\n\n### How was this patch tested?\n\nPass the CIs with the updated test case.\n\nI also tested manually with Apache Spark 4.2.0 and 4.0.4 Spark Connect servers. The Scala Spark Connect client of Apache Spark 4.2.0 returns the same results as the AFTER example.\n\n```\n$ swift test --no-parallel --filter \"DataFrameTests|DataFrameInternalTests\"\n...\nTest run with 178 tests in 3 suites passed after 31.553 seconds.\n```\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #622 from dongjoon-hyun/SPARK-60516.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "7dc5f0b0d13bfb280dae517812a319123d68970e",
      "tree": "a37e05bcd71ffb43f479c00d72a4c417f792bd22",
      "parents": [
        "571058fa0c9dffdcc67023aaadce6b9376ee6a8a"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 23 10:04:28 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 23 10:04:28 2026 -0700"
      },
      "message": "[SPARK-60511] Make SparkSession.Builder.create public\n\n### What changes were proposed in this pull request?\n\nThis PR aims to make `SparkSession.Builder.create()` `public` like Scala and Python clients. In addition, the documentation of `SparkSession.Builder.getOrCreate()` is fixed because it also always creates a new session.\n\n### Why are the changes needed?\n\nSince SPARK-51483, `getOrCreate()` has been `return try await create()`, while `create()` is `internal`. So, every `getOrCreate()` call creates a new session in all releases from [v0.1.0](https://github.com/apache/spark-connect-swift/releases/tag/v0.1.0) (2025-05-04) to [0.7.0](https://github.com/apache/spark-connect-swift/releases/tag/0.7.0) (2026-07-22). However, the documentation added by SPARK-51991 says the following.\n\n\u003e If a session with the same configuration already exists, it will be returned. Otherwise, a new session is created with the specified configuration.\n\nIn [v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0) (2026-07-11), the other Spark Connect clients provide both APIs with different semantics.\n\n| | `create()` | `getOrCreate()` |\n| --- | --- | --- |\n| Scala ([Builder](https://github.com/apache/spark/blob/v4.2.0/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/SparkSession.scala#L1089-L1133)) | A new session | A cached session for the same client configuration |\n| Python ([Builder](https://github.com/apache/spark/blob/v4.2.0/python/pyspark/sql/connect/session.py#L227-L260)) | A new session | The active or default session |\n\nThis PR provides `create()` so that users can explicitly request a new session like the other clients. `getOrCreate()` keeps its behavior because existing applications and tests depend on it, e.g., each caller can `stop()` its own session without affecting the others. Its documentation now says that, unlike the other clients, it doesn\u0027t return an existing session.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, `SparkSession.builder.create()` becomes available as a public API. There is no behavior change.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test case, `BuilderTests.create`.\n\nSince `BuilderTests` uses `testable import SparkConnect`, the visibility was verified manually with a temporary test file which uses `import SparkConnect`. It fails to compile without this change.\n\n```\nerror: \u0027create\u0027 is inaccessible due to \u0027internal\u0027 protection level\n```\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #621 from dongjoon-hyun/SPARK-60511.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "571058fa0c9dffdcc67023aaadce6b9376ee6a8a",
      "tree": "f71f2e96af0d784c7eac101e4d34e83ca7d73bbe",
      "parents": [
        "7225e7337bb4b346e81effa62af06407a901df49"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Oct 22 10:03:58 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Oct 22 10:03:58 2026 -0700"
      },
      "message": "[SPARK-60477] Fix `DataFrameStatFunctions.approxQuantile` to throw instead of crashing on empty results\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix the single-column `DataFrameStatFunctions.approxQuantile(_:_:_:)` to throw `SparkConnectError.invalidState` instead of crashing when the server returns an empty list of quantiles. It uses the existing `Array.firstOrThrow()` helper instead of `[0]`.\n\n### Why are the changes needed?\n\nThe single-column version returns the first element of the multi-column result with `[0]`, which has no bounds check. So, an empty result crashes the client process.\n\n```\nSwift/ContiguousArrayBuffer.swift:695: Fatal error: Index out of range\n```\n\nA Spark Connect server returns one list of quantiles per requested column ([SparkConnectPlanner.transformStatApproxQuantile](https://github.com/apache/spark/blob/v4.2.0/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L576-L587) in [v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0) (2026-07-11)), so this happens only with a misbehaving or incompatible server. As described in SPARK-59193, such a server should cause a catchable error instead of crashing the client application. SPARK-59193 and SPARK-60352 fixed the other traps in this file, but this `[0]` remained. In the same case, the Scala client ([DataFrameStatFunctions.approxQuantile](https://github.com/apache/spark/blob/v4.2.0/sql/api/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala#L74-L79)) and the Python client ([DataFrame.approxQuantile](https://github.com/apache/spark/blob/v4.2.0/python/pyspark/sql/connect/dataframe.py#L1751-L1753)) raise catchable exceptions, `NoSuchElementException` and `IndexError`.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo for normal usage. When the server returns an empty list of quantiles, the single-column `approxQuantile` now throws `SparkConnectError.invalidState` instead of crashing.\n\n```\nThe server returned an empty response for approxQuantile(_:_:_:).\n```\n\n### How was this patch tested?\n\nPass the CIs. No test case is added because a Spark Connect server returns an empty list of quantiles only for an empty list of columns, so the changed code path cannot be tested without a mock server. `firstOrThrow()` on an empty array is covered by `ExtensionTests.firstOrThrowOnEmptyArray`, and the existing `DataFrameStatFunctionsTests.approxQuantile` covers the normal and empty `DataFrame` cases.\n\nIn addition, this was verified manually against an `apache/spark:4.2.0` Spark Connect server with a temporary client interceptor which removes the column from the `StatApproxQuantile` request. The single-column `approxQuantile` crashed with `Fatal error: Index out of range` before this fix, and throws `SparkConnectError.invalidState` after this fix. `DataFrameStatFunctionsTests` and `ExtensionTests` also passed against `apache/spark:4.2.0` and `apache/spark:4.0.4-java21-scala` Spark Connect servers.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #620 from dongjoon-hyun/SPARK-60477.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "7225e7337bb4b346e81effa62af06407a901df49",
      "tree": "d32e22225e7e9cc53cab1a3e2f8583938143b6ea",
      "parents": [
        "9b32afb97425282be5c796625bc9af1129967016"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Oct 22 10:02:32 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Oct 22 10:02:32 2026 -0700"
      },
      "message": "[SPARK-60476] Remove unused `ArrowFlight` files\n\n### What changes were proposed in this pull request?\n\nThis PR aims to remove the following unused files, which were copied from the `ArrowFlight` module of Apache Arrow Swift 19.0.1 by SPARK-51465.\n\n- `Flight.pb.swift`: generated from Arrow Flight\u0027s `Flight.proto`\n- `FlightData.swift`: `public class FlightData`\n- `FlightDescriptor.swift`: `public class FlightDescriptor`\n\nIn addition, `AGENTS.md` no longer lists `Flight.pb.swift` as generated code.\n\n### Why are the changes needed?\n\nSpark Connect doesn\u0027t use Arrow Flight. The server sends query results as serialized Arrow data in [ExecutePlanResponse.ArrowBatch](https://github.com/apache/spark/blob/v4.2.0/sql/connect/common/src/main/protobuf/spark/connect/base.proto#L453-L469) of `v4.2.0` (2026-07-11), and `DataFrame` reads it with `ArrowReader`. The three files only refer to each other, and no source, test, or example has ever used them. Only 3 of the 19 files in [ArrowFlight](https://github.com/apache/arrow/tree/apache-arrow-19.0.1/swift/ArrowFlight/Sources/ArrowFlight) of `apache-arrow-19.0.1` (2025-02-16) were copied, so there is no Flight client or server to use them with. The other vendored files come from the `Arrow` module, which doesn\u0027t depend on `ArrowFlight`.\n\nIn addition, `Flight.pb.swift` was generated by an old version of the SwiftProtobuf plugin and causes all 20 warnings of the library build.\n\n```\nSources/SparkConnect/Flight.pb.swift:639:58: warning: \u0027init(dictionaryLiteral:)\u0027 is deprecated: Please regenerate your .pb.swift files with the current version of the SwiftProtobuf protoc plugin. [#DeprecatedDeclaration]\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. This removes two `public` classes, `FlightData` and `FlightDescriptor` (including `FlightDescriptor.DescriptorType`), so code that refers to them doesn\u0027t compile anymore. They have been `public` since [v0.1.0](https://github.com/apache/spark-connect-swift/releases/tag/v0.1.0) (2025-05-08) and are in the latest release, [0.7.0](https://github.com/apache/spark-connect-swift/releases/tag/0.7.0) (2026-07-25). However, SPARK-51997 hid them from the documentation with `nodoc`, and no other API takes or returns them. There is no behavior change.\n\n### How was this patch tested?\n\nPass the CIs. In addition, I checked that a clean build of the library has no warnings instead of 20, and that all tests pass against an `apache/spark:4.2.0` Connect server.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #619 from dongjoon-hyun/SPARK-60476.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "9b32afb97425282be5c796625bc9af1129967016",
      "tree": "7df1cda284fa1fdcbca2c5fd2134593de22f917b",
      "parents": [
        "6ab8a4342e04666275914a7911dc59bd8b317eba"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Oct 22 08:34:35 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Oct 22 08:34:35 2026 -0700"
      },
      "message": "[SPARK-60472] Fix `collect` to throw an error instead of misreading compressed Arrow batches\n\n### What changes were proposed in this pull request?\n\nThis PR fixes `DataFrame` actions to throw `SparkConnectError.unsupportedOperation` instead of returning wrong values or crashing when the Spark Connect server sends compressed Arrow batches.\n\n- `ArrowReader` returns an error for a record batch with `compression` instead of reading the compressed buffers as they are.\n- `DataFrame` actions check the header of each Arrow batch first and throw `unsupportedOperation` with the codec name and the workaround, `spark.sql.execution.arrow.compression.codec\u003dnone`.\n\n### Why are the changes needed?\n\nSince Spark 4.1.0, the server compresses the buffers of Arrow batches when [spark.sql.execution.arrow.compression.codec](https://github.com/apache/spark/blob/v4.1.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L4046) is `zstd` or `lz4`. [ArrowBatchIterator](https://github.com/apache/spark/blob/v4.1.0/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala#L118) of `v4.1.0` (2025-12-11) passes the codec to `VectorUnloader`. The server then sets `compression` on every record batch, including small ones whose buffers are stored uncompressed after an 8-byte length of `-1`. However, `ArrowReader` ignores `compression` and reads the compressed buffers as they are.\n\n```swift\ntry await spark.conf.set(\"spark.sql.execution.arrow.compression.codec\", \"zstd\")\ntry await spark.range(10).count()    // -1\ntry await spark.range(10).collect()  // Wrong values like -1\ntry await spark.range(3).show()      // Fatal error: UnsafeBufferPointer with negative count\n```\n\nLike the Scala client, which fails with `IllegalArgumentException: Please add arrow-compression module to use CommonsCompressionFactory for ZSTD`, this PR does not support decompression. PySpark decompresses the batches with PyArrow.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. With Spark 4.1+ and a compression codec, `collect()`, `collect(as:)`, `head()`, `first()`, `take()`, `tail()`, `count()`, and `show()` throw the following error instead of returning wrong values or crashing. Results without buffers, e.g., `df.select()`, also throw it.\n\n```\nunsupportedOperation: Unsupported Arrow compression codec: zstd. Set `spark.sql.execution.arrow.compression.codec` to `none`.\n```\n\nThere is no change when the codec is `none`, which is the default.\n\n### How was this patch tested?\n\nAdded `RecordBatchTests.readCompressed`, which does not require a Spark Connect server, and `DataFrameTests.arrowCompression` for Spark 4.1+. Verified against an `apache/spark:4.2.0` Connect server.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #618 from dongjoon-hyun/SPARK-60472.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "6ab8a4342e04666275914a7911dc59bd8b317eba",
      "tree": "4348c72b72e841dd91cfcd3867f214d2844056e9",
      "parents": [
        "972101c622f7658a9a2d8cf3c35adad011fb3eda"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Oct 22 08:06:29 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Oct 22 08:06:29 2026 -0700"
      },
      "message": "[SPARK-60471] Fix `DataFrame.withColumnRenamed` to not crash on duplicate column names\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix `DataFrame.withColumnRenamed(_ colNames: [String], _ newColNames: [String])` to not crash on duplicate column names. It now sends the renames in the given order via the `renames` field of `WithColumnsRenamed`, like the Scala and Python Spark Connect clients.\n\n- Add `SparkConnectClient.getWithColumnRenamed(_:_:_:)`, which builds `WithColumnsRenamed.renames` from the given column name lists.\n- `withColumnRenamed(_ colNames: [String], _ newColNames: [String])` uses it instead of `Dictionary(uniqueKeysWithValues:)` and the deprecated `rename_columns_map` field. If the lists have different lengths, the extra names are ignored as before.\n\n`withColumnRenamed(_ existingName: String, _ newName: String)` and `withColumnRenamed(_ colsMap: [String: String])` are not changed.\n\n### Why are the changes needed?\n\n`Dictionary(uniqueKeysWithValues:)` traps on duplicate keys, so the following crashes the client process although the API does not throw.\n\n```swift\n// Fatal error: Duplicate values for key: \u0027a\u0027\ndf.withColumnRenamed([\"a\", \"a\"], [\"x\", \"y\"])\n```\n\nIn addition, `rename_columns_map` is an unordered `map`, and `swift-protobuf` serializes it in the iteration order of `Dictionary`, which differs across processes. Since the server applies the renames one by one, the result of order-sensitive renames was not deterministic. For example, `df.withColumnRenamed([\"a\", \"x\"], [\"x\", \"y\"])` returned `[\"x\", \"b\", \"c\", \"d\"]` or `[\"y\", \"b\", \"c\", \"d\"]` depending on the process.\n\nThe `renames` field was added by SPARK-46261 in Apache Spark [v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19), so all servers supported by this client accept it. In `v4.2.0` (2026-07-11), the server reads it in [SparkConnectPlanner](https://github.com/apache/spark/blob/v4.2.0/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L1287-L1310) and applies the renames in order in [UnresolvedStarWithColumnsRenames](https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala#L856-L865). The Scala client ([Dataset.withColumnsRenamed](https://github.com/apache/spark/blob/v4.2.0/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala#L828-L846)) and the Python client ([WithColumnsRenamed](https://github.com/apache/spark/blob/v4.2.0/python/pyspark/sql/connect/plan.py#L1687-L1702)) also use `renames`.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, it fixes a bug. Duplicate column names no longer crash, and the renames are applied in the given order, the same as the Scala and Python clients. The API signatures do not change.\n\n- `df.withColumnRenamed([\"a\", \"a\"], [\"x\", \"y\"])`: crash → `[\"x\", \"b\", \"c\", \"d\"]`. The second `a` is ignored because `a` is already renamed to `x`, like `df.withColumnRenamed(\"a\", \"x\").withColumnRenamed(\"a\", \"y\")`.\n- `df.withColumnRenamed([\"a\", \"x\"], [\"x\", \"y\"])`: `[\"x\", \"b\", \"c\", \"d\"]` or `[\"y\", \"b\", \"c\", \"d\"]` → always `[\"y\", \"b\", \"c\", \"d\"]`.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n- `DataFrameInternalTests.withColumnRenamedPlan`, which only inspects the generated plan, so it does not require a running server.\n- `DataFrameTests.withColumnRenamedWithDuplicateNames`\n\nBoth crashed with `Fatal error: Duplicate values for key: \u0027a\u0027` before this fix. In addition, `DataFrameTests` and `DataFrameInternalTests` passed against `apache/spark:4.2.0` and `apache/spark:4.0.4-java21-scala` Spark Connect servers, and the Scala Spark Connect client returned the same results against `apache/spark:4.2.0`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #617 from dongjoon-hyun/SPARK-60471.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "972101c622f7658a9a2d8cf3c35adad011fb3eda",
      "tree": "669b6a1841cf41c5aa2681bebe1b7302ef478087",
      "parents": [
        "ae52ef08f9c278291becca7651876f694a589e04"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Oct 22 08:05:09 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Oct 22 08:05:09 2026 -0700"
      },
      "message": "[SPARK-60464] Fix completed executions to not fail when the final `ReleaseExecute` fails\n\n### What changes were proposed in this pull request?\n\nThis PR aims to send the final `ReleaseExecute(release_all)` of reattachable executions in a best-effort way, like the Scala and Python clients.\n\nPreviously, it was sent inside the reattach loop right after `ResultComplete` arrived, so its failure was handled like a broken response stream. Now, it is sent after the loop by the existing `releaseAll(_:)` helper, which already ignores failures when an execution is abandoned after an error.\n\n### Why are the changes needed?\n\n`ReleaseExecute` only asks the server to free the buffered responses, and the server removes abandoned executions by itself after `spark.connect.execute.manager.detachedTimeout` (5 minutes by default). However, its failure failed the whole execution even after all responses were received and processed.\n\n- `INVALID_HANDLE.SESSION_NOT_FOUND`, e.g., after the server restarted, was reported as `invalidState: OPERATION_NOT_FOUND on the server but responses were already received from it.`\n- `INVALID_HANDLE.SESSION_CLOSED`, e.g., after another task released the session, was thrown as `sessionClosed`.\n- Only retriable errors like `UNAVAILABLE` succeeded, after a needless backoff sleep.\n\nSince `collect()`, `count()`, and the command APIs like `DataFrameWriter.save()` use this path, a finished write could be reported as failed, and retrying it could repeat its side effects. The Scala and Python clients send this request asynchronously and only log its failure.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo for the released versions because the reattachable execution (SPARK-58543) is not released yet. The latest release is [0.7.0](https://github.com/apache/spark-connect-swift/releases/tag/0.7.0) (2026-07-25). In the `main` branch, executions which received all responses no longer fail when the final `ReleaseExecute` fails.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test case, `ReattachableExecuteTests.releaseAllFailureAfterResultComplete`. It releases the session with `ReleaseSession` when `ResultComplete` arrives, so that the final `ReleaseExecute` fails with `SESSION_NOT_FOUND` (`allow_reconnect\u003dtrue`, like after a server restart) and `SESSION_CLOSED`.\n\nWithout this fix, the test case fails like the following.\n\n```\nTest releaseAllFailureAfterResultComplete() recorded an issue at ReattachableExecuteTests.swift:130:4: Caught error: invalidState: OPERATION_NOT_FOUND on the server but responses were already received from it.\n```\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #616 from dongjoon-hyun/SPARK-60464.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "ae52ef08f9c278291becca7651876f694a589e04",
      "tree": "2ad98d7f5e7f496a0d3150a212ea6a0111678e78",
      "parents": [
        "1f55429a3b71626ce642a99ea6ac86eea1bdacaf"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 21 11:59:56 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 21 11:59:56 2026 -0700"
      },
      "message": "[SPARK-60440] Migrate `notify_test_workflow.yml` from `pull_request_target` to `workflow_run`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to migrate `notify_test_workflow.yml` from `pull_request_target` to the signal and `workflow_run` pattern of the [ASF GitHub Actions Policy](https://infra.apache.org/github-actions-policy.html), like https://github.com/apache/spark-kubernetes-operator/pull/976 .\n\n- Add `notify_test_workflow_signal.yml`, an unprivileged `pull_request` workflow with `permissions: {}` and a single `echo` step, for the same `opened`, `reopened` and `synchronize` events. It takes over the general name `On pull request update` and its comment, because the `Build` check is attached to a check suite on the PR head commit, which is now the suite of this run. It uses `ubuntu-slim`, like `notify_test_workflow.yml`.\n- Change `notify_test_workflow.yml` (now named `Notify test workflow`) to run on `workflow_run` of `On pull request update`. GitHub runs it from the default branch, so it never runs the code of a PR. It checks `github.event.workflow_run.event \u003d\u003d \u0027pull_request\u0027`, sets `permissions: {}` at the top level, and keeps `actions: read` and `checks: write` for the job. The script reads the head repository, branch and SHA from `workflow_run` instead of `pull_request.head`.\n\n  | `pull_request_target` | `workflow_run` |\n  | --- | --- |\n  | `pull_request.head.repo.owner.login` | `workflow_run.head_repository.owner.login` |\n  | `pull_request.head.repo.name` | `workflow_run.head_repository.name` |\n  | `pull_request.head.repo.full_name` | `workflow_run.head_repository.full_name` |\n  | `pull_request.head.ref` | `workflow_run.head_branch` |\n  | `pull_request.head.sha` | `workflow_run.head_sha` |\n\n- Since the `workflow_run` run belongs to the `main` commit, the PR does not show it when it fails. So, a new `Report failure` step (`if: failure()`) creates a `completed` / `action_required` `Build` check on the head commit with a link to the failed run. The check run lookup is also made best-effort, as its comment says, so a lookup failure falls back to the Actions URL instead of failing the job.\n- `build_main.yml`, `build_and_test.yml` and `update_build_status.yml` are unchanged. The `Build` check keeps its name and `output.text` (`{owner, repo, run_id}`), so the scheduled updater keeps syncing it and skips `action_required` ones as before.\n\n```mermaid\nflowchart TD\n    subgraph fork[\"Forked repo (contributor)\"]\n        P([push to any branch]) --\u003e BM[\"build_main.yml — job: Run\"]\n        BM --\u003e|workflow_call| BAT[\"build_and_test.yml (reusable)\"]\n        BAT --\u003e CR[\"check runs:\u003cbr/\u003eRun / License Check, ...\"]\n    end\n\n    subgraph up[\"Upstream repo (apache/spark-connect-swift)\"]\n        PR([pull_request]) --\u003e S[\"notify_test_workflow_signal.yml\u003cbr/\u003eOn pull request update, no permissions\"]\n        S --\u003e|\"workflow_run (requested)\"| N[\"notify_test_workflow.yml\u003cbr/\u003eruns from main, checks: write\"]\n        SCH([schedule: every 15 min]) --\u003e U[\"update_build_status.yml\"]\n        N --\u003e|create| B([\"Build check on PR\"])\n        U --\u003e|\"PATCH status / conclusion\"| B\n    end\n\n    BM -.-\u003e|\"① find run (id: build_main.yml)\"| N\n    CR -.-\u003e|\"② link check-run view\"| N\n    BM -.-\u003e|\"③ poll run status\"| U\n```\n\n`notify_test_workflow.yml` uses `types: [requested]` instead of `[completed]`. By default, ASF requires approval for every `pull_request` run from external contributors. GitHub creates such a run as `completed` / `action_required` without jobs and sends `requested` for it at once, but it sends `completed` only after a committer approves the run and the run finishes. `requested` is not sent again when a run is approved or re-run. So, every push gets exactly one `Build` check without approval, as it does now.\n\nGitHub does not run `pull_request` workflows while a PR has merge conflicts, so such a PR gets a new `Build` check after the conflicts are resolved.\n\n### Why are the changes needed?\n\nThe ASF GitHub Actions Policy says, \"From November 2026, the pull_request_target trigger is disabled for ASF repositories.\" Without this PR, PRs from forks get no `Build` check from November 2026.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This only changes the CI workflows that maintain the `Build` status check of PRs.\n\n### How was this patch tested?\n\n- `yq` parses all workflow files. `workflow_run.workflows` matches the name of the signal workflow, and no `pull_request_target` remains under `.github`.\n- Both `github-script` bodies of `notify_test_workflow.yml` compile as async functions, the way `actions/github-script` runs them.\n- Both workflow files are identical to those of https://github.com/apache/spark-kubernetes-operator/pull/976 after SPARK-60408 except the runner label (`ubuntu-slim`), the project name and the image URL.\n- The whole flow takes effect only after the merge to `main`, because GitHub runs `workflow_run` workflows from the default branch.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #615 from dongjoon-hyun/SPARK-60440.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "1f55429a3b71626ce642a99ea6ac86eea1bdacaf",
      "tree": "a255363fc5789f67afb90988adf2774256cbf95b",
      "parents": [
        "2384f38764a4bfcfa4a640c46252eb4efa58ac6d"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 21 11:43:07 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 21 11:43:07 2026 -0700"
      },
      "message": "[SPARK-60435] Enable gRPC keepalive by default and support `grpc_keepalive_*` parameters in connection string\n\n### What changes were proposed in this pull request?\n\nThis PR proposes to enable gRPC keepalive (HTTP/2 PING) by default and to support the following connection string parameters with the same names and meanings as the Scala and Python clients (SPARK-58094).\n\n| Parameter | Default | Scala/Python default |\n| --- | --- | --- |\n| `grpc_keepalive_enabled` | `true` | `true` |\n| `grpc_keepalive_time_ms` | `300000` (5 minutes) | `60000` |\n| `grpc_keepalive_timeout_ms` | `20000` | `20000` |\n| `grpc_keepalive_without_calls` | `true` | `true` |\n\n- `SparkConnectClient.init(remote:)` parses them. Like `grpc_max_message_size`, a non-positive or non-integer `grpc_keepalive_time_ms` or `grpc_keepalive_timeout_ms` throws `SparkConnectError.InvalidArgument`. Like `use_ssl` and the Python client, a boolean parameter is `true` only when its value is `true`, ignoring case.\n- `getGRPCClient()` sets `HTTP2ClientTransport.Config.Connection.keepalive` of the transport. It stays `nil` (no keepalive) with `grpc_keepalive_enabled\u003dfalse`.\n\nThe default keepalive time is 5 minutes instead of 60 seconds because of Apache Spark 4.0, 4.1 and 4.2 servers. They use the gRPC Java defaults, `permitKeepAliveTime\u003d5m` and `permitKeepAliveWithoutCalls\u003dfalse`. On the third early ping since they last sent anything, they send `GOAWAY(ENHANCE_YOUR_CALM, \"too_many_pings\")` and close the connection with all its in-flight RPCs. The Scala and Python clients recover by doubling their keepalive time (gRFC A8), but `grpc-swift-nio-transport` [2.11.0](https://github.com/grpc/grpc-swift-nio-transport/releases/tag/2.11.0) (2026-10-13) doesn\u0027t. With 60 seconds, every attempt of an RPC on which the server stays silent for 3-4 minutes, e.g. `AnalyzePlan` with a long schema inference, would be killed. Apache Spark 4.3 servers allow a ping every 10 seconds even without calls, so their users can set `grpc_keepalive_time_ms\u003d60000` to get the same behavior as the Scala and Python clients.\n\n`grpc_keepalive_without_calls\u003dfalse` is not recommended with `grpc-swift-nio-transport` 2.11.0. It stops the keepalive timer when the last stream of a connection closes and doesn\u0027t restart it for new streams. Since the Swift client shares one connection, keepalive stops after the first RPC.\n\n### Why are the changes needed?\n\nThe Swift client used neither keepalive nor RPC deadlines. When a NAT gateway, a load balancer or a proxy silently drops a connection without TCP RST/FIN, a blocked RPC like `StreamingQuery.awaitTermination()` or a long `DataFrame.collect()` waits forever because nothing detects it.\n\nWith this PR, such a connection is closed within `grpc_keepalive_time_ms + grpc_keepalive_timeout_ms` (5 minutes and 20 seconds by default). The RPC fails with `UNAVAILABLE`, is retried by `RetryPolicy`, and `ExecutePlan` is resumed by `ReattachExecute`.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes.\n- The client sends a keepalive PING every 5 minutes by default, and an RPC on a silently dropped connection is retried or reattached instead of waiting forever. `grpc_keepalive_enabled\u003dfalse` restores the previous behavior.\n- `grpc_keepalive_*` parameters are honored instead of being ignored with `Unknown parameter: ...`. An invalid value such as `grpc_keepalive_time_ms\u003dabc` throws `SparkConnectError.InvalidArgument`.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases in `SparkConnectClientTests`.\n\nIn addition, I tested manually against `apache/spark:4.2.0` through a TCP relay. The relay can freeze its existing connections silently, reading and dropping bytes in both directions while keeping the sockets open, like a NAT gateway which dropped the mapping. New connections are forwarded as usual.\n\n1. `spark.sql(\"SELECT reflect(\u0027java.lang.Thread\u0027, \u0027sleep\u0027, 30000L)\").collect()` while the relay froze the connection 3 seconds later\n\n| Connection string | Result |\n| --- | --- |\n| `grpc_keepalive_enabled\u003dfalse` (previous behavior) | Blocked until the relay closed the frozen connection 63 seconds later |\n| `grpc_keepalive_time_ms\u003d10000;grpc_keepalive_timeout_ms\u003d5000` | Sent a PING, closed the connection 5 seconds later with `GOAWAY(keepalive_expired)`, reattached on a new connection, and returned in 30.4 seconds |\n| The above with `grpc_keepalive_without_calls\u003dfalse` | No PING was sent and blocked like the first row |\n\n2. Keepalive enforcement of the Apache Spark 4.2.0 server\n\n| Case | Result |\n| --- | --- |\n| Idle with `grpc_keepalive_time_ms\u003d10000` | The server closed the connection 30 seconds (3 PINGs) after it opened |\n| `spark.read.json(path).schema` taking 10.2 seconds with the default, with `grpc_keepalive_time_ms\u003d1000` | The server killed each of the first three attempts after 3 seconds, while each abandoned attempt kept running on the server |\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #614 from dongjoon-hyun/SPARK-60435.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "2384f38764a4bfcfa4a640c46252eb4efa58ac6d",
      "tree": "6bf9fb0c99a93869c78ecfa81c79c7bfd34ebb86",
      "parents": [
        "42eb4acad2f0d9390d783c0605e9ead7471630db"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 21 11:42:21 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 21 11:42:21 2026 -0700"
      },
      "message": "[SPARK-60434] Send `client_observed_server_side_session_id` to detect server-side session changes\n\n### What changes were proposed in this pull request?\n\nThis PR aims to send the server-side session ID observed by the client as `client_observed_server_side_session_id` in requests, like the Scala and Python clients. With it, the server rejects requests with `INVALID_HANDLE.SESSION_CHANGED` when the server-side session has changed, e.g., after the server restarted.\n\n- Add `SparkConnectClient.serverSideSessionID`, which `connect(_:)` records from the `server_side_session_id` of its `AnalyzePlan` response.\n- Set `client_observed_server_side_session_id` in all requests which have the field: `ExecutePlan`, `AnalyzePlan`, `Config`, `Interrupt`, `GetStatus`, `CloneSession`, `AddArtifacts`, and `ArtifactStatuses`. `ReattachExecute` and `ReleaseExecute` copy it from the initial `ExecutePlan` request.\n- Remove the unused `SparkSession.serverSideSessionID`.\n\n`FetchErrorDetails` is not changed, like the Scala client, because it only fetches the details of an error which the server already returned.\n\n### Why are the changes needed?\n\nWhen the Spark Connect server restarts, e.g., due to a driver failure, the server-side session is lost. Since the Swift client sends only `session_id`, the server silently creates a new empty session for it. The existing `SparkSession` keeps working without the session state like SQL configurations, temporary views, registered functions, and artifacts.\n\n```swift\nlet spark \u003d try await SparkSession.builder.getOrCreate()\ntry await spark.conf.set(\"spark.sql.session.timeZone\", \"Asia/Seoul\")\ntry await spark.range(3).createTempView(\"v\")\n// The server restarts here.\ntry await spark.conf.get(\"spark.sql.session.timeZone\")\ntry await spark.table(\"v\").count()\ntry await spark.range(3).count()\n```\n\n- Before: `Etc/UTC` (the server default), `TABLE_OR_VIEW_NOT_FOUND`, and `3`.\n- After: All of them fail with `[INVALID_HANDLE.SESSION_CHANGED] The handle ... is invalid. The existing Spark server driver instance has restarted. Please reconnect.`\n\nThe Scala and Python clients have sent this field since Apache Spark [v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19) via SPARK-47380, so all servers supported by this client validate it. The server checks it in [SparkConnectSessionManager](https://github.com/apache/spark/blob/v4.2.0/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectSessionManager.scala#L78-L106) of `v4.2.0` (2026-07-11) before running a request, so a request is never executed in the new session.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. After the server-side session changes, e.g., due to a server restart, the requests of the existing `SparkSession` fail with `INVALID_HANDLE.SESSION_CHANGED` instead of silently running in a new empty session. Users need to create a new `SparkSession` to continue, like the Scala and Python clients. There is no change while the server-side session stays the same.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n- `SparkConnectClientTests.clientObservedServerSideSessionID` and `ReattachableExecuteTests.clientObservedServerSideSessionID` check the request builders. They don\u0027t need a server.\n- `SparkConnectClientTests.serverSideSessionID` checks that `connect(_:)` records the server-side session ID.\n- `SparkConnectClientTests.serverSideSessionChanged` changes the observed server-side session ID to simulate a server restart. It checks that the server rejects every kind of request with `INVALID_HANDLE.SESSION_CHANGED` without running it.\n\nI also checked the above example against `apache/spark:4.2.0` by restarting the server container before and after this PR.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #613 from dongjoon-hyun/SPARK-60434.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "42eb4acad2f0d9390d783c0605e9ead7471630db",
      "tree": "60b66b60f0e7d4cf057d8542a37491ce725087a3",
      "parents": [
        "847c8e338863c30b985c866e73bc2cf9ab4539c9"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 21 10:17:43 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 21 10:17:43 2026 -0700"
      },
      "message": "[SPARK-60426] Make the index-based subscript of Row public\n\n### What changes were proposed in this pull request?\n\nThis PR aims to make the index-based subscript of `Row`, `subscript(index: Int)`, `public`. In addition, doc comments are added to both the index-based and the name-based subscripts.\n\n### Why are the changes needed?\n\nSince SPARK-51815, `subscript(index: Int)` has been `internal`, while the `get(_ i: Int)` method it calls is `public`. Later, SPARK-58496 added `public subscript(name: String)` next to it. As a result, users can use `try row.get(0)` and `try row[\"name\"]`, but not `try row[0]`. Since only the name-based subscript is visible outside the module, the compiler error is also misleading.\n\n```swift\nlet row \u003d try await spark.sql(\"SELECT * FROM VALUES (1, \u0027abc\u0027) T(id, name)\").collect()[0]\nprint(try row.get(0))   // OK\nprint(try row[\"name\"])  // OK\nprint(try row[0])       // error: cannot convert value of type \u0027Int\u0027 to expected argument type \u0027String\u0027\n```\n\nThis PR makes `Row` consistent with the following.\n- Apache Spark\u0027s Scala `Row.apply(i: Int)` and PySpark `row[i]`\n- `StructType`, which already provides both `public subscript(name: String)` and `public subscript(position: Int)`\n\nNote that the existing test code already uses `rows[0][0]` (e.g., `MathFunctionsTests`). It compiles only because of `testable import SparkConnect`.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, `row[i]` becomes available as a public API. There is no behavior change.\n\n### How was this patch tested?\n\nPass the CIs with the updated test case. Since `RowTests` uses `import SparkConnect` instead of `testable import SparkConnect`, the updated test case fails to compile without this change.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #612 from dongjoon-hyun/SPARK-60426.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "847c8e338863c30b985c866e73bc2cf9ab4539c9",
      "tree": "94f86025444f31ac0d8ec4329dbc5e1395c499d9",
      "parents": [
        "2310e8feec1e5b5068ce3d7ed0d3d29684584704"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Oct 20 15:43:17 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Oct 20 15:43:17 2026 -0700"
      },
      "message": "[SPARK-60412] Fix `SparkSession.Builder` to not send spark.(remote|master) to the server\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix `SparkSession.Builder` to not send `spark.remote` and `spark.master` to the server as session configurations, like the Python client.\n\n- Add `SparkSession.Builder.sessionConf`, which is `sparkConf` without `spark.remote` and `spark.master`.\n- `create()` sends `sessionConf` instead of `sparkConf`. It still uses `sparkConf[\"spark.remote\"]` to connect.\n\n### Why are the changes needed?\n\n`remote(_:)` stores the connection string as `spark.remote` in `sparkConf`, and `create()` sends the whole `sparkConf` to the server after connecting. So the connection string is stored in the server-side session configuration. If it has a secret like `;token\u003d...`, the secret is exposed via `spark.conf.get(\"spark.remote\")` and `spark.conf.getAll()`. SQL `SET` output is masked by the server\u0027s `spark.redaction.regex`, but these two APIs return the raw value.\n\n```swift\nlet spark \u003d try await SparkSession.builder.remote(\"sc://localhost:15002/;token\u003dmy-secret-token\").getOrCreate()\ntry await spark.conf.getOption(\"spark.remote\")\n```\n\n- Before: `\"sc://localhost:15002/;token\u003dmy-secret-token\"`\n- After: `nil`\n\nThe other Spark Connect clients don\u0027t send them:\n- The Python client excludes `spark.remote` and `spark.master` (`invalid_opts` in `SparkSession.Builder._apply_options`) because they take no effect after the session creation.\n- The Scala client keeps the connection string of `remote()` in the builder only, and does not send `spark.master` either (`SparkSession.Builder.handleBuilderConfig`).\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. `spark.remote` and `spark.master` given to `SparkSession.Builder` are no longer set in the server-side session configuration. They had no effect on the session. For example, `spark.conf.getOption(\"spark.remote\")` returns `nil` and `spark.conf.get(\"spark.master\")` returns the server\u0027s value.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n- `BuilderTests.sessionConf` checks that `sessionConf` excludes `spark.remote` and `spark.master` while `sparkConf[\"spark.remote\"]` is kept. It doesn\u0027t need a server.\n- `BuilderTests.remoteAndMasterAreNotSentToServer` creates a session via `SparkSession.builder.remote(...)` and checks that the server-side session has no `spark.remote` and that `spark.master` is not overwritten.\n\nI also checked the above example against `apache/spark:4.2.0` before and after this PR.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #611 from dongjoon-hyun/SPARK-60412.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "2310e8feec1e5b5068ce3d7ed0d3d29684584704",
      "tree": "32a58ae695da1bdd8909abcdda8f306d304e1db1",
      "parents": [
        "585f7a37b3b4ab9a90ebeaa188c81f14667a65b7"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Oct 20 15:03:09 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Oct 20 15:03:09 2026 -0700"
      },
      "message": "[SPARK-60409] Fix `createDataFrame` to keep the number of rows of local data without columns\n\n### What changes were proposed in this pull request?\n\nThis PR fixes `SparkSession.createDataFrame` to keep the number of rows of local data without columns, like PySpark.\n\n- `[[Sendable?]]` rows: `ConvertToArrow.toArrowIPCStream` sets the length of the `RecordBatch` to the number of rows.\n- `Encodable` items: `createDataFrame(batch:rowCount:schema:)` sets the length to the number of items when the encoded `RecordBatch` has no columns.\n\n### Why are the changes needed?\n\nLocal data without columns crashes because `RecordBatch.init(_:columns:)` reads `columns[0]`. In addition, `RecordBatch.Builder.finish()` takes the length from the first column, so a `RecordBatch` without columns cannot have any rows.\n\n```swift\nstruct EmptyRecord: Encodable, Sendable {}\n\ntry await spark.createDataFrame([EmptyRecord(), EmptyRecord()]).count()  // crash\ntry await spark.createDataFrame([[], []], \"\").count()                    // crash\ntry await spark.createDataFrame([[], []], \"struct\u003c\u003e\").count()            // crash\ntry await spark.createDataFrame([[], []], StructType()).count()          // crash\n```\n\nPySpark keeps the number of rows in this case. [LocalDataToArrowConversion.convert](https://github.com/apache/spark/blob/v4.2.0/python/pyspark/sql/conversion.py#L912) of `v4.2.0` (2026-07-11) uses `pa.Table.from_struct_array(pa.array([{}] * len(rows)))`. The server accepts a 0-column batch with N rows.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. The above examples return `2` instead of crashing. With SPARK-60405, which fixes reading results without columns, `collect()` returns `[Row(), Row()]` for them.\n\n### How was this patch tested?\n\nAdded `CreateDataFrameTests.emptySchema` and `CreateDataFrameTests.createDataFrameWithEncodableWithoutFields`. Verified against an `apache/spark:4.2.0` Connect server.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #610 from dongjoon-hyun/SPARK-60409.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "585f7a37b3b4ab9a90ebeaa188c81f14667a65b7",
      "tree": "555c674fa6fa5c02862d61da38dbe61ab2acde0b",
      "parents": [
        "dc678a00a6378e76765ec577ac1e015e2671b3ce"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Oct 20 14:50:40 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Oct 20 14:50:40 2026 -0700"
      },
      "message": "[SPARK-60407] Fix command APIs to not return empty results when ExecutionProgress arrives first\n\n### What changes were proposed in this pull request?\n\nThis PR fixes the command APIs to read the result from the first response of the expected `responseType` instead of the first response.\n\n- Add `[ExecutePlanResponse].firstOrThrow(_:)`, which takes the key path of a `responseType` case like `\\.streamingQueryCommandResult` and throws `SparkConnectError.invalidState` if no response has it.\n- Use it at all 14 call sites that read a command result: `StreamingQuery` (7), `StreamingQueryManager` (3), `SparkConnectClient` (3, for `SparkSession.executeCommand`, `DataFrame.checkpoint`/`localCheckpoint`, and `createDataflowGraph`), and `DataStreamWriter.start` (1).\n\nThe other 17 `firstOrThrow()` call sites in `Catalog` and `DataFrameStatFunctions` are unchanged because they work on the `[Row]` of `collect()`, which only reads the schema and Arrow batches.\n\n### Why are the changes needed?\n\nSince Apache Spark 4.0.0 (SPARK-47081), the Spark Connect server sends an `ExecutionProgress` response every `spark.connect.progress.reportInterval` (default: `2s`) while an execution is waiting for its next response. So, a command taking longer than 2 seconds receives `ExecutionProgress` responses before its result. For example, `apache/spark:4.2.0` returns the following.\n\n```\ncheckpoint       : executionProgress (x3), checkpointCommandResult, executionProgress (x2), resultComplete\nawaitTermination : executionProgress, streamingQueryCommandResult, executionProgress, resultComplete\n```\n\nSince the client used the first response, it silently returned the empty default value of the expected result.\n\n- `StreamingQuery.awaitTermination(timeoutMs)` and `StreamingQueryManager.awaitAnyTermination(timeoutMs)` return `false` even when the query terminated.\n- Eager `DataFrame.checkpoint()` and `localCheckpoint()` return a `DataFrame` with an empty cached relation ID, which fails at the next action with `[CONNECT_INVALID_PLAN.DATAFRAME_NOT_FOUND]`.\n\nThe reference clients handle this already. The Scala client ignores `ExecutionProgress` in `SparkSession.execute(command)` and uses `find(_.hasSqlCommandResult)`, and the Python client handles each response by `HasField`.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. Commands taking longer than `spark.connect.progress.reportInterval` return the actual results instead of empty default values. If the server returns no expected result, `SparkConnectError.invalidState` is thrown instead.\n\n### How was this patch tested?\n\nPass the CIs with the newly added `ExtensionTests` cases, which need no Spark Connect server.\n\n- `firstOrThrowCommandResultAfterExecutionProgress`\n- `firstOrThrowEmptyCommandResult`\n- `firstOrThrowWithoutCommandResult`\n\nAlso, manually verified against an `apache/spark:4.2.0` Connect server that eager `localCheckpoint()` of `SELECT reflect(\u0027java.lang.Thread\u0027, \u0027sleep\u0027, 3000L)` works and `awaitTermination(10000)` returns `true` for a query stopped after 3 seconds.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #609 from dongjoon-hyun/SPARK-60407.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "dc678a00a6378e76765ec577ac1e015e2671b3ce",
      "tree": "662f2a640e4547ed783798bed7ca21ad1ebc2196",
      "parents": [
        "6b5d4591bbae4d96e06efbf963da89c8fdb10d8e"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Oct 20 14:49:06 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Oct 20 14:49:06 2026 -0700"
      },
      "message": "[SPARK-60405] Fix `collect` to return empty `Row`s instead of crashing on results without columns\n\n### What changes were proposed in this pull request?\n\nThis PR fixes `DataFrame` actions to return empty `Row`s instead of crashing when a result has no columns, e.g., `spark.range(3).select().collect()`.\n\n- `ArrowReader` creates each `RecordBatch` with the `length` of the Arrow IPC record batch message. An invalid (negative) length now returns an error instead of trapping.\n- `RecordBatch` gets `init(_:columns:length:)`. When there are no columns, `init(_:columns:)` falls back to `0` instead of reading `columns[0]`.\n\n### Why are the changes needed?\n\n`RecordBatch.init` took the number of rows from `columns[0]`, which traps with `Index out of range` on a record batch without columns. The Spark Connect server sends such a batch with `N` rows and no columns, and PySpark returns `N` empty `Row`s for it.\n\n```swift\n// Fatal error: Index out of range\ntry await spark.range(3).select().collect()\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. `collect()`, `head()`, `first()`, `take()`, `tail()`, and `collect(as:)` return `N` rows instead of crashing when the result has no columns. `createDataFrame` with local data without columns no longer crashes either.\n\n### How was this patch tested?\n\nAdded `RecordBatchTests`, which does not require a Spark Connect server, and `DataFrameTests.collectWithoutColumns`. Verified against an `apache/spark:4.2.0` Connect server.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #608 from dongjoon-hyun/SPARK-60405.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "6b5d4591bbae4d96e06efbf963da89c8fdb10d8e",
      "tree": "aea0c1342463cf7ff64872ca8a385d0d800eef3b",
      "parents": [
        "833b62513acaae42112c2951f96a3e8a724ee433"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Oct 20 14:48:13 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Oct 20 14:48:13 2026 -0700"
      },
      "message": "[SPARK-60404] Use lowercase UUIDs as operation IDs to fix ReattachExecute and ReleaseExecute\n\n### What changes were proposed in this pull request?\n\nThis PR aims to use a lowercase UUID string as the operation ID of `ExecutePlanRequest`, like the session ID.\n\n### Why are the changes needed?\n\n`UUID().uuidString` is uppercase. The Spark Connect server normalizes the operation ID of `ExecutePlan` to lowercase, but `ReattachExecute` and `ReleaseExecute` look it up as it is. So the server cannot find the execution:\n\n- `ReleaseExecute(release_all)` is a no-op, and the execution stays on the server until it is abandoned (5 minutes by default).\n- `ReattachExecute` fails with `INVALID_HANDLE.OPERATION_NOT_FOUND`, so actions longer than `spark.connect.execute.reattachable.senderMaxStreamDuration` (2 minutes by default) fail.\n\nThe Scala and Python clients use lowercase UUIDs.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, it fixes a bug. Long-running actions no longer fail, and the server releases executions right after they finish.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test case, `ReattachableExecuteTests.lowercaseOperationID`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #607 from dongjoon-hyun/SPARK-60404.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "833b62513acaae42112c2951f96a3e8a724ee433",
      "tree": "0cc8a1bd6fc408077377db60f0d88333f0669da5",
      "parents": [
        "8b078a547d1b3227f1f6bd5279c0b690deeee303"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Oct 20 11:43:37 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Oct 20 11:43:37 2026 -0700"
      },
      "message": "[SPARK-60390] Fix `collect` to return struct columns as nested `Row`s instead of crashing on non-UTF-8 binary fields\n\n### What changes were proposed in this pull request?\n\nThis PR fixes `DataFrame.collect()` to return `struct` columns as nested `Row`s (like Scala) instead of a `String`.\n\n- `collect()` converts each `struct` field recursively with the same logic as top-level columns.\n- `Row.toString()` and `\u003d\u003d` support nested `Row`s.\n- `BinaryArray.asString` falls back to hex for non-UTF-8 bytes instead of trapping.\n\n### Why are the changes needed?\n\n`collect()` rendered a `struct` via `BinaryArray.asString`, which force-unwrapped `String(data:encoding: .utf8)`. This crashes on non-UTF-8 bytes, e.g. `VARIANT` and `GEOMETRY`/`GEOGRAPHY` columns. Scala keeps these as bytes inside a `Row`.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. A `struct` column from `collect()` is now a `Row` instead of a `String` (e.g. `\"{1,x}\"` becomes a `Row` with `a \u003d 1` and `b \u003d \"x\"`). `VARIANT` and `GEOMETRY`/`GEOGRAPHY` columns no longer crash.\n\n### How was this patch tested?\n\nAdded `ArrowArrayTests`, `RowTests.nestedRow`, `DataFrameTests.collectStruct`, and `DataFrameTests.collectVariant`. Verified against an `apache/spark:4.2.0` Connect server.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Sonnet 5.5\n\nCloses #606 from dongjoon-hyun/SPARK-60390.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "8b078a547d1b3227f1f6bd5279c0b690deeee303",
      "tree": "43fb434019d216673a5b315721199db4e88a2080",
      "parents": [
        "53ffed59c878aff29021648c780fbb9f9c8ee37d"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Oct 20 11:42:45 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Oct 20 11:42:45 2026 -0700"
      },
      "message": "[SPARK-60389] Fix `createDataFrame` to throw instead of crashing on a non-integer `spark.sql.session.localRelationCacheThreshold`\n\n### What changes were proposed in this pull request?\n\nThis PR fixes `SparkSession.createDataFrame` to throw `SparkConnectError.invalidArgument` instead of trapping when `spark.sql.session.localRelationCacheThreshold` returned by the server is not a valid integer.\n\n### Why are the changes needed?\n\n`createDataFrame(ipcStream:schema:)` force-unwraps `Int(...)` of the conf value, so a non-integer value crashes the process. The conf is an `intConf` in Spark (same as the Python client\u0027s `int(...)`), so no unit-suffix parsing is needed.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo for valid servers. A non-integer value now throws `invalidArgument` instead of crashing.\n\n### How was this patch tested?\n\nAdded `SparkSessionTests.parseLocalRelationCacheThreshold`, which needs no server.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Sonnet 5.5\n\nCloses #605 from dongjoon-hyun/SPARK-60389.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "53ffed59c878aff29021648c780fbb9f9c8ee37d",
      "tree": "7d58ef09d718884782e695b21d7bf946ef78fddf",
      "parents": [
        "5276cd8384e3689e59593334c84c9b4e0a5b18e6"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Oct 20 11:41:57 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Oct 20 11:41:57 2026 -0700"
      },
      "message": "[SPARK-60388] Fix `DataFrame.na` and `DataFrame.stat.sampleBy` to not crash on unsupported value types\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix `DataFrame.na.fill`, `DataFrame.na.replace`, and `DataFrame.stat.sampleBy` to not crash on unsupported value types. The public API stays non-throwing.\n\n- Support `Int8` and `Int16` in `na.fill` and `na.replace`.\n- Convert any other type via `ExpressionLiteral.init(_:)`, so `Decimal`, `Date`, etc. are sent as literals.\n- Types that cannot be converted are sent as a string literal of their description instead of `as! String`.\n\n### Why are the changes needed?\n\nThe `default` branches used `value as! String`, which traps and crashes the client process:\n\n```swift\ndf.na.fill([\"a\": Int8(1)])                // crash\ndf.stat.sampleBy(\"key\", [Int16(1): 0.5])  // crash\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, it fixes a bug. These calls used to crash the process. Now they build a plan, and values the server does not accept are reported by the Spark Connect server. The API signatures do not change.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases in `DataFrameInternalTests`. They only inspect the generated plan, so they do not need a running server.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Sonnet 5.5\n\nCloses #604 from dongjoon-hyun/SPARK-60388.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "5276cd8384e3689e59593334c84c9b4e0a5b18e6",
      "tree": "c41fe6c2d9f56e1db51b8ea6bebb876c50e9b91a",
      "parents": [
        "61fa645c4ddb9463ab67542f035fb113d3c266f1"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Oct 19 17:17:37 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Oct 19 17:17:37 2026 -0700"
      },
      "message": "[SPARK-60352] Fix DataFrameStatFunctions.cov/corr/approxQuantile to throw instead of crashing on unexpected result types\n\n### What changes were proposed in this pull request?\n\nThis PR fixes `DataFrameStatFunctions.cov`, `corr`, and `approxQuantile` to throw `SparkConnectError.InvalidType` instead of crashing on an unexpected result.\n\n- `collectDouble` (`cov`, `corr`): use `Row.getAsDouble(0)` instead of `as! Double`.\n- `approxQuantile`: use `as? [[Double]]` with a `throw` instead of nested `as!`.\n\n### Why are the changes needed?\n\n`as!` traps the process when the result is `NULL` or has an unexpected type. A thrown error can be caught by callers.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo for normal usage. An unexpected result now throws `InvalidType` instead of crashing.\n\n### How was this patch tested?\n\nAdded empty-DataFrame, single-row, and null-value cases to `DataFrameStatFunctionsTests`. `swift build` passes, but the new tests were not run because they need a live Spark Connect server.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Sonnet 5.5\n\nCloses #602 from dongjoon-hyun/SPARK-60352.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "61fa645c4ddb9463ab67542f035fb113d3c266f1",
      "tree": "d70c218842c42f853518af908742749d8d95571b",
      "parents": [
        "bc8f8dded05040043d501590e7254d3608495ed4"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Oct 19 12:05:28 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Oct 19 12:05:28 2026 -0700"
      },
      "message": "[SPARK-60353] Fix `Catalog` APIs to not crash on null values returned by the server\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix `Catalog` APIs to not crash on `null` values returned by the server, by removing all 24 force casts (`as!`) in `Catalog.swift`.\n\n- `tableType` of `SparkTable` (`listTables`, `getTable`, `listViews`, `getView`, etc.) falls back to an empty string, like `Catalog.listFunctions`/`listDatabases` already do after SPARK-60063.\n- The other required values (`name`, `isTemporary`, `partition`, table property keys, and the single-value results of `currentCatalog`, `currentDatabase`, `getCreateTableString`, `databaseExists`, etc.) now use the existing `Row.getAsString`/`Row.getAsBool`, which throw `SparkConnectError.InvalidType` instead of crashing the process.\n\n### Why are the changes needed?\n\nPreviously, these APIs force-cast the server results with `as! String`/`as! Bool`, which crashes the client process when the server returns `null`.\n\n```\nCould not cast value of type \u0027Swift.Optional\u003cAny\u003e\u0027 to \u0027Swift.String\u0027.\n```\n\nFor example, Apache Spark\u0027s `Catalog.listTables` returns a `Table` with `tableType \u003d null` when it fails to resolve a table or view, in order to return partial results (see `classic/Catalog.scala`). Third-party catalogs may also return `null` for other fields.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. `SparkTable.tableType` is now an empty string instead of crashing when the server returns `null`. Other `null` required values now throw `SparkConnectError.InvalidType` instead of crashing. There is no API change because the field types of the public structs (`SparkTable`, `Database`, `Function`, `CatalogMetadata`, `TablePartition`) remain the same.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Sonnet 5.5\n\nCloses #603 from dongjoon-hyun/SPARK-60353.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "bc8f8dded05040043d501590e7254d3608495ed4",
      "tree": "72e72311b207c8b35a963ac0ff4b0c562d495284",
      "parents": [
        "4f9d4bb54a2ddd94d8cf3f189b21df6ccaeb0e7f"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Oct 19 12:03:58 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Oct 19 12:03:58 2026 -0700"
      },
      "message": "[SPARK-60351] Fix `getConf(WithDefault|Option)?` and `isModifiable` to not crash on empty Config responses\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix `SparkConnectClient.getConf`, `getConfWithDefault`, `getConfOption` and `isModifiable` to not crash when the server returns a `ConfigResponse` with no pairs.\n\n- `getConf` throws `SparkConnectError.sqlConfNotFound`.\n- `getConfWithDefault` returns the given default value.\n- `getConfOption` returns `nil`.\n- `isModifiable` returns `false`.\n\n### Why are the changes needed?\n\nThese methods read `response.pairs[0]` without a bounds check, so an empty response traps with an index-out-of-range error and crashes the client process.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, it fixes a bug. An empty `ConfigResponse` used to crash the process. Now it is handled as described above.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test case (`SparkConnectClientTests.getConfValue`). The other three methods are not covered because an empty server response cannot be simulated without a mock server.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Sonnet 5.5\n\nCloses #601 from dongjoon-hyun/SPARK-60351.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "4f9d4bb54a2ddd94d8cf3f189b21df6ccaeb0e7f",
      "tree": "2db49c1bb69c0dc02ddba6a3f17ce5d1a782f7d6",
      "parents": [
        "96ed30e625613d5ab7d432012eacda88a594ce4f"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Oct 19 12:02:52 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Oct 19 12:02:52 2026 -0700"
      },
      "message": "[SPARK-60349] Fix `StreamingQuery` creation to throw instead of trapping on non-UUID query ids from the server\n\n### What changes were proposed in this pull request?\n\nReplace the `UUID(uuidString:)!` force unwraps on server-provided query ids with a throwing `StreamingQuery` initializer that throws `SparkConnectError.invalidArgument` for non-UUID `id`/`runId`. It is used in `DataStreamWriter.start()`, `StreamingQueryManager.active`, and `StreamingQueryManager.get(_:)`.\n\n### Why are the changes needed?\n\nA non-UUID `id` or `runId` from the server currently traps and crashes the application. All three call sites already `throws`, so a malformed response can surface as a catchable error instead.\n\n### Does this PR introduce _any_ user-facing change?\n\nOnly for malformed server responses: an error is thrown instead of a crash.\n\n### How was this patch tested?\n\nAdded `StreamingQueryTests.createFromQueryID`, which needs no server. Tests that need a live server were not run locally.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Sonnet 5.5\n\nCloses #600 from dongjoon-hyun/SPARK-60349.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "96ed30e625613d5ab7d432012eacda88a594ce4f",
      "tree": "3c3521c2e4c6021fdd7b10024c386f3908b244ab",
      "parents": [
        "90341ca889d67c8e2cd700a03788985403ce8890"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 16 15:21:21 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 16 15:21:21 2026 -0700"
      },
      "message": "[SPARK-60303] Support `UDFRegistration.registerJava` via `SparkSession.udf`\n\n### What changes were proposed in this pull request?\n\nThis PR adds `spark.udf.registerJava` to register a Java UDF as a SQL function, mirroring the Scala Spark Connect client. If no return type is given, the server infers it. `registerJavaUDAF` is out of scope.\n\n```swift\ntry await spark.addArtifact(\"/path/to/udf.jar\")\ntry await spark.udf.registerJava(\"sconcat\", \"com.example.StringConcat\", .string)\ntry await spark.sql(\"SELECT sconcat(\u0027a\u0027, \u0027b\u0027)\").show()\n```\n\n### Why are the changes needed?\n\nThe Scala Spark Connect client can register Java UDFs by class name, but the Swift client cannot.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, this adds new APIs: `SparkSession.udf` and `UDFRegistration.registerJava`.\n\n### How was this patch tested?\n\n- Added `UDFRegistrationTests`:\n  - `registerJavaWithUnknownClass` checks that registering a nonexistent class fails with a `NoSuchUDF` error.\n  - `registerJavaWithoutUDFInterface` checks that registering `java.lang.String` fails with `NO_UDF_INTERFACE`.\n- Manually verified the positive path against `apache/spark:4.2.0`. I uploaded a `UDF2` jar with `addArtifact` and registered it both with `.string` and with the return type omitted (server-side inference). Both registered functions returned the concatenated string (e.g. `SELECT sconcat(\u0027a\u0027, \u0027b\u0027)` returned `ab`). This is not committed because it needs a jar fixture.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Sonnet 5.5\n\nCloses #599 from dongjoon-hyun/SPARK-60303.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "90341ca889d67c8e2cd700a03788985403ce8890",
      "tree": "25b0840d3223d560d668bacfc902847a338c3973",
      "parents": [
        "63cf898f50ca561c27e38b3ba2dc5f109907d616"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 16 12:34:27 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 16 12:34:27 2026 -0700"
      },
      "message": "[SPARK-60298] Set default `grpc_max_message_size` to 128 MiB and support Arrow batch result chunking\n\n### What changes were proposed in this pull request?\n\nThis PR aims to align the gRPC message size limit and the Arrow result handling with the Scala and Python Spark Connect clients.\n\n1. Set the default `grpc_max_message_size` to 128 MiB (`SparkConnectClient.DEFAULT_GRPC_MAX_MESSAGE_SIZE`). `SparkConnectClient.maxMessageSize` is no longer optional, and the `ServiceConfig` always applies the limit to both request and response messages.\n2. Send `ResultChunkingOptions(allow_arrow_batch_chunking \u003d true)` in every `ExecutePlanRequest` so that the server can split large Arrow batches into chunks.\n3. Assemble the chunks in `DataFrame.execute()` using `chunk_index` and `num_chunks_in_batch` before parsing the Arrow IPC stream. An out-of-order chunk fails with `InvalidArrowData`.\n4. Count `row_count` only for the first chunk (`chunk_index \u003d\u003d 0`) in `executeAndCount()`, because the server sets the same `row_count` on all chunks of a batch.\n5. Add the `ResultChunkingOptions` type alias.\n\n`preferred_arrow_chunk_size` is not exposed in this PR.\n\n### Why are the changes needed?\n\nWithout `grpc_max_message_size`, the client is limited to the gRPC default of 4 MiB, so a response larger than 4 MiB fails with `Received message larger than max`. The Scala client and the Python client both default to 128 MiB. Python has used 128 MiB since [SPARK-42816](https://github.com/apache/spark/commit/da19e0de05b) (2023-03-21, v3.5.0).\n\nEven with a larger limit, a single Arrow batch can exceed it. Result chunking, enabled by default in the Scala and Python clients, lets the server split such batches.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes.\n- When `grpc_max_message_size` is not specified, the limit changes from 4 MiB to 128 MiB. Responses larger than 4 MiB now succeed instead of failing.\n- Large Arrow batches are now received in chunks and transparently assembled.\n\n### How was this patch tested?\n\n- Updated `parameterWithoutGrpcMaxMessageSize` to expect the 128 MiB default in `maxMessageSize` and in the `ServiceConfig`.\n- Added `resultChunking` to `SparkConnectClientTests` to verify the request option.\n- Confirmed that `swift build` and the test target compile.\n- The integration tests were not run locally because no Spark Connect server was available. They are expected to run in CI.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Sonnet 5.5\n\nCloses #597 from dongjoon-hyun/SPARK-60298.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "63cf898f50ca561c27e38b3ba2dc5f109907d616",
      "tree": "5352e57fbd4423740e4b2223146e065575ab8c19",
      "parents": [
        "3fa39dc4b2b12ff7f648e7e59d878ed75bd3bcea"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 16 12:27:23 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 16 12:27:23 2026 -0700"
      },
      "message": "[SPARK-60299] Fix `SparkSession` to reject an invalid `session_id` instead of silently truncating it\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix `SparkSession` to reject an invalid `session_id` in the connection string instead of silently truncating it.\n\n- Read the whole `session_id` value (`[^;]*`) instead of stopping at the first character outside `[a-zA-Z0-9-]`.\n- Throw `SparkConnectError.InvalidSessionID` if the value is not a valid UUID.\n\n### Why are the changes needed?\n\n`;session_id\u003dabc_def` was read as `abc`, and `;session_id\u003d\u003cuuid\u003e_suffix` was accepted as `\u003cuuid\u003e`. Like the Python client, an invalid `session_id` should be an error.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. An invalid `session_id` such as `abc_def` now fails with `InvalidSessionID`. Valid UUIDs behave as before.\n\n### How was this patch tested?\n\nAdded `SparkSessionTests.invalidSessionIDInConnectionString`. It needs no Spark Connect server. I could not run it locally because the `Testing` module is missing, so CI should confirm it.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Sonnet 5.5\n\nCloses #598 from dongjoon-hyun/SPARK-60299.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "3fa39dc4b2b12ff7f648e7e59d878ed75bd3bcea",
      "tree": "dbc93bb407a11c3165c73184b6f7367df0986991",
      "parents": [
        "af54b33eaff2b72b7fd9f2c6ff49f5c959bb00c1"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 16 11:29:09 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 16 11:29:09 2026 -0700"
      },
      "message": "[SPARK-60297] Enforce TLS in `SparkConnectClient` when a token is set\n\n### What changes were proposed in this pull request?\n\nThis PR aims to enable TLS in `SparkConnectClient` whenever an authentication token is set, even if `use_ssl\u003dtrue` is not specified, like the Python client.\n\n- The token can come from the `token\u003d` connection string parameter or the `SPARK_CONNECT_AUTHENTICATE_TOKEN` environment variable.\n- `localhost` is excepted and stays plaintext unless `use_ssl\u003dtrue` is given, like the Python client, which uses local channel credentials in that case.\n- Update `SecurityGuide.md` accordingly.\n- Add the `tokenEnforcesTLS` test.\n\n### Why are the changes needed?\n\nCurrently, the client sends the `Authorization: Bearer` header over a plaintext connection if the connection string has a token but no `use_ssl\u003dtrue`. This exposes the token to interception. The Python client treats a connection as secure when either `use_ssl\u003dtrue` or a token is set (`secure \u003d use_ssl or token is not None`), so this PR aligns the behavior.\n\nThe `localhost` exception keeps the existing CI, which connects to a local server with `SPARK_CONNECT_AUTHENTICATE_TOKEN` over plaintext, working.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. When a token is set and the host is not `localhost`, the client now connects with TLS even without `use_ssl\u003dtrue`. Previously, it connected with plaintext. If the server does not support TLS, the connection fails.\n\n### How was this patch tested?\n\nPassed `swift build`. Added the `tokenEnforcesTLS` test, which does not require a Spark Connect server, and it verifies the following.\n\n- `sc://host1:123/;token\u003dabc` uses TLS.\n- `sc://localhost:123/;token\u003dabc` stays plaintext.\n- `sc://localhost:123/;token\u003dabc;use_ssl\u003dtrue` uses TLS.\n\nThe new test was not run in my local environment because Xcode is not installed (`no such module \u0027Testing\u0027`), so it relies on CI.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Sonnet 5.5\n\nCloses #596 from dongjoon-hyun/SPARK-60297.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "af54b33eaff2b72b7fd9f2c6ff49f5c959bb00c1",
      "tree": "e8e997f38db3a4929ff5d48f85b1f9bf15f9e755",
      "parents": [
        "0cbff0618dc9770697a6f51464513095ffce98f9"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 16 11:00:13 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 16 11:00:13 2026 -0700"
      },
      "message": "[SPARK-60295] Fix flaky `StreamingQueryTests.status` test by waiting for query initialization before stop\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix the flaky `StreamingQueryTests.status` test by calling `query.processAllAvailable()` before `query.stop()`. This makes the test wait until the query has finished initializing.\n\n### Why are the changes needed?\n\n`StreamingQueryTests.status` fails intermittently in CI. For example:\n- https://github.com/apache/spark-connect-swift/actions/runs/38934002527/job/117084779028\n\n```\n✘ Test status() recorded an issue at StreamingQueryTests.swift:128:5: Expectation failed: stoppedStatus.statusMessage \u003d\u003d \"Stopped\"\n↳ stoppedStatus.statusMessage \u003d\u003d \"Stopped\" → false\n↳   stoppedStatus.statusMessage → \"Initializing sources\"\n```\n\nThe test calls `stop()` right after starting the query, which races with server-side initialization. In `StreamExecution.runStream()`, the server first sets the status message to `Initializing sources`. It then tries `state.compareAndSet(INITIALIZING, ACTIVE)`. If `stop()` has already been called, this transition fails and the server leaves the cleanup to the `finally` block without setting the status message to `Stopped`. As a result, the message stays as `Initializing sources`.\n\nThe server\u0027s `processAllAvailable()` returns only after the query has become `ACTIVE` and processed the available data. So stopping the query after it always reaches the code path that sets `Stopped`.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is a test-only change.\n\n### How was this patch tested?\n\nPass the CIs.\n\nI also ran the test 10 times in a row against a local Apache Spark 4.1.3 Connect server, and all runs passed.\n\n```\nswift test --no-parallel --filter \"StreamingQueryTests/status\"\n```\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #595 from dongjoon-hyun/SPARK-60295.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "0cbff0618dc9770697a6f51464513095ffce98f9",
      "tree": "71dae5e1052545d5b5eb8cfe80797e6c5e12fe34",
      "parents": [
        "e4635c545f877a7ee32fe0960c3751097f6d9e07"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 16 10:37:40 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 16 10:37:40 2026 -0700"
      },
      "message": "[SPARK-60289] Upgrade `gRPC Swift NIO Transport` to 2.11.0\n\n### What changes were proposed in this pull request?\n\nThis PR aims to upgrade `grpc-swift-nio-transport` to 2.11.0.\n\n### Why are the changes needed?\n\nTo bring the latest bug fixes and dependency updates.\n\n- https://github.com/grpc/grpc-swift-nio-transport/releases/tag/2.11.0 (2026-10-13)\n  - grpc/grpc-swift-nio-transport#201: Upgrade grpc-swift-2 and swift-nio dependencies (`grpc-swift-2` \u003e\u003d 2.4.0, `swift-nio` \u003e\u003d 2.104.0)\n  - grpc/grpc-swift-nio-transport#200: Declare GRPCNIOTransportCore\u0027s NIOHTTP1 and NIOTLS dependencies\n\nThere is no source change between 2.10.0 and 2.11.0 other than `Package.swift`, and the new minimum `grpc-swift-2` version is satisfied by our existing `2.4.3` pin.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavior change. Only the dependency version and the corresponding `README.md` entry are updated.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nThis patch had conflicts when merged, resolved by\nCommitter: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n\nCloses #592 from dongjoon-hyun/SPARK-60289.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "e4635c545f877a7ee32fe0960c3751097f6d9e07",
      "tree": "89795c19e40a6d4aaa925920f40609339e0657ea",
      "parents": [
        "7493dd7a70fde9e83e0065afec76f2c24aea3dca"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 16 10:24:19 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 16 10:24:19 2026 -0700"
      },
      "message": "[SPARK-60290] Upgrade `gRPC Swift Protobuf` to 2.5.0\n\n### What changes were proposed in this pull request?\n\nThis PR aims to upgrade `gRPC Swift Protobuf` to 2.5.0.\n\n- [2.5.0 (2026-10-14)](https://github.com/grpc/grpc-swift-protobuf/releases/tag/2.5.0)\n  - Support Protobuf Edition 2026 (grpc/grpc-swift-protobuf#106)\n  - Upgrade grpc-swift-2 (grpc/grpc-swift-protobuf#108)\n\nNote that `gRPC Swift Protobuf` 2.5.0 requires `swift-protobuf` 1.39.0+ and `grpc-swift-2` 2.4.0+.\n\n### Why are the changes needed?\n\nTo bring the latest improvements and bug fixes of the dependencies.\n- https://github.com/grpc/grpc-swift-protobuf/compare/2.4.1...2.5.0\n- https://github.com/apple/swift-protobuf/compare/1.38.1...1.39.0\n\nThe checked-in generated code (`*.pb.swift` and `base.grpc.swift`) remains compatible because `swift-protobuf` 1.39.0 keeps the same generated-code API version (`ProtobufAPIVersion_2`).\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavior change. This only updates the dependency version.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nThis patch had conflicts when merged, resolved by\nCommitter: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n\nCloses #593 from dongjoon-hyun/SPARK-60290.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "7493dd7a70fde9e83e0065afec76f2c24aea3dca",
      "tree": "f4993dad2f9b08a958a835b36049639ec1d2932f",
      "parents": [
        "3a8db6493407ea833243484a1079e11d0eaf1037"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 16 09:41:59 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 16 09:41:59 2026 -0700"
      },
      "message": "[SPARK-60292] Upgrade Swift Docker image to 6.4.1\n\n### What changes were proposed in this pull request?\n\nThis PR aims to upgrade the Swift Docker image to 6.4.1.\n\n- `docker run swift:6.4.1` in the CI build and integration test jobs (`build_and_test.yml`)\n- `FROM swift:6.4.1` and the `swift-6.4.1-RELEASE` static Linux SDK URL in the four example Dockerfiles (`app`, `pi`, `spark-sql`, `stream`)\n\nThe minimum requirement (`swift-tools-version: 6.4.0`) is kept as-is because the `xcode-27` macOS runners ship Swift 6.4.0.\n\n### Why are the changes needed?\n\nTo use the latest Swift 6.4.1 release for building and testing.\n- https://github.com/swiftlang/swift/releases/tag/swift-6.4.1-RELEASE (2026-10-15)\n\nSwift 6.4.1 includes optimizer miscompile fixes and Linux fixes that are relevant to the release builds of this project:\n- LoopInvariantCodeMotion: don\u0027t hoist and sink `store [assign]` (swiftlang/swift#92506)\n- AliasAnalysis: consider global effects of a callee if an address is stored as a pointer to an indirect argument (swiftlang/swift#92631)\n- AllocBoxToStack: fix missing `dealloc_stack` for promoted weak capture boxes (swiftlang/swift#92707)\n- Backtracing: fix DWARF parsing issues which crash the backtracer on Linux (swiftlang/swift#92184)\n\nFull commit range: [swift-6.4.0-RELEASE...swift-6.4.1-RELEASE](https://github.com/swiftlang/swift/compare/swift-6.4.0-RELEASE...swift-6.4.1-RELEASE).\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #594 from dongjoon-hyun/SPARK-60292.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "3a8db6493407ea833243484a1079e11d0eaf1037",
      "tree": "afe05017a2c0326fe99e756dcaa8db6daccf726d",
      "parents": [
        "6a1e37a9b19401413b4815aadfd02e6600e2f94b"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 16 09:40:54 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 16 09:40:54 2026 -0700"
      },
      "message": "[SPARK-60288] Upgrade `grpc-swift-2` to 2.4.4\n\n### What changes were proposed in this pull request?\n\nThis PR aims to upgrade `grpc-swift-2` to 2.4.4.\n\n- https://github.com/grpc/grpc-swift-2/releases/tag/2.4.4 (2026-10-13)\n  - https://github.com/grpc/grpc-swift-2/pull/62\n\n### Why are the changes needed?\n\nTo bring the latest bug fixes. `grpc-swift-2` 2.4.4 raises its minimum `swift-protobuf` version to 1.38.0. This avoids SPM dependency resolution failures when an older `swift-protobuf` (before 1.36.0, which has no package traits) gets resolved. There are no source changes in `GRPCCore`.\n\n- https://github.com/grpc/grpc-swift-2/compare/2.4.3...2.4.4\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavior change. Only the dependency version changes.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #591 from dongjoon-hyun/SPARK-60288.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "6a1e37a9b19401413b4815aadfd02e6600e2f94b",
      "tree": "b6f6d7cf5bbdd614e6391562def86bf3ef7bccc1",
      "parents": [
        "4372aad10988b65e3cd925eab2223b41b947b919"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 07 16:16:19 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 07 16:16:19 2026 -0700"
      },
      "message": "[SPARK-60064] Fix DataFrameReader and DataStreamReader to not send empty format when format is not specified\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix `DataFrameReader.load` and `DataStreamReader.load` to set `DataSource.format` only when a format is specified.\n\n### Why are the changes needed?\n\n`DataSource.format` is a proto3 `optional` field. Previously, `load` always assigned `self.source` (initialized to `\"\"`) to it, so the server saw `hasFormat \u003d\u003d true` and called `reader.format(\"\")` instead of falling back to `spark.sql.sources.default`. As a result, `spark.read.load(path)` and `spark.readStream.schema(...).load(path)` without `format(...)` failed.\n\n```\n[DATA_SOURCE_NOT_FOUND] Failed to find the data source: . Make sure the provider name is correct and the package is properly registered and compatible with your Spark version. SQLSTATE: 42K02\n```\n\nThis is consistent with the PySpark Connect client, which sets `data_source.format` only when `format` is not `None`.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. `load` without `format(...)` now uses the default data source (`spark.sql.sources.default`) instead of failing.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases (`DataFrameReaderTests.loadWithoutFormat` and `DataStreamTests.loadWithoutFormat`). Without this fix, the newly added tests fail with the `DATA_SOURCE_NOT_FOUND` error against Apache Spark 4.2.0 Connect server.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #590 from dongjoon-hyun/SPARK-60064.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "4372aad10988b65e3cd925eab2223b41b947b919",
      "tree": "21b347c969718e944ea65d6d388a72c36b77c20e",
      "parents": [
        "b46468db4e7a81aad60e4af5c265a558d240fa45"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 07 16:15:28 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 07 16:15:28 2026 -0700"
      },
      "message": "[SPARK-60063] Fix Catalog.listDatabases/getDatabase/getFunction to not crash on null locationUri and className\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix `Catalog.listDatabases`, `Catalog.getDatabase`, and `Catalog.getFunction` to handle nullable `locationUri` and `className` columns gracefully by falling back to an empty string, like `Catalog.listFunctions` already does.\n\n### Why are the changes needed?\n\nPreviously, these APIs force-cast the `locationUri` and `className` columns with `as! String`, which crashes the client process when the server returns `null`.\n\n- Apache Spark returns `locationUri \u003d null` for V2 catalog namespaces without a location property (e.g., `JDBCTableCatalog`).\n- Apache Spark returns `className \u003d null` for virtual operators such as `case`, `\u003c\u003e`, `!\u003d`, and `||`.\n\n```\nCould not cast value of type \u0027Swift.Optional\u003cAny\u003e\u0027 to \u0027Swift.String\u0027.\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. These APIs now return an empty string instead of crashing. There is no API change because `Database.locationUri` and `Function.className` remain `String`.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases. I also verified them manually against an Apache Spark 4.2.0 Connect server.\n\n```\n$ swift test --no-parallel --filter CatalogTests\n...\nTest run with 41 tests in 1 suite passed\n```\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #589 from dongjoon-hyun/SPARK-60063.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "b46468db4e7a81aad60e4af5c265a558d240fa45",
      "tree": "7073f2ee54880af9adf3cb7e848578ceed5092c3",
      "parents": [
        "dac41bef8197b6869e46ed89b968b02040359eae"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 07 16:14:22 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 07 16:14:22 2026 -0700"
      },
      "message": "[SPARK-60061] Fix Row.getAsTimestampNanos and Row.\u003d\u003d to not crash on out-of-range values\n\n### What changes were proposed in this pull request?\n\nThis PR fixes two crashes in `Row`.\n\n1. `Row.getAsTimestampNanos` converts a `Date` value by rounding to the nearest microsecond and building `TimestampNanos(epochMicros:)`, instead of computing `Int64` epoch nanoseconds. A `Date` that cannot be represented, such as `infinity`, now throws `SparkConnectError.InvalidType`, like the other type-safe getters.\n2. `Row.\u003d\u003d` compares integer values, and arrays of integer values, through `Int128` instead of `Int64`.\n\n### Why are the changes needed?\n\n1. `DataFrame.collect()` returns `TIMESTAMP` and `TIMESTAMP_NTZ` columns as `Date`. `Int64` epoch nanoseconds only cover the years 1677-2262, so `getAsTimestampNanos` crashed on common sentinel values such as `TIMESTAMP\u00270001-01-01\u0027` or `TIMESTAMP\u00279999-12-31\u0027`:\n   ```\n   Fatal error: Double value cannot be converted to Int64 because the result would be less than Int64.min\n   ```\n   The old code also multiplied by `1_000_000_000` in `Double`, which added rounding error and produced wrong sub-microsecond and even microsecond digits. For example, `1_700_000_000.999999` became `999998` microseconds and `976` nanoseconds. `Date` cannot hold nanosecond precision, and Spark `TIMESTAMP` has microsecond precision, so rounding to the nearest microsecond gives the correct value.\n2. `Row(UInt64.max) \u003d\u003d Row(UInt64.max)` crashed because `Int64(UInt64.max)` traps:\n   ```\n   Fatal error: Not enough bits to represent the passed value\n   ```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. These cases no longer crash. `getAsTimestampNanos` on a `Date` value now returns a microsecond-precision `TimestampNanos` (`nanosWithinMicro \u003d\u003d 0`).\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n- `RowTests`: `getAsTimestampNanosOutOfInt64NanosRange`, `getAsTimestampNanosRoundsDateToMicroseconds`, `compareUInt64`, `compareData`\n- `DataFrameTests`: `collectTimestampAsTimestampNanos`, an end-to-end test that collects `TIMESTAMP_NTZ\u00270001-01-01 00:00:00\u0027`, `TIMESTAMP_NTZ\u00272026-01-01 00:00:00.999999\u0027`, and `TIMESTAMP_NTZ\u00279999-12-31 00:00:00\u0027`\n\nI also ran the tests manually against an Apache Spark 4.2.0 Spark Connect server.\n\n```\n$ swift test --no-parallel --filter \u0027RowTests|DataFrameTests\u0027\n...\nTest run with 182 tests in 3 suites passed after 35.407 seconds.\n```\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.\n\nCloses #588 from dongjoon-hyun/SPARK-60061.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "dac41bef8197b6869e46ed89b968b02040359eae",
      "tree": "317afb107971c4819c7bc5db317a4f0799600658",
      "parents": [
        "1187c9b6bacdd8db07fc7918d5c9e8d19576d017"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 07 16:13:27 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 07 16:13:27 2026 -0700"
      },
      "message": "[SPARK-60060] Make StreamingQueryStatus and StreamingQueryException properties public\n\n### What changes were proposed in this pull request?\n\nThis PR aims to make the stored properties of `StreamingQueryStatus` (`statusMessage`, `isDataAvailable`, `isTriggerActive`, `isActive`) and `StreamingQueryException` (`exceptionMessage`, `errorClass`, `stackTrace`) `public`, with brief doc comments.\n\n### Why are the changes needed?\n\n`StreamingQueryStatus` and `StreamingQueryException` are `public` structs returned by the public APIs `StreamingQuery.status()` and `StreamingQuery.exception()`, but their properties were `internal`. As a result, applications outside the `SparkConnect` module could not read any of the returned fields. For example, the following code failed to compile.\n\n```swift\nlet status \u003d try await query.status()\nprint(status.isActive)  // error: \u0027isActive\u0027 is inaccessible due to \u0027internal\u0027 protection level\n```\n\nThis aligns with PySpark, where `StreamingQuery.status` and `StreamingQuery.exception()` expose this information to users.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. Users can now read the fields of `StreamingQueryStatus` and `StreamingQueryException`. This is an additive API change only.\n\n### How was this patch tested?\n\nPass the CIs with the newly added `StreamingQueryTests.status` test case.\n\nIn addition, I manually verified that an out-of-module Swift package depending on `SparkConnect` fails to compile before this PR and compiles after this PR when it accesses these properties.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #587 from dongjoon-hyun/SPARK-60060.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "1187c9b6bacdd8db07fc7918d5c9e8d19576d017",
      "tree": "3b3cffa886aed326e23108e2514b34609f830621",
      "parents": [
        "91e7b8dc3c3a4cb31cae03b920607737404348ca"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 07 14:43:55 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 07 14:43:55 2026 -0700"
      },
      "message": "[SPARK-60055] Fix `dropDuplicatesWithinWatermark` to set `withinWatermark` in `Deduplicate` relation\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix `SparkConnectClient.getDropDuplicates` to set the `within_watermark` field of the `Deduplicate` relation by adding `deduplicate.withinWatermark \u003d withinWatermark`.\n\n### Why are the changes needed?\n\nPreviously, `getDropDuplicates` accepted the `withinWatermark` parameter but never propagated it to `Spark_Connect_Deduplicate`. As a result, the public `DataFrame.dropDuplicatesWithinWatermark(...)` API silently behaved like `dropDuplicates(...)`. For streaming queries, this gives the wrong semantics: the server keeps unbounded deduplication state instead of the watermark-bounded state of `DeduplicateWithinWatermark`.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, this is a bug fix. `dropDuplicatesWithinWatermark` now sends `within_watermark \u003d true` to the Spark Connect server, consistent with the Scala and PySpark Spark Connect clients. As a result, calling it on a batch `DataFrame` now fails with `dropDuplicatesWithinWatermark is not supported with batch DataFrames/DataSets`, the same as other Spark clients. Previously, it silently behaved like `dropDuplicates`.\n\n### How was this patch tested?\n\nPass the CIs with the newly added plan-level test case, `DataFrameInternalTests.dropDuplicatesWithinWatermarkPlan`, which does not require a Spark Connect server. It fails without this fix and passes with it.\n\n```\n$ swift test --no-parallel --filter DataFrameInternalTests/dropDuplicatesWithinWatermarkPlan\n```\n\nIn addition, the existing `DataFrameTests.dropDuplicatesWithinWatermark` and `DataFrameTests.withWatermark` test cases are updated to expect an error for batch `DataFrame`s.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #586 from dongjoon-hyun/SPARK-60055.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "91e7b8dc3c3a4cb31cae03b920607737404348ca",
      "tree": "d85183c6617b603e6f1a5d53d627464f711cc3ff",
      "parents": [
        "be871bbc269b9c081d58e02442994ed5557b05a2"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 07 13:45:09 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 07 13:45:09 2026 -0700"
      },
      "message": "[SPARK-60054] Fix `DataFrame.hint` to not crash on out-of-range integers and unsupported parameter types\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix `DataFrame.hint` so that it no longer crashes on out-of-range integers or unsupported parameter types. The public API stays non-throwing.\n\n`SparkConnectClient.getHint` now converts hint parameters as follows:\n- `Int64` and `Int` values within the `Int32` range are still sent as integer literals, because Spark\u0027s `ResolveHints` expects an `IntegerLiteral` for partition numbers. Values outside that range are sent as long literals, so the Spark Connect server reports the error. This matches how PySpark infers integer literals.\n- Any other type goes through the existing `ExpressionLiteral.init(_:)` conversion, so `Double`, `Float`, `Decimal` and so on are now supported.\n- Types that cannot be converted are sent as a string literal of their description instead of being force-cast with `as! String`.\n\n### Why are the changes needed?\n\n`getHint` used `Int32(value)` and `$0 as! String`, which trap and crash the whole client process. For example:\n\n```swift\ndf.hint(\"rebalance\", 3_000_000_000)  // crash: Int32 overflow\ndf.hint(\"x\", 1.5)                    // crash: Could not cast value of type \u0027Double\u0027 to \u0027String\u0027\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, it fixes a bug. These calls used to crash the process. Now they build a valid plan, and invalid hints are reported by the Spark Connect server instead. The API signature does not change.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases (`SparkConnectClientTests.getHintWithIntegers` and `SparkConnectClientTests.getHintWithOtherTypes`).\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #585 from dongjoon-hyun/SPARK-60054.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "be871bbc269b9c081d58e02442994ed5557b05a2",
      "tree": "be4b0635c470263258318b944d64acc9b434a6c0",
      "parents": [
        "e703d2f8aa3685565970009238a8ac4fb1678857"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 07 13:44:10 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 07 13:44:10 2026 -0700"
      },
      "message": "[SPARK-60053] Fix `MergeIntoWriter.update` to pass assignments to the merge action\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix `MergeIntoWriter.update` to pass the given column assignments (`map`) to `buildMergeAction`.\n\n### Why are the changes needed?\n\nPreviously, `update(_:_:_:)` called `buildMergeAction(ActionType.update, condition)` without `map`, so `assignments` silently defaulted to `[:]`. As a result, the `UPDATE` merge action was sent to the server without any assignments, and `whenMatched().update(map:)` and `whenNotMatchedBySource().update(map:)` ignored the user-provided assignments. In contrast, `insert(_:_:)` passes `map` correctly.\n\n```swift\nawait mergeInto.whenMatched().update(map: [\"t.v\": \"s.v\"])  // assignments were dropped\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. `whenMatched().update(map:)` and `whenNotMatchedBySource().update(map:)` now send the given column assignments as intended.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test case, `MergeIntoWriterTests.updateAssignments`. It checks the built `MergeIntoTableCommand` and does not need a Spark Connect server.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #584 from dongjoon-hyun/SPARK-60053.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "e703d2f8aa3685565970009238a8ac4fb1678857",
      "tree": "f3abb1daee7aca494afcc759dbf03393f124eb9d",
      "parents": [
        "d86f43b28fad64e92a9f5651cf283a176d8e1162"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 07 13:11:52 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Oct 07 13:11:52 2026 -0700"
      },
      "message": "[SPARK-60051] Fix `DataFrame.collect()` to return `Data` for `BINARY` columns instead of crashing\n\n### What changes were proposed in this pull request?\n\nThis PR fixes `DataFrame.collect()` to read `BINARY` column values directly from `BinaryArray` as `Data`, instead of going through `BinaryArray.asString` and returning a `String.UTF8View`. `Row` equality now also compares `Data` values.\n\n### Why are the changes needed?\n\n`BinaryArray.asString` force-unwraps `String(data:encoding: .utf8)`, so collecting any binary value that is not valid UTF-8 (e.g. `X\u0027FF00FE\u0027`) crashes the process. Returning `Data` also matches PySpark, which returns `bytearray`.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. `BINARY` values in collected rows are now `Data` instead of `String.UTF8View`, and non-UTF-8 binary values no longer crash.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Sonnet 5.5\n\nCloses #583 from dongjoon-hyun/claude/fix-binary-collect.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "d86f43b28fad64e92a9f5651cf283a176d8e1162",
      "tree": "d39446af031682ac9c19acd8aa3323a920de45a5",
      "parents": [
        "f1e2187db0019e4331559c79b708cfa7c9088412"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 02 12:41:38 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Oct 02 12:41:38 2026 -0700"
      },
      "message": "[SPARK-59956] Use 4.0.5 RC1 for Spark 4.0 integration tests\n\n### What changes were proposed in this pull request?\n\nThis PR aims to use `Apache Spark 4.0.5 RC1` for the `integration-test-mac` job.\n\n### Why are the changes needed?\n\nTo test the upcoming `Apache Spark 4.0.5` release candidate in advance.\n- https://github.com/apache/spark/releases/tag/v4.0.5-rc1 (2026-10-01)\n- https://dist.apache.org/repos/dist/dev/spark/v4.0.5-rc1-bin/\n\nNote that `Apache Spark 4.0.4` is still tested by `integration-test-ubuntu` and `integration-test-mac-spark4-iceberg`.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a test infra change.\n\n### How was this patch tested?\n\nPass the CIs and check the log.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Sonnet 5.5\n\nCloses #582 from dongjoon-hyun/SPARK-59956.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "f1e2187db0019e4331559c79b708cfa7c9088412",
      "tree": "e88b514bc14c885e81dbbd2d9bd54063c11213e8",
      "parents": [
        "ee0b8ace6f89433c3ebb080cdcd1df5aae0b846a"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Oct 01 17:17:20 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Oct 01 17:17:20 2026 -0700"
      },
      "message": "[SPARK-59940] Use Swift 6.4 as the minimum version and unify CI on `xcode-27`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to use Swift 6.4 as the minimum version and unify CI on `xcode-27`.\n\n- Bump `swift-tools-version` from `6.3.2` to `6.4.0` and update the documentation (`README.md`, `AGENTS.md`, `GettingStarted.md`).\n- Remove duplicate CI jobs: `build-macos-26-swift63` and `integration-test-mac-spark42-swift64`.\n- Move the remaining macOS jobs from `macos-26` (Xcode 26.6) to `xcode-27`.\n\n### Why are the changes needed?\n\n[Swift 6.4.0](https://www.swift.org/blog/swift-6.4-released/) was released on September 15, 2026. The Linux CI jobs and `Examples/*` already use it, so the minimum version should match what CI actually tests. Note that `xcode-27` is still in [public preview](https://github.blog/changelog/2026-09-10-xcode-27-runner-image-now-runs-on-macos-27) (as of September 10, 2026).\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. Building the package now requires Swift 6.4 or later. There is no API or behavior change.\n\n### How was this patch tested?\n\nVerified locally with Swift 6.4.0 (`swift build -c release`). The `xcode-27` jobs are verified by this PR\u0027s CI.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Sonnet 5.5\n\nCloses #581 from dongjoon-hyun/SPARK-59940.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "ee0b8ace6f89433c3ebb080cdcd1df5aae0b846a",
      "tree": "5e9e101befb9cb40a3c7216c24b2f61dbba2d987",
      "parents": [
        "d7ff8aa042e481eb69e4659df5d5049891a99dc2"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 30 13:59:58 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 30 13:59:58 2026 -0700"
      },
      "message": "[SPARK-59904] Upgrade Iceberg to 1.12.0 in integration tests\n\n### What changes were proposed in this pull request?\n\nThis PR aims to upgrade `Iceberg` to 1.12.0 in integration tests.\n\n### Why are the changes needed?\n\nTo test against the latest Apache Iceberg release.\n- https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.12.0 (2026-09-30)\n  - https://github.com/apache/iceberg/pull/14984\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a test-only change.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #580 from dongjoon-hyun/SPARK-59904.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "d7ff8aa042e481eb69e4659df5d5049891a99dc2",
      "tree": "fbfca59b75872ac1470c257cf61b6c5f3487e61a",
      "parents": [
        "bf312a530a1c373636cc53b32703ed8c089a3bf7"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Sep 29 10:50:41 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Sep 29 10:50:41 2026 -0700"
      },
      "message": "[SPARK-59873] Use `ubuntu-26.04` instead of `ubuntu-latest` in GitHub Actions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to use `ubuntu-26.04` (and `ubuntu-26.04-arm`) instead of `ubuntu-latest` (and `ubuntu-24.04-arm`) in GitHub Actions. The `build-ubuntu-latest` job is also renamed to `build-ubuntu`.\n\n### Why are the changes needed?\n\nThe [Ubuntu 26 runner image became generally available](https://github.blog/changelog/2026-09-17-ubuntu-26-generally-available-and-latest-migration) (2026-09-17), and `ubuntu-latest` will be migrated from 24.04 to 26.04 between 2026-10-19 and 2026-11-19. Pinning it explicitly lets us validate the change in advance and keeps the x64 and arm64 jobs on the same OS version.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs. (`publish_image.yml` runs only via `workflow_dispatch` on `main`, so it is not covered by the PR CI.)\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Sonnet 5.5\n\nCloses #579 from dongjoon-hyun/SPARK-59873.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "bf312a530a1c373636cc53b32703ed8c089a3bf7",
      "tree": "bcbaa91447abd3ebbdcf0b42152aff2ef406bd29",
      "parents": [
        "5063acd0bec0d47dd0242a090f40d8549cdf00ce"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Sep 29 09:51:46 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Sep 29 09:51:46 2026 -0700"
      },
      "message": "[SPARK-59867] Update GitHub Actions to the latest versions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to update GitHub Actions to their latest releases.\n\n| Action | Before | After |\n|---|---|---|\n| `apache/skywalking-eyes` | v0.8.0 | [v0.9.0](https://github.com/apache/skywalking-eyes/releases/tag/v0.9.0) (2026-09-01) |\n| `actions/cache` | v4 | [v6](https://github.com/actions/cache/releases/tag/v6.1.0) (v6.1.0, 2026-06-26) |\n| `actions/checkout` | v6 (1 place) | [v7](https://github.com/actions/checkout/releases/tag/v7.0.1) (v7.0.1, 2026-07-20) |\n| `docker/setup-qemu-action` | v4.2.0 | [v4.4.0](https://github.com/docker/setup-qemu-action/releases/tag/v4.4.0) (2026-09-15) |\n| `docker/setup-buildx-action` | v4.3.0 | [v4.4.1](https://github.com/docker/setup-buildx-action/releases/tag/v4.4.1) (2026-09-16) |\n| `docker/build-push-action` | v7.3.0 | [v7.4.0](https://github.com/docker/build-push-action/releases/tag/v7.4.0) (2026-09-15) |\n\nThe other actions (`actions/setup-java`, `actions/github-script`, `docker/login-action`) are already up to date.\n\n### Why are the changes needed?\n\nTo keep the CI up to date. The `docker/*` actions stay pinned to commit hashes listed in the ASF [approved_patterns.yml](https://github.com/apache/infrastructure-actions/blob/main/approved_patterns.yml).\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nVerified that each pinned commit hash matches its release tag and that the `docker/*` hashes are in `approved_patterns.yml`. The updated workflows will be exercised by the CI of this PR.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Sonnet 5.5\n\nCloses #578 from dongjoon-hyun/SPARK-59867.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "5063acd0bec0d47dd0242a090f40d8549cdf00ce",
      "tree": "76c2c317eac0254ece17e74bdc96194d50dfdc08",
      "parents": [
        "32390a97e907dbca4f935ab5000d1434e096f354"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Sep 29 09:51:08 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Sep 29 09:51:08 2026 -0700"
      },
      "message": "[SPARK-59864] Use 4.1.4 RC1 for Spark 4.1 integration tests\n\n### What changes were proposed in this pull request?\n\nThis PR aims to use `Apache Spark 4.1.4 RC1` for the `integration-test-mac-spark41` job.\n\n### Why are the changes needed?\n\nTo test the upcoming `Apache Spark 4.1.4` release candidate in advance.\n- https://github.com/apache/spark/releases/tag/v4.1.4-rc1 (2026-09-28)\n- https://dist.apache.org/repos/dist/dev/spark/v4.1.4-rc1-bin/\n\nNote that `Apache Spark 4.1.3` is still tested by `integration-test-ubuntu-spark41`, `integration-test-token`, and `integration-test-mac-spark41-iceberg`.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a test infra change.\n\n### How was this patch tested?\n\nPass the CIs and check the log.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #577 from dongjoon-hyun/SPARK-59864.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "32390a97e907dbca4f935ab5000d1434e096f354",
      "tree": "bcb3ad0a8952740d1d8833f7200653bddfda0767",
      "parents": [
        "23aa5d9b9169dfa11d8e9c2b75594d517eda1391"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Sep 29 09:50:33 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Sep 29 09:50:33 2026 -0700"
      },
      "message": "[SPARK-59863] Use 4.2.1 RC1 for Spark 4.2 integration tests\n\n### What changes were proposed in this pull request?\n\nThis PR updates the `integration-test-mac-spark42` CI job to use `Apache Spark 4.2.1` RC1.\n\n### Why are the changes needed?\n\nTo test against the latest `Apache Spark 4.2.1` RC1.\n- https://github.com/apache/spark/releases/tag/v4.2.1-rc1 (2026-09-28)\n- https://dist.apache.org/repos/dist/dev/spark/v4.2.1-rc1-bin/\n\nNote that `Apache Spark 4.2.0` is still tested by `integration-test-ubuntu-spark42` and `integration-test-mac-spark42-swift64`.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs and check the log.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #576 from dongjoon-hyun/SPARK-59863.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "23aa5d9b9169dfa11d8e9c2b75594d517eda1391",
      "tree": "0ffa16c32ad2c9051b66cdb063227aaaf7baa35e",
      "parents": [
        "eace5deb15895c669a5ad2f0c7a60cfbe87bfce9"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 23 08:50:49 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 23 08:50:49 2026 -0700"
      },
      "message": "[SPARK-59735] Use `static-debian13` distroless image in `Examples`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to upgrade the base image of the Swift examples (`app`, `pi`, `spark-sql`, `stream`) from `gcr.io/distroless/static-debian12:nonroot` to `gcr.io/distroless/static-debian13:nonroot`.\n\n### Why are the changes needed?\n\nTo use the latest Debian 13 (Trixie)-based distroless image.\n\nSince the example binaries are fully statically linked via the Swift Static Linux SDK (`*-swift-linux-musl`), they don\u0027t depend on the base image\u0027s libc. The two images have the same runtime configuration:\n\n| | `static-debian12:nonroot` | `static-debian13:nonroot` |\n|---|---|---|\n| `linux/amd64`, `linux/arm64` | Yes | Yes |\n| User | `65532` | `65532` |\n| WorkingDir | `/home/nonroot` | `/home/nonroot` |\n| `SSL_CERT_FILE` | `/etc/ssl/certs/ca-certificates.crt` | `/etc/ssl/certs/ca-certificates.crt` |\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavior change. Only the base image of the example Docker images is changed.\n\n### How was this patch tested?\n\nManually built all four example images and ran the `pi` example against an Apache Spark 4.2.0 Spark Connect server.\n\n```\n$ docker build -t spark-connect-swift:pi Examples/pi\n$ docker network create sc\n$ docker run -d --rm --name spark --network sc -e SPARK_NO_DAEMONIZE\u003d1 apache/spark:4.2.0 bash -c /opt/spark/sbin/start-connect-server.sh\n$ docker run --rm --network sc -e SPARK_REMOTE\u003dsc://spark:15002 spark-connect-swift:pi\nPi is roughly 3.142036\n```\n\nThe image size increases by about 141KB (`linux/arm64`, Docker 29.5.3). The `docker images` example outputs in `Examples/*/README.md` (`112MB` / `32MB`) remain valid.\n\n| Image | Base | DISK USAGE | CONTENT SIZE | Size (bytes) |\n|---|---|---|---|---|\n| `pi` | `static-debian12:nonroot` | 112MB | 32.1MB | 32,049,596 |\n| `pi` | `static-debian13:nonroot` | 112MB | 32.2MB | 32,190,818 |\n| `app`, `spark-sql`, `stream` | `static-debian13:nonroot` | 112MB | 32.2MB | - |\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #574 from dongjoon-hyun/SPARK-59735.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "eace5deb15895c669a5ad2f0c7a60cfbe87bfce9",
      "tree": "3fc8570dd7250b35b4f9ac12f31facc99913e83e",
      "parents": [
        "2c95a5d31da7f5a84ede38b90ccb821905c91aaa"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 23 08:47:47 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 23 08:47:47 2026 -0700"
      },
      "message": "[SPARK-59736] Use `Swift 6.4` Docker image in all Linux CI jobs\n\n### What changes were proposed in this pull request?\n\nThis PR aims to use the official `swift:6.4.0` Docker image in all Linux CI jobs.\n\n- `build-ubuntu-latest`, `build-ubuntu-arm`: `swift:6.3.3` → `swift:6.4.0`\n- `integration-test-ubuntu`, `integration-test-ubuntu-spark41`, `integration-test-ubuntu-spark42`: `swift:6.3.3` → `swift:6.4.0`\n- Remove `build-ubuntu-swift64` because it becomes identical to `build-ubuntu-latest`.\n\n### Why are the changes needed?\n\nWe had better build and test with the latest Swift version on Linux.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a CI-only change.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #575 from dongjoon-hyun/SPARK-59736.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "2c95a5d31da7f5a84ede38b90ccb821905c91aaa",
      "tree": "8984505a41d059a29d03320cbb826ca304940fe4",
      "parents": [
        "83a91cd376bde99fe0dc3549f23dd6e61f9dcbe6"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 20 13:32:42 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 20 13:32:42 2026 -0700"
      },
      "message": "[SPARK-59675] Migrate `Examples` to Swift 6.4\n\n### What changes were proposed in this pull request?\n\nThis PR migrates all Swift `Examples` (`app`, `pi`, `spark-sql`, `stream`) to Swift 6.4.\n\n- `Package.swift` (4 files): `swift-tools-version: 6.3.2` → `6.4.0`\n- `Dockerfile` (4 files): `swift:6.3.3` → `swift:6.4.0` and the `swift-6.3.3-RELEASE` → `swift-6.4.0-RELEASE` static Linux SDK\n- `Dockerfile` (4 files): `.build/*-swift-linux-musl/release/\u003cexecutable\u003e` → `.build/release/\u003cexecutable\u003e`\n- `README.md` (4 files): update the `docker images` example output\n\nThe third item is required because Swift 6.4 uses `swiftbuild` as the default build system, which moved the products from `.build/aarch64-swift-linux-musl/release` to `.build/out/Products/Release-staticlinux-aarch64`. `.build/release` is a symlink to the new location and matches `swift build --show-bin-path`.\n\n### Why are the changes needed?\n\n[Swift 6.4.0](https://github.com/swiftlang/swift/releases/tag/swift-6.4.0-RELEASE) (2026-09-15) has been released and CI already builds and tests with it. We had better recommend users the latest Swift version.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is an example-only change.\n\n### How was this patch tested?\n\nPass the CIs.\n\nIn addition, I built all four examples with Swift 6.4.0 (`swiftlang-6.4.0.34.1`) without any warning, and built all four Docker images to confirm that the statically linked `musl` binaries run on the `distroless` image. The image sizes in the `README.md` files come from these builds.\n\n```\n$ docker images | grep spark-connect-swift\napache/spark-connect-swift:app         112MB    32MB\napache/spark-connect-swift:pi          112MB    32MB\napache/spark-connect-swift:spark-sql   112MB    32MB\napache/spark-connect-swift:stream      112MB    32MB\n```\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #573 from dongjoon-hyun/SPARK-59675.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "83a91cd376bde99fe0dc3549f23dd6e61f9dcbe6",
      "tree": "94bb686f10d6000611f6c9cf8fed847c30c1cb25",
      "parents": [
        "27518265329b0f9d5c768d8f5bd06e27988f37fa"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Sep 19 16:01:41 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Sep 19 16:01:41 2026 -0700"
      },
      "message": "[SPARK-59664] Use official `Swift 6.4` Docker image instead of nightly\n\n### What changes were proposed in this pull request?\n\nThis PR aims to use the official `swift:6.4.0` Docker image instead of the nightly `swiftlang/swift:nightly-6.4.x-amazonlinux2023` image in CI, and to build with `-c release` like the other Ubuntu jobs.\n\n### Why are the changes needed?\n\nSwift 6.4.0 was officially released and is available as an official Docker image. Using the released image makes the CI result reproducible, while the nightly image keeps changing.\n\n- https://www.swift.org/blog/swift-6.4-released/\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a CI-only change.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #572 from dongjoon-hyun/SPARK-59664.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "27518265329b0f9d5c768d8f5bd06e27988f37fa",
      "tree": "a9a0b9eb1b45b4dce06ace9d0fac8c3b573431ba",
      "parents": [
        "2dd11124c0e0bb8cf758a3206f57805a8eead4d6"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 18 11:23:43 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 18 11:23:43 2026 -0700"
      },
      "message": "[SPARK-59646] Upgrade `gRPC Swift NIO Transport` to 2.10.0\n\n### What changes were proposed in this pull request?\n\nThis PR upgrades the `grpc-swift-nio-transport` dependency to `2.10.0`.\n\n### Why are the changes needed?\n\nTo adopt the latest `grpc-swift-nio-transport` release (`2.10.0`, 2026-09-17).\n- https://github.com/grpc/grpc-swift-nio-transport/releases/tag/2.10.0 (2026-09-17)\n  - https://github.com/grpc/grpc-swift-nio-transport/pull/195 (Expose the HTTP/2 receive header list size limit)\n  - https://github.com/grpc/grpc-swift-nio-transport/pull/194 (Compute the transport-specific context once per connection)\n  - https://github.com/grpc/grpc-swift-nio-transport/pull/190 (docs/fixups)\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. The newly exposed `maxHeaderListSize` defaults to 16 KiB, which is the same as the previous NIO default, and `SparkConnectClient` uses the default transport configuration.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #571 from dongjoon-hyun/SPARK-59646.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "2dd11124c0e0bb8cf758a3206f57805a8eead4d6",
      "tree": "f90640a4bc12398a29412afd177308351e80a3bf",
      "parents": [
        "a910bb97983933af55b2e5b95592654d5ba3cfa1"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 16 12:34:00 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 16 12:34:00 2026 -0700"
      },
      "message": "[SPARK-59592] Use Java 21 at least in CI\n\n### What changes were proposed in this pull request?\n\nThis PR aims to use Java 21 at least in CI.\n- `integration-test-ubuntu`: `apache/spark:4.0.4-scala` (Java 17) -\u003e `apache/spark:4.0.4-java21-scala`\n- `integration-test-mac-spark4-iceberg`: `java-version: 17` -\u003e `java-version: 21`\n\n### Why are the changes needed?\n\nTo test against Java 21 or newer only. The other jobs already use Java 21+:\n- `apache/spark:4.1.3-scala` is a Java 21 image.\n- `apache/spark:4.2.0-scala-java25` uses Java 25.\n- The default JDK of `macos-26` runners is Java 21.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is a CI-only change.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #570 from dongjoon-hyun/SPARK-59592.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "a910bb97983933af55b2e5b95592654d5ba3cfa1",
      "tree": "b4042b8bbb41a4b1c192ffe106b03bc58d6db81f",
      "parents": [
        "7751e91e1c8b5bdb2a8814141c3bcc0f712dab22"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 16 11:52:49 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 16 11:52:49 2026 -0700"
      },
      "message": "[SPARK-59591] Add integration tests with Swift 6.4\n\n### What changes were proposed in this pull request?\n\nThis PR aims to add a new GitHub Actions job, `integration-test-mac-spark42-swift64`, which runs the integration test suite with Swift 6.4 against Apache Spark 4.2.0.\n\nThe new job is identical to the existing `integration-test-mac-spark42` job except for the following:\n- It runs on the `xcode-27` runner, like `build-macos-26-swift64`.\n- It has no `Select Xcode 26.6` step because the `xcode-27` runner uses Swift 6.4 by default.\n\n### Why are the changes needed?\n\n[Swift 6.4.0](https://github.com/swiftlang/swift/releases/tag/swift-6.4.0-RELEASE) (2026-09-15) has been released. Currently, CI only builds with Swift 6.4 (`build-macos-26-swift64` and `build-ubuntu-swift64`), and all integration tests run with Swift 6.3. This PR prepares for raising the minimum Swift version to 6.4 later by running the integration tests with Swift 6.4 too.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is a test-only CI change.\n\n### How was this patch tested?\n\nPass the CIs with the newly added `integration-test-mac-spark42-swift64` job.\n\nIn addition, I ran the tests manually with Xcode 27 (Swift 6.4) against an Apache Spark 4.2.0 server started with `-c spark.sql.timeType.enabled\u003dtrue`. All 658 tests in 63 suites passed.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #569 from dongjoon-hyun/SPARK-59591.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "7751e91e1c8b5bdb2a8814141c3bcc0f712dab22",
      "tree": "6968fe7a49fd5ceafa944ee596aa7a95740931c2",
      "parents": [
        "c35e69503b693742c0d007e5b87eb3fa453c0500"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 16 11:43:16 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 16 11:43:16 2026 -0700"
      },
      "message": "[SPARK-59590] Remove unnecessary `try` keywords reported by Swift 6.4\n\n### What changes were proposed in this pull request?\n\nThis PR aims to remove unnecessary `try` keywords reported by `Swift 6.4`.\n\n### Why are the changes needed?\n\nTo remove the new `UnnecessaryEffectMarker` warnings of [Swift 6.4 (2026-09-15)](https://github.com/swiftlang/swift/releases/tag/swift-6.4.0-RELEASE). `ArrowReader.swift` now matches the upstream `apache/arrow-swift` code.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #568 from dongjoon-hyun/SPARK-59590.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "c35e69503b693742c0d007e5b87eb3fa453c0500",
      "tree": "8bf168e6e6bcd30372e9b5d3e354cbaae49c181b",
      "parents": [
        "5057ddd73cb5ee438270b8cd1bdbffcfef3d35a0"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 16 11:42:09 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 16 11:42:09 2026 -0700"
      },
      "message": "[SPARK-59583] Upgrade `actions/checkout` to v7\n\n### What changes were proposed in this pull request?\n\nThis PR aims to upgrade `actions/checkout` to v7.\n\n### Why are the changes needed?\n\nTo use the latest major version with safer defaults:\n- https://github.com/actions/checkout/releases/tag/v7.0.0 (2026-06-18)\n  - https://github.com/actions/checkout/pull/2454\n- https://github.com/actions/checkout/releases/tag/v7.0.1 (2026-07-20)\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #567 from dongjoon-hyun/SPARK-59583.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "5057ddd73cb5ee438270b8cd1bdbffcfef3d35a0",
      "tree": "078864d18f98a16cbb90f7c88e39de4a47839b5c",
      "parents": [
        "c0bf10a1480f6a6d5a94ed9ea1e565a9ef7118cc"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 23:39:15 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 23:39:15 2026 -0700"
      },
      "message": "[SPARK-59485] Use server-side aggregation for `DataFrame.count()`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to compute `DataFrame.count()` on the server side via `groupBy().count()`, like the Scala Spark Connect client.\n\nThe previous client-side counting logic remains as an internal `executeAndCount()` for `Catalog` operations and `isEmpty()`.\n\n### Why are the changes needed?\n\nTo match [Dataset.count()](https://github.com/apache/spark/blob/v4.2.0/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala#L1041-L1043) of the Scala Spark Connect client in Apache Spark [v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0) (2026-07-11), and to avoid sending every row to the client just to count them.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. The results are the same, but the server now computes the count instead of sending every row to the client. As in the Scala client, the server can skip expressions that counting doesn\u0027t need. For example, `select(assert_true(lit(false))).count()` no longer raises an error.\n\n### How was this patch tested?\n\nPass the CIs with the newly added and updated test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #566 from dongjoon-hyun/SPARK-59485.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "c0bf10a1480f6a6d5a94ed9ea1e565a9ef7118cc",
      "tree": "c81cb318185d81cf814072c087c7791ec47a3344",
      "parents": [
        "e3427aac36b31c7f53d119c06563c7a10e17b8d1"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 23:38:30 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 23:38:30 2026 -0700"
      },
      "message": "[SPARK-59484] Use `value` as the output column name of `DataFrame.toJSON`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to make `DataFrame.toJSON()` return a single column named `value`, like the Scala Spark Connect client.\n\n```swift\n-    return selectExpr(\"to_json(struct(*))\")\n+    return selectExpr(\"to_json(struct(*)) AS value\")\n```\n\n### Why are the changes needed?\n\nThe Scala Spark Connect client names the `toJSON` output column `value`.\n\n- [Apache Spark 4.2.0 (2026-07-11) `Dataset.toJSON`](https://github.com/apache/spark/blob/v4.2.0/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala#L1277-L1279)\n\n```scala\ndef toJSON: Dataset[String] \u003d {\n  select(to_json(struct(col(\"*\"))).as(\"value\")).as(StringEncoder)\n}\n```\n\nThe Swift client currently names the column after the expression, such as `to_json(struct(id))`. So the column name depends on the input schema, and code ported from Scala that refers to `value` fails.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. The output column is renamed from `to_json(struct(...))` to `value`, matching the Scala Spark Connect client. The JSON content is unchanged.\n\n**BEFORE**\n\n```swift\ntry await spark.range(2).toJSON().columns  // [\"to_json(struct(id))\"]\n```\n\n**AFTER**\n\n```swift\ntry await spark.range(2).toJSON().columns  // [\"value\"]\ntry await spark.range(2).toJSON().schema   // value STRING (nullable)\n```\n\n### How was this patch tested?\n\nPass the CIs with the updated test case. It checks the column name, the schema (`value STRING`, nullable), unchanged JSON content, a multi-column DataFrame, and a nested struct column.\n\nI also tested manually with an Apache Spark 4.2.0 Spark Connect server.\n\n```\n$ swift test --no-parallel --filter DataFrameTests\n...\nTest run with 152 tests in 2 suites passed after 22.965 seconds.\n```\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #565 from dongjoon-hyun/SPARK-59484.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "e3427aac36b31c7f53d119c06563c7a10e17b8d1",
      "tree": "3513c2bad92774e7c9328eed55b7a05a85deba85",
      "parents": [
        "e6d727931c4ab82dc33e98c3ba6cc2d9952b67b4"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 21:00:10 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 21:00:10 2026 -0700"
      },
      "message": "[SPARK-59483] Fix `DataFrame.(head|first)` to throw an error instead of crashing on empty results\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix `DataFrame.head()` and `first()` to throw `SparkConnectError.invalidState` instead of crashing on empty results.\n\n### Why are the changes needed?\n\nCurrently, `head()` and `first()` crash on an empty `DataFrame`.\n\n```swift\ntry await spark.range(0).head()  // Fatal error: Index out of range\n```\n\nApache Spark\u0027s `Dataset.head()` throws `NoSuchElementException: head of empty array` in this case ([v4.2.0](https://github.com/apache/spark/blob/v4.2.0/sql/api/src/main/scala/org/apache/spark/sql/Dataset.scala#L2786), 2026-07-11). This PR uses the same message.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavior change. Previously, `head()` and `first()` crashes and now they throw an error on an empty `DataFrame` . Both methods are already `async throws`, so this is source-compatible.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #564 from dongjoon-hyun/SPARK-59483.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "e6d727931c4ab82dc33e98c3ba6cc2d9952b67b4",
      "tree": "1322831e8647560cab37b833240682f4dafa8a50",
      "parents": [
        "28d0178805e267fe091b7108523da6d17da213f9"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 18:02:24 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 18:02:24 2026 -0700"
      },
      "message": "[SPARK-59481] Fix concurrent commands on the same session to return correct responses\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix `SparkConnectClient.execute` to collect responses per call via a local `Mutex` instead of the actor-shared `result` array.\n\n### Why are the changes needed?\n\nDue to actor reentrancy, concurrent commands on the same session can return the responses of other commands. For example, concurrent `DataStreamWriter.start` calls return the same `StreamingQuery`.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, this fixes concurrent commands on the same `SparkSession`. There is no API change.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test case.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #563 from dongjoon-hyun/SPARK-59481.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "28d0178805e267fe091b7108523da6d17da213f9",
      "tree": "991ce79b3d2fe6be74a6096c3a99d93a882ebaad",
      "parents": [
        "bd359a485d419dd8ece6694ded18dedc17a5e13b"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 16:04:30 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 16:04:30 2026 -0700"
      },
      "message": "[SPARK-59480] Fix concurrent actions on the same DataFrame to return correct results\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix `DataFrame` actions to use a per-call result instead of the actor-shared `batches` buffer, so that concurrent actions on the same `DataFrame` instance return correct results.\n\n- `DataFrame.execute()` now accumulates `RecordBatch`es into a local `Mutex` and returns them.\n- `collect()` and `collect(as:)` iterate the returned batches.\n- The unused `DataFrame.batches` and `addBatches` are removed.\n\n`head`, `take`, `first`, `tail`, and `show` create a new `DataFrame` and call `collect()`, so they are fixed together. `count()` already uses a local `Atomic` counter and is not affected.\n\n### Why are the changes needed?\n\n`execute()` cleared the actor-shared `batches` and then refilled it across `await` suspension points. Due to actor reentrancy, concurrent actions on the same `DataFrame` interleave and append into the same buffer, so they silently return wrong results.\n\n```swift\nlet df \u003d try await spark.range(300_000)\nasync let a \u003d df.collect()\nasync let b \u003d df.collect()\n// Before this PR, each returns more than 300000 rows.\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. Concurrent `collect()` and `collect(as:)` calls on the same `DataFrame` return correct results now. There is no API change.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test case.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #562 from dongjoon-hyun/SPARK-59480.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "bd359a485d419dd8ece6694ded18dedc17a5e13b",
      "tree": "33f4010bb19c51b34a7bf7adc932195e3cd718cc",
      "parents": [
        "0d10cfc948e727c62056d4a2937887b81724322d"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 13:55:59 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 13:55:59 2026 -0700"
      },
      "message": "[SPARK-59474] Support `DECIMAL` type in `createDataFrame` with `[[Sendable?]]` rows\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support the `DECIMAL` type in `createDataFrame` with `[[Sendable?]]` rows and a DDL schema string.\n\n- Add a `.decimal` case to `ConvertToArrow.toArrowColumn`, which uses the precision and scale of the schema.\n- Accept `Decimal` and integer values. Other values throw `SparkConnectError.InvalidType`.\n- Like `createDataFrame` with `Encodable` types, values are rounded `HALF_UP` to the scale, and values which do not fit in the precision throw an error.\n\n### Why are the changes needed?\n\nCurrently, `createDataFrame` throws `SparkConnectError.InvalidType` for `DECIMAL` columns.\n\n```swift\ntry await spark.createDataFrame([[Decimal(1)]], \"id DECIMAL(10, 2)\")\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavior change because previous it wasn\u0027t supported.\n\n`createDataFrame` now works with `DECIMAL(p, s)` columns. The results match Apache Spark 4.2.0 `createDataFrame(rows, schema)` in ANSI mode, which is the default.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #561 from dongjoon-hyun/SPARK-59474.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "0d10cfc948e727c62056d4a2937887b81724322d",
      "tree": "0021f19021966c492a3996b8fe6324345897da7f",
      "parents": [
        "f445890fa322a38a9a8bb88b0a5359b4cb0e67c4"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 12:46:16 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 12:46:16 2026 -0700"
      },
      "message": "[SPARK-59473] Support top-level `Decimal` decoding in `DataFrame.collect(as:)`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support top-level `Decimal` decoding in `DataFrame.collect(as:)`.\n\n### Why are the changes needed?\n\nFoundation\u0027s `Decimal.init(from:)` requests a keyed container of its internal fields. So, the following throws `ArrowError.invalid(\"Column for key \\\"exponent\\\" not found\")`, while Scala `as[BigDecimal]` returns `1.50`.\n\n```swift\ntry await spark.sql(\"SELECT CAST(1.5 AS DECIMAL(10,2)) AS v\").collect(as: Decimal.self)\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. `collect(as: Decimal.self)` and `collect(as: Decimal?.self)` now return values instead of throwing errors.\n\n### How was this patch tested?\n\nPass the CIs with the updated `CreateDataFrameTests.collectAsDecimal`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #560 from dongjoon-hyun/SPARK-59473.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "f445890fa322a38a9a8bb88b0a5359b4cb0e67c4",
      "tree": "2c2bee951bafee57c94842e785e1eab4bd02725a",
      "parents": [
        "87e0ab721af4a7fc3c67ad4ce59c42466b71de73"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 12:23:20 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 12:23:20 2026 -0700"
      },
      "message": "[SPARK-59472] Support `Decimal` properties in `createDataFrame` with `Encodable` types\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `Decimal` properties in `createDataFrame` with `Encodable` types.\n\n- Write the Arrow `Decimal` schema in `ArrowWriterHelper`.\n- Add `Decimal128BufferBuilder` to store the unscaled value as a 128-bit integer, rounded `HALF_UP` like Spark.\n- Throw an error for values which do not fit in `DECIMAL(38, 18)`.\n\n### Why are the changes needed?\n\nCurrently, `createDataFrame` fails with `InvalidArrowData` for `Encodable` types with `Decimal` properties.\n\n```swift\nstruct D: Codable, Sendable { let v: Decimal }\ntry await spark.createDataFrame([D(v: Decimal(string: \"-1.5\")!)])\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, `createDataFrame` now works with `Decimal` and `Decimal?` properties, matching the Scala `Dataset` behavior for `BigDecimal`.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #559 from dongjoon-hyun/SPARK-59472.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "87e0ab721af4a7fc3c67ad4ce59c42466b71de73",
      "tree": "2d40adf74ecffa79551f00e09621e9c6dcad3922",
      "parents": [
        "55ad4b798a49d7b3140c8bd48b6fb55fb1116d74"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 12:03:10 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 12:03:10 2026 -0700"
      },
      "message": "[SPARK-59471] Fix `DECIMAL` decoding to use the full signed 128-bit value\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix `DECIMAL` decoding to use the full signed 128-bit value.\n\n- `Decimal128Array.subscript` now reads the whole Arrow `Decimal128` value, which is a 128-bit little-endian two\u0027s complement integer. Previously, it read only the low 64 bits as an unsigned integer and always used a plus sign.\n- `ArrowArrayHolderImpl.loadArray` now uses `Decimal128Array` for `decimal128` children of `ARRAY` and `STRUCT` columns. Previously, it used `FixedArray\u003cDecimal\u003e`, which read the Arrow buffer as the raw memory of Foundation `Decimal`.\n\n### Why are the changes needed?\n\n`DECIMAL` values are silently decoded wrong when they are negative or do not fit in 64 bits. This affects `DataFrame.collect()`, `Row.getAsDecimal`, and `DataFrame.collect(as:)`.\n\n| SQL | Scala `as[T]` | Swift (before) |\n|---|---|---|\n| `SELECT CAST(-1.5 AS DECIMAL(10,2)) AS v` | `-1.50` | `184467440737095514.66` |\n| `SELECT CAST(12345678901234567890.5 AS DECIMAL(38,2)) AS v` | `12345678901234567890.50` | `170827812586263823.94` |\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. `DECIMAL` values are now decoded correctly, matching Scala `Dataset.as[T]`. This includes negative values, 38-digit values, and `DECIMAL` elements inside `ARRAY` and `STRUCT` columns.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases. The expected values were verified with `spark-shell` 4.2.0.\n\n- `DataFrameTests.decimalValues` covers `collect()`, `Row.getAsDecimal`, and a nested `ARRAY\u003cDECIMAL\u003e`.\n- `CreateDataFrameTests.collectAsDecimal` covers `collect(as:)`.\n\nBoth tests cover the following values:\n- `-1.5` as `DECIMAL(10,2)`\n- `12345678901234567890.50` as `DECIMAL(38,2)`\n- the max and min values of `DECIMAL(38,0)`\n- `-0.000000000000000001` as `DECIMAL(38,18)`\n- `0` and `NULL`\n\n```bash\nswift test --no-parallel --filter \u0027DataFrameTests|CreateDataFrameTests|RowTests|SQLTests\u0027\n```\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #558 from dongjoon-hyun/SPARK-59471.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "55ad4b798a49d7b3140c8bd48b6fb55fb1116d74",
      "tree": "e41b08a47701bfc5ecf944caa34fb5dd0d3edcc3",
      "parents": [
        "3fb5f071e7afcda30a44c79b1f9e64456d349e8c"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 11:13:14 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 11:13:14 2026 -0700"
      },
      "message": "[SPARK-59470] Respect `spark.sql.caseSensitive` when resolving column names in `DataFrame.collect(as:)`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to respect `spark.sql.caseSensitive` when resolving column names in `DataFrame.collect(as:)`, like Scala `Dataset.as[T]`. If more than one column matches a property, it throws an `ArrowError` instead of silently taking the last column.\n\n| `spark.sql.caseSensitive` | Query | Scala `as[Person]` | Swift `collect(as: Person.self)` |\n|---|---|---|---|\n| `false` | `SELECT \u0027Alice\u0027 AS NAME, 30 AS AGE` | `Person(Alice,30)` | `Person(name: \"Alice\", age: 30)` |\n| `false` | `SELECT \u0027a\u0027 AS name, \u0027b\u0027 AS NAME, 30 AS age` | `AMBIGUOUS_REFERENCE` | `Column for key \"name\" is ambiguous, ...` |\n| `true` | `SELECT \u0027Alice\u0027 AS NAME, 30 AS AGE` | `UNRESOLVED_COLUMN` | `Column for key \"name\" not found` |\n| `true` | `SELECT \u0027a\u0027 AS name, \u0027b\u0027 AS NAME, 30 AS age` | `Person(a,30)` | `Person(name: \"a\", age: 30)` |\n\n### Why are the changes needed?\n\nPreviously, `collect(as:)` matched columns by exact name only. A column differing only in case threw `Column for key \"name\" not found`, or silently decoded to `nil` for Optional properties. Scala returns the value because `spark.sql.caseSensitive` is `false` by default.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, `DataFrame.collect(as:)` is not released yet.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #557 from dongjoon-hyun/SPARK-59470.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "3fb5f071e7afcda30a44c79b1f9e64456d349e8c",
      "tree": "f9ac159b56c6f98d7c48ad01bda8e13b074c1b84",
      "parents": [
        "2c151201b332e2f9949746deb085a41f12ad8a90"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 10:26:46 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 10:26:46 2026 -0700"
      },
      "message": "[SPARK-59465] Support integral to floating-point upcasts in `DataFrame.collect(as:)`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support upcasts from integral types to `Float` and `Double` in `DataFrame.collect(as:)`. The rule follows Apache Spark\u0027s numeric precedence (`Byte \u003c Short \u003c Int \u003c Long \u003c Float \u003c Double`).\n\nThese conversions still throw `ArrowError`, as in Scala `Dataset.as[T]`:\n- `DOUBLE` into `Float`\n- floating-point types into integral types\n- `DECIMAL` into `Float` or `Double`\n- integral types into `Bool`\n- `STRING` into numeric types\n\n### Why are the changes needed?\n\nTo match Scala `Dataset.as[T]`. For example, `SELECT 30 AS v` decodes into a `Double` field in Scala, but Swift threw `Cannot decode Double for v`.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavior change. Previous it throws an error.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #556 from dongjoon-hyun/SPARK-59465.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "2c151201b332e2f9949746deb085a41f12ad8a90",
      "tree": "38fbe95e5f868c15a63b471803efd21ab94ad5d0",
      "parents": [
        "1c877f386ff0377719ae5d5cd1624a8e244427af"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Sep 12 22:00:17 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Sep 12 22:00:17 2026 -0700"
      },
      "message": "[SPARK-59464] Fix `DATE` handling for dates before 1970 and after 2106\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix the vendored Arrow `Date32` and `Date64` code to handle Arrow date values as signed integers.\n\n1. `Date32Array` loads `Int32` instead of `UInt32`, and multiplies by `86400` in `TimeInterval`. Loading `Int32` alone is not enough because `Int32 * 86400` overflows for dates after `2038-01-19`.\n2. `Date64Array` loads `Int64` instead of `UInt64`.\n3. `Date32BufferBuilder` uses `.rounded(.down)` instead of truncating toward zero, so pre-epoch values are floored to their day just like post-epoch values.\n\nThe same bugs exist in upstream `apache/arrow-swift`; they will be reported there separately.\n\n### Why are the changes needed?\n\nArrow `Date32` and `Date64` are signed day and millisecond counts since the UNIX epoch, and Spark sends `DATE` columns as `Date32`. Previously:\n\n- `Date32Array` read a day such as `-1` (`1969-12-31`) as `4294967295`, and `UInt32 * 86400` trapped with `Swift runtime failure: arithmetic overflow`. Only `1970-01-01` through `2106-02-07` could be read; any other `DATE` crashed the client process in both debug and release builds, without any error message. This affects `DataFrame.collect()`, `DataFrame.collect(as:)`, and `ARRAY`/`STRUCT` columns containing `DATE`.\n- `Date64Array` read negative milliseconds as huge positive values.\n- `createDataFrame` stored a pre-epoch `Date` that is not at UTC midnight one day later. For example, `1969-12-31T12:00:00Z` was stored as `1970-01-01`.\n\nReproducer (crashes before this fix, passes after):\n\n```swift\nlet spark \u003d try await SparkSession.builder.getOrCreate()\nlet rows \u003d try await spark.sql(\"SELECT DATE\u00271969-12-31\u0027\").collect()\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavior change for dates from `1970-01-01` to `2106-02-07`. This is a bug fix: `DATE` values outside that range can now be collected instead of crashing the client, and `createDataFrame` stores pre-epoch dates on the correct day.\n\n### How was this patch tested?\n\nPass the CIs with newly added test cases.\n\n- `DataFrameTests.collectDate`\n- `CreateDataFrameTests.dateType`\n- `CreateDataFrameTests.collectAsWithDateBeforeEpoch`\n\nI also verified manually against an Apache Spark 4.2.0 Connect server that all three new tests crash with signal 5 without this fix.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #555 from dongjoon-hyun/SPARK-59464.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "1c877f386ff0377719ae5d5cd1624a8e244427af",
      "tree": "44a75676d9bfbaf402f083a0f7693b4e33ddb511",
      "parents": [
        "4c100a2ba08f140ef3ba74b08a64887978e819c1"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Sep 12 20:47:40 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Sep 12 20:47:40 2026 -0700"
      },
      "message": "[SPARK-59463] Fix FlatBuffers nesting assertion on `TIMESTAMP` columns in `createDataFrame`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to create the timezone string offset before `startTimestamp` in `ArrowWriterHelper.toFBType`. This is the same fix as upstream `apache/arrow-swift`.\n- https://github.com/apache/arrow-swift/pull/148 (2026-03-25)\n\n### Why are the changes needed?\n\nFlatBuffers requires strings to be created before starting the table that references them. Otherwise, debug builds crash in `createDataFrame` with `TIMESTAMP` columns.\n\n```\n$ swift test --no-parallel --filter \"CreateDataFrameTests/supportedTypes\"\n...\nFlatBuffers/FlatBufferBuilder.swift:339: Assertion failed: Object serialization must not be nested\n```\n\nRelease builds are not affected because `assert` is disabled there.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavior change in release builds.\n\n### How was this patch tested?\n\nPass the CIs and manually run `CreateDataFrameTests` with a debug build.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #554 from dongjoon-hyun/SPARK-59463.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "4c100a2ba08f140ef3ba74b08a64887978e819c1",
      "tree": "9bda3efd3e2eee7437613c6f725bce783ee8746a",
      "parents": [
        "4cb592c79c029f718ce59d7a8fa2d028e08215e2"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Sep 12 20:46:55 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Sep 12 20:46:55 2026 -0700"
      },
      "message": "[SPARK-59462] Support safe upcasts and throw errors instead of crashing in `DataFrame.collect(as:)`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to throw `ArrowError.invalid` instead of crashing in `DataFrame.collect(as:)` when a value cannot be decoded into the target Swift type. It also allows Scala-style safe numeric upcasts (a narrower signed integer into a wider one, and `Float` into `Double`).\n\n```swift\n// Upcast: INT -\u003e Int64\ntry await spark.sql(\"SELECT 30 AS age\").collect(as: Int64.self)\n\n// Throws ArrowError.invalid(\"Cannot decode Int32 for column 0\")\ntry await spark.sql(\"SELECT 30L AS age\").collect(as: Int32.self)\n```\n\n### Why are the changes needed?\n\n`ArrowDecoder` force-unwraps decoded values and converts signed values with `UInt(val)`. As a result, a type mismatch, a `NULL` value, or a negative value into `UInt` crashes the whole process.\n\n```swift\nspark.sql(\"SELECT CAST(NULL AS STRING) AS name, 1L AS age\").collect(as: Person.self)\n// Fatal error: Unexpectedly found nil while unwrapping an Optional value\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nNo for the released versions because `DataFrame.collect(as:)` exists only in the `main` branch.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #553 from dongjoon-hyun/SPARK-59462.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "4cb592c79c029f718ce59d7a8fa2d028e08215e2",
      "tree": "a2098d27e609333014b5fbb51af5639c67f35d88",
      "parents": [
        "70b6dff0201f235488250b0a2665b6774a9170eb"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 11 21:09:51 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 11 21:09:51 2026 -0700"
      },
      "message": "[SPARK-59450] Support Codable in `createDataFrame` and `DataFrame.collect`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `Codable`-based DataFrame creation and collection in `SparkSession` and `DataFrame`, leveraging the in-tree `ArrowEncoder` and `ArrowDecoder`.\n\nSpecifically:\n1. **`SparkSession.createDataFrame` with `Encodable`**:\n   - `createDataFrame\u003cT: Encodable\u003e(_ data: [T]) async throws -\u003e DataFrame`: Automatically encodes an array of `Encodable` instances into an Apache Arrow `RecordBatch` via `ArrowEncoder`, infers the Spark DDL schema from the batch schema, and builds a `LocalRelation` or `CachedLocalRelation` (for payloads \u003e\u003d 1MiB).\n   - `createDataFrame\u003cT: Encodable\u003e(_ data: [T], _ schema: String)` and `createDataFrame\u003cT: Encodable\u003e(_ data: [T], _ schema: StructType)`: Supports explicit schema overrides and empty datasets with schema.\n2. **`DataFrame.collect(as:)` with `Decodable`**:\n   - `collect\u003cT: Decodable\u003e(as type: T.Type \u003d T.self) async throws -\u003e [T]`: Executes the plan and directly decodes each Arrow `RecordBatch` into `[T]` using `ArrowDecoder`, eliminating intermediate untyped `Row` allocations and dictionary boxing.\n3. **`ArrowEncoder` and `ArrowDecoder` enhancements**:\n   - Added support for Swift native `Int` (mapped to `Int64`) and `UInt` (mapped to `UInt64`) across keyed, unkeyed, and single-value containers.\n   - Added support for decoding `Date` and `TimestampNanos` from Arrow `Timestamp` columns (seconds, milliseconds, microseconds, and nanoseconds).\n4. **Schema mapping utilities**:\n   - Added `DataType.init(_ arrowType: ArrowType) throws` and `StructType.init(_ arrowSchema: ArrowSchema) throws` to translate Arrow schemas to Spark SQL data types.\n\n```swift\nstruct Person: Codable, Sendable, Equatable {\n  let name: String\n  let age: Int\n}\n\nlet people \u003d [Person(name: \"Alice\", age: 20), Person(name: \"Bob\", age: 25)]\n\n// 1. Create DataFrame directly from Swift models (auto-inferred schema)\nlet df \u003d try await spark.createDataFrame(people)\n\n// 2. Collect query results directly into Swift models\nlet results: [Person] \u003d try await df.filter(\"age \u003e\u003d 21\").collect(as: Person.self)\n```\n\n### Why are the changes needed?\n\nCurrently, `SparkSession.createDataFrame` only accepts untyped rows `[[Sendable?]]` with an explicit DDL string or `StructType` schema, and `DataFrame.collect()` only returns untyped `[Row]`.\n\nThese new APIs provide a type-safe, Dataset-like programming experience in Swift, allowing developers to work seamlessly with native Swift `Codable` structs.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavior change because this PR adds only new public generic methods.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Gemini 3.8 Flash (High)\n\nCloses #552 from dongjoon-hyun/SPARK-59450.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "70b6dff0201f235488250b0a2665b6774a9170eb",
      "tree": "86e6d0841a5ae3e392d20b5bbe7f7af7b50c2fc5",
      "parents": [
        "d71ae40685d645ad4379d432baa9a590b345aef4"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 11 15:36:58 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 11 15:36:58 2026 -0700"
      },
      "message": "[SPARK-59449] Support type-safe getters in `Row`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support type-safe getters and null-check inspection on `Row`.\n\nSpecifically, this PR introduces:\n1. **Null checking**:\n   - `isNullAt(_ i: Int) throws -\u003e Bool`\n   - `isNullAt(_ name: String) throws -\u003e Bool`\n2. **Generic `getAs` accessors**:\n   - `getAs\u003cT\u003e(_ i: Int, _ type: T.Type \u003d T.self) throws -\u003e T`\n   - `getAs\u003cT\u003e(_ name: String, _ type: T.Type \u003d T.self) throws -\u003e T`\n   - Supports both contextual type inference (`let val: Int \u003d try row.getAs(0)`) and explicit type passing (`try row.getAs(0, Int.self)`).\n   - Gracefully handles optional types: requesting an optional type (e.g. `String?.self`) on a `nil` column returns `nil` instead of throwing.\n3. **Convenience typed getters (Index and column name overloads)**:\n   - `getAsBool(_ name: String) throws -\u003e Bool` (alongside existing `getAsBool(_ i: Int)`)\n   - `getAsInt(_ i: Int)` / `getAsInt(_ name: String)`: supports type coercion from any `FixedWidthInteger` (`Int8`, `Int16`, `Int32`, `Int64`, `Int`), throwing `InvalidType` on integer overflow.\n   - `getAsInt64(_ i: Int)` / `getAsInt64(_ name: String)`: supports type coercion from any `FixedWidthInteger`.\n   - `getAsDouble(_ i: Int)` / `getAsDouble(_ name: String)`: supports `Double` and `Float` widening.\n   - `getAsString(_ i: Int)` / `getAsString(_ name: String)`\n   - `getAsDate(_ i: Int)` / `getAsDate(_ name: String)`: supports `Date` and `TimestampNanos.date`.\n   - `getAsTimestampNanos(_ i: Int)` / `getAsTimestampNanos(_ name: String)`: supports `TimestampNanos` and `Date`.\n   - `getAsDecimal(_ i: Int)` / `getAsDecimal(_ name: String)`: supports `Decimal` and `FixedWidthInteger`.\n4. **Standardized error handling**:\n   - Throws `SparkConnectError.InvalidArgument` if column index is out of bounds.\n   - Throws `SparkConnectError.UnsupportedOperation` if accessing by column name on a `Row` without a schema.\n   - Throws `SparkConnectError.ColumnNotFound` if accessing by an unknown column name.\n   - Throws `SparkConnectError.InvalidType` if the column value is `nil` (for non-optional `T`), type mismatch, or integer overflow occurs.\n\n### Why are the changes needed?\n\nPreviously, `Row` only provided untyped `get(_ i: Int) throws -\u003e Sendable` and a single typed getter `getAsBool(_ i: Int)`. Users had to manually downcast column values (e.g., `try row.get(i) as? Int` or `try row[\"name\"] as? String`).\n\nFurthermore, query results decoded from Arrow batches in `DataFrame+Actions.swift` hold sized integer types (`Int8`, `Int16`, `Int32`, `Int64`). A direct cast such as `as? Int` or `as? Double` would fail at runtime even if the data represented an integer or floating-point value.\n\nThese new APIs provide idiomatic, type-safe field extraction with automatic type coercion, bringing feature parity with Scala/PySpark `Row`.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavior change because this PR adds only new public methods (`isNullAt`, `getAs`, `getAsInt`, `getAsInt64`, `getAsDouble`, `getAsString`, `getAsDate`, `getAsTimestampNanos`, `getAsDecimal`, and `getAsBool(_ name: String)`) to `Row`.\n\n### How was this patch tested?\n\nPass the CIs with the newly added unit test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Gemini 3.8 Flash (High)\n\nCloses #551 from dongjoon-hyun/SPARK-59449.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "d71ae40685d645ad4379d432baa9a590b345aef4",
      "tree": "98be20ef314bf1341aa3f2d6cfdfc13565f13dee",
      "parents": [
        "538b05da288c482ef0775a8f4db29eb3739d5dae"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 10 21:59:09 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 10 21:59:09 2026 -0700"
      },
      "message": "[SPARK-59429] Pass plan ID in `DataFrame.colRegex`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to pass the plan ID in `DataFrame.colRegex`, like PySpark, now that `DataFrame` relations have plan IDs (SPARK-59414).\n\n- Set `planID` of `UnresolvedRegex` in `colRegex`.\n- Update the doc comment. The server uses the plan ID only for a column name without backticks, which is resolved as a regular column reference bound to the `DataFrame` like `col(_:)`. A regex enclosed in backticks is still not bound to the `DataFrame`.\n\n### Why are the changes needed?\n\nWhen SPARK-59399 added `colRegex`, this client had no plan IDs. As a result, a column name without backticks could not be disambiguated when both sides of a join had it.\n\n```swift\nlet df \u003d try await spark.range(3)\nlet df1 \u003d await df.filter(\"id \u003e 0\")\nlet df2 \u003d await df.select(\"id\")\nlet joined \u003d await df1.join(df2, joinExprs: df1.colRegex(\"id\") \u003d\u003d df2.colRegex(\"id\"))\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. `colRegex` is not released yet.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test case.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #550 from dongjoon-hyun/SPARK-59429.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "538b05da288c482ef0775a8f4db29eb3739d5dae",
      "tree": "881d4890a6280db638a1c7f224216157c9ae51b8",
      "parents": [
        "830b64e6f708e0906e8a6ab6b4207bb1e4db5ccb"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 10 19:44:23 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 10 19:44:23 2026 -0700"
      },
      "message": "[SPARK-59420] Support `withField` and `dropFields` in `Column`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `withField` and `dropFields` in `Column`.\n\n```swift\n// {\"a\":1,\"b\":2,\"c\":3} -\u003e {\"a\":1,\"b\":20,\"c\":3}\ndf.select(col(\"s\").withField(\"b\", lit(20)))\n\n// {\"a\":1,\"b\":2,\"c\":3} -\u003e {\"a\":1}\ndf.select(col(\"s\").dropFields(\"b\", \"c\"))\n\n// Nested fields: {\"a\":{\"a\":1,\"b\":2}} -\u003e {\"a\":{\"a\":1}}\ndf.select(col(\"s\").dropFields(\"a.b\"))\n```\n\nBoth methods are built on the existing `Expression.UpdateFields` proto, following PySpark Connect.\n- `withField` sets `struct_expression`, `field_name`, and `value_expression`.\n- `dropFields` leaves `value_expression` unset, because the server treats an unset value expression as a drop. Multiple field names are nested in order, like `update_fields(update_fields(s, b), c)`.\n\n`dropFields` requires at least one field name through its signature, `dropFields(_ fieldName: String, _ fieldNames: String...)`, so an empty call is rejected at compile time instead of at runtime.\n\n### Why are the changes needed?\n\nFor feature parity with PySpark, which has supported `Column.withField` and `Column.dropFields` since Apache Spark 3.1.0 and Spark Connect since 3.4.0. These APIs let users add, replace, or drop fields of struct columns, including nested fields.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, this PR adds two new public APIs, `Column.withField` and `Column.dropFields`.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #549 from dongjoon-hyun/SPARK-59420.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "830b64e6f708e0906e8a6ab6b4207bb1e4db5ccb",
      "tree": "7c4c088ced0d68c60ae4345be4c8b6f572da51fe",
      "parents": [
        "eaa42ebd0288c2126b1a8be877c020b8e3e748bd"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 10 17:04:56 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 10 17:04:56 2026 -0700"
      },
      "message": "[SPARK-59416] Support `col` and subscript column accessors in `DataFrame`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `col` and subscript column accessors in `DataFrame`.\n\n```swift\npublic nonisolated func col(_ colName: String) -\u003e Column\npublic nonisolated subscript(_ colName: String) -\u003e Column\n```\n\nUnlike the global `col(_:)` function, the returned `Column` carries the plan ID of the `DataFrame` (assigned since SPARK-59414), so the server can tell apart the columns with the same name in multiple `DataFrame`s.\n\n```swift\nlet df \u003d try await spark.range(3)\nlet df1 \u003d await df.filter(\"id \u003e 0\")\nlet df2 \u003d await df.select(\"id\")\nlet joined \u003d await df1.join(df2, joinExprs: df1[\"id\"] \u003d\u003d df2[\"id\"])\ntry await joined.select(df1[\"id\"], df2.col(\"id\")).show()\n```\n\n- `\"*\"` becomes `UnresolvedStar` with `plan_id`.\n- A name ending with `\".*\"`, e.g. `\"s.*\"`, becomes `UnresolvedStar` with `unparsed_target` only, like the global `col(_:)`. The server rejects a star with both a target and a plan ID (`CONNECT_INVALID_PLAN.UNRESOLVED_STAR_WITH_BOTH_TARGET_AND_PLAN_ID`).\n- Other names become `UnresolvedAttribute` with `plan_id`.\n\nThe plan ID of the root relation is stored as an immutable property when a `DataFrame` is created, so both accessors are `nonisolated` and don\u0027t require `await`.\n\n### Why are the changes needed?\n\nTo resolve ambiguous column references like `Dataset.col` and `Dataset.apply` in Scala and `DataFrame.__getitem__` in PySpark. Spark Connect has supported this since Apache Spark [v3.4.0](https://github.com/apache/spark/releases/tag/v3.4.0) (2023-04-07) via SPARK-41812 and SPARK-41823. No version gating is needed for the supported Apache Spark 4.x servers.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this adds only new public APIs, `DataFrame.col` and `DataFrame.subscript`.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #548 from dongjoon-hyun/SPARK-59416.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "eaa42ebd0288c2126b1a8be877c020b8e3e748bd",
      "tree": "c206f8863347b41b639567e01c4efe864c6a6c70",
      "parents": [
        "cd71d18d5d3e446e78b2f898ecc26146dfb06473"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 10 13:39:29 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 10 13:39:29 2026 -0700"
      },
      "message": "[SPARK-59414] Assign plan IDs to `DataFrame` relations\n\n### What changes were proposed in this pull request?\n\nThis PR proposes to assign a unique plan ID to the root relation of every `DataFrame`.\n\n- A file-scope `Atomic\u003cInt64\u003e` generator hands out monotonically increasing plan IDs, like the\n  existing `lambdaVariableID` in `HigherOrderFunctions.swift`.\n- Both SQL-based initializers now delegate to `init(spark:plan:)`, so every `DataFrame` goes\n  through a single place that stamps the plan ID.\n- An existing plan ID is preserved. For example, `toDF()` without arguments reuses `self.plan`,\n  and keeping its ID is what lets column references of the original `DataFrame` keep working.\n  This mirrors Scala\u0027s `SparkSession.newDataset`, which only tags newly created roots.\n- `DataFrameInternalTests.removeCachedRemoteRelation` now disables the server-side plan cache,\n  because a plan cached under the `DataFrame`\u0027s plan ID otherwise keeps answering `count()`\n  after the cached remote relation is removed.\n\n### Why are the changes needed?\n\nThe server uses plan IDs to resolve column references that are bound to a specific `DataFrame`,\nwhich is how ambiguous columns of a self-join are disambiguated. This mechanism landed in Apache\nSpark [v3.4.0](https://github.com/apache/spark/releases/tag/v3.4.0) (2023-04-07) via SPARK-41812\n(server and PySpark) and SPARK-41823 (Scala client), both backported to `branch-3.4`. This PR is\nthe prerequisite for supporting `DataFrame`-bound column accessors in this client.\n\n- apache/spark#39925\n\n### Does this PR introduce _any_ user-facing change?\n\nNo new API.\n\nHowever, since relations now carry plan IDs, the server-side session plan cache (SPARK-47818,\navailable since Apache Spark 4.0.0) applies to this client for the first time, exactly as it\nalready does for the Scala and Python Connect clients. Concretely, when a single `DataFrame`\nobject created by `spark.read.\u003cformat\u003e(path)` is reused after new files are appended to that\npath, it reports the result of its first execution instead of re-listing the files. `DataFrame`s\nof catalog tables, SQL queries, and temp views stay fresh, because only the unresolved plan is\ncached for them.\n\n- apache/spark#46012\n\n### How was this patch tested?\n\nPass the CIs with a new `DataFrameInternalTests.planID` test.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #547 from dongjoon-hyun/SPARK-59414.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "cd71d18d5d3e446e78b2f898ecc26146dfb06473",
      "tree": "163a7aba94977b87ec08f87f00b1cda7f9f76f5c",
      "parents": [
        "702be5097856c57558eeea59912950bd89839382"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 10 12:12:26 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 10 12:12:26 2026 -0700"
      },
      "message": "[SPARK-59399] Support `colRegex` in `DataFrame`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `colRegex` in `DataFrame`.\n\n```swift\npublic nonisolated func colRegex(_ colName: String) -\u003e Column\n```\n\nIt returns a `Column` wrapping the `UnresolvedRegex` expression. The required protobuf definition (`Expression.UnresolvedRegex`) already exists, so no code generation is involved.\n\nExample:\n\n```swift\nlet df \u003d try await spark.sql(\"SELECT 1 a1, 2 a2, 3 b1\")\n// Select the columns whose names start with `a`, i.e., `a1` and `a2`.\ntry await df.select(df.colRegex(\"`a.*`\")).show()\n```\n\nAs with the other Spark clients, only a column name enclosed in backticks is interpreted as a Java regular expression, and its case sensitivity follows `spark.sql.caseSensitive`. A column name without backticks is resolved as a regular column reference.\n\nNote that this client does not assign plan IDs to relations, so the `plan_id` field is not set. The server ignores `plan_id` for backtick-quoted regexes, so the regex behavior is identical to PySpark. The only difference is that a column name without backticks cannot be disambiguated when both sides of a self-join have it. This limitation is documented in the API doc.\n\n### Why are the changes needed?\n\nTo let users select multiple columns by a regex on column names, closing a gap with the other Spark clients.\n\n`Dataset.colRegex` was added in Apache Spark 2.3.0 via SPARK-12139, PySpark exposes `DataFrame.colRegex` since 2.3.0 via SPARK-23081, and Spark Connect supports it since 3.4.0 via SPARK-41438. No version gating is needed for the supported Apache Spark 4.x servers.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, this adds a new public API, `DataFrame.colRegex`.\n\n### How was this patch tested?\n\nPass the CIs with newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #546 from dongjoon-hyun/SPARK-59399.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "702be5097856c57558eeea59912950bd89839382",
      "tree": "42fc6907a7efebdad040bf3a6c930163e9394b47",
      "parents": [
        "23103d671fa86f7aaab7d35f20afd540add3a956"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 09 15:06:03 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 09 15:06:03 2026 -0700"
      },
      "message": "[SPARK-59391] Support `groupingSets` in `DataFrame`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `groupingSets` in `DataFrame`.\n\n```swift\npublic func groupingSets(_ groupingSets: [[String]], _ cols: String...) -\u003e GroupedData\n```\n\n`GroupedData` is extended with an optional `groupingSets` stored property, which\n`buildAggregate` translates into the `Aggregate.grouping_sets` field, mirroring how\nthe existing `pivot` property is handled. The `init` gains a defaulted parameter, so\nthe existing `groupBy`/`rollup`/`cube`/`pivot` call sites are unchanged.\n\nThe required protobuf definitions (`GROUP_TYPE_GROUPING_SETS` and\n`Aggregate.GroupingSets`) already exist, so no code generation is involved.\n\nExample:\n\n```swift\nlet df \u003d try await spark.sql(\"SELECT * FROM dealer\")\ntry await df.groupingSets([[\"city\", \"car_model\"], [\"city\"], []], \"city\", \"car_model\")\n  .agg(\"sum(quantity) sum\").orderBy(\"city\", \"car_model\").show()\n```\n\n### Why are the changes needed?\n\nTo provide a `DataFrame`-level API for multi-dimensional aggregation over an explicit\nlist of group combinations, closing a gap with the other Spark clients.\n\nUnlike `rollup` and `cube`, which derive the combinations automatically, `groupingSets`\nlets users specify arbitrary combinations, including the empty set that aggregates over\nall rows. `Dataset.groupingSets` was added in Apache Spark 4.0.0 via SPARK-45929, and\nPySpark exposes `DataFrame.groupingSets` since 4.0.0.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, this adds a new public API, `DataFrame.groupingSets`. There is no behavior change\nfor the existing `groupBy`, `rollup`, `cube`, and `pivot` APIs.\n\n### How was this patch tested?\n\nPass the CIs with newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #545 from dongjoon-hyun/SPARK-59391.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "23103d671fa86f7aaab7d35f20afd540add3a956",
      "tree": "cac93853fc722a411f0a363f73febc07564133b7",
      "parents": [
        "80e86b661a59babb69b999bc8ec9068851a5d276"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 09 12:34:13 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 09 12:34:13 2026 -0700"
      },
      "message": "[SPARK-59380] Upgrade `actions/setup-java` to v6\n\n### What changes were proposed in this pull request?\n\nThis PR aims to upgrade `actions/setup-java` from v5 to v6 in the `build_and_test.yml` workflow.\n\n### Why are the changes needed?\n\nTo use the latest `actions/setup-java` release.\n\n- [v6.0.0 (2026-08-24)](https://github.com/actions/setup-java/releases/tag/v6.0.0)\n- [v6.0.1 (2026-09-09)](https://github.com/actions/setup-java/releases/tag/v6.0.1)\n\nv6 migrates the Zulu distribution to the Azul Metadata API and moves the action to ESM. The inputs used in this repository (`distribution` and `java-version`) are unchanged.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5.1\n\nCloses #544 from dongjoon-hyun/SPARK-59380.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "80e86b661a59babb69b999bc8ec9068851a5d276",
      "tree": "0fc2e99c88c84ee9363f0ed1121656476362e8b0",
      "parents": [
        "ce19490d269c80f67465fe807f68b05f04f13bdb"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 12:19:24 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 12:19:24 2026 -0700"
      },
      "message": "[SPARK-59264] Cache Apache Spark distributions in GitHub Actions jobs\n\n### What changes were proposed in this pull request?\n\nThis PR adds a `setup-spark` composite action that caches the downloaded Apache Spark distribution tarball, and switches all seven macOS integration test jobs to use it.\n\n- New `.github/actions/setup-spark/action.yml`:\n  - `actions/cachev4` on `~/spark-dist/spark-\u003cversion\u003e-bin-hadoop3.tgz`, keyed by version and SHA-512, so a release candidate and its final release never share a cache entry.\n  - Downloads from the ASF mirror redirector only on a cache miss. A `url` input overrides it for release candidates.\n  - Verifies SHA-512 before extracting into `/tmp/spark`.\n- `.github/workflows/build_and_test.yml`: the duplicated `curl`/`tar`/`mv` preamble in seven jobs is replaced by a single `uses: ./.github/actions/setup-spark` step.\n- SHA-512 verification is now applied to all versions. The values for 4.0.4 and 4.1.3 were taken from the official ASF `.sha512` files:\n  - [spark-4.0.4-bin-hadoop3.tgz.sha512](https://downloads.apache.org/spark/spark-4.0.4/spark-4.0.4-bin-hadoop3.tgz.sha512)\n  - [spark-4.1.3-bin-hadoop3.tgz.sha512](https://downloads.apache.org/spark/spark-4.1.3/spark-4.1.3-bin-hadoop3.tgz.sha512)\n- `tar xvfz` is changed to `tar xfz` to remove thousands of lines of file listings from the CI logs.\n\n### Why are the changes needed?\n\nSeven macOS jobs each download a Spark distribution from an ASF mirror on every workflow run, roughly 2.8GB of mirror traffic per run:\n\n| Version | Jobs |\n| ------- | ---- |\n| 4.0.4 | `integration-test-mac`, `integration-test-mac-spark4-iceberg` |\n| 4.1.3 | `integration-test-mac-spark41`, `integration-test-token`, `integration-test-mac-spark41-iceberg` |\n| 4.2.0 | `integration-test-mac-spark42` |\n| 4.3.0-rc1 | `integration-test-mac-spark43` |\n\n1. macOS runners are billed at a 10x multiplier, so mirror download time is the most expensive kind of CI time in this repository.\n2. `closer.lua` can redirect to a slow mirror, which puts these jobs at risk of hitting the 20-minute timeout for reasons unrelated to the change under test.\n3. It reduces the load this repository puts on ASF mirrors.\n4. Only 4.2.0 and 4.3.0 verified their download. Every version is verified now.\n5. The download and extraction logic was duplicated seven times, so adding a version or fixing the logic meant editing seven places.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is a CI-only change.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #543 from dongjoon-hyun/SPARK-59264.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "ce19490d269c80f67465fe807f68b05f04f13bdb",
      "tree": "78ecabb4a2f733a92fba2ddcb8cf2194cec1fb84",
      "parents": [
        "029dca72ea4d59f83b95f555479808f19192c7e5"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 11:31:15 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 11:31:15 2026 -0700"
      },
      "message": "[SPARK-59260] Index all public functions and types in the DocC catalog\n\n### What changes were proposed in this pull request?\n\nThis PR rebuilds the DocC catalog\u0027s Topics index so that the library\u0027s public API is\ndiscoverable in the generated documentation. No source code is changed.\n\n**SQL functions.** None of the module-level functions were curated. All 753 public top-level\nfunctions fell into DocC\u0027s automatically-generated `Functions` section on the module landing\npage as one flat, unsectioned list. This PR adds 13 article pages:\n\n- `Functions.md`, a hub page linking to the family pages below.\n- `ColumnFunctions.md` (21), `AggregateFunctions.md` (130), `WindowFunctions.md` (13),\n  `MathFunctions.md` (85), `StringFunctions.md` (124), `DateTimeFunctions.md` (115),\n  `CollectionFunctions.md` (81), `SemiStructuredFunctions.md` (53), `SketchFunctions.md` (52),\n  `GeospatialFunctions.md` (10), `ConditionalFunctions.md` (21), `MiscFunctions.md` (48).\n\nEach page has an Overview with a runnable example and splits its Topics into 3-14 semantic\nsections. Pages are organized by source-file family, so adding a function to\n`XmlFunctions.swift` maps to one obvious place in the docs. The single exception is\n`count`/`sum`/`avg`/`mean`/`min`/`max` from `Functions.swift`, curated on the aggregate page\nwhere readers will look for them.\n\nAll 753 declarations are curated exactly once. The 82 overloaded names use DocC\u0027s type\nsignature disambiguation, for example:\n\n```\n- ``lit(_:)-(Bool)``\n- ``parse_url(_:_:)-(_,Column)``\n```\n\n**Missing public types.** `SparkConnect.md` gains `Column`, `SparkLiteral`, `Window`,\n`WindowSpec`, `Trigger`, `StreamingQueryException`, `StreamingQueryStatus`,\n`DataFrameWriterV2`, `WhenMatched`, `WhenNotMatched`, `WhenNotMatchedBySource`,\n`DataFrameNaFunctions`, `DataFrameStatFunctions`, `Observation`, `RowSchema`,\n`SparkConnectError`, `~\u003d(_:_:)`, `RuntimeConf`, and the six catalog value types\n(`CatalogMetadata`, `CatalogColumn`, `Database`, `SparkTable`, `Function`, `TablePartition`),\nacross new `Expressions`, `Window Frames`, and `Error Handling` sections. `LocalTime` and\n`TimestampNanos` move from `DataFrames` to `Data Types`.\n\n`SparkConnectClient`, `CaseInsensitiveDictionary`, `ErrorUtils`, `ProtoUtils`,\n`SparkFileUtils`, `CRC32`, and `SHA256` are curated under `Low-Level and Utility APIs`. These\nare `public`, so DocC lists them whether or not they are curated; the `/// nodoc` marker used\nby the vendored Arrow sources is a jazzy/SPI convention that DocC does not act on, and actually\nhiding them would need the underscored `_documentation(visibility:)` attribute. Grouping them\nunder an explicitly named section is honest about what they are without touching any source.\n\n**`Column.md`.** A new symbol page with an Overview and 15 topic sections covering all 84\npublic members. Operator overloads are separated into Arithmetic, Comparison, and Logical\nsections instead of appearing as repeated identical titles in the automatic member list.\n\n**Bundle metadata.** `Info.plist` drops the `CFBundleVersion` key and updates\n`NSHumanReadableCopyright` to `© 2025 and onwards, The Apache Software Foundation`, matching\nthe \"and onwards\" form used in this project\u0027s `NOTICE` so it needs no yearly edit. The version\nkey was set to `0.1.0` and had never been updated across the `0.2.0` through `0.7.0` releases;\nit is removed rather than bumped because DocC does not consume it. The\n`INFO.PLIST FALLBACKS` section of `docc convert --help` lists only `CFBundleDisplayName`,\n`CFBundleIdentifier`, a default module kind, and a default code-listing language; the version\nnever reaches the generated site, and a stale value produces no diagnostic, which is why it\nwent unnoticed. Keeping it would mean either shipping a wrong value again after the next\nrelease or adding a release step to maintain something nothing reads.\n\n### Why are the changes needed?\n\nThe published documentation on Swift Package Index does not currently make this library\nnavigable. 259 SQL functions were added recently and the catalog index had never been updated,\nso the largest part of the public API was reachable only through one undifferentiated\nauto-generated list, and core types such as `Column` appeared only in DocC\u0027s automatic\n`Structures`/`Classes` groups mixed in with the vendored Arrow implementation.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is a documentation-only change; the generated DocC site is reorganized.\n\n### How was this patch tested?\n\nPass the CI. In addition, DocC was built locally and compared against the pre-change baseline.\nThe repository has no DocC plugin dependency, so symbol graphs were extracted from a release\nbuild and `docc` was invoked directly rather than adding a dependency to `Package.swift`:\n\n```\nswift build -c release \\\n  -Xswiftc -emit-symbol-graph -Xswiftc -emit-symbol-graph-dir -Xswiftc /tmp/sg\n```\n\n```\nxcrun docc convert Sources/SparkConnect/Documentation.docc \\\n  --fallback-display-name SparkConnect \\\n  --fallback-bundle-identifier org.apache.spark.connect.swift.SparkConnect \\\n  --additional-symbol-graph-dir /tmp/sg --output-path /tmp/docs\n```\n\nResults:\n\n- 61 warnings before this change, 61 after; `diff` of the two warning lists is empty. All 61\n  are pre-existing warnings from doc comments in `.swift` sources and are untouched here, so\n  `--warnings-as-errors` is not usable on this catalog yet.\n- Inspecting the rendered JSON confirms 765 topic links across the 13 function pages, all\n  unique, with no duplicate curation and no link resolving to an unintended symbol.\n  `column.json` shows 84 unique links, every one a `Column` member.\n- The auto-generated `Functions` section (753 entries) is gone from the module landing page,\n  which now has 11 curated sections holding 60 entries. The only remaining uncurated top-level\n  symbols are vendored Arrow and FlatBuffers types.\n- Every symbol name written in Topics was cross-checked against the sources; no missing name.\n- `plutil -lint` reports the plist as valid. Converting with and without `CFBundleVersion`\n  produces 3,256 rendered JSON files that are all semantically identical, confirming the key\n  contributes nothing to the site.\n- `markdownlint --config .markdownlint.yaml --ignore-path .markdownlintignore` is clean.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #542 from dongjoon-hyun/SPARK-59260.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "029dca72ea4d59f83b95f555479808f19192c7e5",
      "tree": "610d89aaa5456df06da5793cdb121dc1bfe11448",
      "parents": [
        "ca05b35dbf5b78fed91156ac79daf65f3f53ebf5"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 10:12:09 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 10:12:09 2026 -0700"
      },
      "message": "[SPARK-59258] Support higher-order map functions\n\n### What changes were proposed in this pull request?\n\nThis PR adds the four higher-order map functions to `HigherOrderFunctions.swift`, built on the lambda infrastructure added by SPARK-59254.\n\n| Function | Closure |\n| -------- | ------- |\n| `map_filter(col, f)` | `(Column, Column) -\u003e Column` (key, value) |\n| `transform_keys(col, f)` | `(Column, Column) -\u003e Column` (key, value) |\n| `transform_values(col, f)` | `(Column, Column) -\u003e Column` (key, value) |\n| `map_zip_with(left, right, f)` | `(Column, Column, Column) -\u003e Column` (key, value1, value2) |\n\n`map_zip_with` is the first user of the three-argument `createLambda` overload, which was added by SPARK-59254 but had no caller until now.\n\n### Why are the changes needed?\n\nFor feature parity with Apache Spark. These four functions exist in both the\nScala and Python clients but had no Swift equivalent. They complete the set of\nhigher-order functions, whose array half landed in SPARK-59254.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, this adds new public APIs. For example:\n\n```swift\nlet df \u003d try await spark.range(1)\nlet m \u003d map_from_arrays(array(lit(1), lit(2)), array(lit(10), lit(20)))\n\ntry await df.select(map_filter(m) { _, v in v \u003e 10 }.cast(\"string\")).show()\n// {2 -\u003e 20}\n\ntry await df.select(transform_keys(m) { k, v in k + v }.cast(\"string\")).show()\n// {11 -\u003e 10, 22 -\u003e 20}\n\ntry await df.select(transform_values(m) { k, v in k + v }.cast(\"string\")).show()\n// {1 -\u003e 11, 2 -\u003e 22}\n\nlet m2 \u003d map_from_arrays(array(lit(1), lit(2)), array(lit(100), lit(200)))\ntry await df.select(map_zip_with(m, m2) { _, v1, v2 in v1 + v2 }.cast(\"string\")).show()\n// {1 -\u003e 110, 2 -\u003e 220}\n```\n\nAll four functions were introduced in Spark 3.1.0, so no version gate is needed.\n\n### How was this patch tested?\n\nExtended the existing `HigherOrderFunctionsTests` suite.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #541 from dongjoon-hyun/SPARK-59258.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "ca05b35dbf5b78fed91156ac79daf65f3f53ebf5",
      "tree": "31881da1aef21279c32775af4f2896b4882b9b5d",
      "parents": [
        "2f84649172864b24d3cb22d8b014210e18f07cd2"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 09:57:05 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 09:57:05 2026 -0700"
      },
      "message": "[SPARK-59257] Update docker GitHub Actions to the latest ASF-approved hashes\n\n### What changes were proposed in this pull request?\n\nThis PR aims to update `docker/*` GitHub Actions in `.github/workflows/publish_image.yml` to the latest ASF-approved commit hashes from [apache/infrastructure-actions/approved_patterns.yml](https://github.com/apache/infrastructure-actions/blob/main/approved_patterns.yml).\n\n| Action | Before | After |\n|---|---|---|\n| `docker/setup-qemu-action` | `ce360397` ([v4.0.0](https://github.com/docker/setup-qemu-action/releases/tag/v4.0.0), 2026-03-04) | `96fe6ef7` ([v4.2.0](https://github.com/docker/setup-qemu-action/releases/tag/v4.2.0), 2026-07-01) |\n| `docker/setup-buildx-action` | `4d04d5d9` ([v4.0.0](https://github.com/docker/setup-buildx-action/releases/tag/v4.0.0), 2026-03-05) | `37fe6310` ([v4.3.0](https://github.com/docker/setup-buildx-action/releases/tag/v4.3.0), 2026-08-17) |\n| `docker/login-action` | `4907a6dd` ([v4.1.0](https://github.com/docker/login-action/releases/tag/v4.1.0), 2026-04-02) | `dbcb8138` ([v4.6.0](https://github.com/docker/login-action/releases/tag/v4.6.0), 2026-07-29) |\n| `docker/build-push-action` | `bcafcacb` ([v7.1.0](https://github.com/docker/build-push-action/releases/tag/v7.1.0), 2026-04-09) | `53b7df96` ([v7.3.0](https://github.com/docker/build-push-action/releases/tag/v7.3.0), 2026-07-01) |\n\n### Why are the changes needed?\n\nThe previously pinned hashes are no longer listed in the ASF-approved patterns. This PR brings the workflow back into compliance with the ASF GitHub Actions policy and picks up the latest releases of these actions.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is an infra-only change.\n\n### How was this patch tested?\n\nManual review. Each new hash was verified against `approved_patterns.yml` and its corresponding release tag via the GitHub API.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5.1\n\nCloses #540 from dongjoon-hyun/SPARK-59257.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "2f84649172864b24d3cb22d8b014210e18f07cd2",
      "tree": "62ff891c0cb84d179af1628a0d5256a1505b84da",
      "parents": [
        "53f0bb6ef0430e327dea1c345faf815a69496535"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 09:44:44 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 09:44:44 2026 -0700"
      },
      "message": "[SPARK-59254] Support higher-order array functions\n\n### What changes were proposed in this pull request?\n\nThis PR adds support for lambda expressions and the higher-order array\nfunctions that consume them, in a new `HigherOrderFunctions.swift` file. No\nexisting file is modified.\n\nLambda infrastructure (internal):\n\n- `createLambda` in one-, two-, and three-argument forms, converting a Swift\n  closure into a `LambdaFunction` expression.\n- Lambda variables get a unique name (`x_1`, `y_2`, ...) from an\n  `Atomic\u003cInt\u003e` counter, matching Scala\u0027s\n  `UnresolvedNamedLambdaVariable.apply` and PySpark\u0027s `fresh_var_name`.\n\nNew public functions:\n\n| Function | Closure |\n| -------- | ------- |\n| `transform(col, f)` | `(Column) -\u003e Column` and `(Column, Column) -\u003e Column` (element, index) |\n| `filter(col, f)` | `(Column) -\u003e Column` and `(Column, Column) -\u003e Column` (element, index) |\n| `exists(col, f)` | `(Column) -\u003e Column` |\n| `forall(col, f)` | `(Column) -\u003e Column` |\n| `aggregate(col, initialValue, merge, finish:)` | `(Column, Column) -\u003e Column`, `(Column) -\u003e Column` (`finish:` optional) |\n| `reduce(col, initialValue, merge, finish:)` | `(Column, Column) -\u003e Column`, `(Column) -\u003e Column` (`finish:` optional) |\n| `zip_with(left, right, f)` | `(Column, Column) -\u003e Column` |\n| `array_sort(col, comparator)` | `(Column, Column) -\u003e Column` |\n\nThe closures are non-escaping because they are invoked immediately while the\nexpression is built and never stored.\n\n`finish:` is labeled so the optional finish closure can be passed as a second\ntrailing closure; the first closure of every function stays unlabeled, matching\nthe rest of the API.\n\n### Why are the changes needed?\n\nFor feature parity with Apache Spark. These functions exist in both the Scala\nand Python clients but had no Swift equivalent, because there was no way to\nbuild a Spark Connect lambda expression from a Swift closure. The generated\nprotobuf types (`Expression.LambdaFunction` and\n`Expression.UnresolvedNamedLambdaVariable`) were already available, so no proto\nregeneration is required.\n\nThe unique variable naming is required for correctness, not cosmetics. The\nserver passes `name_parts` through unchanged\n(`SparkConnectPlanner.transformUnresolvedNamedLambdaVariable`), so with fixed\nnames an inner lambda would shadow the variable of an enclosing one and\n`transform(a) { x in transform(b) { y in x + y } }` would silently compute\n`x + x`.\n\nThe functions live in their own file rather than in `CollectionFunctions.swift`\nbecause they share the lambda infrastructure, which stays internal to the module,\nand because the follow-up higher-order map functions (`map_filter`,\n`transform_keys`, `transform_values`, `map_zip_with`) belong next to them. This\nalso keeps `CollectionFunctions.swift` from growing past 900 lines.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, this adds new public APIs. For example:\n\n```swift\nlet df \u003d try await spark.range(1)\nlet arr \u003d array(lit(1), lit(2), lit(3))\n\ntry await df.select(transform(arr) { $0 * 2 }.cast(\"string\")).show()\n// [2, 4, 6]\n\ntry await df.select(aggregate(arr, lit(0)) { acc, x in acc + x }).show()\n// 6\n\ntry await df.select(aggregate(arr, lit(0)) { acc, x in acc + x } finish: { $0 * 10 }).show()\n// 60\n```\n\nAll of these functions were introduced in Spark 3.5 or earlier\n(`array_sort` with a comparator in 3.4.0), so no version gate is needed.\n\n### How was this patch tested?\n\nAdded a new `HigherOrderFunctionsTests` suite.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5.1\n\nCloses #539 from dongjoon-hyun/SPARK-59254.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "53f0bb6ef0430e327dea1c345faf815a69496535",
      "tree": "d3c76b1ad2bbb9475499dec8d0547494da46e25e",
      "parents": [
        "85e362614e2e97a096ca7e7740d3e077ee97b048"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:44:47 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:44:47 2026 -0700"
      },
      "message": "[SPARK-59245] Support HLL sketch and bitmap aggregate functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support 7 HLL sketch and bitmap aggregate functions (12 declarations), which\ncompletes the sketch function coverage of this client.\n\n| Function | Since |\n| --- | --- |\n| `hll_sketch_agg`, `hll_union_agg` | 3.5.0 |\n| `hll_sketch_estimate`, `hll_union` | 3.5.0 |\n| `bitmap_construct_agg`, `bitmap_or_agg` | 3.5.0 |\n| `bitmap_and_agg` | **4.1.0** |\n\nThe 5 aggregate functions go into `AggregateFunctions.swift`, following the `theta_sketch_agg`,\n`kll_sketch_agg_*` and `tuple_sketch_agg_*` precedents. The 2 scalar functions live in a new\n`HllSketchFunctions.swift`, mirroring `ThetaSketchFunctions.swift`.\n\nOptional arguments follow the upstream implementations exactly and are expressed as overloads.\n`lgConfigK` gets an `Int32` overload alongside the `Column` one, since the server requires the\n`INT` type there and the existing `lit(_ value: Int)` produces a `BIGINT` literal that would be\nrejected. `allowDifferentLgConfigK` is `Union[bool, Column]` in `hll_union_agg` and gets both a\n`Bool` and a `Column` overload, while `hll_union` accepts a `Bool` only, matching\n`def hll_union(c1: Column, c2: Column, allowDifferentLgConfigK: Boolean)` in the Scala API.\n\n### Why are the changes needed?\n\nTo improve API coverage. These functions are available in PySpark and the Spark SQL Scala API,\nbut were missing from this Swift client. The scalar `bitmap_bit_position`,\n`bitmap_bucket_number` and `bitmap_count` functions are already supported, and these three\naggregate functions complete that family.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new addition to the API.\n\n```swift\nlet df \u003d try await spark.sql(\"SELECT * FROM VALUES (1), (2), (2), (3) AS T(v)\")\ntry await df.select(hll_sketch_estimate(hll_sketch_agg(col(\"v\")))).show()\n```\n\n```\n+----------------------------------------------+\n|hll_sketch_estimate(hll_sketch_agg(v, 12))    |\n+----------------------------------------------+\n|3                                             |\n+----------------------------------------------+\n```\n\n```swift\ntry await df.select(bitmap_count(bitmap_construct_agg(bitmap_bit_position(col(\"v\"))))).show()\n```\n\n```\n+-------------------------------------------------------+\n|bitmap_count(bitmap_construct_agg(bitmap_bit_position(v)))|\n+-------------------------------------------------------+\n|3                                                      |\n+-------------------------------------------------------+\n```\n\n### How was this patch tested?\n\nPass the CIs with a new test suite.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #538 from dongjoon-hyun/SPARK-59245.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "85e362614e2e97a096ca7e7740d3e077ee97b048",
      "tree": "7a2b74fa88327578455b6b32b2edd18a3da25150",
      "parents": [
        "017b2f95c149b6c6e9785143c9c34d100de370b6"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:11:28 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:11:28 2026 -0700"
      },
      "message": "[SPARK-59241] Support Theta, KLL and Tuple sketch functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support 52 Datasketches sketch functions (86 declarations).\n\nThe KLL functions have three element-type variants (`_bigint`, `_double`, `_float`) and the\nTuple functions have two summary-type variants (`_double`, `_integer`).\n\n| Function | Since |\n| --- | --- |\n| `theta_sketch_agg`, `theta_union_agg`, `theta_intersection_agg` | 4.1.0 |\n| `theta_sketch_estimate`, `theta_union`, `theta_intersection`, `theta_difference` | 4.1.0 |\n| `kll_sketch_agg_\u003cT\u003e` | 4.1.0 |\n| `kll_merge_agg_\u003cT\u003e` | **4.1.2** |\n| `kll_sketch_get_n_\u003cT\u003e`, `kll_sketch_get_quantile_\u003cT\u003e`, `kll_sketch_get_rank_\u003cT\u003e` | 4.1.0 |\n| `kll_sketch_merge_\u003cT\u003e`, `kll_sketch_to_string_\u003cT\u003e` | 4.1.0 |\n| `tuple_sketch_agg_\u003cT\u003e`, `tuple_union_agg_\u003cT\u003e`, `tuple_intersection_agg_\u003cT\u003e` | 4.2.0 |\n| `tuple_sketch_estimate_\u003cT\u003e`, `tuple_sketch_summary_\u003cT\u003e`, `tuple_sketch_theta_\u003cT\u003e` | 4.2.0 |\n| `tuple_union_\u003cT\u003e`, `tuple_union_theta_\u003cT\u003e` | 4.2.0 |\n| `tuple_intersection_\u003cT\u003e`, `tuple_intersection_theta_\u003cT\u003e` | 4.2.0 |\n| `tuple_difference_\u003cT\u003e`, `tuple_difference_theta_\u003cT\u003e` | 4.2.0 |\n\nThe 37 scalar functions live in three new files, `ThetaSketchFunctions.swift`,\n`KllSketchFunctions.swift` and `TupleSketchFunctions.swift`. The 15 aggregate functions go\ninto `AggregateFunctions.swift`, following the existing `count_min_sketch`,\n`schema_of_variant_agg` and `vector_avg` precedents.\n\nOptional arguments follow the upstream PySpark implementations exactly. Most functions omit\n`lgNomEntries` / `k` / `mode` when they are not given, so the argument count varies and they\nare expressed as overloads, like the existing `hmac`. The exceptions are\n`tuple_sketch_agg_*`, `tuple_union_agg_*`, `tuple_union_*` and `tuple_union_theta_*`, which\nalways send `lgNomEntries` and `mode` filled with `12` and `\"sum\"`; these use labeled Swift\ndefault argument values, like the existing `aes_encrypt`.\n\nAn `Int32` overload is provided for `lgNomEntries` and `k` alongside the `Column` one, since\nthe server requires the `INT` type there and the existing `lit(_ value: Int)` produces a\n`BIGINT` literal that would be rejected.\n\n### Why are the changes needed?\n\nTo improve API coverage. These functions are available in PySpark and the Spark SQL Scala\nAPI, but were missing from this Swift client, where `count_min_sketch` was the only\nsupported sketch function.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new addition to the API.\n\n```swift\nlet df \u003d try await spark.sql(\"SELECT * FROM VALUES (1), (2), (2), (3) AS T(v)\")\ntry await df.select(theta_sketch_estimate(theta_sketch_agg(col(\"v\")))).show()\n```\n\n```\n+--------------------------------------------------+\n|theta_sketch_estimate(theta_sketch_agg(v))        |\n+--------------------------------------------------+\n|3                                                 |\n+--------------------------------------------------+\n```\n\n### How was this patch tested?\n\nPass the CIs with three new test suites, `ThetaSketchFunctionsTests`,\n`KllSketchFunctionsTests` and `TupleSketchFunctionsTests` (18 tests).\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #537 from dongjoon-hyun/SPARK-59241.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "017b2f95c149b6c6e9785143c9c34d100de370b6",
      "tree": "3b38b0b4671fde982861053528440d3f99726334",
      "parents": [
        "79af75a3c3d0e898710cf3dd6ce755dc52c71c0f"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:10:44 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:10:44 2026 -0700"
      },
      "message": "[SPARK-59240] Support `call_function` and `call_udf` functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `call_function` and `call_udf` functions in a new `UDFFunctions.swift` file.\n\n| Function | Since | Signature |\n| --- | --- | --- |\n| `call_function` | 3.5.0 | `(String, Column...)` |\n| `call_udf` | 3.4.0 | `(String, Column...)` |\n\nBoth are backed by the `CallFunction` protobuf node rather than the existing\ninternal `fn` helper which builds an `UnresolvedFunction`. This is intentional:\nthe server\u0027s `SparkConnectPlanner.transformUnresolvedFunction` treats the\nfunction name of an `UnresolvedFunction` as a **single identifier** when\n`is_user_defined_function` is `false`, which is what our `fn` helper produces.\nThat would break qualified names such as `call_function(\"db.my_func\", ...)`.\nIn contrast, `SparkConnectPlanner.transformCallFunction` runs the name through\n`parser.parseMultipartIdentifier`, so quoted and qualified names are resolved\ncorrectly. This matches the documented Scala contract, \"function name that\nfollows the SQL identifier syntax (can be quoted, can be qualified)\", and\nmatches PySpark Connect, whose `call_function` also emits a `CallFunction` node.\n\n`call_udf` delegates to `call_function`, mirroring Scala\u0027s\n`functions.scala`, where `call_udf(udfName, cols: _*) \u003d call_function(udfName, cols: _*)`.\n\n### Why are the changes needed?\n\nFor feature parity with Apache Spark.\n\n`call_function` is especially valuable in this client because it is a general\nescape hatch: it lets users invoke any SQL function available on the server,\nincluding the ones this library has not wrapped yet.\n\nAlthough this repository does not provide a UDF registration API, `call_udf`\nremains useful for invoking functions registered on the server through SQL\n`CREATE FUNCTION`.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this simply adds two new functions.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test suite, `UDFFunctionsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #536 from dongjoon-hyun/SPARK-59240.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "79af75a3c3d0e898710cf3dd6ce755dc52c71c0f",
      "tree": "33fc4cb097ed87ab6c3a737778086486200f3d21",
      "parents": [
        "a46c1feddadd17126481810063e5af0cf171df49"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:09:54 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:09:54 2026 -0700"
      },
      "message": "[SPARK-59239] Support `counter_diff` window function\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support the `counter_diff` window function.\n\n```swift\npublic func counter_diff(_ value: Column) -\u003e Column\npublic func counter_diff(_ value: Column, _ startTime: Column) -\u003e Column\n```\n\n### Why are the changes needed?\n\nFor feature parity with Apache Spark, which added `counter_diff` in `4.3.0` It converts a cumulative counter time series into the delta format and gracefully handles counter resets by returning `NULL`.\n- apache/spark#55828\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new function.\n\n### How was this patch tested?\n\nPass the CIs with newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #535 from dongjoon-hyun/SPARK-59239.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "a46c1feddadd17126481810063e5af0cf171df49",
      "tree": "313315f041b30feb1c26ae3106a6409036c17492",
      "parents": [
        "50a9139a4182e0685e999a228384188c2e604497"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:09:05 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:09:05 2026 -0700"
      },
      "message": "[SPARK-59238] Support partition transform functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support 5 partition transform functions in a new file,\n`Sources/SparkConnect/PartitionTransforms.swift`, and to add a `Column`-based\n`DataFrameWriterV2.partitionBy` overload so that these transforms can actually be used.\n\n| Function | Since | Signature |\n| --- | --- | --- |\n| `bucket` | 3.1.0 | `(Column, Column)`, `(Int32, Column)` |\n| `days` | 3.1.0 | `(Column)` |\n| `hours` | 3.1.0 | `(Column)` |\n| `months` | 3.1.0 | `(Column)` |\n| `years` | 3.1.0 | `(Column)` |\n\nScala groups these under `group partition_transforms` in `functions.scala`, so they are\nplaced in a dedicated file instead of `Functions.swift`. `bucket` takes an `Int32` overload\nbecause PySpark accepts `Union[Column, int]` for `numBuckets`; the literal is wrapped with\n`lit(...)`, matching `partitioning.bucket` in\n`python/pyspark/sql/connect/functions/partitioning.py`.\n\nIn addition, `DataFrameWriterV2.partitionBy(_ columns: Column...)` is added. The existing\n`partitionBy(_ columns: String...)` doc comment already claimed to accept \"columns or\ntransforms\", but only string column names could be passed, so there was no way to build a\n`years(col(\"ts\"))`-partitioned table.\n\n### Why are the changes needed?\n\nTo improve API coverage and to make the partition transforms usable end to end.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavior change. Only `DataFrameWriterV2.partitionBy` gains a new `Column`-variadic overload. This is a pure addition; the existing `String`-variadic overload is unchanged, so no existing code breaks. The 5 functions themselves are new additions to the API.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test suite, `PartitionTransformsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #534 from dongjoon-hyun/SPARK-59238.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "50a9139a4182e0685e999a228384188c2e604497",
      "tree": "93506c5659b784a8d49e10a950c25be05a921db4",
      "parents": [
        "55b9b1e12ffdc92f4ac1d9c0f767a4ff93346964"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:08:16 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:08:16 2026 -0700"
      },
      "message": "[SPARK-59237] Support `broadcast` function\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support the `broadcast` function.\n\n```swift\npublic func broadcast(_ df: DataFrame) async -\u003e DataFrame\n```\n\nUnlike most functions in `Functions.swift`, `broadcast` takes a `DataFrame` and returns a `DataFrame` marked with a `broadcast` join hint, mirroring PySpark\u0027s `broadcast(df) -\u003e df.hint(\"broadcast\")` and Scala\u0027s `def broadcast[U](df: Dataset[U]): df.type`.\n\n### Why are the changes needed?\n\nFor feature parity with Apache Spark. `broadcast` has been available since Apache Spark 1.6.0\nin PySpark and is a commonly used hint for broadcast hash joins.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, this PR adds a new public function, `broadcast`.\n\n```swift\nlet joined \u003d await left.join(broadcast(right), \"id\")\n```\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases in `FunctionsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #533 from dongjoon-hyun/SPARK-59237.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "55b9b1e12ffdc92f4ac1d9c0f767a4ff93346964",
      "tree": "e625e4590388169fd1210ff467589b02ae29c173",
      "parents": [
        "c913324d841b8399df5981f0c8f8763de2d02b71"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 19:42:45 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 19:42:45 2026 -0700"
      },
      "message": "[SPARK-59236] Support vector functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support 7 vector functions (11 overloads).\n\n| Function | Since | Signature |\n| --- | --- | --- |\n| `vector_cosine_similarity` | 4.3.0 | `(Column, Column)` |\n| `vector_inner_product` | 4.3.0 | `(Column, Column)` |\n| `vector_l2_distance` | 4.3.0 | `(Column, Column)` |\n| `vector_norm` | 4.3.0 | `(Column)`, `(Column, Float \\| Column)` |\n| `vector_normalize` | 4.3.0 | `(Column)`, `(Column, Float \\| Column)` |\n| `vector_avg` | 4.3.0 | `(Column)` |\n| `vector_sum` | 4.3.0 | `(Column)` |\n\nThe five scalar functions live in a new `VectorFunctions.swift`. The two aggregate functions,\n`vector_avg` and `vector_sum`, are added to the existing `AggregateFunctions.swift`, following\nthe `schema_of_variant_agg` precedent, while their tests stay with the rest of the family in\n`VectorFunctionsTests.swift`.\n\nDespite the name, these functions take no dedicated `VECTOR` type. They operate on `ARRAY\u003cFLOAT\u003e`\ncolumns, and all the vectors involved must have the same dimension.\n\nThe optional `degree` of `vector_norm` and `vector_normalize` is expressed as a Swift overload,\nmirroring the two Scala overloads. In addition to the `Column` overload, a `Float` overload is\nprovided because the server requires exactly a `FLOAT` for this argument:\n\n```\nSELECT vector_norm(ARRAY(3.0f, 4.0f), CAST(1.0 AS DOUBLE))\n[DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] ... The second parameter requires the \"FLOAT\" type,\nhowever \"CAST(1.0 AS DOUBLE)\" has the type \"DOUBLE\".\n```\n\nSince `lit(Double)` produces a `double` literal in this client, a `Double` overload would compile\nand then fail at analysis time on the server. The `Float` overload keeps `vector_norm(v, 2.0)`\ncorrect by construction, the same reasoning behind the `Int32` srid of `st_setsrid`.\n\n### Why are the changes needed?\n\nTo support the vector functions of Apache Spark 4.3.0 and improve the API coverage.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this PR only adds new functions.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test suite, `VectorFunctionsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #532 from dongjoon-hyun/SPARK-59236.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "c913324d841b8399df5981f0c8f8763de2d02b71",
      "tree": "0f4dce5f4b697bc0917599f6358918ed61ce960b",
      "parents": [
        "99496e69148b67cfdc4154a6ae258ec8206b40bf"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 19:39:41 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 19:39:41 2026 -0700"
      },
      "message": "[SPARK-59235] Support error, reflection and bitmap functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support the error, reflection, and bitmap functions.\n\n| Function | Since |\n| -------- | ----- |\n| `assert_true(col)` / `assert_true(col, errMsg)` | 3.1.0 |\n| `raise_error(errMsg)` | 3.1.0 |\n| `reflect(cols...)` | 3.5.0 |\n| `java_method(cols...)` | 3.5.0 |\n| `try_reflect(cols...)` | 4.0.0 |\n| `bitmap_bit_position(col)` | 3.5.0 |\n| `bitmap_bucket_number(col)` | 3.5.0 |\n| `bitmap_count(col)` | 3.5.0 |\n\nAll of them are classified as misc functions by PySpark and the Spark SQL Scala\nAPI, so they are added to the existing `MiscFunctions.swift` in alphabetical order.\n\nTwo details follow the upstream implementations:\n\n- `errMsg` is `Union[Column, str]` in PySpark, where a `str` is wrapped with\n  `lit()`. This is expressed as a `String` overload for `assert_true` and\n  `raise_error`, like the existing `schema_of_json` overload pair.\n- The reflection functions are variadic. Their first argument is the class name,\n  the second one is the method name, and the remaining ones are the method\n  arguments. No `String` overload is added for them because a bare string is a\n  column name rather than a literal in PySpark, so an overload would silently\n  mean the opposite of the upstream API.\n\nSince the reflection functions invoke an arbitrary Java static method on the\nserver, their doc comments note that the class must be on the Spark server\u0027s\nclasspath and that the class and the method names must never be built from\nuntrusted user input.\n\n### Why are the changes needed?\n\nTo improve the API coverage. These functions are available in PySpark and the\nSpark SQL Scala API, but were missing from this Swift client.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new feature.\n\n```swift\ntry await spark.range(1).select(\n  reflect(lit(\"java.util.UUID\"), lit(\"fromString\"), lit(\"a5cf6c42-0c85-418f-af6c-3e4e5b1328f2\")),\n  bitmap_bit_position(lit(123)),\n  bitmap_bucket_number(lit(123)),\n  bitmap_count(unhex(lit(\"FFFF\")))\n).show()\n```\n\n```\n+-------------------------------------------------------------------------+------------------------+-------------------------+-------------------------+\n|reflect(java.util.UUID, fromString, a5cf6c42-0c85-418f-af6c-3e4e5b1328f2)|bitmap_bit_position(123)|bitmap_bucket_number(123)|bitmap_count(unhex(FFFF))|\n+-------------------------------------------------------------------------+------------------------+-------------------------+-------------------------+\n|                                                     a5cf6c42-0c85-418...|                     122|                        1|                       16|\n+-------------------------------------------------------------------------+------------------------+-------------------------+-------------------------+\n```\n\n### How was this patch tested?\n\nPass the CIs with the new test cases in `MiscFunctionsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #531 from dongjoon-hyun/SPARK-59235.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "99496e69148b67cfdc4154a6ae258ec8206b40bf",
      "tree": "641aa40512cea308b59b32d310f9ea6b151e615f",
      "parents": [
        "e5acec92d7dc87ed0ab685a08ef16a39bb8884b9"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 16:10:39 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 16:10:39 2026 -0700"
      },
      "message": "[SPARK-59228] Support AES and HMAC functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support the AES and HMAC functions.\n\n| Function | Since |\n| -------- | ----- |\n| `aes_encrypt(input, key, mode, padding, iv, aad)` | 3.5.0 |\n| `aes_decrypt(input, key, mode, padding, aad)` | 3.5.0 |\n| `try_aes_decrypt(input, key, mode, padding, aad)` | 3.5.0 |\n| `hmac(key, message, algorithm)` | 4.3.0 |\n\nThe new functions live in a new `CryptoFunctions.swift` file. The existing\n`HashFunctions.swift` holds one-way digests, while these functions take a secret\nkey and perform symmetric encryption or message authentication.\n\nOptional arguments follow the upstream PySpark implementations exactly:\n\n- The AES functions fill omitted arguments with their default literals, so\n  `aes_encrypt` always sends 6 arguments and `aes_decrypt` / `try_aes_decrypt`\n  always send 5. Note that `aes_encrypt` has an `iv` argument while the two\n  decrypt functions do not.\n- `hmac` omits the optional `algorithm` argument instead of filling it, so it\n  sends 2 or 3 arguments. Since a Swift default argument value cannot change the\n  argument count, this is expressed as two overloads, matching the two Scala\n  `hmac` overloads.\n\nSince these functions are evaluated by the server and a key passed as a literal\nbecomes part of the query plan, the doc comments note that a literal key can be\nexposed through server logs and plan output.\n\n### Why are the changes needed?\n\nTo improve the API coverage. These functions are available in PySpark and the\nSpark SQL Scala API, but were missing from this Swift client.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new feature.\n\n```swift\nlet key \u003d lit(\"0000000000000000\")\nlet df \u003d try await spark.sql(\"SELECT \u0027Spark\u0027 AS a\")\ntry await df.select(\n  aes_decrypt(aes_encrypt(col(\"a\"), key), key).cast(\"STRING\")\n).show()\n```\n\n```\n+-------------------------------------------------------------+\n|CAST(aes_decrypt(aes_encrypt(a, 0000000000000000, GCM, ...)...|\n+-------------------------------------------------------------+\n|Spark                                                        |\n+-------------------------------------------------------------+\n```\n\n### How was this patch tested?\n\nPass the CIs with a new test suite, `CryptoFunctionsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #530 from dongjoon-hyun/SPARK-59228.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "e5acec92d7dc87ed0ab685a08ef16a39bb8884b9",
      "tree": "e121626a5d2d4ec583dd75e9ccd9eb91f538e250",
      "parents": [
        "bead6a15488d943bc1b2766416b5fbe1e4a59cbd"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 16:09:30 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 16:09:30 2026 -0700"
      },
      "message": "[SPARK-59227] Support base32 functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support 2 BASE32 (RFC 4648) functions.\n\n| Function | Since | Signature |\n| --- | --- | --- |\n| `to_base32` | 4.3.0 | `(Column)` |\n| `from_base32` | 4.3.0 | `(Column)` |\n\nThe functions are added to `StringFunctions.swift` next to the existing `base64` and\n`unbase64` functions, keeping the alphabetical order of the file.\n\n### Why are the changes needed?\n\nTo improve API coverage. This client supported only the BASE64 encoding so far.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new addition to the API.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test case, `StringFunctionsTests.selectBase32Functions`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #529 from dongjoon-hyun/SPARK-59227.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "bead6a15488d943bc1b2766416b5fbe1e4a59cbd",
      "tree": "c95fd048d1badad130cdfd6f77ed653c75ac6bdf",
      "parents": [
        "ed6803adf2075888fa27215b92dc8f341924bc98"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 16:08:47 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 16:08:47 2026 -0700"
      },
      "message": "[SPARK-59226] Support geospatial ST functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support 5 geospatial ST functions (10 overloads).\n\n| Function | Since | Signature |\n| --- | --- | --- |\n| `st_asbinary` | 4.1.0 | `(Column)` |\n| `st_asbinary` | 4.2.0 | `(Column, String \\| Column)` |\n| `st_geogfromwkb` | 4.1.0 | `(Column)` |\n| `st_geomfromwkb` | 4.1.0 | `(Column)` |\n| `st_geomfromwkb` | 4.2.0 | `(Column, Int32 \\| Column)` |\n| `st_setsrid` | 4.1.0 | `(Column, Int32 \\| Column)` |\n| `st_srid` | 4.1.0 | `(Column)` |\n\nThe functions live in a new `GeospatialFunctions.swift`.\n\nWhere the upstream argument is `Union[Column, int]` or `Union[Column, str]`, both a Swift\nprimitive and a `Column` overload are provided, following the existing `regexp_extract`\nprecedent. `srid` is taken as `Int32` because `lit(Int)` produces a `long` literal in this\nclient, while the ST functions expect an `INTEGER`.\n\nNote that the optional arguments of `st_asbinary` and `st_geomfromwkb` were added later than\nthe functions themselves:\n\n- `st_asbinary(geo, endianness)` is `since 4.2.0` (SPARK-56682).\n- `st_geomfromwkb(wkb, srid)` is `since 4.2.0` (SPARK-55295). The PySpark docstring is\n  missing the corresponding `versionchanged:: 4.2.0` note, but the commit is only contained\n  in `v4.2.0` and later tags.\n\n### Why are the changes needed?\n\nTo improve API coverage. `DataType` already models `.geometry(srid:)` and `.geography(srid:)`,\nand `DataFrameTests.dtypesGeospatial` already exercises the server behavior through raw SQL\nstrings, but no ST function was available in this client.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new addition to the API.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test suite, `GeospatialFunctionsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #528 from dongjoon-hyun/SPARK-59226.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "ed6803adf2075888fa27215b92dc8f341924bc98",
      "tree": "83fb3c1f97f76edeb7b10df7a0d2b91732a34ccf",
      "parents": [
        "87e6676f0bbe70c591c15cdd2f7e04bbb18c65df"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 16:07:38 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 16:07:38 2026 -0700"
      },
      "message": "[SPARK-59225] Support URL functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support 5 URL functions (10 overloads).\n\n| Function | Since | Signatures |\n| --- | --- | --- |\n| `parse_url` | 3.5.0 | `(Column, String)`, `(Column, Column)`, `(Column, String, String)`, `(Column, Column, Column)` |\n| `try_parse_url` | 4.0.0 | `(Column, String)`, `(Column, Column)`, `(Column, String, String)`, `(Column, Column, Column)` |\n| `url_encode` | 3.5.0 | `(Column)` |\n| `url_decode` | 3.5.0 | `(Column)` |\n| `try_url_decode` | 4.0.0 | `(Column)` |\n\nThe functions live in a new `UrlFunctions.swift`, matching the `url_funcs` group of the upstream `functions.scala` and the \"URL Functions\" section of the PySpark docs.\n\nWhere the upstream `partToExtract` and `key` arguments are `Union[Column, str]`, both a `String` and a `Column` overload are provided, following the existing `variant_get` precedent, because these arguments are string literals such as `\"HOST\"` or `\"QUERY\"` in practice.\n\n### Why are the changes needed?\n\nTo improve API coverage. No URL function was available in this client.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new addition to the API.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test suite, `UrlFunctionsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #527 from dongjoon-hyun/SPARK-59225.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    }
  ],
  "next": "87e6676f0bbe70c591c15cdd2f7e04bbb18c65df"
}
