)]}'
{
  "log": [
    {
      "commit": "5057ddd73cb5ee438270b8cd1bdbffcfef3d35a0",
      "tree": "078864d18f98a16cbb90f7c88e39de4a47839b5c",
      "parents": [
        "c0bf10a1480f6a6d5a94ed9ea1e565a9ef7118cc"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 23:39:15 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 23:39:15 2026 -0700"
      },
      "message": "[SPARK-59485] Use server-side aggregation for `DataFrame.count()`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to compute `DataFrame.count()` on the server side via `groupBy().count()`, like the Scala Spark Connect client.\n\nThe previous client-side counting logic remains as an internal `executeAndCount()` for `Catalog` operations and `isEmpty()`.\n\n### Why are the changes needed?\n\nTo match [Dataset.count()](https://github.com/apache/spark/blob/v4.2.0/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala#L1041-L1043) of the Scala Spark Connect client in Apache Spark [v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0) (2026-07-11), and to avoid sending every row to the client just to count them.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. The results are the same, but the server now computes the count instead of sending every row to the client. As in the Scala client, the server can skip expressions that counting doesn\u0027t need. For example, `select(assert_true(lit(false))).count()` no longer raises an error.\n\n### How was this patch tested?\n\nPass the CIs with the newly added and updated test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #566 from dongjoon-hyun/SPARK-59485.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "c0bf10a1480f6a6d5a94ed9ea1e565a9ef7118cc",
      "tree": "c81cb318185d81cf814072c087c7791ec47a3344",
      "parents": [
        "e3427aac36b31c7f53d119c06563c7a10e17b8d1"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 23:38:30 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 23:38:30 2026 -0700"
      },
      "message": "[SPARK-59484] Use `value` as the output column name of `DataFrame.toJSON`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to make `DataFrame.toJSON()` return a single column named `value`, like the Scala Spark Connect client.\n\n```swift\n-    return selectExpr(\"to_json(struct(*))\")\n+    return selectExpr(\"to_json(struct(*)) AS value\")\n```\n\n### Why are the changes needed?\n\nThe Scala Spark Connect client names the `toJSON` output column `value`.\n\n- [Apache Spark 4.2.0 (2026-07-11) `Dataset.toJSON`](https://github.com/apache/spark/blob/v4.2.0/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala#L1277-L1279)\n\n```scala\ndef toJSON: Dataset[String] \u003d {\n  select(to_json(struct(col(\"*\"))).as(\"value\")).as(StringEncoder)\n}\n```\n\nThe Swift client currently names the column after the expression, such as `to_json(struct(id))`. So the column name depends on the input schema, and code ported from Scala that refers to `value` fails.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. The output column is renamed from `to_json(struct(...))` to `value`, matching the Scala Spark Connect client. The JSON content is unchanged.\n\n**BEFORE**\n\n```swift\ntry await spark.range(2).toJSON().columns  // [\"to_json(struct(id))\"]\n```\n\n**AFTER**\n\n```swift\ntry await spark.range(2).toJSON().columns  // [\"value\"]\ntry await spark.range(2).toJSON().schema   // value STRING (nullable)\n```\n\n### How was this patch tested?\n\nPass the CIs with the updated test case. It checks the column name, the schema (`value STRING`, nullable), unchanged JSON content, a multi-column DataFrame, and a nested struct column.\n\nI also tested manually with an Apache Spark 4.2.0 Spark Connect server.\n\n```\n$ swift test --no-parallel --filter DataFrameTests\n...\nTest run with 152 tests in 2 suites passed after 22.965 seconds.\n```\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #565 from dongjoon-hyun/SPARK-59484.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "e3427aac36b31c7f53d119c06563c7a10e17b8d1",
      "tree": "3513c2bad92774e7c9328eed55b7a05a85deba85",
      "parents": [
        "e6d727931c4ab82dc33e98c3ba6cc2d9952b67b4"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 21:00:10 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 21:00:10 2026 -0700"
      },
      "message": "[SPARK-59483] Fix `DataFrame.(head|first)` to throw an error instead of crashing on empty results\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix `DataFrame.head()` and `first()` to throw `SparkConnectError.invalidState` instead of crashing on empty results.\n\n### Why are the changes needed?\n\nCurrently, `head()` and `first()` crash on an empty `DataFrame`.\n\n```swift\ntry await spark.range(0).head()  // Fatal error: Index out of range\n```\n\nApache Spark\u0027s `Dataset.head()` throws `NoSuchElementException: head of empty array` in this case ([v4.2.0](https://github.com/apache/spark/blob/v4.2.0/sql/api/src/main/scala/org/apache/spark/sql/Dataset.scala#L2786), 2026-07-11). This PR uses the same message.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavior change. Previously, `head()` and `first()` crashes and now they throw an error on an empty `DataFrame` . Both methods are already `async throws`, so this is source-compatible.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #564 from dongjoon-hyun/SPARK-59483.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "e6d727931c4ab82dc33e98c3ba6cc2d9952b67b4",
      "tree": "1322831e8647560cab37b833240682f4dafa8a50",
      "parents": [
        "28d0178805e267fe091b7108523da6d17da213f9"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 18:02:24 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 18:02:24 2026 -0700"
      },
      "message": "[SPARK-59481] Fix concurrent commands on the same session to return correct responses\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix `SparkConnectClient.execute` to collect responses per call via a local `Mutex` instead of the actor-shared `result` array.\n\n### Why are the changes needed?\n\nDue to actor reentrancy, concurrent commands on the same session can return the responses of other commands. For example, concurrent `DataStreamWriter.start` calls return the same `StreamingQuery`.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, this fixes concurrent commands on the same `SparkSession`. There is no API change.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test case.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #563 from dongjoon-hyun/SPARK-59481.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "28d0178805e267fe091b7108523da6d17da213f9",
      "tree": "991ce79b3d2fe6be74a6096c3a99d93a882ebaad",
      "parents": [
        "bd359a485d419dd8ece6694ded18dedc17a5e13b"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 16:04:30 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 16:04:30 2026 -0700"
      },
      "message": "[SPARK-59480] Fix concurrent actions on the same DataFrame to return correct results\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix `DataFrame` actions to use a per-call result instead of the actor-shared `batches` buffer, so that concurrent actions on the same `DataFrame` instance return correct results.\n\n- `DataFrame.execute()` now accumulates `RecordBatch`es into a local `Mutex` and returns them.\n- `collect()` and `collect(as:)` iterate the returned batches.\n- The unused `DataFrame.batches` and `addBatches` are removed.\n\n`head`, `take`, `first`, `tail`, and `show` create a new `DataFrame` and call `collect()`, so they are fixed together. `count()` already uses a local `Atomic` counter and is not affected.\n\n### Why are the changes needed?\n\n`execute()` cleared the actor-shared `batches` and then refilled it across `await` suspension points. Due to actor reentrancy, concurrent actions on the same `DataFrame` interleave and append into the same buffer, so they silently return wrong results.\n\n```swift\nlet df \u003d try await spark.range(300_000)\nasync let a \u003d df.collect()\nasync let b \u003d df.collect()\n// Before this PR, each returns more than 300000 rows.\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. Concurrent `collect()` and `collect(as:)` calls on the same `DataFrame` return correct results now. There is no API change.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test case.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #562 from dongjoon-hyun/SPARK-59480.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "bd359a485d419dd8ece6694ded18dedc17a5e13b",
      "tree": "33f4010bb19c51b34a7bf7adc932195e3cd718cc",
      "parents": [
        "0d10cfc948e727c62056d4a2937887b81724322d"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 13:55:59 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 13:55:59 2026 -0700"
      },
      "message": "[SPARK-59474] Support `DECIMAL` type in `createDataFrame` with `[[Sendable?]]` rows\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support the `DECIMAL` type in `createDataFrame` with `[[Sendable?]]` rows and a DDL schema string.\n\n- Add a `.decimal` case to `ConvertToArrow.toArrowColumn`, which uses the precision and scale of the schema.\n- Accept `Decimal` and integer values. Other values throw `SparkConnectError.InvalidType`.\n- Like `createDataFrame` with `Encodable` types, values are rounded `HALF_UP` to the scale, and values which do not fit in the precision throw an error.\n\n### Why are the changes needed?\n\nCurrently, `createDataFrame` throws `SparkConnectError.InvalidType` for `DECIMAL` columns.\n\n```swift\ntry await spark.createDataFrame([[Decimal(1)]], \"id DECIMAL(10, 2)\")\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavior change because previous it wasn\u0027t supported.\n\n`createDataFrame` now works with `DECIMAL(p, s)` columns. The results match Apache Spark 4.2.0 `createDataFrame(rows, schema)` in ANSI mode, which is the default.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #561 from dongjoon-hyun/SPARK-59474.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "0d10cfc948e727c62056d4a2937887b81724322d",
      "tree": "0021f19021966c492a3996b8fe6324345897da7f",
      "parents": [
        "f445890fa322a38a9a8bb88b0a5359b4cb0e67c4"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 12:46:16 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 12:46:16 2026 -0700"
      },
      "message": "[SPARK-59473] Support top-level `Decimal` decoding in `DataFrame.collect(as:)`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support top-level `Decimal` decoding in `DataFrame.collect(as:)`.\n\n### Why are the changes needed?\n\nFoundation\u0027s `Decimal.init(from:)` requests a keyed container of its internal fields. So, the following throws `ArrowError.invalid(\"Column for key \\\"exponent\\\" not found\")`, while Scala `as[BigDecimal]` returns `1.50`.\n\n```swift\ntry await spark.sql(\"SELECT CAST(1.5 AS DECIMAL(10,2)) AS v\").collect(as: Decimal.self)\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. `collect(as: Decimal.self)` and `collect(as: Decimal?.self)` now return values instead of throwing errors.\n\n### How was this patch tested?\n\nPass the CIs with the updated `CreateDataFrameTests.collectAsDecimal`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #560 from dongjoon-hyun/SPARK-59473.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "f445890fa322a38a9a8bb88b0a5359b4cb0e67c4",
      "tree": "2c2bee951bafee57c94842e785e1eab4bd02725a",
      "parents": [
        "87e0ab721af4a7fc3c67ad4ce59c42466b71de73"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 12:23:20 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 12:23:20 2026 -0700"
      },
      "message": "[SPARK-59472] Support `Decimal` properties in `createDataFrame` with `Encodable` types\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `Decimal` properties in `createDataFrame` with `Encodable` types.\n\n- Write the Arrow `Decimal` schema in `ArrowWriterHelper`.\n- Add `Decimal128BufferBuilder` to store the unscaled value as a 128-bit integer, rounded `HALF_UP` like Spark.\n- Throw an error for values which do not fit in `DECIMAL(38, 18)`.\n\n### Why are the changes needed?\n\nCurrently, `createDataFrame` fails with `InvalidArrowData` for `Encodable` types with `Decimal` properties.\n\n```swift\nstruct D: Codable, Sendable { let v: Decimal }\ntry await spark.createDataFrame([D(v: Decimal(string: \"-1.5\")!)])\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, `createDataFrame` now works with `Decimal` and `Decimal?` properties, matching the Scala `Dataset` behavior for `BigDecimal`.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #559 from dongjoon-hyun/SPARK-59472.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "87e0ab721af4a7fc3c67ad4ce59c42466b71de73",
      "tree": "2d40adf74ecffa79551f00e09621e9c6dcad3922",
      "parents": [
        "55ad4b798a49d7b3140c8bd48b6fb55fb1116d74"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 12:03:10 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 12:03:10 2026 -0700"
      },
      "message": "[SPARK-59471] Fix `DECIMAL` decoding to use the full signed 128-bit value\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix `DECIMAL` decoding to use the full signed 128-bit value.\n\n- `Decimal128Array.subscript` now reads the whole Arrow `Decimal128` value, which is a 128-bit little-endian two\u0027s complement integer. Previously, it read only the low 64 bits as an unsigned integer and always used a plus sign.\n- `ArrowArrayHolderImpl.loadArray` now uses `Decimal128Array` for `decimal128` children of `ARRAY` and `STRUCT` columns. Previously, it used `FixedArray\u003cDecimal\u003e`, which read the Arrow buffer as the raw memory of Foundation `Decimal`.\n\n### Why are the changes needed?\n\n`DECIMAL` values are silently decoded wrong when they are negative or do not fit in 64 bits. This affects `DataFrame.collect()`, `Row.getAsDecimal`, and `DataFrame.collect(as:)`.\n\n| SQL | Scala `as[T]` | Swift (before) |\n|---|---|---|\n| `SELECT CAST(-1.5 AS DECIMAL(10,2)) AS v` | `-1.50` | `184467440737095514.66` |\n| `SELECT CAST(12345678901234567890.5 AS DECIMAL(38,2)) AS v` | `12345678901234567890.50` | `170827812586263823.94` |\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. `DECIMAL` values are now decoded correctly, matching Scala `Dataset.as[T]`. This includes negative values, 38-digit values, and `DECIMAL` elements inside `ARRAY` and `STRUCT` columns.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases. The expected values were verified with `spark-shell` 4.2.0.\n\n- `DataFrameTests.decimalValues` covers `collect()`, `Row.getAsDecimal`, and a nested `ARRAY\u003cDECIMAL\u003e`.\n- `CreateDataFrameTests.collectAsDecimal` covers `collect(as:)`.\n\nBoth tests cover the following values:\n- `-1.5` as `DECIMAL(10,2)`\n- `12345678901234567890.50` as `DECIMAL(38,2)`\n- the max and min values of `DECIMAL(38,0)`\n- `-0.000000000000000001` as `DECIMAL(38,18)`\n- `0` and `NULL`\n\n```bash\nswift test --no-parallel --filter \u0027DataFrameTests|CreateDataFrameTests|RowTests|SQLTests\u0027\n```\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #558 from dongjoon-hyun/SPARK-59471.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "55ad4b798a49d7b3140c8bd48b6fb55fb1116d74",
      "tree": "e41b08a47701bfc5ecf944caa34fb5dd0d3edcc3",
      "parents": [
        "3fb5f071e7afcda30a44c79b1f9e64456d349e8c"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 11:13:14 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 11:13:14 2026 -0700"
      },
      "message": "[SPARK-59470] Respect `spark.sql.caseSensitive` when resolving column names in `DataFrame.collect(as:)`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to respect `spark.sql.caseSensitive` when resolving column names in `DataFrame.collect(as:)`, like Scala `Dataset.as[T]`. If more than one column matches a property, it throws an `ArrowError` instead of silently taking the last column.\n\n| `spark.sql.caseSensitive` | Query | Scala `as[Person]` | Swift `collect(as: Person.self)` |\n|---|---|---|---|\n| `false` | `SELECT \u0027Alice\u0027 AS NAME, 30 AS AGE` | `Person(Alice,30)` | `Person(name: \"Alice\", age: 30)` |\n| `false` | `SELECT \u0027a\u0027 AS name, \u0027b\u0027 AS NAME, 30 AS age` | `AMBIGUOUS_REFERENCE` | `Column for key \"name\" is ambiguous, ...` |\n| `true` | `SELECT \u0027Alice\u0027 AS NAME, 30 AS AGE` | `UNRESOLVED_COLUMN` | `Column for key \"name\" not found` |\n| `true` | `SELECT \u0027a\u0027 AS name, \u0027b\u0027 AS NAME, 30 AS age` | `Person(a,30)` | `Person(name: \"a\", age: 30)` |\n\n### Why are the changes needed?\n\nPreviously, `collect(as:)` matched columns by exact name only. A column differing only in case threw `Column for key \"name\" not found`, or silently decoded to `nil` for Optional properties. Scala returns the value because `spark.sql.caseSensitive` is `false` by default.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, `DataFrame.collect(as:)` is not released yet.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #557 from dongjoon-hyun/SPARK-59470.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "3fb5f071e7afcda30a44c79b1f9e64456d349e8c",
      "tree": "f9ac159b56c6f98d7c48ad01bda8e13b074c1b84",
      "parents": [
        "2c151201b332e2f9949746deb085a41f12ad8a90"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 10:26:46 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Sep 13 10:26:46 2026 -0700"
      },
      "message": "[SPARK-59465] Support integral to floating-point upcasts in `DataFrame.collect(as:)`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support upcasts from integral types to `Float` and `Double` in `DataFrame.collect(as:)`. The rule follows Apache Spark\u0027s numeric precedence (`Byte \u003c Short \u003c Int \u003c Long \u003c Float \u003c Double`).\n\nThese conversions still throw `ArrowError`, as in Scala `Dataset.as[T]`:\n- `DOUBLE` into `Float`\n- floating-point types into integral types\n- `DECIMAL` into `Float` or `Double`\n- integral types into `Bool`\n- `STRING` into numeric types\n\n### Why are the changes needed?\n\nTo match Scala `Dataset.as[T]`. For example, `SELECT 30 AS v` decodes into a `Double` field in Scala, but Swift threw `Cannot decode Double for v`.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavior change. Previous it throws an error.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #556 from dongjoon-hyun/SPARK-59465.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "2c151201b332e2f9949746deb085a41f12ad8a90",
      "tree": "38fbe95e5f868c15a63b471803efd21ab94ad5d0",
      "parents": [
        "1c877f386ff0377719ae5d5cd1624a8e244427af"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Sep 12 22:00:17 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Sep 12 22:00:17 2026 -0700"
      },
      "message": "[SPARK-59464] Fix `DATE` handling for dates before 1970 and after 2106\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix the vendored Arrow `Date32` and `Date64` code to handle Arrow date values as signed integers.\n\n1. `Date32Array` loads `Int32` instead of `UInt32`, and multiplies by `86400` in `TimeInterval`. Loading `Int32` alone is not enough because `Int32 * 86400` overflows for dates after `2038-01-19`.\n2. `Date64Array` loads `Int64` instead of `UInt64`.\n3. `Date32BufferBuilder` uses `.rounded(.down)` instead of truncating toward zero, so pre-epoch values are floored to their day just like post-epoch values.\n\nThe same bugs exist in upstream `apache/arrow-swift`; they will be reported there separately.\n\n### Why are the changes needed?\n\nArrow `Date32` and `Date64` are signed day and millisecond counts since the UNIX epoch, and Spark sends `DATE` columns as `Date32`. Previously:\n\n- `Date32Array` read a day such as `-1` (`1969-12-31`) as `4294967295`, and `UInt32 * 86400` trapped with `Swift runtime failure: arithmetic overflow`. Only `1970-01-01` through `2106-02-07` could be read; any other `DATE` crashed the client process in both debug and release builds, without any error message. This affects `DataFrame.collect()`, `DataFrame.collect(as:)`, and `ARRAY`/`STRUCT` columns containing `DATE`.\n- `Date64Array` read negative milliseconds as huge positive values.\n- `createDataFrame` stored a pre-epoch `Date` that is not at UTC midnight one day later. For example, `1969-12-31T12:00:00Z` was stored as `1970-01-01`.\n\nReproducer (crashes before this fix, passes after):\n\n```swift\nlet spark \u003d try await SparkSession.builder.getOrCreate()\nlet rows \u003d try await spark.sql(\"SELECT DATE\u00271969-12-31\u0027\").collect()\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavior change for dates from `1970-01-01` to `2106-02-07`. This is a bug fix: `DATE` values outside that range can now be collected instead of crashing the client, and `createDataFrame` stores pre-epoch dates on the correct day.\n\n### How was this patch tested?\n\nPass the CIs with newly added test cases.\n\n- `DataFrameTests.collectDate`\n- `CreateDataFrameTests.dateType`\n- `CreateDataFrameTests.collectAsWithDateBeforeEpoch`\n\nI also verified manually against an Apache Spark 4.2.0 Connect server that all three new tests crash with signal 5 without this fix.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #555 from dongjoon-hyun/SPARK-59464.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "1c877f386ff0377719ae5d5cd1624a8e244427af",
      "tree": "44a75676d9bfbaf402f083a0f7693b4e33ddb511",
      "parents": [
        "4c100a2ba08f140ef3ba74b08a64887978e819c1"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Sep 12 20:47:40 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Sep 12 20:47:40 2026 -0700"
      },
      "message": "[SPARK-59463] Fix FlatBuffers nesting assertion on `TIMESTAMP` columns in `createDataFrame`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to create the timezone string offset before `startTimestamp` in `ArrowWriterHelper.toFBType`. This is the same fix as upstream `apache/arrow-swift`.\n- https://github.com/apache/arrow-swift/pull/148 (2026-03-25)\n\n### Why are the changes needed?\n\nFlatBuffers requires strings to be created before starting the table that references them. Otherwise, debug builds crash in `createDataFrame` with `TIMESTAMP` columns.\n\n```\n$ swift test --no-parallel --filter \"CreateDataFrameTests/supportedTypes\"\n...\nFlatBuffers/FlatBufferBuilder.swift:339: Assertion failed: Object serialization must not be nested\n```\n\nRelease builds are not affected because `assert` is disabled there.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavior change in release builds.\n\n### How was this patch tested?\n\nPass the CIs and manually run `CreateDataFrameTests` with a debug build.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #554 from dongjoon-hyun/SPARK-59463.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "4c100a2ba08f140ef3ba74b08a64887978e819c1",
      "tree": "9bda3efd3e2eee7437613c6f725bce783ee8746a",
      "parents": [
        "4cb592c79c029f718ce59d7a8fa2d028e08215e2"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Sep 12 20:46:55 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Sep 12 20:46:55 2026 -0700"
      },
      "message": "[SPARK-59462] Support safe upcasts and throw errors instead of crashing in `DataFrame.collect(as:)`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to throw `ArrowError.invalid` instead of crashing in `DataFrame.collect(as:)` when a value cannot be decoded into the target Swift type. It also allows Scala-style safe numeric upcasts (a narrower signed integer into a wider one, and `Float` into `Double`).\n\n```swift\n// Upcast: INT -\u003e Int64\ntry await spark.sql(\"SELECT 30 AS age\").collect(as: Int64.self)\n\n// Throws ArrowError.invalid(\"Cannot decode Int32 for column 0\")\ntry await spark.sql(\"SELECT 30L AS age\").collect(as: Int32.self)\n```\n\n### Why are the changes needed?\n\n`ArrowDecoder` force-unwraps decoded values and converts signed values with `UInt(val)`. As a result, a type mismatch, a `NULL` value, or a negative value into `UInt` crashes the whole process.\n\n```swift\nspark.sql(\"SELECT CAST(NULL AS STRING) AS name, 1L AS age\").collect(as: Person.self)\n// Fatal error: Unexpectedly found nil while unwrapping an Optional value\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nNo for the released versions because `DataFrame.collect(as:)` exists only in the `main` branch.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #553 from dongjoon-hyun/SPARK-59462.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "4cb592c79c029f718ce59d7a8fa2d028e08215e2",
      "tree": "a2098d27e609333014b5fbb51af5639c67f35d88",
      "parents": [
        "70b6dff0201f235488250b0a2665b6774a9170eb"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 11 21:09:51 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 11 21:09:51 2026 -0700"
      },
      "message": "[SPARK-59450] Support Codable in `createDataFrame` and `DataFrame.collect`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `Codable`-based DataFrame creation and collection in `SparkSession` and `DataFrame`, leveraging the in-tree `ArrowEncoder` and `ArrowDecoder`.\n\nSpecifically:\n1. **`SparkSession.createDataFrame` with `Encodable`**:\n   - `createDataFrame\u003cT: Encodable\u003e(_ data: [T]) async throws -\u003e DataFrame`: Automatically encodes an array of `Encodable` instances into an Apache Arrow `RecordBatch` via `ArrowEncoder`, infers the Spark DDL schema from the batch schema, and builds a `LocalRelation` or `CachedLocalRelation` (for payloads \u003e\u003d 1MiB).\n   - `createDataFrame\u003cT: Encodable\u003e(_ data: [T], _ schema: String)` and `createDataFrame\u003cT: Encodable\u003e(_ data: [T], _ schema: StructType)`: Supports explicit schema overrides and empty datasets with schema.\n2. **`DataFrame.collect(as:)` with `Decodable`**:\n   - `collect\u003cT: Decodable\u003e(as type: T.Type \u003d T.self) async throws -\u003e [T]`: Executes the plan and directly decodes each Arrow `RecordBatch` into `[T]` using `ArrowDecoder`, eliminating intermediate untyped `Row` allocations and dictionary boxing.\n3. **`ArrowEncoder` and `ArrowDecoder` enhancements**:\n   - Added support for Swift native `Int` (mapped to `Int64`) and `UInt` (mapped to `UInt64`) across keyed, unkeyed, and single-value containers.\n   - Added support for decoding `Date` and `TimestampNanos` from Arrow `Timestamp` columns (seconds, milliseconds, microseconds, and nanoseconds).\n4. **Schema mapping utilities**:\n   - Added `DataType.init(_ arrowType: ArrowType) throws` and `StructType.init(_ arrowSchema: ArrowSchema) throws` to translate Arrow schemas to Spark SQL data types.\n\n```swift\nstruct Person: Codable, Sendable, Equatable {\n  let name: String\n  let age: Int\n}\n\nlet people \u003d [Person(name: \"Alice\", age: 20), Person(name: \"Bob\", age: 25)]\n\n// 1. Create DataFrame directly from Swift models (auto-inferred schema)\nlet df \u003d try await spark.createDataFrame(people)\n\n// 2. Collect query results directly into Swift models\nlet results: [Person] \u003d try await df.filter(\"age \u003e\u003d 21\").collect(as: Person.self)\n```\n\n### Why are the changes needed?\n\nCurrently, `SparkSession.createDataFrame` only accepts untyped rows `[[Sendable?]]` with an explicit DDL string or `StructType` schema, and `DataFrame.collect()` only returns untyped `[Row]`.\n\nThese new APIs provide a type-safe, Dataset-like programming experience in Swift, allowing developers to work seamlessly with native Swift `Codable` structs.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavior change because this PR adds only new public generic methods.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Gemini 3.8 Flash (High)\n\nCloses #552 from dongjoon-hyun/SPARK-59450.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "70b6dff0201f235488250b0a2665b6774a9170eb",
      "tree": "86e6d0841a5ae3e392d20b5bbe7f7af7b50c2fc5",
      "parents": [
        "d71ae40685d645ad4379d432baa9a590b345aef4"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 11 15:36:58 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 11 15:36:58 2026 -0700"
      },
      "message": "[SPARK-59449] Support type-safe getters in `Row`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support type-safe getters and null-check inspection on `Row`.\n\nSpecifically, this PR introduces:\n1. **Null checking**:\n   - `isNullAt(_ i: Int) throws -\u003e Bool`\n   - `isNullAt(_ name: String) throws -\u003e Bool`\n2. **Generic `getAs` accessors**:\n   - `getAs\u003cT\u003e(_ i: Int, _ type: T.Type \u003d T.self) throws -\u003e T`\n   - `getAs\u003cT\u003e(_ name: String, _ type: T.Type \u003d T.self) throws -\u003e T`\n   - Supports both contextual type inference (`let val: Int \u003d try row.getAs(0)`) and explicit type passing (`try row.getAs(0, Int.self)`).\n   - Gracefully handles optional types: requesting an optional type (e.g. `String?.self`) on a `nil` column returns `nil` instead of throwing.\n3. **Convenience typed getters (Index and column name overloads)**:\n   - `getAsBool(_ name: String) throws -\u003e Bool` (alongside existing `getAsBool(_ i: Int)`)\n   - `getAsInt(_ i: Int)` / `getAsInt(_ name: String)`: supports type coercion from any `FixedWidthInteger` (`Int8`, `Int16`, `Int32`, `Int64`, `Int`), throwing `InvalidType` on integer overflow.\n   - `getAsInt64(_ i: Int)` / `getAsInt64(_ name: String)`: supports type coercion from any `FixedWidthInteger`.\n   - `getAsDouble(_ i: Int)` / `getAsDouble(_ name: String)`: supports `Double` and `Float` widening.\n   - `getAsString(_ i: Int)` / `getAsString(_ name: String)`\n   - `getAsDate(_ i: Int)` / `getAsDate(_ name: String)`: supports `Date` and `TimestampNanos.date`.\n   - `getAsTimestampNanos(_ i: Int)` / `getAsTimestampNanos(_ name: String)`: supports `TimestampNanos` and `Date`.\n   - `getAsDecimal(_ i: Int)` / `getAsDecimal(_ name: String)`: supports `Decimal` and `FixedWidthInteger`.\n4. **Standardized error handling**:\n   - Throws `SparkConnectError.InvalidArgument` if column index is out of bounds.\n   - Throws `SparkConnectError.UnsupportedOperation` if accessing by column name on a `Row` without a schema.\n   - Throws `SparkConnectError.ColumnNotFound` if accessing by an unknown column name.\n   - Throws `SparkConnectError.InvalidType` if the column value is `nil` (for non-optional `T`), type mismatch, or integer overflow occurs.\n\n### Why are the changes needed?\n\nPreviously, `Row` only provided untyped `get(_ i: Int) throws -\u003e Sendable` and a single typed getter `getAsBool(_ i: Int)`. Users had to manually downcast column values (e.g., `try row.get(i) as? Int` or `try row[\"name\"] as? String`).\n\nFurthermore, query results decoded from Arrow batches in `DataFrame+Actions.swift` hold sized integer types (`Int8`, `Int16`, `Int32`, `Int64`). A direct cast such as `as? Int` or `as? Double` would fail at runtime even if the data represented an integer or floating-point value.\n\nThese new APIs provide idiomatic, type-safe field extraction with automatic type coercion, bringing feature parity with Scala/PySpark `Row`.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavior change because this PR adds only new public methods (`isNullAt`, `getAs`, `getAsInt`, `getAsInt64`, `getAsDouble`, `getAsString`, `getAsDate`, `getAsTimestampNanos`, `getAsDecimal`, and `getAsBool(_ name: String)`) to `Row`.\n\n### How was this patch tested?\n\nPass the CIs with the newly added unit test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Gemini 3.8 Flash (High)\n\nCloses #551 from dongjoon-hyun/SPARK-59449.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "d71ae40685d645ad4379d432baa9a590b345aef4",
      "tree": "98be20ef314bf1341aa3f2d6cfdfc13565f13dee",
      "parents": [
        "538b05da288c482ef0775a8f4db29eb3739d5dae"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 10 21:59:09 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 10 21:59:09 2026 -0700"
      },
      "message": "[SPARK-59429] Pass plan ID in `DataFrame.colRegex`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to pass the plan ID in `DataFrame.colRegex`, like PySpark, now that `DataFrame` relations have plan IDs (SPARK-59414).\n\n- Set `planID` of `UnresolvedRegex` in `colRegex`.\n- Update the doc comment. The server uses the plan ID only for a column name without backticks, which is resolved as a regular column reference bound to the `DataFrame` like `col(_:)`. A regex enclosed in backticks is still not bound to the `DataFrame`.\n\n### Why are the changes needed?\n\nWhen SPARK-59399 added `colRegex`, this client had no plan IDs. As a result, a column name without backticks could not be disambiguated when both sides of a join had it.\n\n```swift\nlet df \u003d try await spark.range(3)\nlet df1 \u003d await df.filter(\"id \u003e 0\")\nlet df2 \u003d await df.select(\"id\")\nlet joined \u003d await df1.join(df2, joinExprs: df1.colRegex(\"id\") \u003d\u003d df2.colRegex(\"id\"))\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. `colRegex` is not released yet.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test case.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #550 from dongjoon-hyun/SPARK-59429.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "538b05da288c482ef0775a8f4db29eb3739d5dae",
      "tree": "881d4890a6280db638a1c7f224216157c9ae51b8",
      "parents": [
        "830b64e6f708e0906e8a6ab6b4207bb1e4db5ccb"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 10 19:44:23 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 10 19:44:23 2026 -0700"
      },
      "message": "[SPARK-59420] Support `withField` and `dropFields` in `Column`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `withField` and `dropFields` in `Column`.\n\n```swift\n// {\"a\":1,\"b\":2,\"c\":3} -\u003e {\"a\":1,\"b\":20,\"c\":3}\ndf.select(col(\"s\").withField(\"b\", lit(20)))\n\n// {\"a\":1,\"b\":2,\"c\":3} -\u003e {\"a\":1}\ndf.select(col(\"s\").dropFields(\"b\", \"c\"))\n\n// Nested fields: {\"a\":{\"a\":1,\"b\":2}} -\u003e {\"a\":{\"a\":1}}\ndf.select(col(\"s\").dropFields(\"a.b\"))\n```\n\nBoth methods are built on the existing `Expression.UpdateFields` proto, following PySpark Connect.\n- `withField` sets `struct_expression`, `field_name`, and `value_expression`.\n- `dropFields` leaves `value_expression` unset, because the server treats an unset value expression as a drop. Multiple field names are nested in order, like `update_fields(update_fields(s, b), c)`.\n\n`dropFields` requires at least one field name through its signature, `dropFields(_ fieldName: String, _ fieldNames: String...)`, so an empty call is rejected at compile time instead of at runtime.\n\n### Why are the changes needed?\n\nFor feature parity with PySpark, which has supported `Column.withField` and `Column.dropFields` since Apache Spark 3.1.0 and Spark Connect since 3.4.0. These APIs let users add, replace, or drop fields of struct columns, including nested fields.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, this PR adds two new public APIs, `Column.withField` and `Column.dropFields`.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #549 from dongjoon-hyun/SPARK-59420.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "830b64e6f708e0906e8a6ab6b4207bb1e4db5ccb",
      "tree": "7c4c088ced0d68c60ae4345be4c8b6f572da51fe",
      "parents": [
        "eaa42ebd0288c2126b1a8be877c020b8e3e748bd"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 10 17:04:56 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 10 17:04:56 2026 -0700"
      },
      "message": "[SPARK-59416] Support `col` and subscript column accessors in `DataFrame`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `col` and subscript column accessors in `DataFrame`.\n\n```swift\npublic nonisolated func col(_ colName: String) -\u003e Column\npublic nonisolated subscript(_ colName: String) -\u003e Column\n```\n\nUnlike the global `col(_:)` function, the returned `Column` carries the plan ID of the `DataFrame` (assigned since SPARK-59414), so the server can tell apart the columns with the same name in multiple `DataFrame`s.\n\n```swift\nlet df \u003d try await spark.range(3)\nlet df1 \u003d await df.filter(\"id \u003e 0\")\nlet df2 \u003d await df.select(\"id\")\nlet joined \u003d await df1.join(df2, joinExprs: df1[\"id\"] \u003d\u003d df2[\"id\"])\ntry await joined.select(df1[\"id\"], df2.col(\"id\")).show()\n```\n\n- `\"*\"` becomes `UnresolvedStar` with `plan_id`.\n- A name ending with `\".*\"`, e.g. `\"s.*\"`, becomes `UnresolvedStar` with `unparsed_target` only, like the global `col(_:)`. The server rejects a star with both a target and a plan ID (`CONNECT_INVALID_PLAN.UNRESOLVED_STAR_WITH_BOTH_TARGET_AND_PLAN_ID`).\n- Other names become `UnresolvedAttribute` with `plan_id`.\n\nThe plan ID of the root relation is stored as an immutable property when a `DataFrame` is created, so both accessors are `nonisolated` and don\u0027t require `await`.\n\n### Why are the changes needed?\n\nTo resolve ambiguous column references like `Dataset.col` and `Dataset.apply` in Scala and `DataFrame.__getitem__` in PySpark. Spark Connect has supported this since Apache Spark [v3.4.0](https://github.com/apache/spark/releases/tag/v3.4.0) (2023-04-07) via SPARK-41812 and SPARK-41823. No version gating is needed for the supported Apache Spark 4.x servers.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this adds only new public APIs, `DataFrame.col` and `DataFrame.subscript`.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #548 from dongjoon-hyun/SPARK-59416.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "eaa42ebd0288c2126b1a8be877c020b8e3e748bd",
      "tree": "c206f8863347b41b639567e01c4efe864c6a6c70",
      "parents": [
        "cd71d18d5d3e446e78b2f898ecc26146dfb06473"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 10 13:39:29 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 10 13:39:29 2026 -0700"
      },
      "message": "[SPARK-59414] Assign plan IDs to `DataFrame` relations\n\n### What changes were proposed in this pull request?\n\nThis PR proposes to assign a unique plan ID to the root relation of every `DataFrame`.\n\n- A file-scope `Atomic\u003cInt64\u003e` generator hands out monotonically increasing plan IDs, like the\n  existing `lambdaVariableID` in `HigherOrderFunctions.swift`.\n- Both SQL-based initializers now delegate to `init(spark:plan:)`, so every `DataFrame` goes\n  through a single place that stamps the plan ID.\n- An existing plan ID is preserved. For example, `toDF()` without arguments reuses `self.plan`,\n  and keeping its ID is what lets column references of the original `DataFrame` keep working.\n  This mirrors Scala\u0027s `SparkSession.newDataset`, which only tags newly created roots.\n- `DataFrameInternalTests.removeCachedRemoteRelation` now disables the server-side plan cache,\n  because a plan cached under the `DataFrame`\u0027s plan ID otherwise keeps answering `count()`\n  after the cached remote relation is removed.\n\n### Why are the changes needed?\n\nThe server uses plan IDs to resolve column references that are bound to a specific `DataFrame`,\nwhich is how ambiguous columns of a self-join are disambiguated. This mechanism landed in Apache\nSpark [v3.4.0](https://github.com/apache/spark/releases/tag/v3.4.0) (2023-04-07) via SPARK-41812\n(server and PySpark) and SPARK-41823 (Scala client), both backported to `branch-3.4`. This PR is\nthe prerequisite for supporting `DataFrame`-bound column accessors in this client.\n\n- apache/spark#39925\n\n### Does this PR introduce _any_ user-facing change?\n\nNo new API.\n\nHowever, since relations now carry plan IDs, the server-side session plan cache (SPARK-47818,\navailable since Apache Spark 4.0.0) applies to this client for the first time, exactly as it\nalready does for the Scala and Python Connect clients. Concretely, when a single `DataFrame`\nobject created by `spark.read.\u003cformat\u003e(path)` is reused after new files are appended to that\npath, it reports the result of its first execution instead of re-listing the files. `DataFrame`s\nof catalog tables, SQL queries, and temp views stay fresh, because only the unresolved plan is\ncached for them.\n\n- apache/spark#46012\n\n### How was this patch tested?\n\nPass the CIs with a new `DataFrameInternalTests.planID` test.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #547 from dongjoon-hyun/SPARK-59414.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "cd71d18d5d3e446e78b2f898ecc26146dfb06473",
      "tree": "163a7aba94977b87ec08f87f00b1cda7f9f76f5c",
      "parents": [
        "702be5097856c57558eeea59912950bd89839382"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 10 12:12:26 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 10 12:12:26 2026 -0700"
      },
      "message": "[SPARK-59399] Support `colRegex` in `DataFrame`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `colRegex` in `DataFrame`.\n\n```swift\npublic nonisolated func colRegex(_ colName: String) -\u003e Column\n```\n\nIt returns a `Column` wrapping the `UnresolvedRegex` expression. The required protobuf definition (`Expression.UnresolvedRegex`) already exists, so no code generation is involved.\n\nExample:\n\n```swift\nlet df \u003d try await spark.sql(\"SELECT 1 a1, 2 a2, 3 b1\")\n// Select the columns whose names start with `a`, i.e., `a1` and `a2`.\ntry await df.select(df.colRegex(\"`a.*`\")).show()\n```\n\nAs with the other Spark clients, only a column name enclosed in backticks is interpreted as a Java regular expression, and its case sensitivity follows `spark.sql.caseSensitive`. A column name without backticks is resolved as a regular column reference.\n\nNote that this client does not assign plan IDs to relations, so the `plan_id` field is not set. The server ignores `plan_id` for backtick-quoted regexes, so the regex behavior is identical to PySpark. The only difference is that a column name without backticks cannot be disambiguated when both sides of a self-join have it. This limitation is documented in the API doc.\n\n### Why are the changes needed?\n\nTo let users select multiple columns by a regex on column names, closing a gap with the other Spark clients.\n\n`Dataset.colRegex` was added in Apache Spark 2.3.0 via SPARK-12139, PySpark exposes `DataFrame.colRegex` since 2.3.0 via SPARK-23081, and Spark Connect supports it since 3.4.0 via SPARK-41438. No version gating is needed for the supported Apache Spark 4.x servers.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, this adds a new public API, `DataFrame.colRegex`.\n\n### How was this patch tested?\n\nPass the CIs with newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #546 from dongjoon-hyun/SPARK-59399.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "702be5097856c57558eeea59912950bd89839382",
      "tree": "42fc6907a7efebdad040bf3a6c930163e9394b47",
      "parents": [
        "23103d671fa86f7aaab7d35f20afd540add3a956"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 09 15:06:03 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 09 15:06:03 2026 -0700"
      },
      "message": "[SPARK-59391] Support `groupingSets` in `DataFrame`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `groupingSets` in `DataFrame`.\n\n```swift\npublic func groupingSets(_ groupingSets: [[String]], _ cols: String...) -\u003e GroupedData\n```\n\n`GroupedData` is extended with an optional `groupingSets` stored property, which\n`buildAggregate` translates into the `Aggregate.grouping_sets` field, mirroring how\nthe existing `pivot` property is handled. The `init` gains a defaulted parameter, so\nthe existing `groupBy`/`rollup`/`cube`/`pivot` call sites are unchanged.\n\nThe required protobuf definitions (`GROUP_TYPE_GROUPING_SETS` and\n`Aggregate.GroupingSets`) already exist, so no code generation is involved.\n\nExample:\n\n```swift\nlet df \u003d try await spark.sql(\"SELECT * FROM dealer\")\ntry await df.groupingSets([[\"city\", \"car_model\"], [\"city\"], []], \"city\", \"car_model\")\n  .agg(\"sum(quantity) sum\").orderBy(\"city\", \"car_model\").show()\n```\n\n### Why are the changes needed?\n\nTo provide a `DataFrame`-level API for multi-dimensional aggregation over an explicit\nlist of group combinations, closing a gap with the other Spark clients.\n\nUnlike `rollup` and `cube`, which derive the combinations automatically, `groupingSets`\nlets users specify arbitrary combinations, including the empty set that aggregates over\nall rows. `Dataset.groupingSets` was added in Apache Spark 4.0.0 via SPARK-45929, and\nPySpark exposes `DataFrame.groupingSets` since 4.0.0.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, this adds a new public API, `DataFrame.groupingSets`. There is no behavior change\nfor the existing `groupBy`, `rollup`, `cube`, and `pivot` APIs.\n\n### How was this patch tested?\n\nPass the CIs with newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #545 from dongjoon-hyun/SPARK-59391.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "23103d671fa86f7aaab7d35f20afd540add3a956",
      "tree": "cac93853fc722a411f0a363f73febc07564133b7",
      "parents": [
        "80e86b661a59babb69b999bc8ec9068851a5d276"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 09 12:34:13 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 09 12:34:13 2026 -0700"
      },
      "message": "[SPARK-59380] Upgrade `actions/setup-java` to v6\n\n### What changes were proposed in this pull request?\n\nThis PR aims to upgrade `actions/setup-java` from v5 to v6 in the `build_and_test.yml` workflow.\n\n### Why are the changes needed?\n\nTo use the latest `actions/setup-java` release.\n\n- [v6.0.0 (2026-08-24)](https://github.com/actions/setup-java/releases/tag/v6.0.0)\n- [v6.0.1 (2026-09-09)](https://github.com/actions/setup-java/releases/tag/v6.0.1)\n\nv6 migrates the Zulu distribution to the Azul Metadata API and moves the action to ESM. The inputs used in this repository (`distribution` and `java-version`) are unchanged.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5.1\n\nCloses #544 from dongjoon-hyun/SPARK-59380.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "80e86b661a59babb69b999bc8ec9068851a5d276",
      "tree": "0fc2e99c88c84ee9363f0ed1121656476362e8b0",
      "parents": [
        "ce19490d269c80f67465fe807f68b05f04f13bdb"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 12:19:24 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 12:19:24 2026 -0700"
      },
      "message": "[SPARK-59264] Cache Apache Spark distributions in GitHub Actions jobs\n\n### What changes were proposed in this pull request?\n\nThis PR adds a `setup-spark` composite action that caches the downloaded Apache Spark distribution tarball, and switches all seven macOS integration test jobs to use it.\n\n- New `.github/actions/setup-spark/action.yml`:\n  - `actions/cachev4` on `~/spark-dist/spark-\u003cversion\u003e-bin-hadoop3.tgz`, keyed by version and SHA-512, so a release candidate and its final release never share a cache entry.\n  - Downloads from the ASF mirror redirector only on a cache miss. A `url` input overrides it for release candidates.\n  - Verifies SHA-512 before extracting into `/tmp/spark`.\n- `.github/workflows/build_and_test.yml`: the duplicated `curl`/`tar`/`mv` preamble in seven jobs is replaced by a single `uses: ./.github/actions/setup-spark` step.\n- SHA-512 verification is now applied to all versions. The values for 4.0.4 and 4.1.3 were taken from the official ASF `.sha512` files:\n  - [spark-4.0.4-bin-hadoop3.tgz.sha512](https://downloads.apache.org/spark/spark-4.0.4/spark-4.0.4-bin-hadoop3.tgz.sha512)\n  - [spark-4.1.3-bin-hadoop3.tgz.sha512](https://downloads.apache.org/spark/spark-4.1.3/spark-4.1.3-bin-hadoop3.tgz.sha512)\n- `tar xvfz` is changed to `tar xfz` to remove thousands of lines of file listings from the CI logs.\n\n### Why are the changes needed?\n\nSeven macOS jobs each download a Spark distribution from an ASF mirror on every workflow run, roughly 2.8GB of mirror traffic per run:\n\n| Version | Jobs |\n| ------- | ---- |\n| 4.0.4 | `integration-test-mac`, `integration-test-mac-spark4-iceberg` |\n| 4.1.3 | `integration-test-mac-spark41`, `integration-test-token`, `integration-test-mac-spark41-iceberg` |\n| 4.2.0 | `integration-test-mac-spark42` |\n| 4.3.0-rc1 | `integration-test-mac-spark43` |\n\n1. macOS runners are billed at a 10x multiplier, so mirror download time is the most expensive kind of CI time in this repository.\n2. `closer.lua` can redirect to a slow mirror, which puts these jobs at risk of hitting the 20-minute timeout for reasons unrelated to the change under test.\n3. It reduces the load this repository puts on ASF mirrors.\n4. Only 4.2.0 and 4.3.0 verified their download. Every version is verified now.\n5. The download and extraction logic was duplicated seven times, so adding a version or fixing the logic meant editing seven places.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is a CI-only change.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #543 from dongjoon-hyun/SPARK-59264.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "ce19490d269c80f67465fe807f68b05f04f13bdb",
      "tree": "78ecabb4a2f733a92fba2ddcb8cf2194cec1fb84",
      "parents": [
        "029dca72ea4d59f83b95f555479808f19192c7e5"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 11:31:15 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 11:31:15 2026 -0700"
      },
      "message": "[SPARK-59260] Index all public functions and types in the DocC catalog\n\n### What changes were proposed in this pull request?\n\nThis PR rebuilds the DocC catalog\u0027s Topics index so that the library\u0027s public API is\ndiscoverable in the generated documentation. No source code is changed.\n\n**SQL functions.** None of the module-level functions were curated. All 753 public top-level\nfunctions fell into DocC\u0027s automatically-generated `Functions` section on the module landing\npage as one flat, unsectioned list. This PR adds 13 article pages:\n\n- `Functions.md`, a hub page linking to the family pages below.\n- `ColumnFunctions.md` (21), `AggregateFunctions.md` (130), `WindowFunctions.md` (13),\n  `MathFunctions.md` (85), `StringFunctions.md` (124), `DateTimeFunctions.md` (115),\n  `CollectionFunctions.md` (81), `SemiStructuredFunctions.md` (53), `SketchFunctions.md` (52),\n  `GeospatialFunctions.md` (10), `ConditionalFunctions.md` (21), `MiscFunctions.md` (48).\n\nEach page has an Overview with a runnable example and splits its Topics into 3-14 semantic\nsections. Pages are organized by source-file family, so adding a function to\n`XmlFunctions.swift` maps to one obvious place in the docs. The single exception is\n`count`/`sum`/`avg`/`mean`/`min`/`max` from `Functions.swift`, curated on the aggregate page\nwhere readers will look for them.\n\nAll 753 declarations are curated exactly once. The 82 overloaded names use DocC\u0027s type\nsignature disambiguation, for example:\n\n```\n- ``lit(_:)-(Bool)``\n- ``parse_url(_:_:)-(_,Column)``\n```\n\n**Missing public types.** `SparkConnect.md` gains `Column`, `SparkLiteral`, `Window`,\n`WindowSpec`, `Trigger`, `StreamingQueryException`, `StreamingQueryStatus`,\n`DataFrameWriterV2`, `WhenMatched`, `WhenNotMatched`, `WhenNotMatchedBySource`,\n`DataFrameNaFunctions`, `DataFrameStatFunctions`, `Observation`, `RowSchema`,\n`SparkConnectError`, `~\u003d(_:_:)`, `RuntimeConf`, and the six catalog value types\n(`CatalogMetadata`, `CatalogColumn`, `Database`, `SparkTable`, `Function`, `TablePartition`),\nacross new `Expressions`, `Window Frames`, and `Error Handling` sections. `LocalTime` and\n`TimestampNanos` move from `DataFrames` to `Data Types`.\n\n`SparkConnectClient`, `CaseInsensitiveDictionary`, `ErrorUtils`, `ProtoUtils`,\n`SparkFileUtils`, `CRC32`, and `SHA256` are curated under `Low-Level and Utility APIs`. These\nare `public`, so DocC lists them whether or not they are curated; the `/// nodoc` marker used\nby the vendored Arrow sources is a jazzy/SPI convention that DocC does not act on, and actually\nhiding them would need the underscored `_documentation(visibility:)` attribute. Grouping them\nunder an explicitly named section is honest about what they are without touching any source.\n\n**`Column.md`.** A new symbol page with an Overview and 15 topic sections covering all 84\npublic members. Operator overloads are separated into Arithmetic, Comparison, and Logical\nsections instead of appearing as repeated identical titles in the automatic member list.\n\n**Bundle metadata.** `Info.plist` drops the `CFBundleVersion` key and updates\n`NSHumanReadableCopyright` to `© 2025 and onwards, The Apache Software Foundation`, matching\nthe \"and onwards\" form used in this project\u0027s `NOTICE` so it needs no yearly edit. The version\nkey was set to `0.1.0` and had never been updated across the `0.2.0` through `0.7.0` releases;\nit is removed rather than bumped because DocC does not consume it. The\n`INFO.PLIST FALLBACKS` section of `docc convert --help` lists only `CFBundleDisplayName`,\n`CFBundleIdentifier`, a default module kind, and a default code-listing language; the version\nnever reaches the generated site, and a stale value produces no diagnostic, which is why it\nwent unnoticed. Keeping it would mean either shipping a wrong value again after the next\nrelease or adding a release step to maintain something nothing reads.\n\n### Why are the changes needed?\n\nThe published documentation on Swift Package Index does not currently make this library\nnavigable. 259 SQL functions were added recently and the catalog index had never been updated,\nso the largest part of the public API was reachable only through one undifferentiated\nauto-generated list, and core types such as `Column` appeared only in DocC\u0027s automatic\n`Structures`/`Classes` groups mixed in with the vendored Arrow implementation.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is a documentation-only change; the generated DocC site is reorganized.\n\n### How was this patch tested?\n\nPass the CI. In addition, DocC was built locally and compared against the pre-change baseline.\nThe repository has no DocC plugin dependency, so symbol graphs were extracted from a release\nbuild and `docc` was invoked directly rather than adding a dependency to `Package.swift`:\n\n```\nswift build -c release \\\n  -Xswiftc -emit-symbol-graph -Xswiftc -emit-symbol-graph-dir -Xswiftc /tmp/sg\n```\n\n```\nxcrun docc convert Sources/SparkConnect/Documentation.docc \\\n  --fallback-display-name SparkConnect \\\n  --fallback-bundle-identifier org.apache.spark.connect.swift.SparkConnect \\\n  --additional-symbol-graph-dir /tmp/sg --output-path /tmp/docs\n```\n\nResults:\n\n- 61 warnings before this change, 61 after; `diff` of the two warning lists is empty. All 61\n  are pre-existing warnings from doc comments in `.swift` sources and are untouched here, so\n  `--warnings-as-errors` is not usable on this catalog yet.\n- Inspecting the rendered JSON confirms 765 topic links across the 13 function pages, all\n  unique, with no duplicate curation and no link resolving to an unintended symbol.\n  `column.json` shows 84 unique links, every one a `Column` member.\n- The auto-generated `Functions` section (753 entries) is gone from the module landing page,\n  which now has 11 curated sections holding 60 entries. The only remaining uncurated top-level\n  symbols are vendored Arrow and FlatBuffers types.\n- Every symbol name written in Topics was cross-checked against the sources; no missing name.\n- `plutil -lint` reports the plist as valid. Converting with and without `CFBundleVersion`\n  produces 3,256 rendered JSON files that are all semantically identical, confirming the key\n  contributes nothing to the site.\n- `markdownlint --config .markdownlint.yaml --ignore-path .markdownlintignore` is clean.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #542 from dongjoon-hyun/SPARK-59260.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "029dca72ea4d59f83b95f555479808f19192c7e5",
      "tree": "610d89aaa5456df06da5793cdb121dc1bfe11448",
      "parents": [
        "ca05b35dbf5b78fed91156ac79daf65f3f53ebf5"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 10:12:09 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 10:12:09 2026 -0700"
      },
      "message": "[SPARK-59258] Support higher-order map functions\n\n### What changes were proposed in this pull request?\n\nThis PR adds the four higher-order map functions to `HigherOrderFunctions.swift`, built on the lambda infrastructure added by SPARK-59254.\n\n| Function | Closure |\n| -------- | ------- |\n| `map_filter(col, f)` | `(Column, Column) -\u003e Column` (key, value) |\n| `transform_keys(col, f)` | `(Column, Column) -\u003e Column` (key, value) |\n| `transform_values(col, f)` | `(Column, Column) -\u003e Column` (key, value) |\n| `map_zip_with(left, right, f)` | `(Column, Column, Column) -\u003e Column` (key, value1, value2) |\n\n`map_zip_with` is the first user of the three-argument `createLambda` overload, which was added by SPARK-59254 but had no caller until now.\n\n### Why are the changes needed?\n\nFor feature parity with Apache Spark. These four functions exist in both the\nScala and Python clients but had no Swift equivalent. They complete the set of\nhigher-order functions, whose array half landed in SPARK-59254.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, this adds new public APIs. For example:\n\n```swift\nlet df \u003d try await spark.range(1)\nlet m \u003d map_from_arrays(array(lit(1), lit(2)), array(lit(10), lit(20)))\n\ntry await df.select(map_filter(m) { _, v in v \u003e 10 }.cast(\"string\")).show()\n// {2 -\u003e 20}\n\ntry await df.select(transform_keys(m) { k, v in k + v }.cast(\"string\")).show()\n// {11 -\u003e 10, 22 -\u003e 20}\n\ntry await df.select(transform_values(m) { k, v in k + v }.cast(\"string\")).show()\n// {1 -\u003e 11, 2 -\u003e 22}\n\nlet m2 \u003d map_from_arrays(array(lit(1), lit(2)), array(lit(100), lit(200)))\ntry await df.select(map_zip_with(m, m2) { _, v1, v2 in v1 + v2 }.cast(\"string\")).show()\n// {1 -\u003e 110, 2 -\u003e 220}\n```\n\nAll four functions were introduced in Spark 3.1.0, so no version gate is needed.\n\n### How was this patch tested?\n\nExtended the existing `HigherOrderFunctionsTests` suite.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #541 from dongjoon-hyun/SPARK-59258.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "ca05b35dbf5b78fed91156ac79daf65f3f53ebf5",
      "tree": "31881da1aef21279c32775af4f2896b4882b9b5d",
      "parents": [
        "2f84649172864b24d3cb22d8b014210e18f07cd2"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 09:57:05 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 09:57:05 2026 -0700"
      },
      "message": "[SPARK-59257] Update docker GitHub Actions to the latest ASF-approved hashes\n\n### What changes were proposed in this pull request?\n\nThis PR aims to update `docker/*` GitHub Actions in `.github/workflows/publish_image.yml` to the latest ASF-approved commit hashes from [apache/infrastructure-actions/approved_patterns.yml](https://github.com/apache/infrastructure-actions/blob/main/approved_patterns.yml).\n\n| Action | Before | After |\n|---|---|---|\n| `docker/setup-qemu-action` | `ce360397` ([v4.0.0](https://github.com/docker/setup-qemu-action/releases/tag/v4.0.0), 2026-03-04) | `96fe6ef7` ([v4.2.0](https://github.com/docker/setup-qemu-action/releases/tag/v4.2.0), 2026-07-01) |\n| `docker/setup-buildx-action` | `4d04d5d9` ([v4.0.0](https://github.com/docker/setup-buildx-action/releases/tag/v4.0.0), 2026-03-05) | `37fe6310` ([v4.3.0](https://github.com/docker/setup-buildx-action/releases/tag/v4.3.0), 2026-08-17) |\n| `docker/login-action` | `4907a6dd` ([v4.1.0](https://github.com/docker/login-action/releases/tag/v4.1.0), 2026-04-02) | `dbcb8138` ([v4.6.0](https://github.com/docker/login-action/releases/tag/v4.6.0), 2026-07-29) |\n| `docker/build-push-action` | `bcafcacb` ([v7.1.0](https://github.com/docker/build-push-action/releases/tag/v7.1.0), 2026-04-09) | `53b7df96` ([v7.3.0](https://github.com/docker/build-push-action/releases/tag/v7.3.0), 2026-07-01) |\n\n### Why are the changes needed?\n\nThe previously pinned hashes are no longer listed in the ASF-approved patterns. This PR brings the workflow back into compliance with the ASF GitHub Actions policy and picks up the latest releases of these actions.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is an infra-only change.\n\n### How was this patch tested?\n\nManual review. Each new hash was verified against `approved_patterns.yml` and its corresponding release tag via the GitHub API.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5.1\n\nCloses #540 from dongjoon-hyun/SPARK-59257.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "2f84649172864b24d3cb22d8b014210e18f07cd2",
      "tree": "62ff891c0cb84d179af1628a0d5256a1505b84da",
      "parents": [
        "53f0bb6ef0430e327dea1c345faf815a69496535"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 09:44:44 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 09:44:44 2026 -0700"
      },
      "message": "[SPARK-59254] Support higher-order array functions\n\n### What changes were proposed in this pull request?\n\nThis PR adds support for lambda expressions and the higher-order array\nfunctions that consume them, in a new `HigherOrderFunctions.swift` file. No\nexisting file is modified.\n\nLambda infrastructure (internal):\n\n- `createLambda` in one-, two-, and three-argument forms, converting a Swift\n  closure into a `LambdaFunction` expression.\n- Lambda variables get a unique name (`x_1`, `y_2`, ...) from an\n  `Atomic\u003cInt\u003e` counter, matching Scala\u0027s\n  `UnresolvedNamedLambdaVariable.apply` and PySpark\u0027s `fresh_var_name`.\n\nNew public functions:\n\n| Function | Closure |\n| -------- | ------- |\n| `transform(col, f)` | `(Column) -\u003e Column` and `(Column, Column) -\u003e Column` (element, index) |\n| `filter(col, f)` | `(Column) -\u003e Column` and `(Column, Column) -\u003e Column` (element, index) |\n| `exists(col, f)` | `(Column) -\u003e Column` |\n| `forall(col, f)` | `(Column) -\u003e Column` |\n| `aggregate(col, initialValue, merge, finish:)` | `(Column, Column) -\u003e Column`, `(Column) -\u003e Column` (`finish:` optional) |\n| `reduce(col, initialValue, merge, finish:)` | `(Column, Column) -\u003e Column`, `(Column) -\u003e Column` (`finish:` optional) |\n| `zip_with(left, right, f)` | `(Column, Column) -\u003e Column` |\n| `array_sort(col, comparator)` | `(Column, Column) -\u003e Column` |\n\nThe closures are non-escaping because they are invoked immediately while the\nexpression is built and never stored.\n\n`finish:` is labeled so the optional finish closure can be passed as a second\ntrailing closure; the first closure of every function stays unlabeled, matching\nthe rest of the API.\n\n### Why are the changes needed?\n\nFor feature parity with Apache Spark. These functions exist in both the Scala\nand Python clients but had no Swift equivalent, because there was no way to\nbuild a Spark Connect lambda expression from a Swift closure. The generated\nprotobuf types (`Expression.LambdaFunction` and\n`Expression.UnresolvedNamedLambdaVariable`) were already available, so no proto\nregeneration is required.\n\nThe unique variable naming is required for correctness, not cosmetics. The\nserver passes `name_parts` through unchanged\n(`SparkConnectPlanner.transformUnresolvedNamedLambdaVariable`), so with fixed\nnames an inner lambda would shadow the variable of an enclosing one and\n`transform(a) { x in transform(b) { y in x + y } }` would silently compute\n`x + x`.\n\nThe functions live in their own file rather than in `CollectionFunctions.swift`\nbecause they share the lambda infrastructure, which stays internal to the module,\nand because the follow-up higher-order map functions (`map_filter`,\n`transform_keys`, `transform_values`, `map_zip_with`) belong next to them. This\nalso keeps `CollectionFunctions.swift` from growing past 900 lines.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, this adds new public APIs. For example:\n\n```swift\nlet df \u003d try await spark.range(1)\nlet arr \u003d array(lit(1), lit(2), lit(3))\n\ntry await df.select(transform(arr) { $0 * 2 }.cast(\"string\")).show()\n// [2, 4, 6]\n\ntry await df.select(aggregate(arr, lit(0)) { acc, x in acc + x }).show()\n// 6\n\ntry await df.select(aggregate(arr, lit(0)) { acc, x in acc + x } finish: { $0 * 10 }).show()\n// 60\n```\n\nAll of these functions were introduced in Spark 3.5 or earlier\n(`array_sort` with a comparator in 3.4.0), so no version gate is needed.\n\n### How was this patch tested?\n\nAdded a new `HigherOrderFunctionsTests` suite.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5.1\n\nCloses #539 from dongjoon-hyun/SPARK-59254.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "53f0bb6ef0430e327dea1c345faf815a69496535",
      "tree": "d3c76b1ad2bbb9475499dec8d0547494da46e25e",
      "parents": [
        "85e362614e2e97a096ca7e7740d3e077ee97b048"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:44:47 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:44:47 2026 -0700"
      },
      "message": "[SPARK-59245] Support HLL sketch and bitmap aggregate functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support 7 HLL sketch and bitmap aggregate functions (12 declarations), which\ncompletes the sketch function coverage of this client.\n\n| Function | Since |\n| --- | --- |\n| `hll_sketch_agg`, `hll_union_agg` | 3.5.0 |\n| `hll_sketch_estimate`, `hll_union` | 3.5.0 |\n| `bitmap_construct_agg`, `bitmap_or_agg` | 3.5.0 |\n| `bitmap_and_agg` | **4.1.0** |\n\nThe 5 aggregate functions go into `AggregateFunctions.swift`, following the `theta_sketch_agg`,\n`kll_sketch_agg_*` and `tuple_sketch_agg_*` precedents. The 2 scalar functions live in a new\n`HllSketchFunctions.swift`, mirroring `ThetaSketchFunctions.swift`.\n\nOptional arguments follow the upstream implementations exactly and are expressed as overloads.\n`lgConfigK` gets an `Int32` overload alongside the `Column` one, since the server requires the\n`INT` type there and the existing `lit(_ value: Int)` produces a `BIGINT` literal that would be\nrejected. `allowDifferentLgConfigK` is `Union[bool, Column]` in `hll_union_agg` and gets both a\n`Bool` and a `Column` overload, while `hll_union` accepts a `Bool` only, matching\n`def hll_union(c1: Column, c2: Column, allowDifferentLgConfigK: Boolean)` in the Scala API.\n\n### Why are the changes needed?\n\nTo improve API coverage. These functions are available in PySpark and the Spark SQL Scala API,\nbut were missing from this Swift client. The scalar `bitmap_bit_position`,\n`bitmap_bucket_number` and `bitmap_count` functions are already supported, and these three\naggregate functions complete that family.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new addition to the API.\n\n```swift\nlet df \u003d try await spark.sql(\"SELECT * FROM VALUES (1), (2), (2), (3) AS T(v)\")\ntry await df.select(hll_sketch_estimate(hll_sketch_agg(col(\"v\")))).show()\n```\n\n```\n+----------------------------------------------+\n|hll_sketch_estimate(hll_sketch_agg(v, 12))    |\n+----------------------------------------------+\n|3                                             |\n+----------------------------------------------+\n```\n\n```swift\ntry await df.select(bitmap_count(bitmap_construct_agg(bitmap_bit_position(col(\"v\"))))).show()\n```\n\n```\n+-------------------------------------------------------+\n|bitmap_count(bitmap_construct_agg(bitmap_bit_position(v)))|\n+-------------------------------------------------------+\n|3                                                      |\n+-------------------------------------------------------+\n```\n\n### How was this patch tested?\n\nPass the CIs with a new test suite.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #538 from dongjoon-hyun/SPARK-59245.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "85e362614e2e97a096ca7e7740d3e077ee97b048",
      "tree": "7a2b74fa88327578455b6b32b2edd18a3da25150",
      "parents": [
        "017b2f95c149b6c6e9785143c9c34d100de370b6"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:11:28 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:11:28 2026 -0700"
      },
      "message": "[SPARK-59241] Support Theta, KLL and Tuple sketch functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support 52 Datasketches sketch functions (86 declarations).\n\nThe KLL functions have three element-type variants (`_bigint`, `_double`, `_float`) and the\nTuple functions have two summary-type variants (`_double`, `_integer`).\n\n| Function | Since |\n| --- | --- |\n| `theta_sketch_agg`, `theta_union_agg`, `theta_intersection_agg` | 4.1.0 |\n| `theta_sketch_estimate`, `theta_union`, `theta_intersection`, `theta_difference` | 4.1.0 |\n| `kll_sketch_agg_\u003cT\u003e` | 4.1.0 |\n| `kll_merge_agg_\u003cT\u003e` | **4.1.2** |\n| `kll_sketch_get_n_\u003cT\u003e`, `kll_sketch_get_quantile_\u003cT\u003e`, `kll_sketch_get_rank_\u003cT\u003e` | 4.1.0 |\n| `kll_sketch_merge_\u003cT\u003e`, `kll_sketch_to_string_\u003cT\u003e` | 4.1.0 |\n| `tuple_sketch_agg_\u003cT\u003e`, `tuple_union_agg_\u003cT\u003e`, `tuple_intersection_agg_\u003cT\u003e` | 4.2.0 |\n| `tuple_sketch_estimate_\u003cT\u003e`, `tuple_sketch_summary_\u003cT\u003e`, `tuple_sketch_theta_\u003cT\u003e` | 4.2.0 |\n| `tuple_union_\u003cT\u003e`, `tuple_union_theta_\u003cT\u003e` | 4.2.0 |\n| `tuple_intersection_\u003cT\u003e`, `tuple_intersection_theta_\u003cT\u003e` | 4.2.0 |\n| `tuple_difference_\u003cT\u003e`, `tuple_difference_theta_\u003cT\u003e` | 4.2.0 |\n\nThe 37 scalar functions live in three new files, `ThetaSketchFunctions.swift`,\n`KllSketchFunctions.swift` and `TupleSketchFunctions.swift`. The 15 aggregate functions go\ninto `AggregateFunctions.swift`, following the existing `count_min_sketch`,\n`schema_of_variant_agg` and `vector_avg` precedents.\n\nOptional arguments follow the upstream PySpark implementations exactly. Most functions omit\n`lgNomEntries` / `k` / `mode` when they are not given, so the argument count varies and they\nare expressed as overloads, like the existing `hmac`. The exceptions are\n`tuple_sketch_agg_*`, `tuple_union_agg_*`, `tuple_union_*` and `tuple_union_theta_*`, which\nalways send `lgNomEntries` and `mode` filled with `12` and `\"sum\"`; these use labeled Swift\ndefault argument values, like the existing `aes_encrypt`.\n\nAn `Int32` overload is provided for `lgNomEntries` and `k` alongside the `Column` one, since\nthe server requires the `INT` type there and the existing `lit(_ value: Int)` produces a\n`BIGINT` literal that would be rejected.\n\n### Why are the changes needed?\n\nTo improve API coverage. These functions are available in PySpark and the Spark SQL Scala\nAPI, but were missing from this Swift client, where `count_min_sketch` was the only\nsupported sketch function.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new addition to the API.\n\n```swift\nlet df \u003d try await spark.sql(\"SELECT * FROM VALUES (1), (2), (2), (3) AS T(v)\")\ntry await df.select(theta_sketch_estimate(theta_sketch_agg(col(\"v\")))).show()\n```\n\n```\n+--------------------------------------------------+\n|theta_sketch_estimate(theta_sketch_agg(v))        |\n+--------------------------------------------------+\n|3                                                 |\n+--------------------------------------------------+\n```\n\n### How was this patch tested?\n\nPass the CIs with three new test suites, `ThetaSketchFunctionsTests`,\n`KllSketchFunctionsTests` and `TupleSketchFunctionsTests` (18 tests).\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #537 from dongjoon-hyun/SPARK-59241.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "017b2f95c149b6c6e9785143c9c34d100de370b6",
      "tree": "3b38b0b4671fde982861053528440d3f99726334",
      "parents": [
        "79af75a3c3d0e898710cf3dd6ce755dc52c71c0f"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:10:44 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:10:44 2026 -0700"
      },
      "message": "[SPARK-59240] Support `call_function` and `call_udf` functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `call_function` and `call_udf` functions in a new `UDFFunctions.swift` file.\n\n| Function | Since | Signature |\n| --- | --- | --- |\n| `call_function` | 3.5.0 | `(String, Column...)` |\n| `call_udf` | 3.4.0 | `(String, Column...)` |\n\nBoth are backed by the `CallFunction` protobuf node rather than the existing\ninternal `fn` helper which builds an `UnresolvedFunction`. This is intentional:\nthe server\u0027s `SparkConnectPlanner.transformUnresolvedFunction` treats the\nfunction name of an `UnresolvedFunction` as a **single identifier** when\n`is_user_defined_function` is `false`, which is what our `fn` helper produces.\nThat would break qualified names such as `call_function(\"db.my_func\", ...)`.\nIn contrast, `SparkConnectPlanner.transformCallFunction` runs the name through\n`parser.parseMultipartIdentifier`, so quoted and qualified names are resolved\ncorrectly. This matches the documented Scala contract, \"function name that\nfollows the SQL identifier syntax (can be quoted, can be qualified)\", and\nmatches PySpark Connect, whose `call_function` also emits a `CallFunction` node.\n\n`call_udf` delegates to `call_function`, mirroring Scala\u0027s\n`functions.scala`, where `call_udf(udfName, cols: _*) \u003d call_function(udfName, cols: _*)`.\n\n### Why are the changes needed?\n\nFor feature parity with Apache Spark.\n\n`call_function` is especially valuable in this client because it is a general\nescape hatch: it lets users invoke any SQL function available on the server,\nincluding the ones this library has not wrapped yet.\n\nAlthough this repository does not provide a UDF registration API, `call_udf`\nremains useful for invoking functions registered on the server through SQL\n`CREATE FUNCTION`.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this simply adds two new functions.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test suite, `UDFFunctionsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #536 from dongjoon-hyun/SPARK-59240.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "79af75a3c3d0e898710cf3dd6ce755dc52c71c0f",
      "tree": "33fc4cb097ed87ab6c3a737778086486200f3d21",
      "parents": [
        "a46c1feddadd17126481810063e5af0cf171df49"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:09:54 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:09:54 2026 -0700"
      },
      "message": "[SPARK-59239] Support `counter_diff` window function\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support the `counter_diff` window function.\n\n```swift\npublic func counter_diff(_ value: Column) -\u003e Column\npublic func counter_diff(_ value: Column, _ startTime: Column) -\u003e Column\n```\n\n### Why are the changes needed?\n\nFor feature parity with Apache Spark, which added `counter_diff` in `4.3.0` It converts a cumulative counter time series into the delta format and gracefully handles counter resets by returning `NULL`.\n- apache/spark#55828\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new function.\n\n### How was this patch tested?\n\nPass the CIs with newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #535 from dongjoon-hyun/SPARK-59239.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "a46c1feddadd17126481810063e5af0cf171df49",
      "tree": "313315f041b30feb1c26ae3106a6409036c17492",
      "parents": [
        "50a9139a4182e0685e999a228384188c2e604497"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:09:05 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:09:05 2026 -0700"
      },
      "message": "[SPARK-59238] Support partition transform functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support 5 partition transform functions in a new file,\n`Sources/SparkConnect/PartitionTransforms.swift`, and to add a `Column`-based\n`DataFrameWriterV2.partitionBy` overload so that these transforms can actually be used.\n\n| Function | Since | Signature |\n| --- | --- | --- |\n| `bucket` | 3.1.0 | `(Column, Column)`, `(Int32, Column)` |\n| `days` | 3.1.0 | `(Column)` |\n| `hours` | 3.1.0 | `(Column)` |\n| `months` | 3.1.0 | `(Column)` |\n| `years` | 3.1.0 | `(Column)` |\n\nScala groups these under `group partition_transforms` in `functions.scala`, so they are\nplaced in a dedicated file instead of `Functions.swift`. `bucket` takes an `Int32` overload\nbecause PySpark accepts `Union[Column, int]` for `numBuckets`; the literal is wrapped with\n`lit(...)`, matching `partitioning.bucket` in\n`python/pyspark/sql/connect/functions/partitioning.py`.\n\nIn addition, `DataFrameWriterV2.partitionBy(_ columns: Column...)` is added. The existing\n`partitionBy(_ columns: String...)` doc comment already claimed to accept \"columns or\ntransforms\", but only string column names could be passed, so there was no way to build a\n`years(col(\"ts\"))`-partitioned table.\n\n### Why are the changes needed?\n\nTo improve API coverage and to make the partition transforms usable end to end.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavior change. Only `DataFrameWriterV2.partitionBy` gains a new `Column`-variadic overload. This is a pure addition; the existing `String`-variadic overload is unchanged, so no existing code breaks. The 5 functions themselves are new additions to the API.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test suite, `PartitionTransformsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #534 from dongjoon-hyun/SPARK-59238.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "50a9139a4182e0685e999a228384188c2e604497",
      "tree": "93506c5659b784a8d49e10a950c25be05a921db4",
      "parents": [
        "55b9b1e12ffdc92f4ac1d9c0f767a4ff93346964"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:08:16 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Sep 04 00:08:16 2026 -0700"
      },
      "message": "[SPARK-59237] Support `broadcast` function\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support the `broadcast` function.\n\n```swift\npublic func broadcast(_ df: DataFrame) async -\u003e DataFrame\n```\n\nUnlike most functions in `Functions.swift`, `broadcast` takes a `DataFrame` and returns a `DataFrame` marked with a `broadcast` join hint, mirroring PySpark\u0027s `broadcast(df) -\u003e df.hint(\"broadcast\")` and Scala\u0027s `def broadcast[U](df: Dataset[U]): df.type`.\n\n### Why are the changes needed?\n\nFor feature parity with Apache Spark. `broadcast` has been available since Apache Spark 1.6.0\nin PySpark and is a commonly used hint for broadcast hash joins.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, this PR adds a new public function, `broadcast`.\n\n```swift\nlet joined \u003d await left.join(broadcast(right), \"id\")\n```\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases in `FunctionsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #533 from dongjoon-hyun/SPARK-59237.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "55b9b1e12ffdc92f4ac1d9c0f767a4ff93346964",
      "tree": "e625e4590388169fd1210ff467589b02ae29c173",
      "parents": [
        "c913324d841b8399df5981f0c8f8763de2d02b71"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 19:42:45 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 19:42:45 2026 -0700"
      },
      "message": "[SPARK-59236] Support vector functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support 7 vector functions (11 overloads).\n\n| Function | Since | Signature |\n| --- | --- | --- |\n| `vector_cosine_similarity` | 4.3.0 | `(Column, Column)` |\n| `vector_inner_product` | 4.3.0 | `(Column, Column)` |\n| `vector_l2_distance` | 4.3.0 | `(Column, Column)` |\n| `vector_norm` | 4.3.0 | `(Column)`, `(Column, Float \\| Column)` |\n| `vector_normalize` | 4.3.0 | `(Column)`, `(Column, Float \\| Column)` |\n| `vector_avg` | 4.3.0 | `(Column)` |\n| `vector_sum` | 4.3.0 | `(Column)` |\n\nThe five scalar functions live in a new `VectorFunctions.swift`. The two aggregate functions,\n`vector_avg` and `vector_sum`, are added to the existing `AggregateFunctions.swift`, following\nthe `schema_of_variant_agg` precedent, while their tests stay with the rest of the family in\n`VectorFunctionsTests.swift`.\n\nDespite the name, these functions take no dedicated `VECTOR` type. They operate on `ARRAY\u003cFLOAT\u003e`\ncolumns, and all the vectors involved must have the same dimension.\n\nThe optional `degree` of `vector_norm` and `vector_normalize` is expressed as a Swift overload,\nmirroring the two Scala overloads. In addition to the `Column` overload, a `Float` overload is\nprovided because the server requires exactly a `FLOAT` for this argument:\n\n```\nSELECT vector_norm(ARRAY(3.0f, 4.0f), CAST(1.0 AS DOUBLE))\n[DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] ... The second parameter requires the \"FLOAT\" type,\nhowever \"CAST(1.0 AS DOUBLE)\" has the type \"DOUBLE\".\n```\n\nSince `lit(Double)` produces a `double` literal in this client, a `Double` overload would compile\nand then fail at analysis time on the server. The `Float` overload keeps `vector_norm(v, 2.0)`\ncorrect by construction, the same reasoning behind the `Int32` srid of `st_setsrid`.\n\n### Why are the changes needed?\n\nTo support the vector functions of Apache Spark 4.3.0 and improve the API coverage.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this PR only adds new functions.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test suite, `VectorFunctionsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #532 from dongjoon-hyun/SPARK-59236.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "c913324d841b8399df5981f0c8f8763de2d02b71",
      "tree": "0f4dce5f4b697bc0917599f6358918ed61ce960b",
      "parents": [
        "99496e69148b67cfdc4154a6ae258ec8206b40bf"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 19:39:41 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 19:39:41 2026 -0700"
      },
      "message": "[SPARK-59235] Support error, reflection and bitmap functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support the error, reflection, and bitmap functions.\n\n| Function | Since |\n| -------- | ----- |\n| `assert_true(col)` / `assert_true(col, errMsg)` | 3.1.0 |\n| `raise_error(errMsg)` | 3.1.0 |\n| `reflect(cols...)` | 3.5.0 |\n| `java_method(cols...)` | 3.5.0 |\n| `try_reflect(cols...)` | 4.0.0 |\n| `bitmap_bit_position(col)` | 3.5.0 |\n| `bitmap_bucket_number(col)` | 3.5.0 |\n| `bitmap_count(col)` | 3.5.0 |\n\nAll of them are classified as misc functions by PySpark and the Spark SQL Scala\nAPI, so they are added to the existing `MiscFunctions.swift` in alphabetical order.\n\nTwo details follow the upstream implementations:\n\n- `errMsg` is `Union[Column, str]` in PySpark, where a `str` is wrapped with\n  `lit()`. This is expressed as a `String` overload for `assert_true` and\n  `raise_error`, like the existing `schema_of_json` overload pair.\n- The reflection functions are variadic. Their first argument is the class name,\n  the second one is the method name, and the remaining ones are the method\n  arguments. No `String` overload is added for them because a bare string is a\n  column name rather than a literal in PySpark, so an overload would silently\n  mean the opposite of the upstream API.\n\nSince the reflection functions invoke an arbitrary Java static method on the\nserver, their doc comments note that the class must be on the Spark server\u0027s\nclasspath and that the class and the method names must never be built from\nuntrusted user input.\n\n### Why are the changes needed?\n\nTo improve the API coverage. These functions are available in PySpark and the\nSpark SQL Scala API, but were missing from this Swift client.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new feature.\n\n```swift\ntry await spark.range(1).select(\n  reflect(lit(\"java.util.UUID\"), lit(\"fromString\"), lit(\"a5cf6c42-0c85-418f-af6c-3e4e5b1328f2\")),\n  bitmap_bit_position(lit(123)),\n  bitmap_bucket_number(lit(123)),\n  bitmap_count(unhex(lit(\"FFFF\")))\n).show()\n```\n\n```\n+-------------------------------------------------------------------------+------------------------+-------------------------+-------------------------+\n|reflect(java.util.UUID, fromString, a5cf6c42-0c85-418f-af6c-3e4e5b1328f2)|bitmap_bit_position(123)|bitmap_bucket_number(123)|bitmap_count(unhex(FFFF))|\n+-------------------------------------------------------------------------+------------------------+-------------------------+-------------------------+\n|                                                     a5cf6c42-0c85-418...|                     122|                        1|                       16|\n+-------------------------------------------------------------------------+------------------------+-------------------------+-------------------------+\n```\n\n### How was this patch tested?\n\nPass the CIs with the new test cases in `MiscFunctionsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #531 from dongjoon-hyun/SPARK-59235.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "99496e69148b67cfdc4154a6ae258ec8206b40bf",
      "tree": "641aa40512cea308b59b32d310f9ea6b151e615f",
      "parents": [
        "e5acec92d7dc87ed0ab685a08ef16a39bb8884b9"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 16:10:39 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 16:10:39 2026 -0700"
      },
      "message": "[SPARK-59228] Support AES and HMAC functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support the AES and HMAC functions.\n\n| Function | Since |\n| -------- | ----- |\n| `aes_encrypt(input, key, mode, padding, iv, aad)` | 3.5.0 |\n| `aes_decrypt(input, key, mode, padding, aad)` | 3.5.0 |\n| `try_aes_decrypt(input, key, mode, padding, aad)` | 3.5.0 |\n| `hmac(key, message, algorithm)` | 4.3.0 |\n\nThe new functions live in a new `CryptoFunctions.swift` file. The existing\n`HashFunctions.swift` holds one-way digests, while these functions take a secret\nkey and perform symmetric encryption or message authentication.\n\nOptional arguments follow the upstream PySpark implementations exactly:\n\n- The AES functions fill omitted arguments with their default literals, so\n  `aes_encrypt` always sends 6 arguments and `aes_decrypt` / `try_aes_decrypt`\n  always send 5. Note that `aes_encrypt` has an `iv` argument while the two\n  decrypt functions do not.\n- `hmac` omits the optional `algorithm` argument instead of filling it, so it\n  sends 2 or 3 arguments. Since a Swift default argument value cannot change the\n  argument count, this is expressed as two overloads, matching the two Scala\n  `hmac` overloads.\n\nSince these functions are evaluated by the server and a key passed as a literal\nbecomes part of the query plan, the doc comments note that a literal key can be\nexposed through server logs and plan output.\n\n### Why are the changes needed?\n\nTo improve the API coverage. These functions are available in PySpark and the\nSpark SQL Scala API, but were missing from this Swift client.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new feature.\n\n```swift\nlet key \u003d lit(\"0000000000000000\")\nlet df \u003d try await spark.sql(\"SELECT \u0027Spark\u0027 AS a\")\ntry await df.select(\n  aes_decrypt(aes_encrypt(col(\"a\"), key), key).cast(\"STRING\")\n).show()\n```\n\n```\n+-------------------------------------------------------------+\n|CAST(aes_decrypt(aes_encrypt(a, 0000000000000000, GCM, ...)...|\n+-------------------------------------------------------------+\n|Spark                                                        |\n+-------------------------------------------------------------+\n```\n\n### How was this patch tested?\n\nPass the CIs with a new test suite, `CryptoFunctionsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #530 from dongjoon-hyun/SPARK-59228.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "e5acec92d7dc87ed0ab685a08ef16a39bb8884b9",
      "tree": "e121626a5d2d4ec583dd75e9ccd9eb91f538e250",
      "parents": [
        "bead6a15488d943bc1b2766416b5fbe1e4a59cbd"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 16:09:30 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 16:09:30 2026 -0700"
      },
      "message": "[SPARK-59227] Support base32 functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support 2 BASE32 (RFC 4648) functions.\n\n| Function | Since | Signature |\n| --- | --- | --- |\n| `to_base32` | 4.3.0 | `(Column)` |\n| `from_base32` | 4.3.0 | `(Column)` |\n\nThe functions are added to `StringFunctions.swift` next to the existing `base64` and\n`unbase64` functions, keeping the alphabetical order of the file.\n\n### Why are the changes needed?\n\nTo improve API coverage. This client supported only the BASE64 encoding so far.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new addition to the API.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test case, `StringFunctionsTests.selectBase32Functions`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #529 from dongjoon-hyun/SPARK-59227.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "bead6a15488d943bc1b2766416b5fbe1e4a59cbd",
      "tree": "c95fd048d1badad130cdfd6f77ed653c75ac6bdf",
      "parents": [
        "ed6803adf2075888fa27215b92dc8f341924bc98"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 16:08:47 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 16:08:47 2026 -0700"
      },
      "message": "[SPARK-59226] Support geospatial ST functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support 5 geospatial ST functions (10 overloads).\n\n| Function | Since | Signature |\n| --- | --- | --- |\n| `st_asbinary` | 4.1.0 | `(Column)` |\n| `st_asbinary` | 4.2.0 | `(Column, String \\| Column)` |\n| `st_geogfromwkb` | 4.1.0 | `(Column)` |\n| `st_geomfromwkb` | 4.1.0 | `(Column)` |\n| `st_geomfromwkb` | 4.2.0 | `(Column, Int32 \\| Column)` |\n| `st_setsrid` | 4.1.0 | `(Column, Int32 \\| Column)` |\n| `st_srid` | 4.1.0 | `(Column)` |\n\nThe functions live in a new `GeospatialFunctions.swift`.\n\nWhere the upstream argument is `Union[Column, int]` or `Union[Column, str]`, both a Swift\nprimitive and a `Column` overload are provided, following the existing `regexp_extract`\nprecedent. `srid` is taken as `Int32` because `lit(Int)` produces a `long` literal in this\nclient, while the ST functions expect an `INTEGER`.\n\nNote that the optional arguments of `st_asbinary` and `st_geomfromwkb` were added later than\nthe functions themselves:\n\n- `st_asbinary(geo, endianness)` is `since 4.2.0` (SPARK-56682).\n- `st_geomfromwkb(wkb, srid)` is `since 4.2.0` (SPARK-55295). The PySpark docstring is\n  missing the corresponding `versionchanged:: 4.2.0` note, but the commit is only contained\n  in `v4.2.0` and later tags.\n\n### Why are the changes needed?\n\nTo improve API coverage. `DataType` already models `.geometry(srid:)` and `.geography(srid:)`,\nand `DataFrameTests.dtypesGeospatial` already exercises the server behavior through raw SQL\nstrings, but no ST function was available in this client.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new addition to the API.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test suite, `GeospatialFunctionsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #528 from dongjoon-hyun/SPARK-59226.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "ed6803adf2075888fa27215b92dc8f341924bc98",
      "tree": "83fb3c1f97f76edeb7b10df7a0d2b91732a34ccf",
      "parents": [
        "87e6676f0bbe70c591c15cdd2f7e04bbb18c65df"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 16:07:38 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 16:07:38 2026 -0700"
      },
      "message": "[SPARK-59225] Support URL functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support 5 URL functions (10 overloads).\n\n| Function | Since | Signatures |\n| --- | --- | --- |\n| `parse_url` | 3.5.0 | `(Column, String)`, `(Column, Column)`, `(Column, String, String)`, `(Column, Column, Column)` |\n| `try_parse_url` | 4.0.0 | `(Column, String)`, `(Column, Column)`, `(Column, String, String)`, `(Column, Column, Column)` |\n| `url_encode` | 3.5.0 | `(Column)` |\n| `url_decode` | 3.5.0 | `(Column)` |\n| `try_url_decode` | 4.0.0 | `(Column)` |\n\nThe functions live in a new `UrlFunctions.swift`, matching the `url_funcs` group of the upstream `functions.scala` and the \"URL Functions\" section of the PySpark docs.\n\nWhere the upstream `partToExtract` and `key` arguments are `Union[Column, str]`, both a `String` and a `Column` overload are provided, following the existing `variant_get` precedent, because these arguments are string literals such as `\"HOST\"` or `\"QUERY\"` in practice.\n\n### Why are the changes needed?\n\nTo improve API coverage. No URL function was available in this client.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new addition to the API.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test suite, `UrlFunctionsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #527 from dongjoon-hyun/SPARK-59225.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "87e6676f0bbe70c591c15cdd2f7e04bbb18c65df",
      "tree": "e91d68261ee151cf687563369a3850079ab18626",
      "parents": [
        "650c58180a3e5dacbb0c78d4433c3c7ca432f3ae"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 14:30:37 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 14:30:37 2026 -0700"
      },
      "message": "[SPARK-59223] Support VARIANT functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support 16 `VARIANT` functions (24 overloads).\n\n| Function | Since | Signature |\n| --- | --- | --- |\n| `parse_json` | 4.0.0 | `(Column)` |\n| `try_parse_json` | 4.0.0 | `(Column)` |\n| `to_variant_object` | 4.0.0 | `(Column)` |\n| `is_variant_null` | 4.0.0 | `(Column)` |\n| `variant_get` | 4.0.0 | `(Column, String \\| Column, String)` |\n| `try_variant_get` | 4.0.0 | `(Column, String \\| Column, String)` |\n| `schema_of_variant` | 4.0.0 | `(Column)` |\n| `schema_of_variant_agg` | 4.0.0 | `(Column)` |\n| `is_valid_variant` | 4.2.0 | `(Column)` |\n| `variant_insert` | 4.3.0 | `(Column, String \\| Column, Column)` |\n| `try_variant_insert` | 4.3.0 | `(Column, String \\| Column, Column)` |\n| `variant_set` | 4.3.0 | `(Column, String \\| Column, Column, Bool \u003d true)` |\n| `try_variant_set` | 4.3.0 | `(Column, String \\| Column, Column, Bool \u003d true)` |\n| `variant_array_append` | 4.3.0 | `(Column, String \\| Column, Column)` |\n| `try_variant_array_append` | 4.3.0 | `(Column, String \\| Column, Column)` |\n| `variant_strip_nulls` | 4.3.0 | `(Column, Bool \u003d true)` |\n\nThe functions live in a new `VariantFunctions.swift`, except `schema_of_variant_agg`\nwhich is an aggregate function and is placed in `AggregateFunctions.swift`.\n\nWhere the upstream `path` argument is `Union[Column, str]`, both a `String` and a `Column`\noverload are provided, following the existing `schema_of_json` precedent. Like PySpark\u0027s\nSpark Connect client, the `createIfMissing` and `includeArrays` flags are always sent as\nliteral arguments.\n\n`variant_delete` is out of scope because it is introduced in Spark 5.0.0.\n\n### Why are the changes needed?\n\nTo improve API coverage. `DataType` already models `.variant`, but no `VARIANT` function\nwas available in this client.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new addition to the API.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test suite, `VariantFunctionsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #526 from dongjoon-hyun/SPARK-59223.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "650c58180a3e5dacbb0c78d4433c3c7ca432f3ae",
      "tree": "d2807ca9aef68bf51f947ee7d364a1555b8eac00",
      "parents": [
        "9fcec3d392f26e4fed23ac885063523a50384a8a"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 13:56:13 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 13:56:13 2026 -0700"
      },
      "message": "[SPARK-59222] Support XML functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support the following 12 XML functions in a new file, `XmlFunctions.swift`.\n\n**Conversion functions**\n\n| Function | Overloads added | Since |\n| -------- | --------------- | ----- |\n| `from_xml` | `schema: String` (DDL), `schema: StructType`, `schema: Column` | 4.0.0 |\n| `to_xml` | | 4.0.0 |\n| `schema_of_xml` | `xml: String`, `xml: Column` | 4.0.0 |\n\n**XPath functions**\n\n| Function | Result type | Since |\n| -------- | ----------- | ----- |\n| `xpath` | `ARRAY\u003cSTRING\u003e` | 3.5.0 |\n| `xpath_boolean` | `BOOLEAN` | 3.5.0 |\n| `xpath_double`, `xpath_number` | `DOUBLE` | 3.5.0 |\n| `xpath_float` | `FLOAT` | 3.5.0 |\n| `xpath_short`, `xpath_int`, `xpath_long` | `SMALLINT`, `INT`, `BIGINT` | 3.5.0 |\n| `xpath_string` | `STRING` | 3.5.0 |\n\nLike the JSON and CSV functions added by SPARK-59216, `from_xml`, `to_xml` and\n`schema_of_xml` take an option map which Apache Spark encodes as a plain trailing\n`map(k1, v1, ...)` argument, so this PR reuses the existing internal\n`fn(_ name:options:_ args:)` helper. When there is no option, no extra argument is\nappended at all.\n\nFor the `schema` argument, `from_xml` follows the upstream signature\n(`sql/api/src/main/scala/org/apache/spark/sql/functions.scala`) which accepts a\n`StructType`, a DDL `String`, or a `Column` such as a `schema_of_xml` result. Unlike\n`from_json`, XML has no `MapType`/`ArrayType` form in the upstream API. The `StructType`\noverload delegates to the DDL string form via `StructType.toDDL`; the server resolves it\nthrough `ExprUtils.evalSchemaExpr` / `DataType.fromDDL`.\n\nThe `path` argument of the XPath functions is a `Column`, matching\n`def xpath(xml: Column, path: Column)` upstream. Callers pass a literal with\n`lit(\"a/b/text()\")`. This mirrors the existing `get_json_object(_ col: Column, _ path: String)`\nprecedent, which takes a `String` only because the upstream `get_json_object` does.\n\n### Why are the changes needed?\n\nTo improve the API coverage of the Swift Spark Connect client. Currently, no XML function\nis supported, so Swift users cannot parse, produce, or query XML strings inside a `Column`\nexpression without falling back to `selectExpr`. Note that `DataFrameReader.xml` and\n`DataFrameWriter.xml` are already supported; this PR adds the column-level counterparts.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new feature which adds 12 new functions.\n\n```swift\nlet df \u003d try await spark.sql(\"SELECT 1 AS id, \u0027a\u0027 AS name\")\nlet parsed \u003d from_xml(to_xml(`struct`(col(\"id\"), col(\"name\"))), \"id INT, name STRING\")\ntry await df.select(parsed.alias(\"parsed\")).selectExpr(\"parsed.id\", \"parsed.name\").show()\n\ntry await spark.range(1).select(schema_of_xml(\"\u003cp\u003e\u003ca\u003e1\u003c/a\u003e\u003cb\u003ex\u003c/b\u003e\u003c/p\u003e\")).show()\n// STRUCT\u003ca: BIGINT, b: STRING\u003e\n\ntry await spark.range(1).select(\n  xpath(lit(\"\u003ca\u003e\u003cb\u003eb1\u003c/b\u003e\u003cb\u003eb2\u003c/b\u003e\u003c/a\u003e\"), lit(\"a/b/text()\"))\n).show()\n// [b1, b2]\n```\n\n### How was this patch tested?\n\nPass the CIs with the newly added test suite, `XmlFunctionsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #525 from dongjoon-hyun/SPARK-59222.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "9fcec3d392f26e4fed23ac885063523a50384a8a",
      "tree": "bbc742eb346e429e98d4aa2fe5c18608c3b4433b",
      "parents": [
        "2969e64a31c45eed4c23bb17ff3dd47f8c2c2fd1"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 13:55:15 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 13:55:15 2026 -0700"
      },
      "message": "[SPARK-59221] Support linear regression aggregate functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support 9 linear regression aggregate functions.\n\n| Function | Since | Description |\n| -------- | ----- | ----------- |\n| `regr_avgx(y, x)` | 3.5.0 | The average of the independent variable for non-null pairs |\n| `regr_avgy(y, x)` | 3.5.0 | The average of the dependent variable for non-null pairs |\n| `regr_count(y, x)` | 3.5.0 | The number of non-null number pairs |\n| `regr_intercept(y, x)` | 3.5.0 | The intercept of the univariate linear regression line |\n| `regr_r2(y, x)` | 3.5.0 | The coefficient of determination |\n| `regr_slope(y, x)` | 3.5.0 | The slope of the linear regression line |\n| `regr_sxx(y, x)` | 3.5.0 | `REGR_COUNT(y, x) * VAR_POP(x)` |\n| `regr_sxy(y, x)` | 3.5.0 | `REGR_COUNT(y, x) * COVAR_POP(y, x)` |\n| `regr_syy(y, x)` | 3.5.0 | `REGR_COUNT(y, x) * VAR_POP(y)` |\n\nAll of them take `(y, x)` where `y` is the dependent variable and `x` is the\nindependent variable, matching Apache Spark\u0027s `functions.scala`.\n\n### Why are the changes needed?\n\nTo provide feature parity with the other Apache Spark clients. These functions\nhave been available on the server side since Apache Spark 3.5.0, so no version\ngate is required.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new addition to the API.\n\n### How was this patch tested?\n\nPass the CIs with newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #524 from dongjoon-hyun/SPARK-59221.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "2969e64a31c45eed4c23bb17ff3dd47f8c2c2fd1",
      "tree": "1f4fd6f5e370260fd48ee15f1cb76a6adeb89905",
      "parents": [
        "fb9dcebb7848a24b84386853749c85890bd59c3b"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 12:41:28 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 12:41:28 2026 -0700"
      },
      "message": "[SPARK-59217] Support interval and timestamp constructor functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support 10 interval and timestamp constructor functions in `DateTimeFunctions`.\n\n| Function | Swift signature | Since |\n| --- | --- | --- |\n| `make_interval` | `make_interval(years:months:weeks:days:hours:mins:secs:)` | 3.5.0 |\n| `make_dt_interval` | `make_dt_interval(days:hours:mins:secs:)` | 3.5.0 |\n| `make_ym_interval` | `make_ym_interval(years:months:)` | 3.5.0 |\n| `make_timestamp` | `make_timestamp(_:_:_:_:_:_:)` | 3.5.0 |\n| `make_timestamp` | `make_timestamp(_:_:_:_:_:_:_:)` (with `timezone`) | 3.5.0 |\n| `make_timestamp_ltz` | `make_timestamp_ltz(_:_:_:_:_:_:)` | 3.5.0 |\n| `make_timestamp_ltz` | `make_timestamp_ltz(_:_:_:_:_:_:_:)` (with `timezone`) | 3.5.0 |\n| `make_timestamp_ntz` | `make_timestamp_ntz(_:_:_:_:_:_:)` | 3.5.0 |\n| `try_make_interval` | `try_make_interval(years:months:weeks:days:hours:mins:secs:)` | 4.0.0 |\n| `try_make_timestamp` | `try_make_timestamp(_:_:_:_:_:_:)` | 4.0.0 |\n| `try_make_timestamp` | `try_make_timestamp(_:_:_:_:_:_:_:)` (with `timezone`) | 4.0.0 |\n| `try_make_timestamp_ltz` | `try_make_timestamp_ltz(_:_:_:_:_:_:)` | 4.0.0 |\n| `try_make_timestamp_ltz` | `try_make_timestamp_ltz(_:_:_:_:_:_:_:)` (with `timezone`) | 4.0.0 |\n| `try_make_timestamp_ntz` | `try_make_timestamp_ntz(_:_:_:_:_:_:)` | 4.0.0 |\n\nThe `try_*` variants return `NULL` instead of raising an error when the value\ncannot be created.\n\n**On optional arguments.** The interval constructors have *only* optional fields\n(7 for `make_interval`, 4 for `make_dt_interval`, 2 for `make_ym_interval`), so\nthey are declared with Swift default argument values instead of arity overloads:\n\n```swift\npublic func make_interval(\n  years: Column \u003d lit(0), months: Column \u003d lit(0), weeks: Column \u003d lit(0),\n  days: Column \u003d lit(0), hours: Column \u003d lit(0), mins: Column \u003d lit(0), secs: Column \u003d lit(0)\n) -\u003e Column\n```\n\nModeling these as Scala-style prefix overloads would need 8 declarations for\n`make_interval` alone (16 with `try_make_interval`) and still could not express a\ncall that sets only `hours`. For that reason, and unlike the rest of this\ncodebase, the interval constructors keep their argument labels: without labels,\ndefault values could only be dropped from the tail, which defeats the purpose.\nThe labels also match how these functions are called in PySpark\n(`make_interval(hours\u003d...)`).\n\nFollowing the PySpark implementation, omitted fields are filled with a zero\nliteral, so the full argument list is always sent to the server, e.g.\n`make_interval(hours: col(\"h\"))` sends `make_interval(0, 0, 0, 0, h, 0, 0)`.\nPySpark uses `lit(decimal.Decimal(0))` for `secs`; a `long` zero is implicitly\ncoerced to `DECIMAL(18,6)` by the server, so no new `lit` overload is required.\n\nThe timestamp constructors take six required fields, so they follow the existing\nconvention of unlabeled arguments used by `make_date` and `make_time`. Their\noptional `timezone` field is expressed as an overload pair, matching the existing\n`to_timestamp_ltz`/`to_timestamp_ntz` precedent.\n\n### Why are the changes needed?\n\nFor feature parity with Apache Spark\u0027s Scala and Python clients. These are the\nonly remaining `make_*` datetime constructors missing from this client;\n`make_date` and `make_time` are already supported.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this only adds new functions.\n\n### How was this patch tested?\n\nPass the CIs with newly added test cases in `DateTimeFunctionsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #523 from dongjoon-hyun/SPARK-59217.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "fb9dcebb7848a24b84386853749c85890bd59c3b",
      "tree": "b98744aeb9c42ddaed0ed45dfba1b900acc4969a",
      "parents": [
        "9e01045f757b9a77e436d772a92cf193fb7761d2"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 12:40:35 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 12:40:35 2026 -0700"
      },
      "message": "[SPARK-59216] Support JSON and CSV functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support the following 10 JSON and CSV functions in two new files,\n`JsonFunctions.swift` and `CsvFunctions.swift`.\n\n**JSON functions (`JsonFunctions.swift`)**\n\n| Function | Overloads added | Since |\n| -------- | --------------- | ----- |\n| `from_json` | `schema: String` (DDL), `schema: StructType` | 2.1.0 |\n| `from_json` | `schema: Column` | 2.4.0 |\n| `to_json` | | 2.1.0 |\n| `schema_of_json` | `json: String`, `json: Column` | 2.4.0 (options since 3.0.0) |\n| `get_json_object` | | 1.6.0 |\n| `json_tuple` | | 1.6.0 |\n| `json_array_length` | | 3.5.0 |\n| `json_object_keys` | | 3.5.0 |\n\n**CSV functions (`CsvFunctions.swift`)**\n\n| Function | Overloads added | Since |\n| -------- | --------------- | ----- |\n| `from_csv` | `schema: String` (DDL), `schema: Column` | 3.0.0 |\n| `to_csv` | | 3.0.0 |\n| `schema_of_csv` | `csv: String`, `csv: Column` | 3.0.0 |\n\nUnlike the previous function batches, these functions take an option map. Apache Spark\nencodes it as a plain trailing argument rather than a dedicated proto field: both\n`Column.fnWithOptions` in `sql/api/src/main/scala/org/apache/spark/sql/Column.scala` and\n`_options_to_col` in `python/pyspark/sql/connect/functions/builtin.py` flatten the map into\n`k1, v1, k2, v2, ...`, wrap it in a `map(...)` function call, and append it as the last\nargument. When there is no option, no extra argument is appended at all. This PR mirrors\nthat exactly with a new internal `fn(_ name:options:_ args:)` helper in `Functions.swift`,\nsorting the keys so that the generated expression is deterministic (Swift `Dictionary` is\nunordered).\n\nFor the `schema` argument, this PR follows the existing\n`DataFrameReader.schema(_ schema: String)` / `schema(_ schema: StructType)` precedent:\n`from_json` accepts a DDL `String`, a `StructType` (converted via `StructType.toDDL`), or a\n`Column` such as a `schema_of_json` result. `from_csv` accepts a DDL `String` or a `Column`,\nmatching the Spark Connect Python client. `ArrayType` and `MapType` schemas are covered by\nthe DDL string form, e.g. `ARRAY\u003cSTRUCT\u003ca: INT\u003e\u003e`.\n\n`json_tuple` is a generator function, and is implemented like the existing `explode` /\n`posexplode` / `inline` functions.\n\n### Why are the changes needed?\n\nTo improve the API coverage of the Swift Spark Connect client. Currently, no JSON or CSV\nfunction is supported, so Swift users cannot parse or produce JSON and CSV strings inside a\n`Column` expression without falling back to `selectExpr`.\n\nAll the added functions were introduced in Apache Spark 3.5.0 or earlier, so no server\nversion gate is needed.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new feature which adds 10 new functions.\n\n```swift\nlet df \u003d try await spark.sql(\"SELECT 1 AS id, \u0027a\u0027 AS name\")\nlet parsed \u003d from_json(to_json(struct(col(\"id\"), col(\"name\"))), \"id INT, name STRING\")\ntry await df.select(parsed.alias(\"parsed\")).selectExpr(\"parsed.id\", \"parsed.name\").show()\n\ntry await spark.range(1).select(schema_of_json(\"{\\\"a\\\": 1, \\\"b\\\": \\\"x\\\"}\")).show()\n// STRUCT\u003ca: BIGINT, b: STRING\u003e\n\ntry await df.select(to_csv(struct(col(\"id\"), col(\"name\")), [\"sep\": \";\"])).show()\n// 1;a\n```\n\n### How was this patch tested?\n\nPass the CIs with the newly added test suites, `JsonFunctionsTests` and `CsvFunctionsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #522 from dongjoon-hyun/SPARK-59216.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "9e01045f757b9a77e436d772a92cf193fb7761d2",
      "tree": "8762831dd349e8603e788fe1dc00eec59e13a7f9",
      "parents": [
        "8f94a3dcb7bd549b5df0f548fb3f11d4fc3b984f"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 12:29:45 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 12:29:45 2026 -0700"
      },
      "message": "[SPARK-59215] Support general aggregate functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support 21 general aggregate functions in the Swift Spark Connect\nclient by adding them to `Sources/SparkConnect/AggregateFunctions.swift`, keeping\nthe file\u0027s alphabetical order:\n\n| Function | Since | Notes |\n| --- | --- | --- |\n| `approx_percentile(col, percentage, accuracy \u003d 10000)` | 3.5.0 | `percentage` is a `Double` or `[Double]` |\n| `array_agg(col)` | 3.5.0 | Alias of `collect_list` |\n| `bit_and(col)` | 3.5.0 | Bitwise `AND` of all non-null input values |\n| `bit_or(col)` | 3.5.0 | Bitwise `OR` of all non-null input values |\n| `bit_xor(col)` | 3.5.0 | Bitwise `XOR` of all non-null input values |\n| `collect_union(col)` | 4.3.0 | Distinct union of array elements across rows |\n| `count_min_sketch(col, eps, confidence, seed?)` | 3.5.0 | Returns a binary sketch |\n| `every(col)` | 3.5.0 | Alias of `bool_and` |\n| `first_value(col, ignoreNulls?)` | 3.5.0 | |\n| `histogram_numeric(col, nBins)` | 3.5.0 | |\n| `last_value(col, ignoreNulls?)` | 3.5.0 | |\n| `listagg(col, delimiter?)` | 4.0.0 | |\n| `listagg_distinct(col, delimiter?)` | 4.0.0 | |\n| `percentile(col, percentage, frequency \u003d 1)` | 3.5.0 | `percentage` is a `Double` or `[Double]` |\n| `product(col)` | 3.2.0 | |\n| `some(col)` | 3.5.0 | Alias of `bool_or` |\n| `std(col)` | 3.5.0 | Alias of `stddev_samp` |\n| `string_agg(col, delimiter?)` | 4.0.0 | Alias of `listagg` |\n| `string_agg_distinct(col, delimiter?)` | 4.0.0 | Alias of `listagg_distinct` |\n| `try_avg(col)` | 3.5.0 | `null` on overflow |\n| `try_sum(col)` | 3.5.0 | `null` on overflow |\n\nAll of these belong to the upstream \"Aggregate Functions\" documentation group,\nincluding `bit_and`/`bit_or`/`bit_xor` (which are aggregates, unlike the scalar\nbitwise functions in `BitwiseFunctions.swift`) and `try_avg`/`try_sum`.\n\nThree details are worth calling out, since the naive mapping does not work:\n\n1. `listagg_distinct` and `string_agg_distinct` are **not** separate function\n   names on the server. `FunctionRegistry` registers only `listagg` and\n   `string_agg`, and both `sql/api` `functions.scala`\n   (`Column.fn(\"listagg\", isDistinct \u003d true, e)`) and the Python Spark Connect\n   client (`UnresolvedFunction(\"listagg\", _exprs, is_distinct\u003dTrue)`) send the\n   base name with the `is_distinct` flag set. This PR follows the same pattern\n   already used by `count_distinct` and `sum_distinct`. Sending\n   `listagg_distinct` as a function name fails with `UNRESOLVED_ROUTINE`.\n\n2. `count_min_sketch` always requires four arguments:\n   `CountMinSketchAggExpressionBuilder` declares `seed` without a default, so a\n   three-argument call fails with `REQUIRED_PARAMETER_NOT_FOUND`. The overload\n   without a seed therefore generates a random `Int64` seed on the client, which\n   is what `functions.count_min_sketch(e, eps, confidence)` does in Scala with\n   `lit(SparkClassUtils.random.nextLong)`.\n\n3. `approx_percentile` is a distinct registered function from the existing\n   `percentile_approx`, so it is added rather than aliased.\n\nFor the array form of `percentage`, the `[Double]` overloads build an `array(...)`\nof literals, which stays foldable and satisfies the analyzer\u0027s requirement that\nthe percentage expression be a constant.\n\n### Why are the changes needed?\n\nTo improve API coverage and feature parity with PySpark and Spark SQL. These\naggregate functions were entirely missing from the Swift client, including the\nSQL-standard aliases (`array_agg`, `std`, `every`, `some`, `first_value`,\n`last_value`) that users coming from other SQL engines are most likely to reach\nfor first.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is an additive change that introduces new APIs.\n\n```swift\nlet df \u003d try await spark.sql(\"SELECT * FROM VALUES (1), (2), (3) T(v)\")\ntry await df.select(bit_and(col(\"v\")), bit_or(col(\"v\")), bit_xor(col(\"v\"))).show()\ntry await df.select(percentile(col(\"v\"), 0.5), approx_percentile(col(\"v\"), [0.0, 1.0])).show()\ntry await df.select(try_avg(col(\"v\")), try_sum(col(\"v\")), product(col(\"v\"))).show()\n```\n\nNon-`Column` arguments such as `accuracy`, `frequency`, `nBins`, `delimiter`, and\n`seed` are taken as Swift primitives and wrapped with `lit(...)`, following the\nexisting convention (e.g. `approx_count_distinct(_:_:)` and\n`regexp_extract(_:_:_:)`). Optional arguments are expressed as overloads, except\n`accuracy` and `frequency`, which use their upstream default values.\n\n### How was this patch tested?\n\nPass the CIs with new test cases added to `AggregateFunctionsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #521 from dongjoon-hyun/SPARK-59215.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "8f94a3dcb7bd549b5df0f548fb3f11d4fc3b984f",
      "tree": "756c7675fa639ff18ce6d5f784314c7ceafa157f",
      "parents": [
        "78752136b42d8fa9c344413d3dfd645e6e096b15"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 11:37:22 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 11:37:22 2026 -0700"
      },
      "message": "[SPARK-59214] Support bitwise functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support bitwise functions in the Swift Spark Connect client by\nadding a new `BitwiseFunctions.swift` source file:\n\n| Function | Since | Notes |\n| --- | --- | --- |\n| `bit_count(col)` | 3.5.0 | Number of set bits, as an unsigned 64-bit integer |\n| `bit_get(col, pos)` | 3.5.0 | Bit (0 or 1) at `pos`, numbered from right to left |\n| `getbit(col, pos)` | 3.5.0 | Alias of `bit_get` |\n| `bitwise_not(col)` | 3.2.0 | Bitwise `NOT` of the input |\n\nIn addition, `Column` gains the three bitwise methods that PySpark and Spark SQL\nexpose: `bitwiseAND`, `bitwiseOR`, and `bitwiseXOR`, each with a `Column` and a\nliteral overload.\n\nThe existing `shiftleft`, `shiftright`, and `shiftrightunsigned` functions are\nmoved from `MathFunctions.swift` into the new file, so that `BitwiseFunctions.swift`\nholds exactly the seven functions of the upstream \"Bitwise Functions\" documentation\ngroup, in the same order. This follows the per-category layout established by\n`HashFunctions.swift`, `PredicateFunctions.swift`, `MiscFunctions.swift`, and\n`ConditionalFunctions.swift`.\n\nNote that `bitwise_not` sends `~` as the unresolved function name, matching\n`FunctionRegistry` (`expression[BitwiseNot](\"~\")`), `sql/api` `functions.scala`\n(`Column.fn(\"~\", e)`), and the Python Spark Connect client\n(`_invoke_function_over_columns(\"~\", col)`). The name `bitwise_not` is not\nregistered on the server side.\n\nThe deprecated camelCase alias `bitwiseNOT` is intentionally not added.\n\n### Why are the changes needed?\n\nTo improve API coverage and feature parity with PySpark and Spark SQL. These\nbitwise functions were entirely missing from the Swift client, as were the\n`Column` bitwise methods. Grouping them with the shift functions keeps the whole\nupstream bitwise category in a single file.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is an additive change that introduces new APIs. The relocated\n`shiftleft`, `shiftright`, and `shiftrightunsigned` functions keep their\nsignatures and remain part of the same `SparkConnect` module, so existing code\nis unaffected.\n\n```swift\nlet df \u003d try await spark.range(1)\ntry await df.select(bit_count(lit(7)), bit_get(lit(5), lit(0))).show()\ntry await df.select(lit(170).bitwiseAND(lit(75))).show()\n```\n\n`Column` exposes these as named methods rather than overloading Swift\u0027s `\u0026`,\n`|`, and `^` operators. Both PySpark and Spark SQL use named methods, and in\nPySpark `\u0026` and `|` on a `Column` are *logical* operators, so overloading them\nas bitwise operators in Swift would give the same syntax the opposite meaning.\n\n### How was this patch tested?\n\nPass the CIs with the newly added `BitwiseFunctionsTests` suite.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #520 from dongjoon-hyun/SPARK-59214.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "78752136b42d8fa9c344413d3dfd645e6e096b15",
      "tree": "0be28744aecbe6f55f67a396271f64092a67987a",
      "parents": [
        "66e8a6bacc07dd1bb2a8f5cd67c659a5b247d796"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 11:36:28 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 11:36:28 2026 -0700"
      },
      "message": "[SPARK-59213] Support null ordering in sort functions and `Column`\n\n### What changes were proposed in this pull request?\n\nThis PR adds null-ordering sort APIs.\n\n- `Functions.swift`: `asc_nulls_first`, `asc_nulls_last`, `desc_nulls_first`, `desc_nulls_last`\n- `Column`: `ascNullsFirst()`, `ascNullsLast()`, `descNullsFirst()`, `descNullsLast()`\n\nFree functions use `snake_case` to mirror Spark/PySpark (like the existing `any_value`,\n`count_if`), while `Column` methods use `camelCase` (like the existing `isNull`,\n`eqNullSafe`). The free functions delegate to the new `Column` methods, which reuse the\nexisting private `sortOrder(_:_:)` helper.\n\n### Why are the changes needed?\n\nApache Spark has provided these in both `functions` and `Column` since 2.1.0. This client\nonly had `asc` and `desc`, so null placement could not be controlled without a raw SQL\nstring like `expr(\"v ASC NULLS LAST\")`.\n\nThe existing `asc()`/`desc()` already match the upstream defaults (`asc \u003d\u003d asc_nulls_first`,\n`desc \u003d\u003d desc_nulls_last`) and are unchanged.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo existing behavior changes because this is new public APIs.\n\n```swift\nlet df \u003d try await spark.sql(\"SELECT * FROM VALUES (1), (null), (2) AS T(v)\")\ntry await df.orderBy(asc_nulls_last(\"v\")).show()   // 1, 2, NULL\ntry await df.sort(col(\"v\").descNullsFirst()).show()  // NULL, 2, 1\n```\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #519 from dongjoon-hyun/SPARK-59213.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "66e8a6bacc07dd1bb2a8f5cd67c659a5b247d796",
      "tree": "bc70cf9d8c9c09d31e66eac8a472079e0f346bb1",
      "parents": [
        "7e6488b89fd4a0e26c6a7b79d72bf32d2037621c"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 10:52:49 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 10:52:49 2026 -0700"
      },
      "message": "[SPARK-59211] Support conditional functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support the following 8 conditional SQL functions.\n\n- `coalesce`\n- `ifnull`\n- `nanvl`\n- `nullif`\n- `nullifzero`\n- `nvl`\n- `nvl2`\n- `zeroifnull`\n\nIn addition, this PR moves the existing `when` functions from `Functions.swift` into the new `ConditionalFunctions.swift` file together with their test cases.\n\n### Why are the changes needed?\n\nTo improve the feature parity with the other Apache Spark Connect clients like PySpark and Scala.\n\nLike `HashFunctions.swift`, these functions are grouped into a dedicated file instead of `Functions.swift` in order to follow the existing per-category layout of this repository. The resulting file matches the `Conditional Functions` group of the PySpark API reference, which consists of `coalesce`, `ifnull`, `nanvl`, `nullif`, `nullifzero`, `nvl`, `nvl2`, `when`, and `zeroifnull`.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, the new functions are added to the unreleased branch, and moving `when` to another file does not change the public API.\n\n### How was this patch tested?\n\nPass the CIs with newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #518 from dongjoon-hyun/SPARK-59211.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "7e6488b89fd4a0e26c6a7b79d72bf32d2037621c",
      "tree": "2b48275617a5e651ddd6aafc64cbaa4f68dccf9d",
      "parents": [
        "39cc035bdc59ff4bfe1166c3ac0f79fef29c0c79"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 10:51:59 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 10:51:59 2026 -0700"
      },
      "message": "[SPARK-59210] Support misc metadata functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support the following 15 misc metadata functions by adding a new\n`MiscFunctions.swift` file.\n\n| Function | Since (Apache Spark) |\n| -------- | -------------------- |\n| `monotonically_increasing_id()`, `spark_partition_id()`, `input_file_name()` | 1.6.0 |\n| `input_file_block_start()`, `input_file_block_length()` | 3.5.0 |\n| `current_catalog()`, `current_database()`, `current_schema()` | 3.5.0 |\n| `current_user()`, `user()`, `version()`, `typeof(col)`, `uuid()` | 3.5.0 |\n| `session_user()` | 4.0.0 |\n| `uuid(seed)` | 4.1.0 |\n| `current_path()` | 4.2.0 |\n\nNote that `uuid()` is sent without any argument like PySpark\u0027s\n`sf.uuid()` does, so that it keeps working against Apache Spark 4.0 servers.\n\n### Why are the changes needed?\n\nTo improve the API coverage. These functions expose session, partition, and file\nmetadata, and none of them was available in this client before.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new addition to the public API.\n\n```swift\nlet df \u003d try await spark.range(1)\ntry await df.select(current_catalog(), current_database(), version()).show()\n```\n\n### How was this patch tested?\n\nPass the CIs with newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #517 from dongjoon-hyun/SPARK-59210.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "39cc035bdc59ff4bfe1166c3ac0f79fef29c0c79",
      "tree": "bf45e2dc9715c6d4fafa276ba05f4081324c2806",
      "parents": [
        "ff2a3a00ea458270064a4b36f944bb070d228c3c"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 10:15:17 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 10:15:17 2026 -0700"
      },
      "message": "[SPARK-59209] Support `greatest`, `least`, `random` and `try_*` arithmetic functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support the following 10 functions in `MathFunctions`.\n\n- Comparison: `greatest`, `least`\n- Random: `rand`, `randn`, `uniform`\n- Try arithmetic: `try_add`, `try_subtract`, `try_multiply`, `try_divide`, `try_mod`\n\n`greatest` and `least` take at least two `Column`s, which is expressed as two\nrequired parameters followed by a variadic one, and `rand`, `randn` and\n`uniform` provide an additional overload taking a random seed.\n\n### Why are the changes needed?\n\nTo provide the same set of math functions with Apache Spark and PySpark.\n\n- `greatest`, `least`, `rand` and `randn` exist since Apache Spark 1.4.0/1.5.0.\n- `try_add`, `try_subtract`, `try_multiply` and `try_divide` exist since Apache Spark 3.5.0.\n- `try_mod` and `uniform` exist since Apache Spark 4.0.0.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this PR adds new functions only.\n\n```swift\nlet df \u003d try await spark.range(1)\ntry await df.select(greatest(lit(1), lit(2), lit(3)), least(lit(1), lit(2))).show()\ntry await df.select(try_divide(lit(1), lit(0)), try_add(lit(Int64.max), lit(1))).show()\ntry await df.select(rand(42), randn(42), uniform(lit(5), lit(105), lit(3))).show()\n```\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #516 from dongjoon-hyun/SPARK-59209.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "ff2a3a00ea458270064a4b36f944bb070d228c3c",
      "tree": "c410ee2dad019ee8f1b3d041478569911c92bbfc",
      "parents": [
        "6b53750edfd208a9c83c87e7927be5defb5b46b6"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 10:14:33 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 10:14:33 2026 -0700"
      },
      "message": "[SPARK-59207] Support predicate functions\n\n### What changes were proposed in this pull request?\n\nThis PR adds the 9 missing predicate functions to the Swift Spark Connect client, in a new\n`PredicateFunctions.swift`:\n\n- `equal_null`, `isnan`, `isnotnull`, `isnull`, `like`, `ilike`, `regexp`, `regexp_like`, `rlike`\n\n`like` and `ilike` are provided as two overloads each, with and without the optional `escapeChar`\nargument, mirroring the Scala API.\n\nThe grouping follows upstream: these are exactly the members of the `Predicate Functions` section\nin PySpark\u0027s `functions.rst`, and of the `group predicate_funcs` annotation in Scala\u0027s\n`functions.scala` (whose only other member, `not`, is already covered here by the `!` prefix\noperator on `Column`). The new file also matches the existing per-category layout of\n`MathFunctions.swift`, `StringFunctions.swift` and `CollectionFunctions.swift`.\n\nFollowing the existing convention in this repository, all new functions accept `Column` only;\nno `String` overloads are added.\n\n### Why are the changes needed?\n\nThese are among the most frequently used functions in the PySpark API and have no Swift\nequivalent today. `Column` already exposes `isNull()`, `isNotNull()`, `like()`, `rlike()` and\n`ilike()` as methods, but PySpark and the Scala API additionally provide them as free functions,\nwhich is what this PR fills in.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo because there are new public APIs. For example:\n\n```swift\nlet df \u003d try await spark.sql(\"SELECT * FROM VALUES (\u0027Alice\u0027, 20), (\u0027Bob\u0027, NULL) T(name, age)\")\ntry await df.filter(isnotnull(col(\"age\"))).show()\ntry await df.filter(like(col(\"name\"), lit(\"Al%\"))).show()\n```\n\n### How was this patch tested?\n\nPass the CIs with the newly added `PredicateFunctionsTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #515 from dongjoon-hyun/SPARK-59207.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "6b53750edfd208a9c83c87e7927be5defb5b46b6",
      "tree": "946b1e398d16ce10221b51b3d78e75a3006cd8d7",
      "parents": [
        "ab4d7b8c69dc6626a50f0a020771a378ca179946"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 09:55:51 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 09:55:51 2026 -0700"
      },
      "message": "[SPARK-59206] Support hash functions\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support the following 7 hash functions in a new\n`HashFunctions.swift` file.\n\n| Function | Signature | Since |\n| -------- | --------- | ----- |\n| `md5` | `(Column) -\u003e Column` | 1.5.0 |\n| `sha1` | `(Column) -\u003e Column` | 1.5.0 |\n| `sha` | `(Column) -\u003e Column` | 3.5.0 |\n| `sha2` | `(Column, Int32) -\u003e Column` | 1.5.0 |\n| `crc32` | `(Column) -\u003e Column` | 1.5.0 |\n| `hash` | `(Column...) -\u003e Column` | 2.0.0 |\n| `xxhash64` | `(Column...) -\u003e Column` | 3.0.0 |\n\nNote that the existing `CRC32` and `SHA256` structs are internal utilities used\nby `SparkConnectClient+Artifact.swift` to compute client-side checksums during\nartifact upload. They are unrelated to these SQL functions and are untouched.\n\n### Why are the changes needed?\n\nFor feature parity with Apache Spark\u0027s `functions.scala` and PySpark\u0027s\n`pyspark.sql.functions`. The `hash_funcs` group was previously not covered by\nthis Swift client at all.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a new feature addition to the public API. There is no behavior\nchange in the existing APIs.\n\n### How was this patch tested?\n\nPass the CIs with newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #514 from dongjoon-hyun/SPARK-59206.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "ab4d7b8c69dc6626a50f0a020771a378ca179946",
      "tree": "7f529ac2179dbfdacf4113986c0275012e41b217",
      "parents": [
        "9ae3226a81160fa6da530390041e23c2d0041ce3"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 09:04:06 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 09:04:06 2026 -0700"
      },
      "message": "[SPARK-59205] Support `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` in `createDataFrame`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` (`p` in `7...9`) columns in `SparkSession.createDataFrame` with `TimestampNanos` values.\n\n```swift\nlet ts \u003d TimestampNanos(epochMicros: 1_706_000_000_123_456, nanosWithinMicro: 789)!\nlet df \u003d try await spark.createDataFrame([[ts], [nil]], \"t TIMESTAMP_NTZ(9)\")\ntry await df.dtypes    // [(\"t\", \"timestamp_ntz(9)\")]\ntry await df.collect() // [Row(2024-01-23 08:53:20.123456789), Row(nil)]\n```\n\n- `ConvertToArrow` maps both types to an `Apache Arrow` `Timestamp(NANOSECOND)` column (no timezone for `TIMESTAMP_NTZ(p)`, `UTC` for `TIMESTAMP_LTZ(p)`), writing `TimestampNanos` as `Int64` epoch nanoseconds, symmetrically with the `collect()` decoding path added by SPARK-59198. A value outside the `Int64` nanosecond range throws `SparkConnectError.InvalidType` instead of overflowing silently.\n- The precision is sent as the `SPARK::timestampNanos::precision` field metadata key, which `ArrowUtils.fromArrowField` reads on the server, like `SPARK::time::precision` for `TIME(p)` (SPARK-59155).\n\n### Why are the changes needed?\n\nThis is the last missing piece of nanosecond timestamp support after `DataType` (SPARK-59197), `collect()` (SPARK-59198), and `lit`/SQL parameters (SPARK-59203). `createDataFrame` rejected these schemas with `SparkConnectError.InvalidType`.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavior change. Only `createDataFrame` now accepts `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` columns with `TimestampNanos` values against Apache Spark 4.3.0 and later.\n\n### How was this patch tested?\n\nPass the CIs with a newly added test case, `CreateDataFrameTests.timestampNanosTypes`, which verifies `dtypes` and a `createDataFrame` → `collect` round trip for both types and precision 7 to 9, including sub-microsecond digits, a pre-1970 value, `nil`, and the out-of-range error.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5.1\n\nCloses #513 from dongjoon-hyun/SPARK-59205.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "9ae3226a81160fa6da530390041e23c2d0041ce3",
      "tree": "a24ae154e67c84d6430cf42a4f7b7b00f62bcf1c",
      "parents": [
        "add1f17730746ff48eeb267740c508b70eb8555c"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 08:27:50 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 08:27:50 2026 -0700"
      },
      "message": "[SPARK-59203] Support nanosecond timestamp literals in `lit` and SQL parameter binding\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support nanosecond timestamp literals in `lit` and SQL parameter binding.\n\n- Add `lit(_ value: TimestampNanos)` which builds `Expression.Literal.TimestampLTZNanos` with `epochMicros`, `nanosWithinMicro`, and `precision \u003d 9`.\n- Conform `TimestampNanos` to `SparkLiteral` so it can be used directly as an operand of `Column` operators, e.g. `col(\"t\") \u003d\u003d ts`.\n- Handle `TimestampNanos` in `ExpressionLiteral.init(_:)` so it can be bound as a positional (`?`) or named (`:name`) parameter of `spark.sql`.\n\nLike `Date`, which maps to `TIMESTAMP` (`TIMESTAMP_LTZ`), `TimestampNanos` is an absolute point in time and maps to `TIMESTAMP_LTZ(9)`.\n\n```swift\ntry await spark.conf.set(\"spark.sql.timestampNanosTypes.enabled\", \"true\")\nlet ts \u003d TimestampNanos(epochNanos: 1_767_225_600_123_456_789)\nspark.range(1).select(lit(ts))                     // timestamp_ltz(9)\ntry await spark.sql(\"SELECT ?\", ts)\ntry await spark.sql(\"SELECT :t\", args: [\"t\": ts])\n```\n\n### Why are the changes needed?\n\nSPARK-59198 covered only the read path (`DataFrame.collect`). This completes the write path for expressions so that Swift users can send nanosecond timestamp values to the server via `lit`, `Column` operators, and parameterized SQL without losing sub-microsecond digits.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, `TimestampNanos` is newly accepted by `lit`, `Column` operators, and `spark.sql` parameters. There is no behavior change for existing code.\n\n### How was this patch tested?\n\nPass the CIs with newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5.1\n\nCloses #512 from dongjoon-hyun/SPARK-59203.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "add1f17730746ff48eeb267740c508b70eb8555c",
      "tree": "f396aabb296fb8d6ad4c92053adb7159def07bf1",
      "parents": [
        "ac568a63db4768e25d74277441870ea707baeeee"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 06:47:56 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 06:47:56 2026 -0700"
      },
      "message": "[SPARK-59198] Support nanosecond timestamp values in `DataFrame.collect`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support nanosecond timestamp values in `DataFrame.collect` by introducing a new `TimestampNanos` struct.\n\n- A new public struct `TimestampNanos` represents a timestamp with nanosecond precision. Like `TimestampNanosVal` of Apache Spark, it stores `epochMicros: Int64` and `nanosWithinMicro: Int16`, so it is not limited to the `1677 ~ 2262` range of Arrow\u0027s 64-bit nanosecond timestamps. It provides `init(epochNanos:)`, a lossy `date` conversion, and a UTC `description`.\n- `DataFrame.collect` now decodes Arrow `timestamp(NANOSECOND)` columns into `TimestampNanos` values. Apache Spark serializes both `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` (`p` in `[7, 9]`) columns this way. Microsecond timestamp columns still return `Date`.\n- `Row.\u003d\u003d` supports `TimestampNanos` value comparison.\n\n`lit`, SQL parameter binding, and `createDataFrame` for this type will be handled separately.\n\n### Why are the changes needed?\n\n`Date` stores `Double` seconds, so `DataFrame.collect` lost sub-microsecond digits of the nanosecond timestamp types added in Apache Spark 4.3.0.\n\n```swift\ntry await spark.conf.set(\"spark.sql.timestampNanosTypes.enabled\", \"true\")\nlet df \u003d try await spark.sql(\"SELECT CAST(TIMESTAMP_NTZ\u00272026-01-01 00:00:00.123456789\u0027 AS TIMESTAMP_NTZ(9))\")\ntry await df.collect()  // Before: [Row(2026-01-01 00:00:00.123456717)] as `Date`\n                        // After:  [Row(2026-01-01 00:00:00.123456789)] as `TimestampNanos`\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, `DataFrame.collect` returns `TimestampNanos` values instead of lossy `Date` values for `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` columns, and a new public type `TimestampNanos` is added. Existing `TIMESTAMP` and `TIMESTAMP_NTZ` columns still return `Date`.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases. Manually verified the full test suite against Apache Spark 4.3.0 RC1 and 4.2.0.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5.1\n\nCloses #511 from dongjoon-hyun/SPARK-59198.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "ac568a63db4768e25d74277441870ea707baeeee",
      "tree": "fc8f5bcb2b0f6d9ea021a5b71ccb4544d152a1bd",
      "parents": [
        "be0fd6f6f79dd044ccf15ca3369f0d7302c9027a"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 00:32:12 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Thu Sep 03 00:32:12 2026 -0700"
      },
      "message": "[SPARK-59197] Support `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` nanosecond timestamp types in `DataType`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support the nanosecond timestamp types `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` (`p` in `[7, 9]`, Apache Spark 4.3) in `DataType`.\n\n- Add `DataType.timestampNtzNanos(precision:)` and `DataType.timestampLtzNanos(precision:)`.\n- `simpleString` follows Apache Spark\u0027s `typeName`: `timestamp_ntz(p)` / `timestamp_ltz(p)`.\n- Convert both ways with the `timestamp_ntz_nanos` / `timestamp_ltz_nanos` proto kinds. A missing `precision` defaults to `9`.\n\n### Why are the changes needed?\n\nSince Apache Spark 4.3.0, the server sends these proto types in `AnalyzePlan` and `ExecutePlan` responses.\n- https://github.com/apache/spark/pull/57699 ([SPARK-57161] Convert nanosecond-capable timestamp types and literals between proto and Catalyst in Spark Connect)\n\n`DataType.init(_ proto:)` threw `SparkConnectError.InvalidType` for them, so `DataFrame.schema`/`dtypes`/`printSchema`, `parseDDL`, and `collect()` failed on any DataFrame with such a column.\n\n```swift\nlet df \u003d try await spark.sql(\"SELECT CAST(TIMESTAMP_NTZ\u00272026-01-01 00:00:00.123456789\u0027 AS TIMESTAMP_NTZ(9)) ts\")\ntry await df.schema  // Before: SparkConnectError.invalidType\n                     // After : struct\u003cts:timestamp_ntz(9)\u003e\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, this adds two new `DataType` cases.\n\n### How was this patch tested?\n\nPass the CIs with newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5.1\n\nCloses #510 from dongjoon-hyun/SPARK-59197.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "be0fd6f6f79dd044ccf15ca3369f0d7302c9027a",
      "tree": "b965678368085de5ab9fd36f20b19a670755fdd5",
      "parents": [
        "5840405dc19cabba3a547228d47f834d56d79b4d"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 02 23:09:50 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 02 23:09:50 2026 -0700"
      },
      "message": "[SPARK-59195] Create a gRPC channel per session instead of per RPC\n\n### What changes were proposed in this pull request?\n\nThis PR makes `SparkConnectClient` create its gRPC channel once and reuse it for\nevery RPC of a session, instead of creating and tearing down a `GRPCClient` per RPC.\n\n- A new `getGRPCClient()` lazily creates a `GRPCClient` and runs its connections in\n  a detached task. `stop()` shuts it down with `beginGracefulShutdown()`.\n- `DataFrame.withGPRC` no longer builds its own transport; it uses the shared client.\n  This removes a duplicated transport configuration that had to be kept in sync in\n  two places.\n\nKeepalive, compression, proxy support, and custom TLS are left as follow-ups.\n\n### Why are the changes needed?\n\nEvery RPC paid for a full connection setup: DNS resolution, TCP handshake, TLS\nhandshake, and the HTTP/2 preface, and retries paid it again on every attempt.\nReusing the channel removes that cost and matches how the Scala and Python Spark\nConnect clients work.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs.\n\nReconnection was checked by killing the server container while a session was idle\nand starting a new one on the same port; the next query on the same `SparkSession`\nsucceeded, so the shared channel reconnects on its own.\n\nPerformance, timed on loopback with plaintext transport over three runs each on two\nfreshly started servers. `spark.conf.get(...)` is a single unary RPC per iteration,\nso it isolates the connection cost:\n\n| benchmark | before | after |\n| --- | --- | --- |\n| `spark.conf.get(...)` x 100 | 2.58-4.00 ms/RPC | 0.95-1.52 ms/RPC |\n| `spark.catalog.currentDatabase()` x 50 | 9.18-12.88 ms | 6.74-9.57 ms |\n| `spark.range(1).count()` x 50 | 16.34-31.19 ms | 12.85-22.34 ms |\n\nThat is roughly 1.5-2.5 ms saved per RPC. Loopback with plaintext is the most\nfavorable case for the old code; against a remote server over TLS, each RPC\npreviously paid about three extra round trips that are now paid once per session.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #509 from dongjoon-hyun/SPARK-59195.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "5840405dc19cabba3a547228d47f834d56d79b4d",
      "tree": "da884109923a82bc3386a64132b9274feb476365",
      "parents": [
        "276134a115d3a872cb9d4ba01ea7b4562fa45cc1"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 02 22:31:22 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 02 22:31:22 2026 -0700"
      },
      "message": "[SPARK-59194] Use numeric Spark version comparison in tests\n\n### What changes were proposed in this pull request?\n\nThis PR replaces the lexicographic `String` comparisons in the test version gates with a numeric one.\n\n- Adds `isSparkVersionAtLeast(_:_:)` in a new `Tests/SparkConnectTests/SparkVersionUtils.swift`, covered by `SparkVersionUtilsTests`.\n- Migrates all 60 version gates across 12 test files to it.\n- Turns the prefix checks left over from SPARK-59177 into lower bounds: `starts(with: \"4.\")` becomes a `4.0` bound, and `starts(with: \"4.1\")` in `ConstraintTests` a `4.1` bound.\n\n### Why are the changes needed?\n\n`String` comparison is lexicographic, so the gates break from Apache Spark 4.10 on:\n\n```swift\n\"4.10\" \u003e\u003d \"4.2\"              // false\n\"4.10\".starts(with: \"4.1\")   // true, for a 4.1-only gate\n```\n\nEvery `\u003e\u003d \"4.2\"` gate would silently skip its body on a 4.10 server, and the 4.1 gates in `ConstraintTests` would incorrectly run. The lower-bound style from SPARK-59177 is right, but it only holds if the comparison is numeric. The remaining prefix checks match the 4.x line only, so they would also skip on Spark 5; each guards a feature available since Apache Spark 4.0.\n\n`SparkSessionTests.version()` keeps its `starts(with: \"4.\")`, which asserts the server is in the supported 4.x range rather than gating a test.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is a test-only change.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #508 from dongjoon-hyun/SPARK-59194.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "276134a115d3a872cb9d4ba01ea7b4562fa45cc1",
      "tree": "b9dd704d824ca89f8307b76400b3089b1c33b6fa",
      "parents": [
        "6aec224f413293f9b89e63ec4f0a63d975af1af3"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 02 22:13:38 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 02 22:13:38 2026 -0700"
      },
      "message": "[SPARK-59193] Add `firstOrThrow` to avoid trapping on empty server responses\n\n### What changes were proposed in this pull request?\n\nThis PR replaces force unwraps on server-provided response arrays with an explicit `throw`,\nso an empty server response surfaces as a Swift error instead of trapping the process.\n\nA single internal helper, `Array.firstOrThrow()`, is added to `Extension.swift`. It throws\n`SparkConnectError.invalidState` on an empty array and uses a `#function` default argument\nso the error message names the failing operation without any per-call-site boilerplate.\n\n31 call sites are converted across `Catalog` (15), `StreamingQuery` (7),\n`StreamingQueryManager` (3), `SparkConnectClient` (3), `DataFrameStatFunctions` (2), and\n`DataStreamWriter` (1). Both spellings of the same trap are covered, because `Catalog` uses\nthem interchangeably for the same operation: `response.first!` on `[ExecutePlanResponse]`,\nand `df.collect()[0]` / `df.collect().first!` on `[Row]`.\n\n### Why are the changes needed?\n\n`.first!` on a server response array traps and kills the process when the Spark Connect\nserver returns an empty response stream. A misbehaving or incompatible server should not be\nable to crash a client application; it should raise a catchable error.\n\nThis is the same class of protocol violation that `SparkConnectClient+ReattachableExecute`\nalready handles by throwing `invalidState` when the server side session ID changes, so this\nPR follows that precedent instead of adding a new error case.\n\nPractically every public accessor of `StreamingQuery` was affected, as well as the main SQL\nexecution path in `SparkConnectClient.getExecuteExternalCommand`.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, in the failure path only. Where the client previously trapped on an empty server\nresponse, it now throws `SparkConnectError.invalidState`:\n\n```\nThe server returned an empty response for getDatabase(_:).\n```\n\nThere is no behavior change when the server responds normally.\n\n### How was this patch tested?\n\nPass the CIs with a new `ExtensionTests` suite covering `firstOrThrow()`, which requires no\nSpark Connect server. The remaining call sites cannot be unit-tested without a server that\nreturns an empty response stream, so they are covered by the existing integration tests.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #507 from dongjoon-hyun/SPARK-59193.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "6aec224f413293f9b89e63ec4f0a63d975af1af3",
      "tree": "8a6181b7a94ac073756703598c1fb1e5dd69fbcb",
      "parents": [
        "0530fcb4f24eafa25f7ab50b52149f08bee0ca66"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 02 21:41:24 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 02 21:41:24 2026 -0700"
      },
      "message": "[SPARK-59192] Avoid force casts in `DataFrame.collect`\n\n### What changes were proposed in this pull request?\n\nThis PR removes the force casts (`as!`) from `DataFrame.collect()` and `DataFrame.show(_:_:_:)`.\n\n- Per-value casts (`int64`, `date32`) use `as?`, like their sibling branches.\n- Per-column casts (`ArrowTypeTime64`, `ArrowTypeTimestamp`, `AsString` for `binary`/`struct`) throw `SparkConnectError.invalidArrowData`.\n- `show(_:_:_:)` folds the `String` cast into its existing `guard`.\n\n### Why are the changes needed?\n\nA force cast traps and kills the process. Since these casts apply to Arrow data received from a server, unexpected data crashes the host application. Both methods already `throws`, so callers can handle the error instead.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, only where it previously crashed: `int64`/`date32` cells become `nil`, and the other cases throw `SparkConnectError.invalidArrowData`. No change for valid Arrow data.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #506 from dongjoon-hyun/SPARK-59192.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "0530fcb4f24eafa25f7ab50b52149f08bee0ca66",
      "tree": "9c5f8ebb26d3c7aa365c5b3bc373f32e5dec7ced",
      "parents": [
        "8ef00ec6775168c0c2b98151c1291a49f9fe0aa8"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 02 21:39:14 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 02 21:39:14 2026 -0700"
      },
      "message": "[SPARK-59191] Support `grpc_max_message_size` in connection string\n\n### What changes were proposed in this pull request?\n\nThis PR proposes to support the `grpc_max_message_size` parameter in the Spark\nConnect connection string.\n\n- `SparkConnectClient.init(remote:)` parses it now. Previously it fell through to\n  the `default` branch and only printed `Unknown parameter: ...`, leaving the\n  existing `URIParams.PARAM_GRPC_MAX_MESSAGE_SIZE` constant unused.\n- A non-positive or non-integer value throws `SparkConnectError.InvalidArgument`,\n  like the other invalid inputs in the same initializer.\n- The value is applied to the gRPC transport through a default `MethodConfig`\n  (an empty service name applies to all services and methods) in a `ServiceConfig`,\n  at both places creating the transport: `SparkConnectClient.withGPRC` and\n  `DataFrame.withGPRC`. Both `maxRequestMessageBytes` and `maxResponseMessageBytes`\n  are set, matching PySpark which sets both `grpc.max_send_message_length` and\n  `grpc.max_receive_message_length`.\n\nWhen the parameter is absent, an empty `ServiceConfig` is used and the behavior is\nunchanged (gRPC\u0027s 4MiB default).\n\n### Why are the changes needed?\n\n`grpc_max_message_size` is supported by the PySpark and Scala clients and is used to\navoid hitting the default gRPC message size limit when collecting large results.\nThe Swift client silently dropped it, so users had no way to raise the limit.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. `sc://localhost:15002/;grpc_max_message_size\u003d268435456` is now honored instead\nof ignored, and an invalid value such as `grpc_max_message_size\u003dabc` now throws\n`SparkConnectError.InvalidArgument` instead of being ignored.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases in `SparkConnectClientTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #505 from dongjoon-hyun/SPARK-59191.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "8ef00ec6775168c0c2b98151c1291a49f9fe0aa8",
      "tree": "233bd26dd9e29bc511ce3c5c83fa9b07a2a15b70",
      "parents": [
        "57f286dd0e31a8a5dec1c1765250a76f86f5f2d5"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 02 21:38:07 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 02 21:38:07 2026 -0700"
      },
      "message": "[SPARK-59190] Make `Row.getAsBool` throw instead of trapping on type mismatch\n\n### What changes were proposed in this pull request?\n\nThis PR makes `Row.getAsBool(_:)` throw `SparkConnectError.InvalidType` instead of\ntrapping when the value is not a `Bool`, and adds a doc comment describing its\nthrowing behavior.\n\n### Why are the changes needed?\n\n`getAsBool(_:)` is a public API declared as `throws`, but a forced cast traps\n(crashes the process) rather than throwing when the value is not a `Bool`.\nLibrary users cannot catch a trap.\n\nThis is reachable from server-controlled data: `Catalog` calls `getAsBool(0)` on\nthe first column of a collected query result in `databaseExists`, `tableExists`,\n`functionExists`, `dropTempView`, `dropGlobalTempView`, `isCached`, and\n`clearCache`. If a server returns an unexpected column type, the client crashes\ninstead of surfacing a catchable error.\n\n`SparkConnectError.InvalidType` already exists and is used for the same kind of\ntype mismatch elsewhere (`DataType.swift`, `ConvertToArrow.swift`,\n`SparkSession.parseDDL`), so no new error case is introduced.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, this is a behavior change.\n\n`Row.getAsBool(_:)` previously crashed (trapped) on a type mismatch; it now\nthrows `SparkConnectError.InvalidType`. Code that relied on the process\nterminating will now observe a thrown error instead. Since the function is\nalready declared `throws`, no source-level signature change is required for\ncallers. The success path is unchanged.\n\n### How was this patch tested?\n\nPass the CIs with a newly added test case in `RowTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #504 from dongjoon-hyun/SPARK-59190.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "57f286dd0e31a8a5dec1c1765250a76f86f5f2d5",
      "tree": "7a897aeab8ef27ac1a28b2c462abd0ed2761ce9f",
      "parents": [
        "97d3e1eb37a45f0218b4718716bcea0d032f57fc"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 02 10:07:45 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 02 10:07:45 2026 -0700"
      },
      "message": "[SPARK-59179] Add `Xcode 27` (Swift 6.4) GitHub Action job\n\n### What changes were proposed in this pull request?\n\nThis PR aims to add a new `build-macos-26-swift64` GitHub Action job which uses the `xcode-27` runner image.\n\n### Why are the changes needed?\n\nTo validate Apache Spark Connect Swift with the next Swift toolchain on Apple platforms.\n\nCurrently, all macOS jobs are pinned to `Xcode 26.5` (Swift 6.3), while the next toolchain is only validated on Linux via the existing `build-ubuntu-swift64` job (`swiftlang/swift:nightly-6.4.x`). The [`xcode-27` runner image](https://github.com/actions/runner-images/blob/main/images/macos/xcode-27-arm64-Readme.md) ([public preview](https://github.blog/changelog/2026-07-16-xcode-27-runner-image-now-in-public-preview/) since 2026-07-16) provides `Xcode 27.0 beta` with `Swift 6.4` and the macOS/iOS 27 SDKs, which makes the toolchain coverage symmetric across Linux and Apple platforms.\n\nNote that `Xcode 27` is the only installed and default Xcode in that image, so no `xcode-select` step is required.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a CI-only change.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #503 from dongjoon-hyun/SPARK-59179.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "97d3e1eb37a45f0218b4718716bcea0d032f57fc",
      "tree": "a7304763774c15e3752f92ab157d0dec46caecd5",
      "parents": [
        "bc83c441fe7af8b542b4ec4d183dcea520f65594"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 02 10:06:58 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 02 10:06:58 2026 -0700"
      },
      "message": "[SPARK-59178] Use `Xcode 26.6` in CIs\n\n### What changes were proposed in this pull request?\n\nThis PR aims to upgrade Xcode to `26.6` in GitHub Actions.\n\n### Why are the changes needed?\n\nTo validate Apache Spark Connect Swift with the latest Xcode toolchain on CI.\n\n- https://github.com/actions/runner-images/blob/main/images/macos/macos-26-Readme.md\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #502 from dongjoon-hyun/SPARK-59178.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "bc83c441fe7af8b542b4ec4d183dcea520f65594",
      "tree": "df295ac00986fa2ecbdfbc6e8c89bf69223e01e2",
      "parents": [
        "65cdc2949724b053c08a32b6255b6a4646554ae3"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 02 09:44:42 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Sep 02 09:44:42 2026 -0700"
      },
      "message": "[SPARK-59177] Use lower-bound version checks in `SQLTests` and `SparkConnectClientTests`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to replace the exact version-prefix checks in the test suites with lower-bound version comparisons, so that the version-gated tests also run on newer Apache Spark servers.\n\n- `SQLTests`: `queriesForSpark42Only` is renamed to `queriesForSpark42AndLater` and the skip condition becomes `spark.version \u003c \"4.2\"` instead of `!spark.version.starts(with: \"4.2\")`.\n- `SparkConnectClientTests`: the five Declarative Pipelines API gates become `response.sparkVersion.version \u003e\u003d \"4.1\"` instead of `response.sparkVersion.version.starts(with: \"4.1\")`.\n\n### Why are the changes needed?\n\nThe existing conditions match a single feature version only, so the tests are silently skipped on any later server.\n\n| Test | Spark 4.0 | Spark 4.1 | Spark 4.2 | Spark 4.3 |\n| --- | --- | --- | --- | --- |\n| `SQLTests`: `geography.sql`, `geometry.sql`, `time.sql` | skipped | skipped | run | **skipped** |\n| `SparkConnectClientTests`: `createDataflowGraph`, `defineSqlGraphElements` | skipped | run | **skipped** | **skipped** |\n\nIn other words, the newly added `integration-test-mac-spark43` job (SPARK-59156) loses the `TIME`/geospatial SQL coverage, and the Declarative Pipelines APIs have not been covered by any job since Apache Spark 4.2.0. Since these features are available in all later versions, the gates should be lower bounds. This also matches the style already used across the test suites, e.g. `if await spark.version \u003e\u003d \"4.2\"`.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is a test-only change.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #501 from dongjoon-hyun/SPARK-59177.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "65cdc2949724b053c08a32b6255b6a4646554ae3",
      "tree": "cd3b328ea338fb16f62ef2ed2e3d7c7f614a0a09",
      "parents": [
        "7e21abc12e153858c36f63dc3d3cd33700039042"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Sep 01 12:57:16 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Sep 01 12:57:16 2026 -0700"
      },
      "message": "[SPARK-59159] Remove `queriesForSpark4Only` from `SQLTests`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to remove the `queriesForSpark4Only` list and its skip logic from `SQLTests`.\n\n### Why are the changes needed?\n\nApache Spark 3 is no longer supported by this client, so the Spark 3 skip branch is dead code. All the listed queries are now always executed.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #500 from dongjoon-hyun/SPARK-59159.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "7e21abc12e153858c36f63dc3d3cd33700039042",
      "tree": "cc908c40eb5c7480556d5d36874944dac0cf7497",
      "parents": [
        "b579214e96b15e1a0f6a2a34b66aa97424d33f9b"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Sep 01 12:13:13 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Sep 01 12:13:13 2026 -0700"
      },
      "message": "[SPARK-59157] Upgrade `grpc-swift-2` to 2.4.3\n\n### What changes were proposed in this pull request?\n\nThis PR aims to upgrade `grpc-swift-2` to 2.4.3.\n\n### Why are the changes needed?\n\nTo bring the latest bug fixes.\n- https://github.com/grpc/grpc-swift-2/releases/tag/2.4.3 (2026-09-01)\n  - https://github.com/grpc/grpc-swift-2/pull/52\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #499 from dongjoon-hyun/SPARK-59157.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "b579214e96b15e1a0f6a2a34b66aa97424d33f9b",
      "tree": "1fc1b3d73d8b8b17dd7b681908d82904ede0f8b2",
      "parents": [
        "3fe2e798ffba179f5469e8d6ad21e8c627a06807"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Sep 01 12:11:58 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Sep 01 12:11:58 2026 -0700"
      },
      "message": "[SPARK-59156] Add `MacOS` integration test with Apache Spark `4.3.0` RC1\n\n### What changes were proposed in this pull request?\n\nThis PR aims to add an `Apache Spark 4.3.0` RC1 `MacOS` integration test job,\n`integration-test-mac-spark43`.\n\n- https://dist.apache.org/repos/dist/dev/spark/v4.3.0-rc1-bin/\n- https://github.com/apache/spark/releases/tag/v4.3.0-rc1 (2026-08-31)\n\nThe new job mirrors the existing `integration-test-mac-spark42` job, and additionally\nverifies the SHA512 checksum of the downloaded binary distribution.\n\n### Why are the changes needed?\n\nTo be ready for the `Apache Spark 4.3.0` release.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is a CI-only change.\n\n### How was this patch tested?\n\nPass the CIs, especially the new `integration-test-mac-spark43` job.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #498 from dongjoon-hyun/SPARK-59156.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "3fe2e798ffba179f5469e8d6ad21e8c627a06807",
      "tree": "ed0ef53abcc87839b9c45dfbbb5f98ac4ddc04a4",
      "parents": [
        "c9d613f7816355cbd34dd7d87132052cfedcb78f"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Sep 01 11:00:42 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Sep 01 11:00:42 2026 -0700"
      },
      "message": "[SPARK-59155] Support `TIME(p)` precision in `createDataFrame`\n\n### What changes were proposed in this pull request?\n\nThis PR proposes to send the precision of `TIME(p)` columns as `Apache Arrow` field metadata when `createDataFrame` serializes local data into an `Apache Arrow` IPC stream.\n\nThe `Apache Arrow` `Time` type has no fractional-second precision field, so `Apache Spark` carries it in the field metadata under the `SPARK::time::precision` key. See `toPrecisionTaggedArrowField` / `fromArrowField` in [ArrowUtils.scala](https://github.com/apache/spark/blob/v4.3.0-rc1/sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala).\n\n- `ArrowField` gains an optional `metadata`, defaulting to `nil`.\n- `ArrowWriter.writeField` writes it into the FlatBuffers `Field.custom_metadata` vector, sorted by key so that the same input yields a byte-identical stream.\n- `ConvertToArrow` attaches the key to `TIME` columns.\n\n### Why are the changes needed?\n\nApache Spark 4.3.0 implemented this in the following.\n- https://github.com/apache/spark/pull/56778\n\nWithout the metadata, an `Apache Spark` 4.3.0 server reads a `time64[ns]` field as the canonical `TIME(6)` and rejects any other declared precision:\n\n```swift\ntry await spark.createDataFrame([[time]], \"t TIME(0)\")\n```\n\n```\ninternalError: \"[INVALID_COLUMN_OR_FIELD_DATA_TYPE] Column or field `col_0` is of type \"TIME(6)\" while it\u0027s required to be \"TIME(0)\". SQLSTATE: 42000\"\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. `createDataFrame` with a `TIME(p)` schema now works for every precision against `Apache Spark` 4.3.0 and later, where it previously failed for any `p` other than `6`.\n\n### How was this patch tested?\n\nPass the CIs with the existing tests first. Apache Spark 4.3.0 integration test will be added soon.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #497 from dongjoon-hyun/SPARK-59155.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "c9d613f7816355cbd34dd7d87132052cfedcb78f",
      "tree": "c2e8becf773ecf6da76723c65ca933be3b3a1ea3",
      "parents": [
        "6630082a9eaf3a850d2b1c0aa3d31ac37d818207"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Aug 23 20:08:58 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Aug 23 20:08:58 2026 -0700"
      },
      "message": "[SPARK-58957] Upgrade `gRPC Swift NIO Transport` to 2.9.2\n\n### What changes were proposed in this pull request?\n\nThis PR upgrades the `grpc-swift-nio-transport` dependency to `2.9.2`.\n\n### Why are the changes needed?\n\nTo adopt the latest `grpc-swift-nio-transport` release (`2.9.2`, 2026-08-20).\n- https://github.com/grpc/grpc-swift-nio-transport/releases/tag/2.9.2\n  - https://github.com/grpc/grpc-swift-nio-transport/pull/191\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #496 from dongjoon-hyun/SPARK-58957.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "6630082a9eaf3a850d2b1c0aa3d31ac37d818207",
      "tree": "4a186e825e06a5da7897fcb08890663529482203",
      "parents": [
        "03a9f69fe78818f99866ff746e043888b1ab50c4"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Aug 23 20:08:12 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sun Aug 23 20:08:12 2026 -0700"
      },
      "message": "[SPARK-58956] Upgrade `swift-system` to 1.8.1\n\n### What changes were proposed in this pull request?\n\nThis PR aims to upgrade `swift-system` to 1.8.1.\n\n### Why are the changes needed?\n\nTo bring the latest bug fixes.\n\n- https://github.com/apple/swift-system/releases/tag/1.8.1 (2026-08-14)\n  - https://github.com/apple/swift-system/pull/378 (fix(wasi): use mock TLS for all non-threaded WASI targets)\n  - https://github.com/apple/swift-system/pull/379 (fix(MachPort): silence concurrency check for `mach_task_self_`)\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5\n\nCloses #495 from dongjoon-hyun/SPARK-58956.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "03a9f69fe78818f99866ff746e043888b1ab50c4",
      "tree": "8c0536a7ce968ac568ac1604253931dddf41eaf6",
      "parents": [
        "7682a68df6081f5fc2f2be89c00e5539d95269d2"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Aug 08 20:26:11 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Aug 08 20:26:11 2026 -0700"
      },
      "message": "[SPARK-58667] Upgrade `swift-system` to 1.8.0\n\n### What changes were proposed in this pull request?\n\nThis PR aims to upgrade `swift-system` to 1.8.0.\n\n### Why are the changes needed?\n\nTo bring the latest improvements including the Swift 6 language mode support.\n\n- https://github.com/apple/swift-system/releases/tag/1.8.0 (2026-08-04)\n  - https://github.com/apple/swift-system/pull/369\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #494 from dongjoon-hyun/SPARK-58667.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "7682a68df6081f5fc2f2be89c00e5539d95269d2",
      "tree": "dd4e8d29a31b924105d6b56f19b8024612451435",
      "parents": [
        "3a5f20c118bc3fbac0a85168e1e22cf2bb33aff1"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Aug 08 20:25:32 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Aug 08 20:25:32 2026 -0700"
      },
      "message": "[SPARK-52747] Test timezone variations\n\n### What changes were proposed in this pull request?\n\nThis PR aims to test timezone variations in the `timestamp` test of `DataFrameTests` by removing `TODO(SPARK-52747)` and verifying the behavior explicitly under multiple session timezones.\n\n- A timestamp literal with a timezone offset (`Z`) is verified to denote a fixed instant independent of `spark.sql.session.timeZone` (`UTC` and `America/Los_Angeles`).\n- A timestamp literal without a timezone offset is verified to be interpreted in the session timezone.\n- The original `spark.sql.session.timeZone` setting is restored at the end of the test.\n\n### Why are the changes needed?\n\n`TODO(SPARK-52747)` was added by SPARK-52744 as a workaround when the `MacOS` integration test started to run a Spark Connect server directly on the host whose timezone is not `UTC`. This is not a server behavior change but the documented Spark semantics: a timestamp literal without a timezone offset is interpreted in the session timezone. Since the Swift client converts Arrow timestamp values into absolute instants (`Date(timeIntervalSince1970:)`), it already handles timezone variations correctly. This PR closes the TODO by testing those semantics deterministically regardless of the server\u0027s host timezone.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is a test-only change.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #493 from dongjoon-hyun/dongjoon/suspicious-bohr-3d780f.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "3a5f20c118bc3fbac0a85168e1e22cf2bb33aff1",
      "tree": "3888692c8ed5f0ee6a6096ebf68dc3f0630397cc",
      "parents": [
        "2d2c053aab41b63f17cde11b2095b74ddd4142be"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Aug 08 20:24:53 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Aug 08 20:24:53 2026 -0700"
      },
      "message": "[SPARK-52746] Test `isLocal` behavior explicitly for Spark 4.0 and 4.1+ servers\n\n### What changes were proposed in this pull request?\n\nThis PR updates the `isLocal` test in `DataFrameTests` to verify the version-specific expected behavior explicitly, replacing the previous `TODO(SPARK-52746)` guard that simply skipped the checks on Spark 4.1/4.2 servers.\n\n- Spark 4.0.x servers: `SHOW DATABASES`/`SHOW TABLES` plans are expected to be local (`isLocal() \u003d\u003d true`).\n- Spark 4.1+ servers: they are expected to be non-local (`isLocal() \u003d\u003d false`), which is now asserted instead of skipped.\n- The version branch uses `version \u003c \"4.1\"` (following the existing string-comparison pattern in this file) instead of enumerating `\"4.1\"`/`\"4.2\"` prefixes, so it keeps working for Spark 4.3+.\n\n### Why are the changes needed?\n\nThe behavior difference is not a server bug awaiting a fix, but an intentional change in Spark 4.1 by [SPARK-51818](https://issues.apache.org/jira/browse/SPARK-51818) (\"Move QueryExecution creation to AnalyzeHandler and don\u0027t Execute for AnalyzePlanRequests\"). Up to 4.0, `AnalyzePlan` executed commands eagerly, so the analyzed plan became a `CommandResult` and `Dataset.isLocal` returned `true`. Since 4.1, commands are no longer executed during analysis (`CommandExecutionMode.SKIP`), so the plan stays an unexecuted command and `isLocal` returns `false`. Therefore `isLocal() \u003d\u003d false` on 4.1+ is the expected behavior going forward and should be asserted, not skipped.\n- https://github.com/apache/spark/pull/50605\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is a test-only change.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #492 from dongjoon-hyun/dongjoon/magical-archimedes-157671.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "2d2c053aab41b63f17cde11b2095b74ddd4142be",
      "tree": "4c1224f2e80557dd4ba7949ecca0c655d683cdbf",
      "parents": [
        "1650218bf4c2dff094f53ef4b618d08e9e2f29f7"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Aug 07 13:48:15 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Aug 07 13:48:15 2026 -0700"
      },
      "message": "[SPARK-58652] Update `Spark Connect`-generated `Swift` source code with `branch-4.3 (2026-08-07)`\n\n### What changes were proposed in this pull request?\n\nThis PR updates the `Spark Connect`-generated Swift source code by regenerating with `branch-4.3 (2026-08-07)`\n- apache/spark#56300\n- apache/spark#56909\n- apache/spark#57412\n\n### Why are the changes needed?\n\nTo keep the generated Swift source code in sync with Apache Spark `branch-4.3` protobuf definitions.\n\n```\n$ git clone -b branch-4.3 https://github.com/apache/spark.git\n$ cd spark/sql/connect/common/src/main/protobuf/\n$ protoc --swift_out\u003d. spark/connect/*.proto\n$ protoc --grpc-swift_out\u003d. spark/connect/*.proto\n\n// Remove empty GRPC files\n$ cd spark/connect\n$ grep \u0027This file contained no services\u0027 * | awk -F: \u0027{print $1}\u0027 | xargs rm\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #490 from dongjoon-hyun/SPARK-58652.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "1650218bf4c2dff094f53ef4b618d08e9e2f29f7",
      "tree": "e8a14e51a48908c87c16631c9c5ef4990d71303f",
      "parents": [
        "aaccf2680e4a275e38c70f601cde9c0ede52abca"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Aug 07 13:28:21 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Aug 07 13:28:21 2026 -0700"
      },
      "message": "[SPARK-58653] Support typed `StreamingQueryProgress` in `(last|recent)Progress`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support typed `StreamingQueryProgress` in `StreamingQuery.lastProgress` and `recentProgress`.\n\n- `lastProgress` now returns `StreamingQueryProgress?` (previously `String?`) and `recentProgress` returns `[StreamingQueryProgress]` (previously `[String]`).\n- The unused internal structs in `StreamingQueryManager.swift` are moved to a new file `StreamingQueryProgress.swift` and promoted to public `Sendable` and `Codable` types: `StreamingQueryProgress` (renamed from the typo `StreamingQueryProcess`), `SourceProgress`, `SinkProgress`, and `StateOperatorProgress`.\n- JSON decoding is lenient like the Scala/PySpark clients: unknown fields are ignored, missing fields fall back to defaults, and offset fields preserve arbitrary JSON as strings. The original JSON remains accessible via the `json` property.\n\n### Why are the changes needed?\n\nFor feature parity with the Scala and PySpark clients which return typed `StreamingQueryProgress` objects.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, `lastProgress` and `recentProgress` now return typed values instead of raw JSON strings. The raw JSON is still available via the `json` property.\n\n### How was this patch tested?\n\nPass the CIs with the newly added unit tests.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #491 from dongjoon-hyun/SPARK-58653.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "aaccf2680e4a275e38c70f601cde9c0ede52abca",
      "tree": "09013aa41d70b2a54654e4b137942e40e0ba0287",
      "parents": [
        "3a3a0f39bc577bfb84e353dd2d02cbcbfaa17fb2"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Aug 07 13:27:01 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Aug 07 13:27:01 2026 -0700"
      },
      "message": "[SPARK-58651] Support `parseDDL` in `SparkSession`\n\n### What changes were proposed in this pull request?\n\nThis PR adds `SparkSession.parseDDL` which parses a DDL-formatted schema string into the public `StructType` model using the server-side parser.\n\n```swift\nlet schema \u003d try await spark.parseDDL(\"id BIGINT NOT NULL, name STRING\")\n// StructType(fields: [\n//   StructField(name: \"id\", dataType: .long, nullable: false),\n//   StructField(name: \"name\", dataType: .string),\n// ])\n```\n\nInternally, it reuses the existing `AnalyzePlan` `ddlParse` round-trip (already used by `createDataFrame` and `DataFrameReader.schema`) and converts the returned protobuf `DataType` via the `StructType` initializer added in SPARK-58647. A non-struct top-level type (e.g. `array\u003cint\u003e`) throws `SparkConnectError.InvalidType`, and an invalid DDL string throws `SparkConnectError.ParseSyntaxError` through the existing `ErrorInfo`-based error conversion.\n\nNote that this follows the PySpark Connect behavior where `DataType.fromDDL` delegates to the server-side parser (`SparkSession._parse_ddl` via the `ddl_parse` analyze method), instead of implementing a client-local SQL type parser.\n\n### Why are the changes needed?\n\nFor feature parity with Scala (`StructType.fromDDL`) and PySpark (`DataType.fromDDL`), so users can obtain a traversable `StructType` from a DDL string. This is the missing inverse of `StructType.toDDL`.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This adds a new API without changing existing behavior.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases .\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #489 from dongjoon-hyun/SPARK-58651.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "3a3a0f39bc577bfb84e353dd2d02cbcbfaa17fb2",
      "tree": "2abbead754aac9b3e4be56afd6468363333c55a9",
      "parents": [
        "b0e60bd28ab962b36859d48354a8c51df507f1b9"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Aug 07 12:07:33 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Aug 07 12:07:33 2026 -0700"
      },
      "message": "[SPARK-58648] Support StructType in schema, createDataFrame, and to APIs\n\n### What changes were proposed in this pull request?\n\nThis PR adds `StructType` overloads to the four schema-taking APIs which previously accepted only DDL strings.\n\n- `DataFrameReader.schema(_ schema: StructType)`\n- `DataStreamReader.schema(_ schema: StructType)`\n- `SparkSession.createDataFrame(_ data: [[Sendable?]], _ schema: StructType)`\n- `DataFrame.to(_ schema: StructType)`\n\nFollowing the Scala client implementation, the reader and `createDataFrame` overloads convert the schema via the newly added public `StructType.toDDL`/`StructField.toDDL` (Catalyst `toDDL` format with backtick-quoting and `NOT NULL`) and reuse the existing DDL-string paths, while `DataFrame.to` converts the model directly to the protobuf `DataType` (like `DataTypeProtoConverter.toConnectProtoType`) and requires no server round-trip.\n\n```swift\nlet schema \u003d StructType(fields: [\n  StructField(name: \"id\", dataType: .integer, nullable: false),\n  StructField(name: \"name\", dataType: .string),\n])\nlet df \u003d try await spark.createDataFrame([[1, \"Alice\"], [2, nil]], schema)\n```\n\n### Why are the changes needed?\n\nFor feature parity with the Scala/PySpark clients where all schema-taking APIs accept a `StructType` object, so users can build and pass schemas programmatically instead of assembling DDL strings by hand.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This adds new APIs (`StructType` overloads and `toDDL`) without changing existing behavior.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #488 from dongjoon-hyun/SPARK-58648.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "b0e60bd28ab962b36859d48354a8c51df507f1b9",
      "tree": "cac6a507810500fcc9572d2102f02d3e06bb1ce3",
      "parents": [
        "b1e8af5778af5b0d3ada5e83e4a2bc18102b03b5"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Aug 07 11:03:59 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Aug 07 11:03:59 2026 -0700"
      },
      "message": "[SPARK-58647] Add public `DataType` and `StructType`\n\n### What changes were proposed in this pull request?\n\nThis PR adds public schema model types — `DataType`, `StructType`, and `StructField` — and changes `DataFrame.schema` to return a `StructType` instead of a JSON string.\n\n- `DataType` is an `indirect enum` covering all Spark Connect type kinds, with `simpleString` for Spark SQL-style representations like `decimal(10,2)` and `array\u003cstring\u003e`.\n- `StructType` holds an ordered list of `StructField`s (`name`/`dataType`/`nullable`/`metadata`) and supports subscripts and iteration. All types are `Sendable` and `Equatable`.\n\n```swift\nlet schema \u003d try await df.schema\nfor field in schema {\n  print(\"\\(field.name): \\(field.dataType.simpleString) (nullable\u003d\\(field.nullable))\")\n}\n```\n\n### Why are the changes needed?\n\nFor feature parity with Scala/PySpark where `df.schema` returns a traversable `StructType` instead of a raw JSON string.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. `DataFrame.schema` now returns a `StructType` instead of a JSON `String`.\n\n### How was this patch tested?\n\nPass the CIs with newly added `DataTypeTests` and `StructTypeTests` suites and the revised `DataFrameTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #487 from dongjoon-hyun/SPARK-58647.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "b1e8af5778af5b0d3ada5e83e4a2bc18102b03b5",
      "tree": "d1af5c9cdcaf4ccdfda50440469222c4631c4636",
      "parents": [
        "044cc6c0934ff34e2595c16ce939fd503a5cac42"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Aug 07 09:45:12 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Fri Aug 07 09:45:12 2026 -0700"
      },
      "message": "[SPARK-58644] Document intentional ignore of `queryStartedEventJson` in `DataStreamWriter`\n\n### What changes were proposed in this pull request?\n\nThis PR replaces the dead `// TODO: post` block in `DataStreamWriter.start()` with a comment\ndocumenting that `WriteStreamOperationStartResult.queryStartedEventJson` is intentionally\nignored because this client does not support `StreamingQueryListener` yet.\n\n### Why are the changes needed?\n\nThe previous code checked `result.hasQueryStartedEventJson` and did nothing, leaving a\nmisleading `TODO` that suggested an unfinished code path. Reference clients (PySpark and\nScala Connect) parse this JSON into a `QueryStartedEvent` and post it to their\n`StreamingQueryListener` bus, which is a no-op when no listener is registered. Since this\nclient has no listener subsystem, dropping the event is semantically equivalent; the comment\nnow records that decision and marks the posting point for future listener support.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs. This is a comment-only change with no behavior difference.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #486 from dongjoon-hyun/SPARK-58644.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "044cc6c0934ff34e2595c16ce939fd503a5cac42",
      "tree": "75f70e77333310b5a795ff0d38aadf122574696c",
      "parents": [
        "65e60b228e019f79e315023ef49715d88c9ae8e8"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Aug 05 22:59:53 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Aug 05 22:59:53 2026 -0700"
      },
      "message": "[SPARK-58610] Support `cloneSession` in `SparkSession`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `cloneSession` API in `SparkSession` by using the `CloneSession` RPC.\n\n```swift\n// Clone with a server-generated session ID\nlet cloned \u003d try await spark.cloneSession()\n\n// Clone with a custom session ID (must be a valid UUID)\nlet cloned \u003d try await spark.cloneSession(\"00112233-4455-6677-8899-aabbccddeeff\")\n```\n\n- `SparkSession.cloneSession(_ newSessionID: String? \u003d nil)` follows the PySpark API shape (`cloneSession(new_session_id\u003dNone)`) and covers both Scala overloads (`cloneSession()` / `cloneSession(sessionId:)`) via the optional parameter.\n- `SparkConnectClient.cloneSession` issues the `CloneSession` RPC and validates that the returned session ID matches the requested one when provided, like the Scala and Python clients.\n- The existing `newSession()` semantics are unchanged, consistent with the Scala client where `newSession()` (fresh empty session) and `cloneSession()` (server-side state clone) are distinct APIs.\n\n### Why are the changes needed?\n\nThe `CloneSession` RPC was added via [SPARK-53455](https://github.com/apache/spark/commit/264ca4dc320c9de52ce53fdbdca8146ea9d1a5b6) and is available since [Apache Spark 4.1.0 (2025-12-11)](https://github.com/apache/spark/releases/tag/v4.1.0). This allows users to create a new independent session that inherits the original session\u0027s server-side state (SQL configurations, temporary views, registered functions, catalog state), which `newSession()` cannot provide.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, this adds a new API, `SparkSession.cloneSession`. For old servers without `CloneSession` RPC support (Apache Spark 4.0.x), the server error is propagated like the Scala and Python clients.\n\n### How was this patch tested?\n\nPass the CIs with the newly added integration tests, `cloneSession` and `cloneSessionWithSessionID`, which verify that the cloned session inherits SQL configurations and temporary views and that subsequent changes to either session are isolated from the other. The tests are guarded to run on Apache Spark 4.1+ servers, and were verified manually against a local Apache Spark 4.2.0 Connect server.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #485 from dongjoon-hyun/SPARK-58610.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "65e60b228e019f79e315023ef49715d88c9ae8e8",
      "tree": "ce247f2f4fbe3d4ddfc316640e8199b61e49b004",
      "parents": [
        "84a6cc5de6759da44406e573de856c82d094f64a"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Aug 05 20:14:17 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Wed Aug 05 20:14:17 2026 -0700"
      },
      "message": "[SPARK-58608] Support `Observation` to retrieve observed metrics\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `Observation` to retrieve the observed metrics of `DataFrame.observe`.\n\n```swift\nlet observation \u003d Observation(\"my_metrics\")\nlet observedDf \u003d try await df.observe(observation, count(col(\"*\")), max(col(\"id\")))\ntry await observedDf.count()\nlet metrics \u003d try await observation.get  // [\"count(1)\": Int64(10), \"max(id)\": ...]\n```\n\n- Add a new `Observation` actor whose `get` property returns the observed metrics keyed by the metric column names, or throws `SparkConnectError.invalidState` if no action has been performed yet.\n- Add a `DataFrame.observe(_ observation: Observation, _ expr: Column, _ exprs: Column...)` overload which registers the observation to `SparkConnectClient`.\n- Collect `ExecutePlanResponse.observed_metrics` inside `executePlanWithReattach` and deliver the values to the registered observations by matching `ObservedMetrics.name`, like the PySpark Connect client. Since a reattached stream resumes after `last_response_id`, no response is delivered twice.\n- Add an `ExpressionLiteral.toSwiftValue` helper converting metric value literals back to Swift values.\n\n### Why are the changes needed?\n\nSPARK-58425 added `DataFrame.observe` which builds the `CollectMetrics` relation, but the observed metric values returned by the server via `ExecutePlanResponse.observed_metrics` were discarded, so users had no way to read them. This provides the `Observation` API with the same usage as Scala and PySpark.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is a new feature addition.\n\n### How was this patch tested?\n\nPass the CIs with a newly added test case.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #484 from dongjoon-hyun/SPARK-58608.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "84a6cc5de6759da44406e573de856c82d094f64a",
      "tree": "e698286dfa2515e3bd25ed4fe142f5bb87bffc15",
      "parents": [
        "434c5b9c72f7488eb80fbb899c63ee16dc083941"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Aug 04 17:58:24 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Aug 04 17:58:24 2026 -0700"
      },
      "message": "[SPARK-58579] Support `TIME` type in `SparkSession.createDataFrame`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `TIME` type in `SparkSession.createDataFrame`.\n\n```swift\nlet time \u003d LocalTime(hour: 12, minute: 34, second: 56, nanosecond: 123_456_000)!\nlet df \u003d try await spark.createDataFrame([[time], [nil]], \"t TIME\")\ntry await df.collect()  // [Row(12:34:56.123456), Row(nil)]\n```\n\n- `ConvertToArrow` maps the `time` DDL type to an Arrow `time64(NANOSECOND)` column and\n  writes `LocalTime.nanoOfDay` values as-is, symmetrically with the `collect()` decoding\n  path added by SPARK-58571. `nil` is mapped to `NULL` like the other types.\n- Fix the vendored `ArrowWriterHelper` to serialize the `bitWidth` field (64) of the\n  FlatBuffers `Time` type for `time64`. Without it, the field defaults to 32 and the\n  server rejects the uploaded column with\n  `[UNSUPPORTED_ARROWTYPE] Unsupported arrow type Time(NANOSECOND, 32)`.\n  (The same omission exists in upstream `apache/arrow-swift`; `time32` is unaffected\n  because 32 is the correct default there.)\n\n### Why are the changes needed?\n\nThis is the last missing piece of `TIME` type support. `DataType.simpleString`\n(SPARK-58568), `collect()` (SPARK-58571), and `lit`/SQL parameter binding (SPARK-58574)\nalready support `TIME`, but `createDataFrame` rejected `t TIME` schemas with\n`SparkConnectError.InvalidType`.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, `createDataFrame` now accepts `TIME` columns with `LocalTime` values.\nPreviously it threw `SparkConnectError.InvalidType`.\n\n### How was this patch tested?\n\nPass the CIs with a newly added test case, `CreateDataFrameTests.timeType`, which\nverifies a `createDataFrame` → `collect` round trip including `LocalTime` values and\n`nil`, and the `time(6)`/`time(0)` schema mapping. The test is guarded by\n`spark.version \u003e\u003d \"4.2\"` and `spark.sql.timeType.enabled`, and was verified manually\nagainst both Apache Spark 4.2.0 (runs) and 4.1.3 (skips) Spark Connect servers.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #483 from dongjoon-hyun/dongjoon/silly-lederberg-905677.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "434c5b9c72f7488eb80fbb899c63ee16dc083941",
      "tree": "b48c71ad2dc35aaea0c82706d102bbe017c4e6dc",
      "parents": [
        "ac834d8636eb727d8ca8af4207a67c74586986d8"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Aug 04 12:58:43 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Aug 04 12:58:43 2026 -0700"
      },
      "message": "[SPARK-58574] Support `TIME` type literals in `lit` and SQL parameter binding\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `TIME` type literals in `lit` and SQL parameter binding.\n\n- Add `lit(_ value: LocalTime)` which builds `Expression.Literal.Time` with `nano \u003d nanoOfDay` and `precision \u003d 6`, matching the Scala client (`TimeTypeConnectOps`) which always uses `DEFAULT_PRECISION`.\n- Conform `LocalTime` to `SparkLiteral` so it can be used directly as an operand of `Column` operators, e.g. `col(\"t\") \u003d\u003d LocalTime(hour: 12, minute: 34, second: 56)!`.\n- Handle `LocalTime` in `ExpressionLiteral.init(_:)` so it can be bound as a positional (`?`) or named (`:name`) parameter of `spark.sql`.\n\n```swift\nlet time \u003d LocalTime(hour: 12, minute: 34, second: 56)!\nspark.range(1).select(lit(time))\ntry await spark.sql(\"SELECT ?\", time)\ntry await spark.sql(\"SELECT :t\", args: [\"t\": time])\n```\n\n### Why are the changes needed?\n\nSPARK-58571 covered only the read path (`DataFrame.collect`). This completes the write path for expressions so that Swift users can send `TIME` values to the server via `lit`, `Column` operators, and parameterized SQL, on par with the Scala client.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, `LocalTime` is newly accepted by `lit`, `Column` operators, and `spark.sql` parameters. There is no behavior change for existing code.\n\n### How was this patch tested?\n\nPass the CIs with newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #482 from dongjoon-hyun/SPARK-58574.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "ac834d8636eb727d8ca8af4207a67c74586986d8",
      "tree": "5c65af1a4ad2e128bef4baab4b17cfdd63f4e00c",
      "parents": [
        "2ecd79c24600bb5f999c89c397987d130446ed38"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Aug 04 11:29:17 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Aug 04 11:29:17 2026 -0700"
      },
      "message": "[SPARK-58571] Support `TIME` type values in `DataFrame.collect`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `TIME` type values in `DataFrame.collect` by introducing a new\n`LocalTime` struct, following up SPARK-58568 which covered the schema path.\n\n- A new public struct `LocalTime` represents a time of day without a date or a time zone,\n  like `java.time.LocalTime` of the Scala client and `datetime.time` of PySpark. It stores\n  `nanoOfDay` and exposes `hour`/`minute`/`second`/`nanosecond` components.\n- `DataFrame.collect` now decodes Arrow `time64` columns into `LocalTime` values. Apache\n  Spark serializes `TIME(n)` columns as Arrow `time64(NANOSECOND)` regardless of precision.\n- `Row.\u003d\u003d` supports `LocalTime` value comparison.\n\nNo vendored `Arrow*.swift` file is touched; the Arrow layer already decodes `time64`.\n\n### Why are the changes needed?\n\n`DataFrame.collect` silently returned `nil` for every `TIME` column value because the\nArrow-to-Row conversion had no `time64` case and fell through to the `String` cast.\n\n```swift\nlet df \u003d try await spark.sql(\"SELECT TIME\u002712:34:56\u0027\")\ntry await df.collect()  // Before: [Row(nil)]\n                        // After:  [Row(LocalTime(hour: 12, minute: 34, second: 56))]\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, `DataFrame.collect` returns `LocalTime` values instead of `nil` for `TIME` columns,\nand a new public type `LocalTime` is added. This is a bug fix from the unreleased perspective.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #481 from dongjoon-hyun/SPARK-58571.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "2ecd79c24600bb5f999c89c397987d130446ed38",
      "tree": "aa4485391bf15ee07d983be3ee5065beb534cd9d",
      "parents": [
        "71f2d3b82107efe2f01a790541d245449cb77fc9"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Aug 04 10:40:37 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Aug 04 10:40:37 2026 -0700"
      },
      "message": "[SPARK-58568] Support `TIME` type in `DataType.simpleString`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `TIME` type (added in Apache Spark 4.1) in `DataType.simpleString`\nso that `DataFrame.schema`/`dtypes`/`printSchema` work on `TIME` columns.\n\nFollowing Apache Spark\u0027s `TimeType.typeName` convention, the mapping uses `time(\u003cprecision\u003e)`.\nWhen the proto `precision` field is absent, it falls back to `time(6)` like Apache Spark\u0027s\n`TimeType.DEFAULT_PRECISION` (`MICROS_PRECISION`).\n\n### Why are the changes needed?\n\n`Spark_Connect_DataType`\u0027s `time` kind fell through to the `default:` branch of\n`DataType.simpleString` and threw `SparkConnectError.InvalidType`. As a result,\n`DataFrame.dtypes` failed entirely on any DataFrame containing a `TIME` column.\n\n```swift\nlet df \u003d try await spark.sql(\"SELECT TIME\u002712:34:56\u0027\")\ntry await df.dtypes  // Before: throws SparkConnectError.InvalidType\n                     // After: [(\"TIME \u002712:34:56\u0027\", \"time(6)\")]\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, `DataFrame.dtypes` no longer throws on `TIME` columns and returns Spark-style type\nstrings instead. This is a bug fix from the unreleased perspective.\n\n### How was this patch tested?\n\nPass the CIs with a newly added test case. The new test enables `spark.sql.timeType.enabled`\nat the session level and is guarded to run on Apache Spark 4.2+ only.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #480 from dongjoon-hyun/SPARK-58568.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "71f2d3b82107efe2f01a790541d245449cb77fc9",
      "tree": "2c28c3256c5afd7c470567448deaf6f51a594bc2",
      "parents": [
        "d4b38b463f13bbbe6da9a0fd71e2260e0da7e506"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Aug 04 09:29:40 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Aug 04 09:29:40 2026 -0700"
      },
      "message": "[SPARK-58563] Support server-side error chain in `SparkConnectError` via `FetchErrorDetails`\n\n### What changes were proposed in this pull request?\n\nThis PR improves `SparkConnectError` to carry the server-side error chain by calling the\n`FetchErrorDetails` RPC, like the PySpark (`_fetch_enriched_error`) and Scala\n(`GrpcExceptionConverter.fetchEnrichedError`) clients.\n\n1. A new `SparkConnectError.ServerError` struct carries the un-truncated error message, the\n   exception class hierarchy, the server-side stack trace frames, and the per-error\n   `errorClass`/`sqlState`/`messageParameters`. `SparkConnectError.Details` gains a `serverErrors`\n   array (the thrown error first, followed by the cause chain flattened via `causeIdx`) and a\n   `serverStackTrace` convenience property that renders the chain in the JVM\n   `Caused by:`/`at ...` style.\n2. When the `google.rpc.ErrorInfo` metadata contains an `errorId`, `GrpcErrorConverter` sends one\n   `FetchErrorDetails` request over the already-open gRPC connection. The call is best-effort\n   at-most-once without retries, and any failure silently falls back to the previous\n   `ErrorInfo`-only conversion so that the original error is never masked.\n3. The `errorClass` and `sqlState` from `ErrorInfo` take precedence to keep the error\n   classification stable, while the truncated `message` and `messageParameters` are replaced by\n   the un-truncated values of the root error.\n\nThe server-side stack trace is populated only when the SQL configuration\n`spark.sql.connect.serverStacktrace.enabled` is true, like the other clients.\n\n```swift\n} catch SparkConnectError.tableOrViewNotFound(let details) {\n  print(details.serverErrors.first?.errorTypeHierarchy ?? [])\n  // [\"org.apache.spark.sql.catalyst.ExtendedAnalysisException\",\n  //  \"org.apache.spark.sql.AnalysisException\", ...]\n  print(details.serverStackTrace ?? \"\")\n}\n```\n\n### Why are the changes needed?\n\nUsers could not access the server-side stack trace, the cause chain, or the un-truncated error\nmessage from a caught error although the server provides them via the `FetchErrorDetails` RPC.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, `SparkConnectError` additionally provides the server-side error chain. The existing code\ncontinues to compile and behave the same because the new `Details.serverErrors` field has a\ndefault value and the error classification is unchanged.\n\n### How was this patch tested?\n\nPass the CIs with the newly added `serverErrors` and `serverStackTrace` tests in\n`SparkConnectErrorTests`. All existing test suites pass unmodified.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #479 from dongjoon-hyun/SPARK-58563.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "d4b38b463f13bbbe6da9a0fd71e2260e0da7e506",
      "tree": "a453bdedac29a2de163fe2d052d45d015d97e5ec",
      "parents": [
        "c627124ccbc05927aa0af4a42924154a41f2e6ce"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Aug 04 07:56:35 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Aug 04 07:56:35 2026 -0700"
      },
      "message": "[SPARK-58560] Use Apache Spark `4.0.4` and `4.1.3` in CI\n\n### What changes were proposed in this pull request?\n\nThis PR aims to use Apache Spark `4.0.4` and `4.1.3` in CI instead of `4.0.3` and `4.1.2`.\n\nIn addition, this PR updates the tested server versions in `AGENTS.md` accordingly.\n\n### Why are the changes needed?\n\nTo test with the latest Apache Spark maintenance releases:\n\n- [v4.1.3](https://github.com/apache/spark/releases/tag/v4.1.3) (2026-07-11)\n- [v4.0.4](https://github.com/apache/spark/releases/tag/v4.0.4) (2026-07-12)\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a test infra update.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #478 from dongjoon-hyun/SPARK-58560.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "c627124ccbc05927aa0af4a42924154a41f2e6ce",
      "tree": "23aff92a51c381e51e72c7df6e6db2a39cf6443d",
      "parents": [
        "02e33e08ae9874d0c89354182f71f016d86c9e93"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Aug 04 07:12:17 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Tue Aug 04 07:12:17 2026 -0700"
      },
      "message": "[SPARK-58543] Support reattachable execution with `ReattachExecute` and `ReleaseExecute`\n\n### What changes were proposed in this pull request?\n\nThis PR supports reattachable execution in `SparkConnectClient` like the Scala side\n`org.apache.spark.sql.connect.client.ExecutePlanResponseReattachableIterator` and the Python\nside `pyspark.sql.connect.client.reattach.ExecutePlanResponseReattachableIterator`.\n\n1. A new `executePlanWithReattach` helper executes `ExecutePlan` with\n   `ReattachOptions.reattachable\u003dtrue` and tracks `response_id`, `server_side_session_id`, and\n   `ResultComplete` of the response stream.\n2. When the response stream is broken by a retriable RPC error (governed by the existing\n   `RetryPolicy.defaultPolicy` with exponential backoff, reset whenever an attempt receives a\n   new response), or ends gracefully without a `ResultComplete` message, the stream is resumed\n   with `ReattachExecute` after the last received response on a fresh channel.\n3. When `ReattachExecute` fails with `INVALID_HANDLE.OPERATION_NOT_FOUND` or\n   `INVALID_HANDLE.SESSION_NOT_FOUND` before receiving any response, the initial `ExecutePlan`\n   didn\u0027t reach the server and is started over. After receiving responses, it is surfaced as\n   the new `SparkConnectError.invalidState`, which is also thrown when the server side session\n   ID changes in the middle of a stream.\n4. The server side buffer is released with a `release_all` `ReleaseExecute` when\n   `ResultComplete` arrives, and in a best-effort way when the execution is abandoned after an\n   error. Incremental `release_until` calls are a server-side optimization hint (the server\n   auto-releases the sent responses beyond `spark.connect.execute.reattachable.observerRetryBufferSize`)\n   and are left as a follow-up.\n5. All three `ExecutePlan` consumption paths (`DataFrame.execute`, `DataFrame.count`, and\n   `SparkConnectClient.execute`) use the new helper, superseding the temporary\n   retry-until-the-first-response pattern introduced by SPARK-58541.\n\n### Why are the changes needed?\n\n`ExecutePlan` was a single non-resumable server stream. Once the first response arrived, any\nbroken connection failed the whole query without a recovery path, and the temporary retry of\nSPARK-58541 could not help because re-executing the plan can have side effects. Reattachable\nexecution is the mechanism the Scala and Python clients use by default (Spark 3.5+ servers),\nand this PR brings the Swift client to parity.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, executions now survive transient response stream breaks by resuming after the last\nreceived response instead of failing, and the server releases its response buffers explicitly\nat the end of each execution. Successful executions return the same results as before.\n\n### How was this patch tested?\n\nPass the CIs with the newly added `ReattachableExecuteTests` test suite covering the\nreattachable `ExecutePlan`, `ReattachExecute`, and `ReleaseExecute` request builders. All\nexisting test suites pass with a live Spark Connect server 4.2.0, now exercising every\n`ExecutePlan` through the reattachable path. Note that a real mid-stream disconnection is not\nreproducible reliably in the integration tests (the server ends idle reattachable streams only\nafter `spark.connect.execute.reattachable.senderMaxStreamDuration`, 2 minutes by default), so\nthe reattach path is verified indirectly through the graceful stream-end handling.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #477 from dongjoon-hyun/SPARK-58543.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "02e33e08ae9874d0c89354182f71f016d86c9e93",
      "tree": "e46a63e12e8e2fa596a8ad3652b899a808e3ff02",
      "parents": [
        "55a31f10877ea59f9812b69bd2f8420ea23e47ad"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Aug 03 23:46:19 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Aug 03 23:46:19 2026 -0700"
      },
      "message": "[SPARK-58542] Upgrade `gRPC Swift NIO Transport` to 2.9.1\n\n### What changes were proposed in this pull request?\n\nThis PR upgrades the `grpc-swift-nio-transport` dependency to `2.9.1`.\n\n### Why are the changes needed?\n\nTo adopt the latest `grpc-swift-nio-transport` release (`2.9.1`, 2026-08-03).\n- https://github.com/grpc/grpc-swift-nio-transport/releases/tag/2.9.1\n  - https://github.com/grpc/grpc-swift-nio-transport/pull/185\n  - https://github.com/grpc/grpc-swift-nio-transport/pull/187\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nPass the CIs.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #476 from dongjoon-hyun/SPARK-58542.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "55a31f10877ea59f9812b69bd2f8420ea23e47ad",
      "tree": "dc905357e95f61d16c719600f62dbab662db3f15",
      "parents": [
        "b3e3f1bd530338d6716c747410ed2815617956b8"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Aug 03 23:45:25 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Aug 03 23:45:25 2026 -0700"
      },
      "message": "[SPARK-58541] Support retry with exponential backoff in `SparkConnectClient`\n\n### What changes were proposed in this pull request?\n\nThis PR supports automatic retries with exponential backoff for RPC calls in `SparkConnectClient`.\n\n1. A new `RetryPolicy` implements the same default policy as the Scala side\n   `org.apache.spark.sql.connect.client.RetryPolicy` and the Python side\n   `pyspark.sql.connect.client.retries.DefaultPolicy`: `maxRetries\u003d15`, `initialBackoff\u003d50ms`,\n   `maxBackoff\u003d60s`, `backoffMultiplier\u003d4.0`, `jitter\u003d500ms`, `minJitterThreshold\u003d2s`, and\n   `maxServerRetryDelay\u003d10min`, guaranteeing a maximum tolerated wait of at least 10 minutes.\n2. Retriable errors are `UNAVAILABLE`, `INTERNAL` caused by `INVALID_CURSOR.DISCONNECTED`, and\n   any error containing `google.rpc.RetryInfo` in its gRPC status details. A server-provided\n   `RetryInfo.retry_delay` overrides the client\u0027s backoff, limited by `maxServerRetryDelay`.\n3. A `withRetry` helper wraps both `withGPRC` implementations, so every non-streaming RPC\n   (`AnalyzePlan`, `Config`, `Interrupt`, `GetStatus`, artifact RPCs, etc.) is retried with a\n   fresh channel per attempt. `Task` cancellation stops the retry loop immediately, and the last\n   error is rethrown when the retries are exhausted.\n4. Since `ExecutePlan` can have side effects on the server, it is retried only until the first\n   response arrives (the operation is not started before that). A complete solution is\n   `ReattachExecute`, which is a follow-up.\n\n### Why are the changes needed?\n\nEvery RPC call failed immediately on transient network errors, server restarts, or `UNAVAILABLE`\nresponses. The PySpark and Scala Spark Connect clients retry these errors by default, and this PR\nbrings the Swift client to parity with their `DefaultPolicy`.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, transient RPC failures are now retried automatically for up to 15 attempts instead of\nfailing immediately. Non-retriable errors are still thrown as before.\n\n### How was this patch tested?\n\nPass the CIs with the newly added `RetryPolicyTests` test suite covering the retriable-error\nclassification, the exponential backoff sequence and its cap, the server-provided retry delay and\nits limit, the retry loop attempt counts, and the cancellation propagation. All existing test\nsuites pass with a live Spark Connect server.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #475 from dongjoon-hyun/SPARK-58541.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "b3e3f1bd530338d6716c747410ed2815617956b8",
      "tree": "2afdcd39ac40cd3c5c91b980aaac1b80ece981f2",
      "parents": [
        "40db9c0a19f6c833ace32f0aa75d62c621708eb7"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Aug 03 22:28:31 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Aug 03 22:28:31 2026 -0700"
      },
      "message": "[SPARK-58539] Support error details in `SparkConnectError` using gRPC `ErrorInfo`\n\n### What changes were proposed in this pull request?\n\nThis PR improves `SparkConnectError` to carry error details and replaces message-substring-based\nerror classification with `google.rpc.ErrorInfo` parsing.\n\n1. Each `SparkConnectError` case now carries a `Details` payload (`message`, `errorClass`,\n   `sqlState`, `messageParameters`). The cases are renamed to `lowerCamelCase` while the original\n   spellings are kept as compatibility static properties, so the existing `throw`/`catch`/\n   `#expect(throws:)` call sites continue to compile unchanged.\n2. A new `GrpcErrorConverter` classifies errors by the Spark error class from `ErrorInfo` of the\n   gRPC status details, like the PySpark and Scala clients. Message-substring matching remains only\n   as a fallback.\n3. The three duplicated substring-matching sites (`SparkConnectClient.withGPRC`,\n   `DataFrame.withGPRC`, `ddlParse`) are unified into the converter.\n\n### Why are the changes needed?\n\nUsers could not access the server error message, error class, or SQLSTATE from a caught error, and\nthe classification relied on fragile substring matching duplicated in three places.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, `SparkConnectError` provides detailed error information now.\n\n```swift\n} catch SparkConnectError.tableOrViewNotFound(let details) {\n  print(details.errorClass)  // Optional(\"TABLE_OR_VIEW_NOT_FOUND\")\n  print(details.sqlState)    // Optional(\"42P01\")\n}\n```\n\nThe existing code using the original case spellings continues to compile and behave the same.\n\n### How was this patch tested?\n\nPass the CIs with the newly added `SparkConnectErrorTests` test suite. All existing test suites pass\nunmodified, which proves the source compatibility of this change.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #474 from dongjoon-hyun/SPARK-58539.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "40db9c0a19f6c833ace32f0aa75d62c621708eb7",
      "tree": "706b9ee734eb71dd90b24d9d270709e70b44d96e",
      "parents": [
        "dbb0e2f2144f2a87abddb87ef8f440d13ca446d0"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Aug 03 15:34:57 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Aug 03 15:34:57 2026 -0700"
      },
      "message": "[SPARK-58530] Support large local data in `createDataFrame` via `CachedLocalRelation`\n\n### What changes were proposed in this pull request?\n\nThis PR supports large local data in `SparkSession.createDataFrame` via `CachedLocalRelation`.\nWhen the serialized Arrow IPC stream is equal to or larger than\n`spark.sql.session.localRelationCacheThreshold` (1MiB by default), the serialized\n`LocalRelation` is uploaded to the server as a `cache/\u003csha256\u003e` artifact and the plan\nreferences it with a `CachedLocalRelation` message, instead of inlining the data into the\nplan. This follows the PySpark client behavior.\n\n`CachedLocalRelation` (supported since Spark 3.5) is used instead of the newer\n`ChunkedCachedLocalRelation` (added in Spark 4.1.0 by SPARK-53917) in order to support all\nSpark Connect servers this library targets, including 4.0.x.\n\n### Why are the changes needed?\n\nPreviously, `createDataFrame` inlined the data into the plan, so it was limited by the gRPC\nmessage size limit and threw `LocalRelationTooLarge` for larger data. With this PR, larger\ndata is transparently uploaded via the artifact API (SPARK-58528) and re-uploading identical\ndata is skipped based on its SHA-256 hash.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. `createDataFrame` now succeeds for data whose serialized size exceeds the gRPC message\nsize limit. The API signature is unchanged.\n\n### How was this patch tested?\n\nPass the CIs with a new test case (`CreateDataFrameTests.largeData`) which creates a\nDataFrame with 10,000 rows (~2MiB serialized, above the default 1MiB threshold), verifies\n`count`/`sum`/`filter` round trips against a live Spark Connect server, and verifies that a\nsecond `createDataFrame` call with the same data skips the upload.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #473 from dongjoon-hyun/SPARK-58530.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "dbb0e2f2144f2a87abddb87ef8f440d13ca446d0",
      "tree": "9288d01c071f80551d031a2889c7dc3f29dda465",
      "parents": [
        "c579369b76cdcd870c5083e0519993959335793d"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Aug 03 15:19:04 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Aug 03 15:19:04 2026 -0700"
      },
      "message": "[SPARK-58528] Support cache artifact upload in `SparkConnectClient`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support cache artifact upload in `SparkConnectClient` by adding a new file, `SparkConnectClient+Artifact.swift`, with the following internal APIs.\n\n- `artifactExists`: checks whether an artifact exists in the server-side session via the `ArtifactStatus` RPC.\n- `cacheArtifact`: caches the given data as a `cache/\u003csha256\u003e` artifact and returns its SHA-256 hash. The upload is skipped if the artifact already exists. Data up to 32KiB is uploaded as a single-chunk batch, and larger data as a chunked artifact stream, like PySpark\u0027s `ArtifactManager.cache_artifact`.\n\nThe existing `addArtifact` is moved to the new file to group all artifact operations together.\n\n### Why are the changes needed?\n\nThis is an infrastructure for supporting large `createDataFrame` data via `CachedLocalRelation`.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, these are internal APIs.\n\n### How was this patch tested?\n\nPass the CIs with a new test suite, `SparkConnectClientArtifactTests`.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #472 from dongjoon-hyun/SPARK-58528.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "c579369b76cdcd870c5083e0519993959335793d",
      "tree": "ddba2a1870ee8a6e499f62382814308411494fe3",
      "parents": [
        "40b282229bc339d492ff779a85842eba235152cc"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Aug 03 14:00:31 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Aug 03 14:00:31 2026 -0700"
      },
      "message": "[SPARK-58526] Fix `VariableBufferBuilder` to handle large first value and offsets truncation\n\n### What changes were proposed in this pull request?\n\nThis PR aims to fix two bugs in the vendored Arrow `VariableBufferBuilder` (used for `STRING`/`BINARY` serialization in `SparkSession.createDataFrame`).\n\n1. **Heap buffer overflow in `append()`**: the values-buffer capacity check was inside the `if index \u003e 0` branch, so the very first value was written without any capacity check. When the first string/binary value is longer than the initial 64-byte buffer, the write overflows the heap allocation (intermittent segfaults or data corruption). The fix moves the capacity check out of the `if index \u003e 0` branch so it applies to the first value as well.\n\n2. **Truncated offsets buffer in `finish()`**: the Arrow format requires `length + 1` offset entries because value `i` is read from the range `offsets[i]..\u003coffsets[i + 1]`, but `finish()` allocated only `length` entries, dropping the final offset. The fix allocates `(length + 1) * 4` bytes for the offsets buffer while keeping its `.length` as `length`, because `ArrowData` uses the offsets buffer\u0027s `.length` as the array length.\n\nThe same bugs exist in upstream `apache/arrow-swift`; they will be reported there separately.\n\n### Why are the changes needed?\n\nBoth bugs corrupt Arrow IPC data produced by `createDataFrame`:\n\n- Bug 1 causes intermittent crashes or corrupted values when the first string exceeds 64 bytes.\n- Bug 2 truncates the last offsets entry whenever `4 * rowCount` is a multiple of 64 (e.g., 16 rows), making the server fail with `Cannot grow BufferHolder by size \u003cnegative\u003e`. In other cases, the 64-byte alignment padding happens to preserve the final offset, which is why the existing small-scale tests passed.\n\nReproducer (fails or crashes before this fix, passes after):\n\n```swift\nlet spark \u003d try await SparkSession.builder.getOrCreate()\nlet value \u003d String(repeating: \"x\", count: 200)\nlet data: [[Sendable?]] \u003d (0..\u003c16).map { (i: Int) in [i, value + String(i)] }\nlet rows \u003d try await spark.createDataFrame(data, \"id INT, value STRING\").collect()\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is a bug fix. `createDataFrame` now works correctly with string/binary data that previously crashed or was rejected by the server.\n\n### How was this patch tested?\n\nPass the CIs with a newly added test case.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #471 from dongjoon-hyun/SPARK-58526.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "40b282229bc339d492ff779a85842eba235152cc",
      "tree": "157ef068cfda0bb8ca1e2bc79356bb1a174d48b7",
      "parents": [
        "f62880e0c2b9d0a6a1ea30a550235a321e6d0384"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Aug 03 12:57:45 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Mon Aug 03 12:57:45 2026 -0700"
      },
      "message": "[SPARK-58524] Add `SHA256` struct\n\n### What changes were proposed in this pull request?\n\nThis PR aims to add a pure Swift `SHA256` utility (FIPS 180-4) to `SparkConnect` module.\n\n- `SHA256.digest(data:)` returns the 32-byte SHA-256 digest as `[UInt8]`.\n- `SHA256.hexString(data:)` returns the digest as a 64-character lowercase hex `String`.\n\nNote that this is based on the following `Secure Hash Standard` documentation.\n\n- https://doi.org/10.6028/NIST.FIPS.180-4\n\n### Why are the changes needed?\n\nThis is a prerequisite for uploading cache artifacts whose names are derived from the SHA-256 hash of the serialized data (`cache/\u003csha256-hex\u003e`), which will be used by `CachedLocalRelation`-based large `createDataFrame` support.\n\nThe implementation is dependency-free pure Swift in the same style as the existing `CRC32.swift`:\n- `CryptoKit` is Apple-platform-only and unavailable on Linux.\n- `swift-crypto` exists only as a transitive dependency and this PR avoids adding a new direct dependency.\n\nThe hex conversion uses a lookup table instead of `String(format:)` for `FoundationEssentials` compatibility on Linux.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This adds a new utility only.\n\n### How was this patch tested?\n\nPass the CIs with the newly added `SHA256Tests` suite.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #470 from dongjoon-hyun/SPARK-58524.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "f62880e0c2b9d0a6a1ea30a550235a321e6d0384",
      "tree": "fb380632d358939bc33eec7ff3c36b5161cb3d8b",
      "parents": [
        "ab62a1766e540d283cc4d634c44a071e752025a5"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Aug 01 20:06:16 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Aug 01 20:06:16 2026 -0700"
      },
      "message": "[SPARK-58498] Support `geometry/geography` types in `DataType.simpleString`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `GEOMETRY` and `GEOGRAPHY` types (added in Apache Spark 4.1) in\n`DataType.simpleString` so that `DataFrame.dtypes` works on geospatial columns.\n\nFollowing Apache Spark\u0027s `GeometryType`/`GeographyType.typeName` convention, the mapping uses\n`geometry(\u003csrid\u003e)`/`geography(\u003csrid\u003e)` for fixed-SRID types and `geometry(any)`/`geography(any)`\nfor the mixed-SRID case (`srid \u003d\u003d -1`).\n\n### Why are the changes needed?\n\n`Spark_Connect_DataType`\u0027s `geometry` and `geography` kinds fell through to the `default:` branch\nof `DataType.simpleString` and threw `SparkConnectError.InvalidType`. As a result,\n`DataFrame.dtypes` failed entirely on any DataFrame containing a geospatial column.\n\n```swift\nlet df \u003d try await spark.sql(\"SELECT st_geomfromwkb(X\u00270101000000000000000000F03F0000000000000040\u0027)\")\ntry await df.dtypes  // Before: throws SparkConnectError.InvalidType\n                     // After: [(\"st_geomfromwkb(...)\", \"geometry(0)\")]\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, `DataFrame.dtypes` no longer throws on geospatial columns and returns Spark-style type strings instead. This is a bug fix from the unreleased perspective.\n\n### How was this patch tested?\n\nPass the CIs with a newly added test case.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #469 from dongjoon-hyun/SPARK-58498.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "ab62a1766e540d283cc4d634c44a071e752025a5",
      "tree": "26861d7859ef510093a72f9ba8202dc0dede6fea",
      "parents": [
        "98035c23cc34ec7fac3b6f21e85238896dc968c3"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Aug 01 15:57:51 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Aug 01 15:57:51 2026 -0700"
      },
      "message": "[SPARK-58497] Support `createDataFrame` in `SparkSession`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support `SparkSession.createDataFrame(_:_:)` to create a `DataFrame` from local Swift data with a DDL-formatted schema.\n\n```swift\nlet df \u003d try await spark.createDataFrame(\n  [[1, \"Alice\"], [2, \"Bob\"], [3, nil]], \"id INT, name STRING\")\n```\n\nThe data is serialized into an `Apache Arrow` IPC stream via the in-tree `ArrowWriter` and embedded into the plan as a `LocalRelation`. Supported types are `BOOLEAN`, `TINYINT`, `SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `STRING`, `BINARY`, `DATE`, and `TIMESTAMP`. `nil` is mapped to `NULL`. Data larger than 128MiB throws the new `SparkConnectError.LocalRelationTooLarge`.\n\n### Why are the changes needed?\n\nPreviously, there was no way to create a `DataFrame` from local Swift data. This is one of the fundamental `SparkSession` APIs which all other Spark Connect clients provide.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is a new API addition.\n\n### How was this patch tested?\n\nPass the CIs with the newly added `CreateDataFrameTests` test suite.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #468 from dongjoon-hyun/SPARK-58497.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    },
    {
      "commit": "98035c23cc34ec7fac3b6f21e85238896dc968c3",
      "tree": "7ec5da3ed7f8f44a1fa93b1a0c1529d81db311a4",
      "parents": [
        "14d1ddb66d8116bfab4a0a19223984d134d8ddb6"
      ],
      "author": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Aug 01 15:56:52 2026 -0700"
      },
      "committer": {
        "name": "Dongjoon Hyun",
        "email": "dongjoon@apache.org",
        "time": "Sat Aug 01 15:56:52 2026 -0700"
      },
      "message": "[SPARK-58496] Support field-name-based access in `Row`\n\n### What changes were proposed in this pull request?\n\nThis PR aims to support field-name-based access in `Row` like Scala\u0027s `GenericRowWithSchema`.\n\n- A new `RowSchema` class holds the field names and a name-to-index dictionary, shared by reference across all `Row`s of the same batch.\n- `Row` gains `fieldIndex(_:)`, `get(_ name:)`, `subscript(name:)`, and `asDict()`. For duplicate field names, the last one wins like Scala.\n- `DataFrame.collect()` attaches a `RowSchema` created from the Arrow schema to every `Row`.\n\n### Why are the changes needed?\n\nPreviously, `Row` only supported position-based access. This provides the equivalent of PySpark\u0027s `row[\"name\"]`/`asDict()` and Scala\u0027s `fieldIndex(name)`.\n\n```swift\nlet rows \u003d try await spark.sql(\"SELECT * FROM VALUES (1, \u0027abc\u0027) T(id, name)\").collect()\nlet id \u003d try rows[0].get(\"id\") as! Int32\nlet name \u003d try rows[0][\"name\"] as! String\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, this adds new public APIs. All existing APIs and behaviors are unchanged.\n\n### How was this patch tested?\n\nPass the CIs with newly added test cases.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Fable 5\n\nCloses #467 from dongjoon-hyun/SPARK-58496.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n"
    }
  ],
  "next": "14d1ddb66d8116bfab4a0a19223984d134d8ddb6"
}