)]}'
{
  "commit": "d0d77493f17d058f684ff04a1e15ee79d43bee8a",
  "tree": "de7816b219a293af70bdf69a9d1e2e68d15c3ff7",
  "parents": [
    "ec77694d4cceff155f974bde0efef1d136664ddb"
  ],
  "author": {
    "name": "YangJie",
    "email": "yangjie01@baidu.com",
    "time": "Tue Jul 28 13:52:02 2026 +0800"
  },
  "committer": {
    "name": "GitHub",
    "email": "noreply@github.com",
    "time": "Tue Jul 28 13:52:02 2026 +0800"
  },
  "message": "[#2771] feat(spark): Support Spark 4.2 (#2772)\n\n### What changes were proposed in this pull request?\n\nAdd a mutually-exclusive `spark4.2` Maven profile that builds the existing `client-spark/spark4` module against **Apache Spark 4.2.0**, mirroring the `spark4.1` profile added in #2751.\n\n- **Root `pom.xml`**: new `spark4.2` profile — a structural clone of `spark4.1`, differing only in `spark.version\u003d4.2.0`, `scala.version\u003d2.13.18`, `netty.version\u003d4.2.13.Final`, `jackson.version\u003d2.21.2`, and the lz4 coordinate (`lz4.groupId` + `lz4.version`, see below).\n- **`client-spark/spark4/pom.xml`**: `spark4.2` profile reuses the `src/main/java-spark4_1` compat source root (the 4.1 `Spark4Compat` shim).\n- **`client-spark/extension/pom.xml`**: `spark4.2` profile reuses the `scala-jakarta` servlet source root.\n- **`client-spark/extension` — `ShufflePage.scala`**: the Uniffle UI tab\u0027s collapsible sections use `onClick\u003d\"collapseTable(...)\"`, relying on a global `collapseTable()` that Spark 3.x/4.0/4.1 ship in `webui.js`. Spark 4.2 removed that function (its own pages moved to Bootstrap 5 `data-bs-toggle` collapse), so under `-Pspark4.2` the sections could not be expanded. Inline a fallback that defines a compatible `collapseTable` only when the host Spark didn\u0027t provide one — 3.x/4.0/4.1 keep Spark\u0027s implementation, 4.2 gets a working equivalent, reusing the `.collapsed` / `arrow-open|closed` CSS every Spark line still ships.\n- **lz4**: introduce `${lz4.groupId}` (`org.lz4` by default, `at.yawk.lz4` under `spark4.2`) and `${lz4.version}` (`1.8.1` by default, `1.11.0` under `spark4.2`); the two integration-test modules declare `${lz4.groupId}:lz4-java`, and the `at.yawk.lz4` root-`dependencyManagement` entry is pinned to `${lz4.version}`. The default build and every non-`spark4.2` profile stay on `org.lz4:1.8.1` — the bump is scoped to `spark4.2` only.\n- **CI**: `parallel.yml` gains a `spark4.2 / java 17` matrix entry; `sequential.yml` gains a `-Pspark4.2` step gated on `java-version \u003d\u003d 17`.\n\nNo new Maven module. `client-spark/spark4-shaded` needs no relocation change (its bouncycastle/jctools relocations are already global); a comment was added there noting its lz4 exclusion must stay `org.lz4` — Maven matches exclusions on the as-declared coordinate, so parameterizing it would leak `at.yawk.lz4` into the shaded jar under `-Pspark4.2`.\n\n### Why are the changes needed?\n\nFix: #2771\n\nUsers should be able to build and run Uniffle\u0027s Spark client against Spark 4.2.0 with full parity to the existing 4.1 support.\n\nThe three cross-major APIs called from `client-spark/spark4` are **byte-identical** between 4.1.1 and 4.2.0 (verified with `javap`), so no shim change is required:\n\n| API | 4.1.1 | 4.2.0 |\n|---|---|---|\n| `MapStatus.apply(BlockManagerId, long[], long, long)` | 4-arg | identical |\n| `ExternalSorter(…, RowBasedChecksum[])` ctor | 6-arg | identical |\n| `WebUIPage.render(jakarta…HttpServletRequest)` | jakarta | identical |\n\nDependency notes:\n- **jackson → 2.21.2 (required):** Spark 4.2.0 ships `jackson-module-scala_2.13` 2.21.2, which validates the classpath `jackson-databind` on registration and throws in `RDDOperationScope`\u0027s static initializer if it is outside `[2.21.0, 2.22.0)`. Same class of failure the 4.1 work hit.\n- **netty → 4.2.13.Final:** matches what Spark 4.2.0 was compiled/tested against (still Netty 4.2, same major as 4.1\u0027s 4.2.7, so #2751\u0027s `NioIoHandler`/refCnt handling still applies — no code change).\n- **lz4 → at.yawk.lz4:1.11.0, scoped to spark4.2 (required):** Spark 4.2.0 bumped lz4 to 1.11.0 and calls `LZ4BlockInputStream.newBuilder()` (added in the 1.11 line); without it the integration tests fail at runtime with `NoSuchMethodError`. Because `org.lz4:1.8.1` is a relocation stub that Maven rewrites to `at.yawk.lz4:1.8.1`, a globally-managed `at.yawk.lz4:1.11.0` would silently bump lz4 across the whole reactor (server/client/coordinator/…), which is out of this PR\u0027s scope and would override the `org.lz4:1.8.1` pin from #2693 (CVE-2025-12183). The `${lz4.groupId}`/`${lz4.version}` properties confine the bump to the `spark4.2` profile; every other build stays on 1.8.1.\n- hadoop / jetty / log4j / slf4j / commons-lang3 deltas are deferred; the profile keeps the 4.1 values.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo behavioral change to existing profiles. Adds a new opt-in `-Pspark4.2` build profile, and fixes the Uniffle Spark UI tab so its collapsible sections expand under Spark 4.2. Existing `-Pspark4` (4.0.2) and `-Pspark4.1` (4.1.1) are unchanged.\n\n### How was this patch tested?\n\nVerified locally on JDK 17 (Zulu 17.0.18):\n\n1. `mvn clean install -Pspark4.2 -DskipTests` — compiles + shades against 4.2.0 (spark-core/sql/catalyst 4.2.0, scala 2.13.18, netty 4.2.13.Final, jackson 2.21.2, lz4 `at.yawk.lz4:1.11.0` resolved).\n2. `mvn clean install -Pspark4.1 -DskipTests` + integration tests — 4.1.1 build and `AQERepartitionTest` / `MapSideCombineTest` still green (lz4 stays `org.lz4:1.8.1`); confirms no regression. Also confirmed the default build (no profile) and other spark profiles keep lz4 at 1.8.1.\n3. `AQERepartitionTest`, `MapSideCombineTest` pass under `-Pspark4.2`.\n4. `TransportFrameDecoderTest` (2 tests) passes on Netty 4.2.13 under `-Pspark4.2`.\n5. `RepartitionWithLocalFileRssTest` (NOOP/ZSTD/**LZ4** codecs) passes under `-Pspark4.2` — exercises the lz4 runtime path.\n6. `dev/scripts/checkshade.sh` on the spark4.2 shaded jar — \"check success\", 0 unshaded classes; bouncycastle/jctools relocated, no `net.jpountz` (lz4) leak.\n7. **End-to-end on a real Spark 4.2.0 distribution** against a local Uniffle cluster (coordinator + shuffle server): a shuffle job runs through `RssShuffleManager`, and the Uniffle Spark UI tab renders and its collapsible sections (Build Info / Shuffle Server / Assignment / …) expand with no page JS errors.\n\nCI runs the build + unit + integration tests via the new `spark4.2` matrix entry / `-Pspark4.2` step.",
  "tree_diff": [
    {
      "type": "modify",
      "old_id": "6c0034abcf2ad93a97b60a99311c2bcd531cbc5c",
      "old_mode": 33188,
      "old_path": ".github/workflows/parallel.yml",
      "new_id": "41dd56ee4ba353094b56337723fa07ca1cfb5520",
      "new_mode": 33188,
      "new_path": ".github/workflows/parallel.yml"
    },
    {
      "type": "modify",
      "old_id": "9b579e9d5c29c3b46883ddeeb9f62a93778d62e7",
      "old_mode": 33188,
      "old_path": ".github/workflows/sequential.yml",
      "new_id": "a7a3f1d0e337b9ba61334be092d2731aaea5fa61",
      "new_mode": 33188,
      "new_path": ".github/workflows/sequential.yml"
    },
    {
      "type": "modify",
      "old_id": "0ade60c6ffb9fed3c8ad25fd5cb5f1c7c250e953",
      "old_mode": 33188,
      "old_path": "client-spark/extension/pom.xml",
      "new_id": "c1609868ed65dd5eb3ce94d53d68827929d56d87",
      "new_mode": 33188,
      "new_path": "client-spark/extension/pom.xml"
    },
    {
      "type": "modify",
      "old_id": "dcc700cbaea86546702bf10874aae9ab22f93d45",
      "old_mode": 33188,
      "old_path": "client-spark/extension/src/main/scala/org/apache/spark/ui/ShufflePage.scala",
      "new_id": "cd0c15118e26a75a9e181acc04ced732c99f5af1",
      "new_mode": 33188,
      "new_path": "client-spark/extension/src/main/scala/org/apache/spark/ui/ShufflePage.scala"
    },
    {
      "type": "modify",
      "old_id": "48d4441465d24398f24899a86db8af8c5a23f01b",
      "old_mode": 33188,
      "old_path": "client-spark/spark4-shaded/pom.xml",
      "new_id": "49c6fd9404cc2416e97420086dabada6b08e05c2",
      "new_mode": 33188,
      "new_path": "client-spark/spark4-shaded/pom.xml"
    },
    {
      "type": "modify",
      "old_id": "435460139b82b46f0b9bf6f8f17cdfcbfd3f0476",
      "old_mode": 33188,
      "old_path": "client-spark/spark4/pom.xml",
      "new_id": "5d73ed86b661c647ef001568e326a1a367ced1e7",
      "new_mode": 33188,
      "new_path": "client-spark/spark4/pom.xml"
    },
    {
      "type": "modify",
      "old_id": "5744b74144bec3db322e7b4b8e199cf3a07d8b8d",
      "old_mode": 33188,
      "old_path": "integration-test/spark-common/pom.xml",
      "new_id": "ff8eae9355ac45ebb31c4381ab09605525095b0e",
      "new_mode": 33188,
      "new_path": "integration-test/spark-common/pom.xml"
    },
    {
      "type": "modify",
      "old_id": "99fd341775aa738f7559efa882dcf131c7566300",
      "old_mode": 33188,
      "old_path": "integration-test/spark4/pom.xml",
      "new_id": "20e89993838c80ecb190f0db041c28abef0f1abc",
      "new_mode": 33188,
      "new_path": "integration-test/spark4/pom.xml"
    },
    {
      "type": "modify",
      "old_id": "449e20689f83a6594aaccc0dc3f2ded912b82ba9",
      "old_mode": 33188,
      "old_path": "pom.xml",
      "new_id": "b725d62611c9fa32d9e47dcb049fea99a34484d2",
      "new_mode": 33188,
      "new_path": "pom.xml"
    }
  ]
}