)]}'
{
  "log": [
    {
      "commit": "2b5dc8d5f14ed575be8d4a3f4d72f718fd945701",
      "tree": "089c81207fddaec40eb51424bd617a6db906bccb",
      "parents": [
        "f0c8139621f3da73ccf30566fb344ffb2ff47cf9"
      ],
      "author": {
        "name": "sychen",
        "email": "sychen@ctrip.com",
        "time": "Tue Jul 28 20:27:45 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Tue Jul 28 20:27:45 2026 +0800"
      },
      "message": "[CELEBORN-2308] Bump commons-lang3 version from 3.17.0 to 3.20.0\n\n### What changes were proposed in this pull request?\n\nhttps://commons.apache.org/proper/commons-lang/changes.html#a3.20.0\n\nhttps://issues.apache.org/jira/browse/LANG-1786\n\n### Why are the changes needed?\nJDK25\n```\nWARNING: Use of the three-letter time zone ID \"IET\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"JST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"IST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"BET\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"ACT\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"HST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"AET\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"AGT\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"VST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"CNT\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"EST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"NET\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"PLT\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"CST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"SST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"CTT\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"PNT\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"BST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"MIT\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"ART\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"AST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"PRT\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"PST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"NST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"EAT\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"MST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"ECT\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"CAT\" is deprecated and it will be removed in a future release\n```\n\n```java\n\tat java.base/java.util.TimeZone.getTimeZone(TimeZone.java:600)\n\tat java.base/java.util.TimeZone.getTimeZone(TimeZone.java:549)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateParser$TimeZoneStrategy.\u003cinit\u003e(FastDateParser.java:920)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateParser.getLocaleSpecificStrategy(FastDateParser.java:661)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateParser.getStrategy(FastDateParser.java:630)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateParser.access$200(FastDateParser.java:75)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateParser$StrategyParser.letterPattern(FastDateParser.java:234)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateParser$StrategyParser.getNextStrategy(FastDateParser.java:220)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateParser.init(FastDateParser.java:163)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateParser.\u003cinit\u003e(FastDateParser.java:149)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateFormat.\u003cinit\u003e(FastDateFormat.java:399)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateFormat.\u003cinit\u003e(FastDateFormat.java:383)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:105)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:102)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FormatCache.getInstance(FormatCache.java:79)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateFormat.getInstance(FastDateFormat.java:162)\n\tat org.apache.celeborn.common.util.Utils$.\u003cinit\u003e(Utils.scala:1238)\n\tat org.apache.celeborn.common.util.Utils$.\u003cclinit\u003e(Utils.scala)\n```\n\nhttps://bugs.openjdk.org/browse/JDK-8342550\n\n### Does this PR resolve a correctness bug?\nNo\n\n### Does this PR introduce _any_ user-facing change?\nNo\n\n### How was this patch tested?\nGHA\n\nCloses #3663 from cxzl25/CELEBORN-2308.\n\nAuthored-by: sychen \u003csychen@ctrip.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "f0c8139621f3da73ccf30566fb344ffb2ff47cf9",
      "tree": "93e6454121150c93d542589c91a313fddf9c767b",
      "parents": [
        "76ce74504ba66e022e4b7950db6af75e5e0089e2"
      ],
      "author": {
        "name": "Chao Sun",
        "email": "chao@openai.com",
        "time": "Fri Jul 24 10:14:00 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Fri Jul 24 10:14:00 2026 +0800"
      },
      "message": "[CELEBORN-2371] Bound Spark batch-open client creation retries and stop them on interruption\n\n## Why are the changes needed?\n\n[CELEBORN-2371](https://issues.apache.org/jira/browse/CELEBORN-2371) follows up on the parallel Spark batch-open client creation added by [#3692](https://github.com/apache/celeborn/pull/3692).\n\nBatch-open locations are grouped by `host:fetchPort`, but each worker task previously walked multiple `PartitionLocation` entries. Since every `createClient` invocation already receives `TransportClientFactory`\u0027s full retry budget, that outer loop could multiply connection latency without targeting a different endpoint.\n\nCancellation also needs to terminate consistently. A direct or wrapped `InterruptedException` must stop reader setup and fetch retry flow before it is recorded as a worker failure, added to shared exclusion state, retried against a peer, or reported as a shuffle fetch failure. The unlimited-timeout transport branch also needed explicit cleanup on cancelled or failed connection futures.\n\n## What changes were proposed in this PR?\n\n### Keep one retry budget per worker endpoint\n\nEach grouped `host:fetchPort` now uses one representative location for client creation. `TransportClientFactory` remains the owner of connection retries, controlled by the existing `celeborn.data.io.maxRetries`; there is no second outer retry setting or multiplicative retry budget.\n\n### Propagate cancellation through reader paths\n\n`CelebornShuffleReader` now uses a shared interrupt-aware client-creation helper in both sequential and parallel batch-open paths. It checks a pre-existing interrupt flag, restores the flag when `createClient` throws `InterruptedException`, and exits without invoking the ordinary failure callback.\n\n`CelebornInputStream` now detects interruption throughout reader creation, failed-stream cleanup, reconnect, and buffer-fill paths. It exits before exclusion, retry, peer failover, or shuffle-fetch-failure reporting. If reader cleanup itself fails while propagating cancellation, the cleanup failure is retained as a suppressed exception without masking the original interruption.\n\n### Preserve interruption and cleanup in transport bootstrap\n\n`TransportClientFactory` uses Guava\u0027s `Throwables.getCausalChain()` to detect wrapped `InterruptedException`, restores the interrupt flag, and stops retrying immediately. TCP-connect and TLS-handshake waits close the in-progress channel before propagating interruption. The unlimited-timeout connection path now also closes cancelled and failed channel futures explicitly.\n\n## How was this PR tested?\n\nFormatting was applied with:\n\n```text\n./build/mvn --no-transfer-progress -DskipTests spotless:apply\n```\n\nThe focused transport tests passed (11 tests):\n\n```text\n./build/mvn --no-transfer-progress -pl common -am \\\n  -Dtest\u003dTransportClientFactorySuiteJ,TransportClientFactoryInterruptSuiteJ \\\n  -DwildcardSuites\u003dnone clean test\n```\n\nThe focused input-stream peer-failover and interruption tests passed (6 tests):\n\n```text\n./build/mvn --no-transfer-progress -pl client -am \\\n  -Dtest\u003dCelebornInputStreamPeerFailoverTest \\\n  -DwildcardSuites\u003dnone clean test\n```\n\nThe Spark 3.5 reader suite passed (7 tests):\n\n```text\n./build/mvn --no-transfer-progress -Pspark-3.5 -pl client-spark/spark-3 -am \\\n  -Dtest\u003dnone \\\n  -DwildcardSuites\u003dorg.apache.spark.shuffle.celeborn.CelebornShuffleReaderSuite \\\n  clean test\n```\n\nSpark 4.0 / Scala 2.13 production and test compilation also passed:\n\n```text\n./build/mvn --no-transfer-progress -Pspark-4.0 -pl client-spark/spark-3 -am \\\n  -DskipTests clean test\n```\n\nCloses #3746 from sunchao/dev/chao/codex/bound-batch-open-client-retries-oss.\n\nAuthored-by: Chao Sun \u003cchao@openai.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "76ce74504ba66e022e4b7950db6af75e5e0089e2",
      "tree": "7735e851225e90c3b78c07dc61759f7ec8e62f4b",
      "parents": [
        "59413d040f7e9a6c6145ce13ced3e1707058682f"
      ],
      "author": {
        "name": "senthh",
        "email": "senthil.kumar@acceldata.io",
        "time": "Fri Jul 24 10:06:42 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Fri Jul 24 10:06:42 2026 +0800"
      },
      "message": "[CELEBORN-2380] Support JmxSink for exposing metrics as JMX MBeans\n\n### What changes were proposed in this pull request?\n\nThis PR adds a new `JmxSink` metrics sink that exposes Celeborn metrics as JMX MBeans.\n\n- Add `JmxSink` (`common/src/main/scala/org/apache/celeborn/common/metrics/sink/JmxSink.scala`), backed by Dropwizard Metrics\u0027 `JmxReporter`. It follows the existing `Sink` contract and is loaded reflectively by `MetricsSystem` via the standard `(Properties, MetricRegistry)` constructor, so no changes to `MetricsSystem` are required.\n- Add the `io.dropwizard.metrics:metrics-jmx` dependency (which contains `JmxReporter` in Dropwizard Metrics 4.x) to both the Maven build (`pom.xml` dependency management + `common/pom.xml`) and the SBT build (`project/CelebornBuild.scala`), pinned to the existing `${codahale.metrics.version}` (4.2.25).\n- Add a commented-out example to `conf/metrics.properties.template` and `charts/celeborn/files/conf/metrics.properties` showing how to enable the sink.\n- Document `JmxSink` in `docs/monitoring.md`.\n- Register `metrics-jmx-4.2.25.jar` in all `dev/deps/dependencies-*` manifests (server + client profiles) and add `io.dropwizard.metrics:metrics-jmx` to `LICENSE-binary`, since the artifact is now bundled in the distributions.\n\nEnabling the sink is opt-in:\n\n```properties\n*.sink.jmx.class\u003dorg.apache.celeborn.common.metrics.sink.JmxSink\n\nTest Output:\n\n\u003cimg width\u003d\"1315\" height\u003d\"800\" alt\u003d\"Celeborn_jmx\" src\u003d\"https://github.com/user-attachments/assets/7af729f8-7eb9-4749-af9c-e2af64d48fb8\" /\u003e\n\nCloses #3758 from senthh/CELEBORN-2380.\n\nLead-authored-by: senthh \u003csenthil.kumar@acceldata.io\u003e\nCo-authored-by: senthh \u003csenthh@gmail.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "59413d040f7e9a6c6145ce13ced3e1707058682f",
      "tree": "675b7043fa198479b2e89dec91fadb2e87e3a8c1",
      "parents": [
        "17f7d4f9f46fb23fd668f3b57f6053d651123693"
      ],
      "author": {
        "name": "sychen",
        "email": "sychen@ctrip.com",
        "time": "Thu Jul 23 10:12:19 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Thu Jul 23 10:12:19 2026 +0800"
      },
      "message": "[CELEBORN-2385] Guard ByteBufAllocator cast in ChannelsLimiter trim path\n\n### What changes were proposed in this pull request?\n\n`ChannelsLimiter#userEventTriggered` unconditionally casts `ctx.alloc()` to\n`PooledByteBufAllocator` when handling a `TrimCache` event:\n\n```java\n((PooledByteBufAllocator) ctx.alloc()).trimCurrentThreadCache();\n```\n\nThe concrete type of ctx.alloc() is determined by\nceleborn.network.memory.allocator.pooled. When it is set to false (the\ndocumented escape hatch for workers whose direct memory stays high even after\ntrimming), the allocator is an UnpooledByteBufAllocator, so this cast throws a\nClassCastException on the Netty EventLoop thread the next time memory pressure\ntriggers MemoryManager#trimAllListeners → ChannelsLimiter#onTrim →\ntrimCache().\n\n### Why are the changes needed?\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nCloses #3763 from cxzl25/CELEBORN-2385.\n\nAuthored-by: sychen \u003csychen@ctrip.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "17f7d4f9f46fb23fd668f3b57f6053d651123693",
      "tree": "32db933aaaeba7fc34133a298b9a8cac46b28aca",
      "parents": [
        "d77543ef4bbd3a62aa8f247b6434e7c438d3850a"
      ],
      "author": {
        "name": "sychen",
        "email": "sychen@ctrip.com",
        "time": "Thu Jul 16 13:58:25 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Thu Jul 16 13:58:25 2026 +0800"
      },
      "message": "[CELEBORN-2379] Keep sbt and maven dependency lists in sync and cross-check both in CI\n\n### What changes were proposed in this pull request?\n\nKeep the sbt and maven client dependency lists in sync and cross-check both build systems in CI, and fix the `paranamer` dependency convergence for the Spark 4.x clients.\n\n- `dev/dependencies.sh`: drop the process-wide `export LC_ALL\u003dC` and apply `LC_ALL\u003dC sort` locally to the two sort calls, so the sort order stays machine-independent without forcing the locale globally.\n- `.github/workflows/deps.yml`: merge the separate `sbt` and `maven-jdk{8,11,17}` jobs into `jdk{8,11,17}` jobs that fan out over a `tool: [maven, sbt]` matrix (with the `sbt` + `tez` combination excluded, since there is no sbt tez project), so every module is cross-checked by both maven and sbt on its matching JDK.\n- `project/CelebornBuild.scala`: gate the `jdkTools` (`maven-jdk-tools-wrapper`) dependency to JDK 8 only, matching the `JDKTools` plugin and the maven `jdk-8` profile; the wrapper is a no-op on JDK 9+ where the tools API ships with the JDK, and including it unconditionally desynced the sbt/maven classpaths.\n- Fix the `paranamer` version conflict — `spark-sql` declares `2.8.3` while the transitive `jackson-module-scala 2.15.3` pulls `2.8` — for the Spark 4.x clients, keeping maven and sbt in sync:\n  - spark-4.0: `spark-sql 4.0.3` itself declares `2.8`, so the manifest is deduped to `paranamer 2.8` with no override.\n  - spark-4.1: pin `paranamer` to `2.8.3` via the `spark-4.1` maven profile `dependencyManagement` and the `Spark41` sbt `dependencyOverrides`.\n  - spark-4.2: same conflict (`spark-sql 4.2.0` → `2.8.3`); pin to `2.8.3` via the `spark-4.2` maven profile `dependencyManagement` and the `Spark42` sbt `dependencyOverrides`, and drop the stray `paranamer 2.8` line from the manifest.\n\n### Why are the changes needed?\n\nThe sbt and maven dependency manifests could silently drift apart; CI previously validated only one toolchain per module, so divergence went unnoticed. The `paranamer` conflict left two versions on the runtime classpath and two entries in the manifest, breaking dependency convergence. Pinning the conflict and cross-checking both builds in CI keeps the two dependency lists identical. The unconditional `jdkTools` dependency likewise left a stale `tools.jar` entry on JDK 9+ sbt classpaths that had no maven counterpart.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\nGHA `deps` workflow (maven + sbt on JDK 8/11/17); `./dev/dependencies.sh --module spark-4.2 --check` passes locally.\n\nCloses #3757 from cxzl25/CELEBORN-2379.\n\nAuthored-by: sychen \u003csychen@ctrip.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "d77543ef4bbd3a62aa8f247b6434e7c438d3850a",
      "tree": "9845953e44e39988c28f9d20b55c808832e7e710",
      "parents": [
        "3aa25932fad50a2193a5f1958f49a82a8da22598"
      ],
      "author": {
        "name": "Kalvin2077",
        "email": "wk.huang2077@outlook.com",
        "time": "Wed Jul 15 17:56:32 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Wed Jul 15 17:56:32 2026 +0800"
      },
      "message": "[CELEBORN-2382] Authorize `PbReviseLostShuffles` to prevent cross-application metadata modification\n\n### What changes were proposed in this pull request?\n\nCall `checkAuth(context, pb.getAppId)` before serving `PbReviseLostShuffles` in the Master, consistent with the other application-scoped handlers.\n\n### Why are the changes needed?\n\nThe Master handled `PbReviseLostShuffles` without an authorization check. With authentication enabled, an application could specify another application\u0027s ID and add shuffle IDs to its `registeredAppAndShuffles` metadata.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\n- New case in `MasterApplicationMetaAuthSuite` verifying same-application requests succeed and cross-application requests are rejected with authentication enabled\n- New case in `MasterSuite` verifying same-application requests succeed, cross-application requests are rejected, and authentication-disabled behavior is preserved.\n\nCloses #3761 from Kalvin2077/fix/pbrevise-checkauth.\n\nAuthored-by: Kalvin2077 \u003cwk.huang2077@outlook.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "3aa25932fad50a2193a5f1958f49a82a8da22598",
      "tree": "2a3c55d1337ff868c920f8bb0433609736e32c0e",
      "parents": [
        "50c18eb6b66c1926d037441d954270b60f6f638f"
      ],
      "author": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Wed Jul 15 13:37:16 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Wed Jul 15 13:37:16 2026 +0800"
      },
      "message": "[CELEBORN-2383] Support Spark 4.2\n\n### What changes were proposed in this pull request?\n\nSupport Spark 4.2.\n\nAdd a Spark 4.2 build target for the Celeborn Spark client, pinning Spark 4.2.0 on Scala 2.13.18:\n\n- Add the `spark-4.2` Maven profile (`pom.xml`, `tests/spark-it/pom.xml`) and the `Spark42` sbt project (`project/CelebornBuild.scala`), and register Scala 2.13.18 in `ALL_SCALA_VERSIONS`.\n- Add the `dev/deps/dependencies-client-spark-4.2` dependency manifest.\n- Wire `spark-4.2` into `build/make-distribution.sh`, `dev/reformat`, and the deps/maven/sbt CI matrices, and bump the release deploy profile from `spark-4.1` to `spark-4.2`.\n- Document Spark 4.2 compatibility in `README.md` and `docs/developers/sbt.md`.\n\n### Why are the changes needed?\n\nSpark 4.2.0 has already released, which release notes refer to [Spark Release 4.2.0](https://spark.apache.org/releases/spark-release-4-2-0.html).\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nCI.\n\nCloses #3760 from SteNicholas/CELEBORN-2383.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "50c18eb6b66c1926d037441d954270b60f6f638f",
      "tree": "78552548688fabb25e20c70609728712b1da743c",
      "parents": [
        "da8c1cec328110452458ee4174f03570ba8dddea"
      ],
      "author": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Tue Jul 14 21:58:10 2026 +0800"
      },
      "committer": {
        "name": "Shuang",
        "email": "lvshuang.xjs@alibaba-inc.com",
        "time": "Tue Jul 14 21:58:10 2026 +0800"
      },
      "message": "[CELEBORN-2381] Authorize PbApplicationMetaRequest to prevent secret disclosure\n\n### What changes were proposed in this pull request?\n\nCall `checkAuth(context, appId)` before serving `PbApplicationMetaRequest` in the Master, consistent with the other application-scoped handlers (`RequestSlots`, `UnregisterShuffle`, `ApplicationLost`).\n\nWorkers fetch application meta over the internal channel, where the connection has no per-application client id, so the check is a no-op for them; it only rejects an external application that asks for another application\u0027s secret.\n\n### Why are the changes needed?\n\nThe Master served `PbApplicationMetaRequest` by returning the requested application\u0027s SASL secret without an authorization check. With authentication enabled, a caller on the external application port could read another application\u0027s secret and impersonate it.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\n- New `MasterApplicationMetaAuthSuite` verifying `PbApplicationMetaRequest` is authorized against the registered application.\n- New case in `MasterSuite` verifying a caller requesting another application\u0027s secret is rejected.\n\nCloses #3759 from SteNicholas/CELEBORN-2381.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Shuang \u003clvshuang.xjs@alibaba-inc.com\u003e\n"
    },
    {
      "commit": "da8c1cec328110452458ee4174f03570ba8dddea",
      "tree": "d586366364cbb7a355bc3c02c54a4b7cbdee214a",
      "parents": [
        "b2f1da858fa374b6755b7e50b9fd875b517962a5"
      ],
      "author": {
        "name": "afterincomparableyum",
        "email": "224495379+afterincomparableyum@users.noreply.github.com",
        "time": "Tue Jul 14 15:36:53 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Tue Jul 14 15:36:53 2026 +0800"
      },
      "message": "[CELEBORN-2362] Fix Flaky CI/CD\n\n### What changes were proposed in this pull request?\n\n  Address several independent root causes of CI/CD flakiness, spanning the\n  build tooling, JaCoCo instrumentation, the test mini-cluster, and the\n  process-wide shuffle client.\n\nCI infrastructure\n\n  - build/mvn — resilient Maven bootstrap. Default to archive.apache.org instead of the\n  closer.lua mirror redirector, and validate that each download is a real gzip tarball before\n  extracting (closer.lua intermittently returns an HTML mirror-chooser page with HTTP 200).\n  Retry up to 3 times, backing off only between attempts. A valid\n  pre-staged/previously-downloaded tarball is reused (keeps offline/air-gapped builds working),\n  and curl falls through to wget when curl is absent or produced no valid file (proxy/TLS/CA\n  failures wget would survive).\n  - pom.xml — exclude io/netty/** from JaCoCo. JaCoCo\u0027s agent rejects Netty 4.2\u0027s\n  already-enhanced JFR event classes, tearing down channels mid-write and flaking the Flink\n  integration tests.\n\n  Process-wide ShuffleClient made per-application\n\n  The static single-slot ShuffleClient singleton couldn\u0027t represent the multiple applications a\n  reused spark-it JVM runs with overlapping lifecycles: on an app switch it evicted the\n  previous app\u0027s client without teardown, and a lock-free get() fast path could hand a live app\n  the wrong app\u0027s client — surfacing as celebornShuffleId 0 corruption\n  (ArrayIndexOutOfBoundsException / CommitMetadata CRC mismatch).\n\n  - Replaced the single slot with a per-appUniqueId registry (ConcurrentHashMap\u003cString,\n  ShuffleClient\u003e). Each application gets its own isolated client; only fully-initialized\n  instances are ever published, closing the torn read.\n  - Added ShuffleClient.removeInstance(...), and SparkShuffleManager.stop() now removes+shuts\n  down only its own app\u0027s client instead of calling the global reset() (which would tear down\n  other live applications\u0027 clients in a multi-app JVM). This also eliminates the\n  orphaned-client leak (RpcEnv, Netty factory, retry pool, reviveManager) that previously\n  accumulated across app switches.\n\n  Spark-it lifecycle isolation\n\n  - SparkTestBase — stop any SparkSession/SparkContext still alive in the JVM in afterAll\n  (triggering SparkShuffleManager.stop()) and reset the client, so a straggler task from a\n  leaked context can\u0027t bind to a later suite\u0027s LifecycleManager through the shared client.\n  Reduced worker count from 5 to 3 (the MiniCluster default): the suites run serially in one\n  JVM, so each surplus worker multiplies the long-lived thread/CPU footprint and, under CI\n  contention, starves RPC/fetch handlers past the 240s network timeout.\n\n  Cluster-setup test robustness\n\n  - MiniClusterFeature — retry worker startup with exponential backoff, recreating the worker\n  each attempt. Record a worker in the returned set only once it has registered, so a torn-down\n  failed worker (distinct identity — Worker has no equals/hashCode) doesn\u0027t leak into the\n  returned set and get double-stopped. Tear failed workers down locally (stop +\n  rpcEnv.shutdown) instead of exitImmediately(), which issues a blocking WorkerLost master RPC\n  and pollutes the master\u0027s excluded list inside the retry loop.\n  - Random port selection — draw ports below the ephemeral floor (Utils.selectRandomPort() /\n  MAX_SELECTABLE_PORT) to avoid a TOCTOU with OS-assigned/TIME_WAIT ports that look free at\n  selection time but fail to bind. Consolidated the duplicated picker into a shared\n  RandomPortSupport test trait.\n  - RatisMasterStatusSystemSuiteJ — re-point each server to a fresh storage directory on every\n  start attempt; Ratis releases the directory lock asynchronously, so a retry would otherwise\n  hit \"directory is already locked\".\n  - JVMQuake / JVMQuakeSuite — extract checkAndDump(...) so the threshold/heap-dump logic can\n  be exercised deterministically without inducing real GC pressure.\n  - LifecycleManagerUnregisterShuffleSuite — shorten the expired-check interval to 5s so the\n  shuffle unregister runs within the eventually() window with retry margin.\n\n### Why are the changes needed?\n\nCI/CD is always failing. With this, CI/CD rarely fails.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nCI/CD\n\nCloses #3737 from afterincomparableyum/flaky-cicd.\n\nAuthored-by: afterincomparableyum \u003c224495379+afterincomparableyum@users.noreply.github.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "b2f1da858fa374b6755b7e50b9fd875b517962a5",
      "tree": "75d0b9fcfa805dc1c49bb3959e0cfa7385d67ded",
      "parents": [
        "1e715b2cf2936a3704c6a44da463628f7e737f35"
      ],
      "author": {
        "name": "yz",
        "email": "yz@U-YVQK6W4G-2059.local",
        "time": "Thu Jul 09 19:59:16 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Thu Jul 09 19:59:16 2026 +0800"
      },
      "message": "[CELEBORN-2377] Support Heartbeat message decoding in C++ client\n\n### What changes were proposed in this pull request?\nThis PR adds Heartbeat message decoding support to the C++ client network.\n\n### Why are the changes needed?\n\nThe HEARTBEAT type (id 22) is already declared in the Message::Type enum and recognized by decodeType, but Message::decodeFrom had no case for it, so any heartbeat frame received by the C++ client hit the unsupported Message decode type failure path. As the protocol/server side begins to send heartbeat messages, the C++ client must be able to decode them instead of throwing.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce any user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nAdded MessageTest.decodeHeartbeat, which constructs a heartbeat wire frame and asserts that Message::decodeFrom returns a non-null message with type HEARTBEAT. Built and ran the test with -- gtest_filter\u003d\"MessageTest.decodeHeartbeat\" inside the jraaaay/celeborn-cpp-dev:0.4 container; it passed.\n\nCloses #3755 from Kalvin2077/main.\n\nAuthored-by: yz \u003cyz@U-YVQK6W4G-2059.local\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "1e715b2cf2936a3704c6a44da463628f7e737f35",
      "tree": "a2f314d78596321978cf46eab535fe96b0dfc4ed",
      "parents": [
        "a9055aeadc7725e7e14b2de4a5befd043c647365"
      ],
      "author": {
        "name": "Aravind Patnam",
        "email": "akpatnam25@gmail.com",
        "time": "Wed Jul 08 13:13:12 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Wed Jul 08 13:13:12 2026 +0800"
      },
      "message": "[CELEBORN-2329][CIP22] Encryption at Rest Spark Impl\n\n### What changes were proposed in this pull request?\n\nAdds EAR support for Spark side.\nSee more details in [doc](https://docs.google.com/document/d/1xBrLtpb8bk8CdJENiM3aLJCbFThRKLor8uAjSsfXoxE/edit?usp\u003dsharing).\n\n### Why are the changes needed?\n\nSee more details in [doc](https://docs.google.com/document/d/1xBrLtpb8bk8CdJENiM3aLJCbFThRKLor8uAjSsfXoxE/edit?usp\u003dsharing).\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes.\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes.\n\n### How was this patch tested?\n\nUnit tests and tested in production internally.\n\nCloses #3689 from akpatnam25/ear-spark-impl.\n\nLead-authored-by: Aravind Patnam \u003cakpatnam25@gmail.com\u003e\nCo-authored-by: Aravind Patnam \u003capatnam@linkedin.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "a9055aeadc7725e7e14b2de4a5befd043c647365",
      "tree": "687fc9eb2ca03467ef937b535dbfd178a3dd2d0b",
      "parents": [
        "17159eb34ac8132e53e8f19b2c8d38d63d7853cb"
      ],
      "author": {
        "name": "sychen",
        "email": "sychen@ctrip.com",
        "time": "Thu Jul 02 16:27:08 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Thu Jul 02 16:27:08 2026 +0800"
      },
      "message": "[CELEBORN-2374] Bump maven version from 3.9.14 to 3.9.16\n\n### What changes were proposed in this pull request?\nhttps://maven.apache.org/docs/3.9.16/release-notes.html\n\n### Why are the changes needed?\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\nGHA\n\nCloses #3752 from cxzl25/CELEBORN-2374.\n\nAuthored-by: sychen \u003csychen@ctrip.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "17159eb34ac8132e53e8f19b2c8d38d63d7853cb",
      "tree": "dfbb44b130d5ae319c61f8b0d9bf1621e18f1607",
      "parents": [
        "63e28aedd7f0703ffeba438d49ea6f3809160892"
      ],
      "author": {
        "name": "Chao Sun",
        "email": "chao@openai.com",
        "time": "Tue Jun 30 11:23:05 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Tue Jun 30 11:23:05 2026 +0800"
      },
      "message": "[CELEBORN-2369] Retry stopped master Outbox failures\n\n## Why are the changes needed?\n\nWhen a Celeborn client loses its cached master connection, the associated Outbox can be stopped before queued or late master RPC messages are sent. Those requests fail with `CelebornException: Message is dropped because Outbox is stopped`.\n\n`MasterClient` currently retries I/O and RPC timeout failures, but it does not recognize this stopped-Outbox failure when it is wrapped by the normal `awaitResult` path. In HA mode, a request can therefore fail instead of clearing the stale `RpcEndpointRef` and reconnecting to another available master.\n\nThe same legacy message is also used when the local `RpcEnv` shuts down permanently. Treating every occurrence as retryable without distinguishing shutdown would cause futile reconnect attempts through an already-stopped environment.\n\nJIRA: https://issues.apache.org/jira/browse/CELEBORN-2369\n\n## What changes were proposed in this PR?\n\n- Recognize the exact legacy stopped-Outbox failure through its cause chain and reset the cached master endpoint so the existing HA retry path can reconnect.\n- Use `RpcEnvStoppedException` for terminal local shutdown so shutdown failures remain non-retryable.\n- Preserve each Outbox\u0027s original stop cause for queued and late messages.\n- Fail messages submitted after `RpcEnv` shutdown immediately with the terminal cause.\n- Add focused coverage for HA reconnection, terminal no-reconnect behavior, and transient versus terminal Outbox causes.\n\n### Does this PR resolve a correctness bug?\n\n- [x] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes. Master RPCs can now fail over after a transient stopped-Outbox failure instead of failing immediately. No public API or configuration changes are introduced.\n\n## How was this PR tested?\n\nUsing JDK 17:\n\n- `build/mvn --no-transfer-progress -DskipTests spotless:apply`\n- `build/mvn --no-transfer-progress -pl common -DskipTests test-compile`\n- `build/mvn --no-transfer-progress -pl common -DargLine\u003d -Dtest\u003dorg.apache.celeborn.common.client.MasterClientSuiteJ surefire:test` — 17 tests passed.\n- `build/mvn --no-transfer-progress -pl common -DargLine\u003d -DwildcardSuites\u003dorg.apache.celeborn.common.rpc.netty.OutboxSuite scalatest:test` — 3 tests passed.\n- `build/mvn --no-transfer-progress -pl common -DargLine\u003d -DwildcardSuites\u003dorg.apache.celeborn.common.rpc.netty.NettyRpcEnvSuite scalatest:test` — 33 tests passed.\n- `build/mvn --no-transfer-progress -DskipTests spotless:check` — all 10 reactor modules passed.\n\nCloses #3744 from sunchao/codex/retry-stopped-master-outbox-oss.\n\nAuthored-by: Chao Sun \u003cchao@openai.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "63e28aedd7f0703ffeba438d49ea6f3809160892",
      "tree": "dfd450abc3aba76a96bd1d83ff796d24d8a531be",
      "parents": [
        "38f823141cbb794d754185d9d24623535fd63302"
      ],
      "author": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Tue Jun 30 10:35:14 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Tue Jun 30 10:35:14 2026 +0800"
      },
      "message": "[CELEBORN-2373] Support Flink 2.3\n\n### What changes were proposed in this pull request?\n\nThis PR adds an Apache Flink 2.3 client for Celeborn:\n\n- Add the `client-flink/flink-2.3` and `client-flink/flink-2.3-shaded` modules.\n- Introduce `CelebornFlinkShimV23` and `tiered/CelebornTierFactory` for Flink 2.3, registered via the `org.apache.celeborn.plugin.flink.CelebornFlinkShim` service file.\n- Adapt the common `RemoteShuffleInputGate` to the Flink 2.3 `IndexedInputGate`/`InputGate` API surface: add `getBufferFilteringCompleteFuture()` and the `setCheckpointingDuringRecoveryEnabled`/`isCheckpointingDuringRecoveryEnabled` hooks, and pass the extra recovered-state argument (`new ArrayDeque\u003c\u003e()`) introduced by the recovery-checkpointing work in Flink 2.3.\n- Port the existing Flink client UT suites to the new module (`RemoteShuffleMasterSuiteJV23`, `RemoteShuffleResultPartitionSuiteJV23`, `RemoteShuffleResultPartitionFactorySuiteJV23`, `RemoteShuffleServiceFactorySuiteJV23`, `ShuffleResourceTrackerSuiteJV23`, `tiered/CelebornTierMasterAgentSuiteJV23`).\n- Wire the new module into the build/release/CI: `pom.xml`, `project/CelebornBuild.scala`, `build/make-distribution.sh`, `build/release/release.sh`, `dev/dependencies.sh`, `dev/reformat`, the `deps`/`license`/`style` workflows, and add the `dev/deps/dependencies-client-flink-2.3` dependency list plus the shaded `LICENSE`/`NOTICE` files.\n- Update docs (`README.md`, `docs/README.md`, `docs/deploy.md`, `docs/developers/overview.md`, `docs/developers/sbt.md`) to include Flink 2.3.\n\n### Why are the changes needed?\n\n[Apache Flink 2.3.0](https://flink.apache.org/2026/06/25/apache-flink-2.3.0-release-announcement/) was released on June 25, 2026. Among other things it adds support for checkpointing during unaligned-checkpoint recovery, which changed the `InputGate` API that the Celeborn Flink client integrates against:\n\n- [FLINK-38543] [apache/flink#27862](https://github.com/apache/flink/pull/27862) — change the overall UC restore process, JM and task initialization (introduces `bufferFilteringCompleteFuture` and the checkpointing-during-recovery transition).\n- [FLINK-38930] [apache/flink#27783](https://github.com/apache/flink/pull/27783) — filter records before processing without a spilling strategy during recovery.\n- [FLINK-39018] [apache/flink#27861](https://github.com/apache/flink/pull/27861) — support checkpoint for `LocalInputChannel`.\n\nCeleborn needs a dedicated Flink 2.3 client so users running on Flink 2.3 can use remote shuffle.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes\n\nA new Flink 2.3 client is available via the `-Pflink-2.3` profile, the release binary now ships a `celeborn-client-flink-2.3-shaded` jar, and the docs list Flink 2.3 as a supported version.\n\n### How was this patch tested?\n\nThe existing Flink client UT suites are ported to the new `flink-2.3` module and run under the `-Pflink-2.3` profile. The `deps`, `license`, and `style` CI workflows are extended to cover `flink-2.3`.\n\nCloses #3751 from SteNicholas/CELEBORN-2373.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "38f823141cbb794d754185d9d24623535fd63302",
      "tree": "c63b7bac1dacabd4d24d74e8d9488ac183a347af",
      "parents": [
        "a1ecf1926e21590a0c1df816a9e6093b2ebbf69c"
      ],
      "author": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Tue Jun 30 10:15:03 2026 +0800"
      },
      "committer": {
        "name": "Shuang",
        "email": "lvshuang.xjs@alibaba-inc.com",
        "time": "Tue Jun 30 10:15:03 2026 +0800"
      },
      "message": "[CELEBORN-2355] Return an empty stream from C++ ShuffleClientImpl::readPartition for an empty partition\n\n### What changes were proposed in this pull request?\n\nThe C++ `ShuffleClientImpl::readPartition` now returns an empty stream when a shuffle has no locations for the requested partition, instead of constructing a `CelebornInputStream` over zero locations. Specifically:\n\n- Add a `CelebornInputStream::empty()` factory that builds a no-op stream whose `read()` immediately yields `-1`.\n- In `readPartition`, when `locations` is empty, log a warning and return `CelebornInputStream::empty()`; the shuffle key is now built only on the non-empty path.\n\nTests:\n- A unit test for `CelebornInputStream::empty()` (immediate EOF, repeated reads, zero-length read).\n- Two `readPartition` tests (empty file groups, missing partition) backed by a `VisibleForTesting` seam that seeds the cached file-group response so no RPC to the `LifecycleManager` is needed.\n\n### Why are the changes needed?\n\nFor a partition with no shuffle data, the previous code constructed a full reader over an empty location list (also building the decompressor when compression is enabled). Returning an empty stream short-circuits this and makes the empty-shuffle case explicit, consistent with the read path\u0027s intent.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nAdded unit tests under `cpp/celeborn/client/tests/`:\n- `CelebornInputStreamTest.emptyStreamReadsReturnEof`\n- `ShuffleClientImplTest.readPartitionEmptyWhenFileGroupsEmpty`\n- `ShuffleClientImplTest.readPartitionEmptyWhenPartitionMissing`\n\nRun with:\n\n```\ncd cpp \u0026\u0026 mkdir -p build \u0026\u0026 cd build \u0026\u0026 cmake .. \u0026\u0026 make celeborn_client_test\n./celeborn/client/tests/celeborn_client_test \\\n  --gtest_filter\u003d\u0027CelebornInputStreamTest.*:ShuffleClientImplTest.readPartition*\u0027\n```\n\nCloses #3748 from SteNicholas/CELEBORN-2355.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Shuang \u003clvshuang.xjs@alibaba-inc.com\u003e\n"
    },
    {
      "commit": "a1ecf1926e21590a0c1df816a9e6093b2ebbf69c",
      "tree": "402f375826a314df632e39eeebc9a5f0d3452b77",
      "parents": [
        "38202449af6652f2dcb86061c509daa3bedaceac"
      ],
      "author": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Sun Jun 28 14:34:52 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Sun Jun 28 14:34:52 2026 +0800"
      },
      "message": "[CELEBORN-2372] Remove Hadoop 2 client support\n\n### What changes were proposed in this pull request?\n\nThis PR removes Hadoop 2 client support from Celeborn. Specifically:\n\n- Remove the `hadoop-2` Maven profile from the root `pom.xml`, which was activated when `hadoop.version` matched `2.*` and pulled in `hadoop-client` with a long list of exclusions.\n- Remove the corresponding `hadoop-2` profiles (activated via the `hadoop-2-deps` property) from `common/pom.xml`, `master/pom.xml`, and `client-mr/mr/pom.xml`.\n- Update `README.md` to reflect that Celeborn supports Hadoop MapReduce 3 (instead of 2/3).\n\n### Why are the changes needed?\n\nHadoop 2 is an old release line that is no longer actively maintained. Dropping it reduces the build/dependency matrix and maintenance burden, allowing the project to focus on Hadoop 3.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes\n\nCeleborn no longer provides the `hadoop-2` build profile. Users building against Hadoop 2 must either upgrade to Hadoop 3 or stay on a previous Celeborn release.\n\n### How was this patch tested?\n\nExisting CI. No new functionality is introduced; the change only removes build profiles and references.\n\nCloses #3747 from SteNicholas/CELEBORN-2372.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "38202449af6652f2dcb86061c509daa3bedaceac",
      "tree": "45dc1228e687589437bbe06aea0c712ee2d9694e",
      "parents": [
        "a956feea9fee33009cd300f3d24024f4f79688b6"
      ],
      "author": {
        "name": "sychen",
        "email": "sychen@ctrip.com",
        "time": "Thu Jun 25 22:25:45 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Thu Jun 25 22:25:45 2026 +0800"
      },
      "message": "[CELEBORN-2368][FOLLOWUP] Remove leftover dependency list files for Spark 2.4\n\n### What changes were proposed in this pull request?\n\nRemove leftover dependency list files for Spark 2.4.\n\n### Why are the changes needed?\n\nRemove leftover dependency list files for Spark 2.4.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nGHA\n\nCloses #3750 from cxzl25/CELEBORN-2368_f.\n\nAuthored-by: sychen \u003csychen@ctrip.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "a956feea9fee33009cd300f3d24024f4f79688b6",
      "tree": "96f352235f7104f1740a542d5232e613814afc3d",
      "parents": [
        "581a0c0f00a3f050d819ce67344eb24f4a3525f0"
      ],
      "author": {
        "name": "sychen",
        "email": "sychen@ctrip.com",
        "time": "Thu Jun 25 22:24:11 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Thu Jun 25 22:24:11 2026 +0800"
      },
      "message": "[CELEBORN-2367][FOLLOWUP] Remove leftover dependency list files for Flink 1.16 and 1.17\n\n### What changes were proposed in this pull request?\n\nRemove leftover dependency list files for Flink 1.16 and 1.17.\n\n### Why are the changes needed?\n\nRemove leftover dependency list files for Flink 1.16 and 1.17.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nGHA\n\nCloses #3749 from cxzl25/CELEBORN-2367_f.\n\nAuthored-by: sychen \u003csychen@ctrip.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "581a0c0f00a3f050d819ce67344eb24f4a3525f0",
      "tree": "16ec15b6df64cd00058d262ea29ded26119f4964",
      "parents": [
        "5f49ed90f7fefcde81876ea056241dd9340b7788"
      ],
      "author": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Thu Jun 25 12:46:50 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Thu Jun 25 12:46:50 2026 +0800"
      },
      "message": "[CELEBORN-2368] Remove Spark 2.4 client support\n\n### What changes were proposed in this pull request?\n\nThis PR removes Spark 2.4 client support from Celeborn. Specifically:\n\n- Delete the `client-spark/spark-2` and `client-spark/spark-2-shaded` modules, including the shuffle manager/reader/writer implementations (`SparkShuffleManager`, `CelebornShuffleReader`, `HashBasedShuffleWriter`, `SortBasedShuffleWriter`), `SparkUtils`, the fallback policy runner, shuffle handle, license/notice resources, and test suites.\n- Remove the `spark-2.4` Maven profile from `pom.xml` (and the related `tests/spark-it` profile) and the `Spark24` SBT project from `project/CelebornBuild.scala`.\n- Drop `spark-2.4` from the build and release scripts (`build/make-distribution.sh`, `build/release/release.sh`, `dev/dependencies.sh`, `dev/reformat`) and from CI workflows (`deps.yml`, `license.yml`, `maven.yml`, `sbt.yml`, `style.yml`).\n- Update documentation (`README.md`, `docs/developers/sbt.md`) to reflect the supported Spark versions (3.0/3.1/3.2/3.3/3.4/3.5/4.0).\n\n### Why are the changes needed?\n\nSpark 2.4 is an old release that is no longer maintained upstream. Dropping it reduces the build/test matrix, maintenance burden, and release artifacts, allowing the project to focus on currently supported Spark versions (3.0 and above).\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes\n\nCeleborn no longer ships or builds the Spark 2.4 client jar. Users running Spark 2.4 must either upgrade to a supported Spark version (3.0+) or stay on a previous Celeborn release.\n\n### How was this patch tested?\n\nExisting CI for the remaining Spark versions (3.0/3.1/3.2/3.3/3.4/3.5/4.0) via the updated Maven/SBT/style/license/deps workflows. No new functionality is introduced; the change only removes code, build profiles, and references.\n\nCloses #3743 from SteNicholas/CELEBORN-2368.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "5f49ed90f7fefcde81876ea056241dd9340b7788",
      "tree": "d5cc48744c59fc29fa40381ef3cd5cabce720037",
      "parents": [
        "441d6b45a5b5e71310c90ef6099626f97c99ef47"
      ],
      "author": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Tue Jun 23 20:44:31 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Tue Jun 23 20:48:51 2026 +0800"
      },
      "message": "[CELEBORN-2367] Remove Flink 1.16 and 1.17 client support\n\n### What changes were proposed in this pull request?\n\nThis PR removes Flink 1.16 and Flink 1.17 client support from Celeborn. Specifically:\n\n- Delete the `client-flink/flink-1.16`, `client-flink/flink-1.16-shaded`, `client-flink/flink-1.17`, and `client-flink/flink-1.17-shaded` modules, including their shims (`CelebornFlinkShimV116`/`CelebornFlinkShimV117`), service descriptors, license/notice resources, and test suites.\n- Remove the `flink-1.16` and `flink-1.17` Maven profiles from `pom.xml` and the `Flink116`/`Flink117` SBT projects from `project/CelebornBuild.scala`.\n- Drop `flink-1.16`/`flink-1.17` from the build and release scripts (`build/make-distribution.sh`, `build/release/release.sh`, `dev/dependencies.sh`, `dev/reformat`) and from CI workflows (`deps.yml`, `license.yml`, `maven.yml`, `sbt.yml`, `style.yml`).\n- Remove the `v1_16`/`v1_17` entries from the `FlinkVersion` test enum.\n- Update documentation (`README.md`, `docs/README.md`, `docs/deploy.md`, `docs/developers/overview.md`, `docs/developers/sbt.md`) to reflect the supported Flink versions (1.18/1.19/1.20/2.0/2.1/2.2).\n\n### Why are the changes needed?\n\nFlink 1.16 and 1.17 are older releases that are no longer maintained upstream. Dropping them reduces the build/test matrix, maintenance burden, and release artifacts, allowing the project to focus on currently supported Flink versions (1.18 and above).\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes\n\nCeleborn no longer ships or builds Flink 1.16/1.17 client jars. Users running Flink 1.16 or 1.17 must either upgrade to a supported Flink version (1.18+) or stay on a previous Celeborn release.\n\n### How was this patch tested?\n\nExisting CI for the remaining Flink versions (1.18/1.19/1.20/2.0/2.1/2.2) via the updated Maven/SBT/style/license/deps workflows. No new functionality is introduced; the change only removes code, build profiles, and references.\n\nCloses #3742 from SteNicholas/CELEBORN-2367.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "441d6b45a5b5e71310c90ef6099626f97c99ef47",
      "tree": "ab404d940df3939b0af297c1704992ba62f038f0",
      "parents": [
        "8903a31d81dedf3b9bd52e5773cbb98220d6b14d"
      ],
      "author": {
        "name": "Zemin Piao",
        "email": "pzm6391@gmail.com",
        "time": "Tue Jun 23 17:38:30 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Tue Jun 23 17:38:30 2026 +0800"
      },
      "message": "[CELEBORN-2354] Fix Helm chart indentation, PodMonitor flags and cluster-scoped PriorityClass namespace\n\n### What changes were proposed in this pull request?\n\nFix helm chart indentation, podmonitor flags and remove the namespace option for priorityClass.\n\n### Why are the changes needed?\n\nCleanup the helm chart .\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nNo.\n\nCloses #3723 from zemin-piao/fix_helm_chart_indentation_podmonitor_flags_priorityclass.\n\nAuthored-by: Zemin Piao \u003cpzm6391@gmail.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "8903a31d81dedf3b9bd52e5773cbb98220d6b14d",
      "tree": "c0001b55ce53d3da1941b96e55a7e016dd8f39e5",
      "parents": [
        "0198bcf53a602f585df3a72cc1e5f147dff419a1"
      ],
      "author": {
        "name": "afterincomparableyum",
        "email": "224495379+afterincomparableyum@users.noreply.github.com",
        "time": "Tue Jun 23 10:56:50 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Tue Jun 23 10:56:50 2026 +0800"
      },
      "message": "[CELEBORN-2360] Fix ReviseLostShuffles RPC deserialization\n\n### What changes were proposed in this pull request?\n\nCELEBORN-1601 added ReviseLostShuffles support but only wired up the encode side (toTransportMessage), the master handler, and the client sender. The corresponding decode cases in ControlMessages.fromTransportMessage were never added. Since that match has no default case, receiving a REVISE_LOST_SHUFFLES (89) request on the master or a REVISE_LOST_SHUFFLES_RESPONSE (90) on the client throws a MatchError, so the feature is broken over RPC. The existing test exercises handleReviseLostShuffles directly and bypasses the RPC round-trip, which is why this was probably not caught.\n\nTo fix, I added the missing REVISE_LOST_SHUFFLES_VALUE and REVISE_LOST_SHUFFLES_RESPONSE_VALUE cases so the messages deserialize into PbReviseLostShuffles / PbReviseLostShufflesResponse.\n\nI also added a round trip test in UtilsSuite covering both messages.\n\n### Why are the changes needed?\n\nTo fix an existing bug.\n\n### Does this PR resolve a correctness bug?\n\n- [x] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nCI/CD\n\nCloses #3735 from afterincomparableyum/CELEBORN-2360.\n\nAuthored-by: afterincomparableyum \u003c224495379+afterincomparableyum@users.noreply.github.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "0198bcf53a602f585df3a72cc1e5f147dff419a1",
      "tree": "a680389c17823de8300db252a387dcf3ea8d7fc3",
      "parents": [
        "d17489bc6310616a1cbd614cdba49ef10c84307d"
      ],
      "author": {
        "name": "AmandeepSingh285",
        "email": "mailto.amandeep.singh.28@gmail.com",
        "time": "Mon Jun 22 10:48:46 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Mon Jun 22 10:48:46 2026 +0800"
      },
      "message": "[CELEBORN-2334] Automatically restore RocksDB in case of failures\n\n### What changes were proposed in this pull request?\n\nThe patch re-instantiates RocksDB in case of failures. In the current implementation, when RocksDB enters a read-only mode due to failures, Celeborn metadata operations fail and remain blocked until manual intervention or restart. This pull request adds logic to detect such RocksDB failures and re-instantiate the RocksDB instance so that metadata operations can recover automatically and continue functioning without prolonged disruption. RocksDB can enter a read-only or unusable state under scenarios such as: corruption in files, errors from underlying file system. In such cases, RocksDB prevents further writes to protect data consistency, which causes Celeborn metadata operations to fail.\n\n### Why are the changes needed?\n\nOnce RocksDB enters a read-only or error state, Celeborn metadata operations become unavailable because the existing RocksDB instance remains unusable, which could lead to failures in metadata updates.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nUnit tests.\n\nCloses #3695 from AmandeepSingh285/auto-recover-rocks-db.\n\nLead-authored-by: AmandeepSingh285 \u003cmailto.amandeep.singh.28@gmail.com\u003e\nCo-authored-by: amandeeps.28 \u003camandeeps.28@uber.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "d17489bc6310616a1cbd614cdba49ef10c84307d",
      "tree": "32f41a04fb5e3ca7d9b1214b16145c0d88564483",
      "parents": [
        "3a11d1e11950314c3692b84996af2e76b261b971"
      ],
      "author": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Tue Jun 16 21:46:20 2026 +0800"
      },
      "committer": {
        "name": "Shuang",
        "email": "lvshuang.xjs@alibaba-inc.com",
        "time": "Tue Jun 16 21:46:20 2026 +0800"
      },
      "message": "[CELEBORN-2353] Align C++ shuffle client push/merge data path with Java ShuffleClient\n\n### What changes were proposed in this pull request?\n\nThis PR aligns the C++ shuffle client\u0027s push/merge data path with the Java `ShuffleClient`:\n\n- Refactor `pushData`/`mergeData`/`pushMergedData` to share one push prologue (`prepareBatch`) and a uniform synchronous-failure route (`pushWithFailureRouting`), and move throttling (`limitMaxInFlight`) to the caller before the initial push so the retry path never re-throttles.\n- Exclude push workers on connection/timeout failures (`celeborn.client.push.excludeWorkerOnFailure.enabled`), routing pushes away from them until a successful revive or re-assignment lifts the exclusion, and clear it wholesale on shutdown.\n- Track push-failed batches and report them at `MapperEnd`; the reader dedups duplicate batches for the adaptive skewed-partition read optimization.\n- Derive specific failure causes from transport error messages (`getPushDataFailCause`) and propagate them through revive/retry; add `StatusCode::toString`.\n\n### Why are the changes needed?\n\nThe C++ shuffle client lacked the push-side resiliency the Java `ShuffleClient` already provides: excluding failed push workers and tracking data-push failures so the reader can dedup duplicate batches for the adaptive skewed-partition read optimization. Bringing the C++ push/merge data path to parity keeps the native and JVM clients behaviorally consistent and honors the existing `celeborn.client.push.excludeWorkerOnFailure.enabled` / adaptive skewed-partition read configs.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nCovered by new/extended C++ unit tests:\n\n- `PushDataCallbackTest`, `PushMergedDataCallbackTest` — push / merged-push callback failure routing and cause propagation.\n- `ShuffleClientImplTest` — push worker exclusion and data-push-failure tracking.\n- `CelebornInputStreamRetryTest` — reader-side dedup of duplicate push-failed batches.\n- `PushStateTest`, `ControlMessagesTest` — failed-batch state and `MapperEnd` serialization round-trip.\n\nCloses #3724 from SteNicholas/CELEBORN-2353.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Shuang \u003clvshuang.xjs@alibaba-inc.com\u003e\n"
    },
    {
      "commit": "3a11d1e11950314c3692b84996af2e76b261b971",
      "tree": "1b59fba93f31b6a4b00b9b9e8d56eb3d4ceb634a",
      "parents": [
        "3eee443fe6e4f66a878150b19bc1a4f71d61e238"
      ],
      "author": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Tue Jun 16 10:41:53 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Tue Jun 16 10:41:53 2026 +0800"
      },
      "message": "[CELEBORN-2365] Bump Spark version from 4.0.1 to 4.0.3\n\n### What changes were proposed in this pull request?\n\nBump the Spark 4.0 dependency version from `4.0.1` to `4.0.3` in both the Maven build (`pom.xml`) and the SBT build (`project/CelebornBuild.scala`).\n\n### Why are the changes needed?\n\n[Apache Spark 4.0.3](https://spark.apache.org/news/spark-4-0-3-released.html) is a maintenance release containing security and correctness fixes, and the Spark community strongly recommends all 4.0 users upgrade to this stable release. See the [release notes](https://spark.apache.org/releases/spark-release-4-0-3.html) for details.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nPass GitHub Actions.\n\nCloses #3741 from SteNicholas/CELEBORN-2365.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "3eee443fe6e4f66a878150b19bc1a4f71d61e238",
      "tree": "ce74f51bc739d072ecf889c6187597adde319aed",
      "parents": [
        "8c3108ddbfe0e260af6115fd087fc0a5b981704a"
      ],
      "author": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Mon Jun 15 16:24:41 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Mon Jun 15 16:24:41 2026 +0800"
      },
      "message": "[CELEBORN-2363] Bump Flink version to 1.20.5, 2.0.2 and 2.1.3\n\n### What changes were proposed in this pull request?\n\nBump the supported Flink versions to their latest patch releases:\n\n- Flink 1.20: `1.20.3` → `1.20.5` (and the default `flink.version` property `1.20.1` → `1.20.5`)\n- Flink 2.0: `2.0.1` → `2.0.2`\n- Flink 2.1: `2.1.1` → `2.1.3`\n\nUpdated in both `pom.xml` and `project/CelebornBuild.scala`.\n\n### Why are the changes needed?\n\nKeep the Flink clients aligned with the latest patch releases, which include bug fixes and security fixes from upstream Flink.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nExisting CI (Maven and SBT builds plus the Flink integration tests `tests/flink-it`).\n\nCloses #3738 from SteNicholas/CELEBORN-2363.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "8c3108ddbfe0e260af6115fd087fc0a5b981704a",
      "tree": "7b4ba5dc2727ccf2916c57bcef27bee4d0a4053b",
      "parents": [
        "638270effc4751a72fa0a999855ca226b63c39df"
      ],
      "author": {
        "name": "Aurélien Pupier",
        "email": "apupier@ibm.com",
        "time": "Mon Jun 15 10:32:56 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Mon Jun 15 10:32:56 2026 +0800"
      },
      "message": "[MINOR] Configure GitHub workflows to use concurrency cancel-in-progress for pull requests\n\nsee recommended best practices at Apache\nhttps://cwiki.apache.org/confluence/pages/viewpage.action?spaceKey\u003dINFRA\u0026title\u003dGitHub+Actions+Recommended+Practices\n\n### What changes were proposed in this pull request?\n\nConfigure GitHub workflows to use concurrency cancel-in-progress for pull requests. it means that if a PR is modified before the end of the execution of PR checks, the current runnign oe are cancelled to let more room for the new needed one.\n\n### Why are the changes needed?\n\nIt will help to reduce the usage of GitHub runners. The pool of github runners is shared across whole Apache oganization, and the limit is regularly reached these days.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nsimilar changes were applied to several other Apache repositories\n\nCloses #3729 from apupier/addConcurrencyCancelInProgressForPRGitHubWorkflows.\n\nAuthored-by: Aurélien Pupier \u003capupier@ibm.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "638270effc4751a72fa0a999855ca226b63c39df",
      "tree": "7c1e811183bede2dae1297d8877697ee1a2a405e",
      "parents": [
        "90b9b44eb18228a580b5fbce9293e2d750f01d2f"
      ],
      "author": {
        "name": "Sanskar Modi",
        "email": "sanskarmodi97@gmail.com",
        "time": "Fri Jun 12 16:41:03 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Fri Jun 12 16:41:03 2026 +0800"
      },
      "message": "[CELEBORN-2016] Fix the worker decommission and graceful shutdown condition\n\n### What changes were proposed in this pull request?\n\nIn current code, we set the shutdown hook for `timeout` time and condition tries to check `timeSpent \u003c timeout` when this condition is will become true, shutdown hook timer is already passed and VM will exit without executing the code below this point.\n\nNew condition will be `timeSpent + interval \u003c timeout`, so we will get (0, interval] time to execute the below code.\n\n### Why are the changes needed?\n\nWe current shutdown logic we have seen worker getting shutdown abruptly with timeout exception without completely executing the shutdown hook because of which Celeborn is unable to print unreleased partition location and unreleased shuffle on decommission and graceful shutdown.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nNA\n\nCloses #3727 from s0nskar/shutdown_fix.\n\nAuthored-by: Sanskar Modi \u003csanskarmodi97@gmail.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "90b9b44eb18228a580b5fbce9293e2d750f01d2f",
      "tree": "332aaeb5383fb9681ae0a588cea7cdcdf3fa7b90",
      "parents": [
        "61499a0c9ac40786c0e846009f0ea77e1b1d42ed"
      ],
      "author": {
        "name": "Sanskar Modi",
        "email": "sanskarmodi97@gmail.com",
        "time": "Thu Jun 11 22:38:03 2026 +0800"
      },
      "committer": {
        "name": "Shuang",
        "email": "lvshuang.xjs@alibaba-inc.com",
        "time": "Thu Jun 11 22:38:03 2026 +0800"
      },
      "message": "[CELEBORN-2351] Partition file sorting should only be paused for PUSH_AND_REPLICATE_PAUSED\n\n### What changes were proposed in this pull request?\n\nPartition file sorting should only be paused for `PUSH_AND_REPLICATE_PAUSED`, which represent very high memory pressure and cause OOM for workers. Sorting should be allowed for `PUSH_PAUSED` state.\n\n### Why are the changes needed?\n\nCurrently even for push pause state we stop the sorting for partition files. If pause is sustained for a longer time then sorting can timeout and reader waiting for sorting will fail or be delayed.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [X] Yes\n\n### How was this patch tested?\n\nExisting UTs\n\nCloses #3720 from s0nskar/sort_memory_ready.\n\nAuthored-by: Sanskar Modi \u003csanskarmodi97@gmail.com\u003e\nSigned-off-by: Shuang \u003clvshuang.xjs@alibaba-inc.com\u003e\n"
    },
    {
      "commit": "61499a0c9ac40786c0e846009f0ea77e1b1d42ed",
      "tree": "5a9037ff6d490900504e5a5e8db5a497ac383fd5",
      "parents": [
        "b1a7cb4964f55a05544334befb0e49f78001f872"
      ],
      "author": {
        "name": "夷羿",
        "email": "yiyi.zt@alibaba-inc.com",
        "time": "Thu Jun 11 21:55:41 2026 +0800"
      },
      "committer": {
        "name": "Shuang",
        "email": "lvshuang.xjs@alibaba-inc.com",
        "time": "Thu Jun 11 21:55:41 2026 +0800"
      },
      "message": "[CELEBORN-2319] Standalone LifecycleManager \u0026\u0026 rust sdk\n\n### What changes were proposed in this pull request?\n\nThis PR introduces two major features to support **non-JVM (C++/Rust) clients** using Apache Celeborn for shuffle:\n\n**1. Standalone LifecycleManager Daemon (Scala/JVM)**\n\n- Added `LifecycleManagerDaemon` — a standalone JVM process that hosts a `LifecycleManager` independently from any compute engine (Spark/Flink) Driver. It installs a shutdown hook (with a watchdog that force-halts if graceful stop exceeds the timeout) and blocks until SIGINT/SIGTERM.\n- Added `LifecycleManagerDaemonArguments` for CLI argument parsing (`--app-id`, `--master-endpoints`, `--port`/`-p`, `--host`, `--properties-file`, `-h`/`--help`). Parsing is a pure function that throws `ArgumentParseException` (carrying an exit code) so every branch is unit-testable; `parseOrExit` wraps it for the process entry point.\n- Added `sbin/start-lifecycle-manager.sh` launch script with classpath assembly, environment loading, required-argument validation, automatic free-port selection, and **RPC-port polling** to confirm the daemon is actually bound before reporting success.\n- Added a new **`lifecycle-manager` Maven/sbt module** that depends on `celeborn-service`, `celeborn-client` and `celeborn-common`. Registered the module in both the root `pom.xml` and `project/CelebornBuild.scala` (`projectDefinitions`), and wired it into `build/make-distribution.sh` for both the Maven and sbt build paths.\n- **Security note**: the standalone LM runs without authentication; it logs a warning on startup and the code documents that operators must bind it to a trusted network only.\n\n**2. Rust SDK via C++ FFI (`rust/` directory)**\n\n- `celeborn-client-sys`: Low-level FFI crate bridging Rust ↔ C++ via a **plain C ABI** (no `cxx`). The C++ side exposes `celeborn_ffi_*` functions (`create_client`, `setup_lifecycle_manager`, `shutdown`, `push_data`, `mapper_end`, partition reader open/read/close, etc.) returning status codes plus heap-allocated error strings.\n  - `build.rs`: links the single aggregated shared library `libceleborn_client.{so,dylib}` (which whole-archives all internal static libs and hides non-`celeborn` symbols), so downstream Rust never sees folly / protobuf / glog / abseil.\n- `celeborn-client`: Safe, ergonomic Rust wrapper providing `ShuffleClient` with:\n  - Input validation extracted into a pure `validate_connect_args` (app_id non-empty, port \u003e 0, codec ∈ {NONE, LZ4, ZSTD}) — testable without a live cluster.\n  - Documented `Send + Sync` rationale (the C++ `ShuffleClientImpl` synchronizes internally), enabling `Arc\u003cShuffleClient\u003e` sharing for concurrent `\u0026self` push/read.\n  - `Drop`-safe shutdown that nulls the handle to avoid a double `celeborn_ffi_shutdown`. The native handle is **intentionally leaked** after shutdown to dodge a folly `EventBase` teardown race; this implies a **per-process-client** usage model, which is documented prominently on the type.\n- Two example programs (`data_sum_writer.rs`, `data_sum_reader.rs`) mirroring the existing C++ `DataSumWithWriterClient` / `DataSumWithReaderClient` test programs. The writer seeds its RNG **per mapper thread** so each thread emits a distinct byte stream, genuinely exercising the concurrent push path.\n\n**3. C++ build portability (`cpp/CMakeLists.txt`)**\n\n- Discover the Homebrew prefix via `brew --prefix` instead of hard-coding `/opt/homebrew` (so Intel macOS under `/usr/local` works too).\n- Discover Abseil via `find_package(absl CONFIG)`, falling back to a toolchain-derived GNU multiarch dir (covers `aarch64-linux-gnu`, which the README advertises); a missing Abseil is now a `FATAL_ERROR` rather than a silent warning.\n- Guard the x86-only `-msse4.2` flag by architecture so aarch64 builds compile.\n\n### Why are the changes needed?\n\nCurrently, `LifecycleManager` can only run **embedded inside a JVM-based compute engine Driver** (e.g., Spark Driver). This makes it impossible for non-JVM applications (Daft engine, etc.) to use Celeborn as their shuffle service, because:\n\n1. The C++ client requires a running `LifecycleManager` to coordinate shuffle metadata (register shuffles, allocate slots, manage partition locations) with Celeborn Masters and Workers.\n2. Without a standalone `LifecycleManager`, non-JVM applications have no way to bootstrap this coordination layer.\n\nBy decoupling the `LifecycleManager` into a **standalone daemon process**, any client — regardless of language runtime — can connect to it via RPC. The Rust SDK then leverages this architecture to provide first-class Rust support by bridging to the existing, battle-tested C++ client implementation via FFI.\n\n### Does this PR resolve a correctness bug?\n\nNo\n\n### Does this PR introduce _any_ user-facing change?\n\nYes.\n\n- **New component**: Users can now start a standalone `LifecycleManager` daemon via `sbin/start-lifecycle-manager.sh --app-id \u003cid\u003e --master-endpoints \u003ceps\u003e [--port \u003cport\u003e] [--host \u003chost\u003e]`.\n- **New SDK**: Rust applications can now use the `celeborn-client` crate to perform shuffle read/write operations against a Celeborn cluster.\n- **Limitation**: The standalone `LifecycleManager` does **not** support auth (`celeborn.auth.enabled` must be `false`), as the C++/Rust clients lack SASL support. Deploy it on a trusted network only.\n\n### How was this patch tested?\n\n- **Unit tests (JVM)**: `LifecycleManagerDaemonArgumentsSuite` covers the happy paths plus help / unknown-arg / missing-arg / invalid-port branches and `applyArgsToConf` (13 tests). Run via Maven (`mvn -pl lifecycle-manager test`) and compiled/style-checked under sbt as a first-class module.\n- **Unit tests (Rust)**: `validate_connect_args` is covered by `cargo test -p celeborn-client` (codec/app_id/port validation) without requiring a cluster.\n- **Integration**: The Rust SDK was validated using the `data_sum_writer` / `data_sum_reader` examples (Rust ports of `DataSumWithWriterClient.cpp` / `DataSumWithReaderClient.cpp`), which write random numeric data across partitions and verify correctness by comparing partition sums between writer and reader. The `LifecycleManagerDaemon` was tested by starting it against a local Celeborn cluster (Master + Workers) and verifying the Rust examples connect, push, and read through the daemon.\n\nCloses #3677 from gavin9402/standalone_and_rust.\n\nLead-authored-by: 夷羿 \u003cyiyi.zt@alibaba-inc.com\u003e\nCo-authored-by: Zhou \u003cgavin9402@163.com\u003e\nSigned-off-by: Shuang \u003clvshuang.xjs@alibaba-inc.com\u003e\n"
    },
    {
      "commit": "b1a7cb4964f55a05544334befb0e49f78001f872",
      "tree": "5166bf1502fba8a73e9973967c2bed4210e8b9bc",
      "parents": [
        "4a9112a594ddc3f55a98172e643af1d08d72740d"
      ],
      "author": {
        "name": "James Xu",
        "email": "xumingmingv@gmail.com",
        "time": "Thu Jun 11 16:15:31 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Thu Jun 11 16:15:31 2026 +0800"
      },
      "message": "[CELEBORN-2315][FOLLOWUP] Change assertIteratorFullyConsumed to throw CelebornIOException\n\n### What changes were proposed in this pull request?\n\nReplace TaskKilledException with CelebornIOException in assertIteratorFullyConsumed. CelebornIOException extends IOException and fits the existing throws IOException contract of all write() methods cleanly, without needing an unchecked exception workaround.\n\nAlso revert the throwTaskKillException(String message) overload added to TaskInterruptedHelper in CELEBORN-2315, which is now dead code.\n\n### Why are the changes needed?\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nCloses #3728 from xumingming/celeborn-2315-followup-celebornioexception.\n\nAuthored-by: James Xu \u003cxumingmingv@gmail.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "4a9112a594ddc3f55a98172e643af1d08d72740d",
      "tree": "75e8385b7dc6182982983f18561878476832985f",
      "parents": [
        "b131a400a239486fe36c659ed6b4397a27272a82"
      ],
      "author": {
        "name": "James Xu",
        "email": "xumingmingv@gmail.com",
        "time": "Thu Jun 11 14:30:07 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Thu Jun 11 14:30:07 2026 +0800"
      },
      "message": "[CELEBORN-2315] Add iterator fully-consumed validation after shuffle write\n\n### What changes were proposed in this pull request?\n\nAdds a post-write safety check to HashBasedShuffleWriter and SortBasedShuffleWriter: after the write loop completes, verify the input iterator was fully consumed. If records remain, kill the task with TaskKilledException. This guards against silent data loss.\n\n### Why are the changes needed?\n\nIt could give another layer of correctness guarantee.\n\n### Does this PR resolve a correctness bug?\n\nEnhance correctness guarantee.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nUT.\n\nCloses #3672 from xumingming/iterator-fully-consumed-check.\n\nAuthored-by: James Xu \u003cxumingmingv@gmail.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "b131a400a239486fe36c659ed6b4397a27272a82",
      "tree": "654ccf3ba5e5d9832fafd4f959a70fa3304dfa50",
      "parents": [
        "e5099f90d55e7b70292b0677c3a889f72c9b7efb"
      ],
      "author": {
        "name": "Sanskar Modi",
        "email": "sanskarmodi97@gmail.com",
        "time": "Tue Jun 09 03:31:17 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Tue Jun 09 03:31:17 2026 +0800"
      },
      "message": "[MINOR] Fix PbSerDeUtilsTest failure\n\n### What changes were proposed in this pull request?\n\nFix PbSerDeUtilsTest failure for spark4 jobs\n\n```\nError:  /home/runner/work/celeborn/celeborn/common/src/test/scala/org/apache/celeborn/common/util/PbSerDeUtilsTest.scala:851: comparing values of types Boolean and Boolean using `equals` unsafely bypasses cooperative equality; use `\u003d\u003d` instead\nError: [ERROR] one error found\n```\nhttps://github.com/apache/celeborn/actions/runs/27126367363/job/80055898287?pr\u003d3720\n\n### Why are the changes needed?\n\nAfter this change https://github.com/apache/celeborn/pull/3675, some of tests are failing.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nExisting UTs.\n\nCloses #3722 from s0nskar/CELEBORN-1577_fix_test.\n\nAuthored-by: Sanskar Modi \u003csanskarmodi97@gmail.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "e5099f90d55e7b70292b0677c3a889f72c9b7efb",
      "tree": "b07d9652d7b1b4817088d7c3eb63709dd5d0585c",
      "parents": [
        "a0f098f75eff99ae4b914678c5eeacc089223889"
      ],
      "author": {
        "name": "James Xu",
        "email": "xumingmingv@gmail.com",
        "time": "Tue Jun 09 03:18:57 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Tue Jun 09 03:18:57 2026 +0800"
      },
      "message": "[CELEBORN-2313] Extend E2E checked zone to batch assembly point\n\n### What changes were proposed in this pull request?\n\nCeleborn\u0027s E2E integrity check computes CRC_M inside `ShuffleClientImpl.pushOrMergeData()`, which runs in the async `DataPusher` thread. This leaves the segment from batch assembly in the writer thread through the `DataPusher` queue entirely outside the checked zone — meaning any corruption that occurs in that window is invisible to the integrity check and reaches reducers silently.\n\nThis change closes that gap and enables detection of a class of correctness bugs where data corruption occurs between batch assembly and async push dispatch, including bugs involving shared buffer pool references.\n\nIntroduce `ShuffleClient.computeBatchCRC()` and consolidate its invocation into two choke points:\n\n- `DataPusher.addTask()`: covers all async push paths. The CRC is recorded on the writer thread immediately before the buffer is enqueued, so `DataPusher.pushData()` intentionally uses the bare `client.pushData()` to avoid double-counting the same batch into CommitMetadata.\n\n- `ShuffleClient.pushDataWithCRC()` and `ShuffleClient.mergeDataWithCRC()`: new concrete convenience methods that call `computeBatchCRC()` then delegate to the abstract `pushData()`/`mergeData()`. These cover all synchronous push paths (`pushGiantRecord`, `close()` flush). The abstract `pushData()`/`mergeData()` are now documented as internal-use-only; all writer call sites across `HashBasedShuffleWriter` (spark-2/3), `SortBasedShuffleWriter` (spark-2/3), and `SortBasedPusher` use the `WithCRC` variants instead.\n\nThe now-redundant CRC computation inside `pushOrMergeData()` is removed.\n\nThis consolidation eliminates 7 scattered `computeBatchCRC` call sites that previously had to be manually paired with each push/merge call, reducing the risk of a future call site omitting the CRC step.\n\n### Why are the changes needed?\n\nEnhance E2E Integrity Check, so it can cover more code path.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nUnit Test.\n\nCloses #3716 from xumingming/extend-e2e-checked-zone-v2.\n\nAuthored-by: James Xu \u003cxumingmingv@gmail.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "a0f098f75eff99ae4b914678c5eeacc089223889",
      "tree": "0fe8b39856689adfde2af291490d8a53b7474178",
      "parents": [
        "93de9b3042f419fa914144cd640e18e8995a3b8b"
      ],
      "author": {
        "name": "Sanskar Modi",
        "email": "sanskarmodi97@gmail.com",
        "time": "Mon Jun 08 14:25:00 2026 +0800"
      },
      "committer": {
        "name": "Shuang",
        "email": "lvshuang.xjs@alibaba-inc.com",
        "time": "Mon Jun 08 14:25:00 2026 +0800"
      },
      "message": "[CELEBORN-2347] Change get reducer file group cache to expireAfterAccess\n\n### What changes were proposed in this pull request?\n\nChange get reducer file group cache to `expireAfterAccess`.\n\n### Why are the changes needed?\n\nCurrently the policy is expireAfterWrite which is not efficient, as it strictly clears the cache after the timeout, without considering that that entry was hot or not. `expireAfterAccess` will make sure to only clear if it was not actively not being accessed.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nExisting UTs.\n\nCloses #3717 from s0nskar/cache_policy.\n\nAuthored-by: Sanskar Modi \u003csanskarmodi97@gmail.com\u003e\nSigned-off-by: Shuang \u003clvshuang.xjs@alibaba-inc.com\u003e\n"
    },
    {
      "commit": "93de9b3042f419fa914144cd640e18e8995a3b8b",
      "tree": "643e58bbd6306bad39a633eeb87f29d0fd0ed975",
      "parents": [
        "b10546448657d430cc33d8607012a34135217498"
      ],
      "author": {
        "name": "afterincomparableyum",
        "email": "224495379+afterincomparableyum@users.noreply.github.com",
        "time": "Mon Jun 08 14:03:26 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Mon Jun 08 14:14:15 2026 +0800"
      },
      "message": "[CELEBORN-2332] Fix self join deadlock in C++ WorkerPartitionReader fetch callbacks\n\n### What changes were proposed in this pull request?\n\npush-merged data, which exercises this fetch path more aggressively and reliably triggered an EDEADLK abort.\n\nThe onSuccess_/onFailure_ callbacks are invoked on the TransportClient\u0027s IO thread and capture a weak_ptr that is lifted to a shared_ptr inside the callback body. When that local shared_ptr happens to hold the last reference, dropping it inline runs WorkerPartitionReader on the IO executor\u0027s own thread, which transitively destroys the embedded TransportClient and its IOThreadPoolExecutor. The executor then attempts to pthread_join the thread that is currently executing the callback and the join fails with EDEADLK, aborting the process.\n\nHand the final reference off to the global CPU executor so destruction of the reader (and the IO executor underneath it) always happens on a different thread than the one running the callback.\n\n### Why are the changes needed?\n\nWhen running the bytedance bolt celeborn e2e tests for push-merged data I am working on, I run into this error.\n\n### Does this PR resolve a correctness bug?\n\nYes\n\n### Does this PR introduce _any_ user-facing change?\n\nNo\n\n### How was this patch tested?\n\nI ran the Celeborn bolt e2e tests with this change and the tests passed with push merged data support.\n\nCloses #3693 from afterincomparableyum/celeborn-2332.\n\nAuthored-by: afterincomparableyum \u003c224495379+afterincomparableyum@users.noreply.github.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "b10546448657d430cc33d8607012a34135217498",
      "tree": "24139c7ea5c3cbf105f687043f2e35e5a6968a58",
      "parents": [
        "bd2523fa5900e90063cdf3916aa5e9e9d48a49e7"
      ],
      "author": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Fri Jun 05 10:42:12 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Fri Jun 05 10:42:12 2026 +0800"
      },
      "message": "[CELEBORN-2348] Support end-to-end shuffle integrity check for Flink\n\n### What changes were proposed in this pull request?\n\nThis PR extends the end-to-end shuffle integrity checks introduced in CELEBORN-894 (Spark-only) to Flink workloads, covering both the regular and the tiered (hybrid) read paths. When the check is enabled, the write side records a per subpartition CRC32 + byte count and the driver validates it against what the reader actually consumed, failing the read on a mismatch.\n\n- **Write side**: `FlinkShuffleClientImpl` hashes each push payload (the body after the batch header) into `PushState` via a zero-copy `ByteBuffer` view and reports the per-subpartition CRC32/bytes at `MapperEnd`, reusing the existing `crc32PerPartition` / `bytesWrittenPerPartition` plumbing. The constructor fails fast if the write-side `BATCH_HEADER_SIZE` ever diverges from the read-side `BufferUtils.HEADER_LENGTH_PREFIX`.\n- **Read side**: `RemoteBufferStreamReader` and `CelebornChannelBufferReader` accumulate the read CRC32/bytes through a shared `ReadIntegrityTracker` and report them at the last partition\u0027s stream end. The tracker owns the per-path framing/stripping and disables itself on any unexpected buffer shape (wrong component count, or a buffer shorter than the batch header) rather than risk a false mismatch.\n- **Driver side**: `ReadReducerPartitionEnd` is reused for MAP partitions, and `MapPartitionCommitHandler.finishPartition` combines the recorded write-side checksums over the consumed subpartition range, failing closed on a mismatch or missing metadata.\n- Add zero-copy `ByteBuffer` overloads to `CelebornCRC32`, `CommitMetadata` and `PushState`.\n- Minor: drop a stray (cosmetic, no-op) unary plus in `handleReducerPartitionEnd`\u0027s failure branch and update the client config doc.\n\n### Why are the changes needed?\n\nCELEBORN-894 added end-to-end integrity verification only for Spark. Flink workloads — including hybrid/tiered shuffle — had no equivalent guard, so silent shuffle data corruption (bit flips, truncation, mis-framing) could go undetected and surface as wrong results rather than a failed task. This PR brings the same write-vs-read checksum/byte-count validation to Flink so such corruption fails the read instead of being silently consumed.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes\n\nThe existing `celeborn.client.shuffle.integrityCheck.enabled` config now also applies to Flink (previously Spark-only); its documentation is updated accordingly. The default remains `false`, so there is no behavior change unless the check is explicitly enabled.\n\n### How was this patch tested?\n\nAdded unit and integration tests:\n\n- `CelebornCRC32Test` / `CommitMetadataTest`: the new `ByteBuffer` overloads (single and split header/data), order-independence, and corruption / byte-count-mismatch detection.\n- `MapPartitionCommitHandlerTest`: `finishPartition` success and all failure branches (no metadata, missing map partition, out-of-bounds range, checksum mismatch, byte-count mismatch), concurrent recording, and the expired-shuffle race.\n- `ReadIntegrityTrackerTest`: report-once / disable semantics and per-path framing for both the regular and tiered read paths.\n- `RemoteBufferStreamReaderTest`: the stream-end-after-close race (a failed report must not notify the failure listener on a closed channel).\n- `CelebornBufferStreamTest`: the `hasRemainingPartitions` location-index boundary.\n- `WordCountTest` (`WordCountTestWithIntegrityCheck`) and `HybridShuffleWordCountTest`: end-to-end Flink runs with the check enabled, on both the regular and hybrid shuffle paths.\n\nCloses #3718 from SteNicholas/CELEBORN-2348.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "bd2523fa5900e90063cdf3916aa5e9e9d48a49e7",
      "tree": "9b0eaa5e3d675c701ba42be8e5fd2d909cffcf12",
      "parents": [
        "bf0a1e8fc281d622fdf176076706d4dc4a52d80e"
      ],
      "author": {
        "name": "Sanskar Modi",
        "email": "sanskarmodi97@gmail.com",
        "time": "Fri Jun 05 10:39:25 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Fri Jun 05 10:39:25 2026 +0800"
      },
      "message": "[CELEBORN-1577][FOLLOWUP] Fix backward compatiblity issue with interrupt shuffle\n\n### What changes were proposed in this pull request?\n\nFix backward compatibility issue with interrupt shuffle by checking if the reason is nonEmpty.\n\n### Why are the changes needed?\n\nIf someone uses a new client with old server which is not sending `CheckQuotaResponse` in `HeartbeatFromApplicationResponse` then proto uses default value to build CheckQuotaResponse with isAvailable\u003dfalse and reason\u003d\"\". In this case the job will always fails without breaching the quota, we should not fail the job if the reason is empty to make it backward compatible.\n\n### Does this PR resolve a correctness bug?\n\nNo\n\n### Does this PR introduce _any_ user-facing change?\n\nNo\n\n### How was this patch tested?\n\nTested in local setup.\n\nCloses #3675 from s0nskar/fix_interrupt.\n\nAuthored-by: Sanskar Modi \u003csanskarmodi97@gmail.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "bf0a1e8fc281d622fdf176076706d4dc4a52d80e",
      "tree": "e8e42a80ea628463af9dc87c9325c5602e9fe150",
      "parents": [
        "221129bb858f14743412a1d7a25886540333d94e"
      ],
      "author": {
        "name": "Chao Sun",
        "email": "chao@openai.com",
        "time": "Wed Jun 03 17:38:19 2026 +0800"
      },
      "committer": {
        "name": "Nicholas Jiang",
        "email": "programgeek@163.com",
        "time": "Wed Jun 03 17:38:19 2026 +0800"
      },
      "message": "[CELEBORN-2346] Add RequestSlots failure metrics\n\n### What changes were proposed in this pull request?\n\nExport a new master `RequestSlotsFailed` counter with a bounded `status` label for `SLOT_NOT_AVAILABLE` and `WORKER_EXCLUDED`.\n\nThis patch also makes metric-specific labels override configured `celeborn.metrics.extraLabels`, while preserving reserved `role` and `instance` labels. Configured extra labels are snapshotted when a metrics source is constructed so labeled metric keys remain stable.\n\n### Why are the changes needed?\n\n`RequestSlots` placement failures directly affect shuffle registration, but existing master metrics do not expose whether applications are receiving these failure responses. The new counter provides a direct monitoring signal.\n\nWithout the label precedence change, an extra label such as `status\u003dprod` shadows the metric-specific failure reason and collapses both series.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes\n\n### How was this patch tested?\n\n- `JAVA_HOME\u003d$(brew --prefix openjdk17) dev/reformat`\n- `JAVA_HOME\u003d$(brew --prefix openjdk17) build/mvn --no-transfer-progress -DskipTests -DprotocPluginExecutable\u003d/tmp/protoc-gen-grpc-java-noop-proto3-optional -pl master -am install`\n- `JAVA_HOME\u003d$(brew --prefix openjdk17) build/mvn --no-transfer-progress -DprotocPluginExecutable\u003d/tmp/protoc-gen-grpc-java-noop-proto3-optional -Dsuites\u003dorg.apache.celeborn.common.metrics.source.CelebornSourceSuite -pl common test-compile scalatest:test`\n- `JAVA_HOME\u003d$(brew --prefix openjdk17) build/mvn --no-transfer-progress -DprotocPluginExecutable\u003d/tmp/protoc-gen-grpc-java-noop-proto3-optional -Dsuites\u003dorg.apache.celeborn.service.deploy.master.MasterSourceSuite,org.apache.celeborn.service.deploy.master.MasterSuite -pl master test-compile scalatest:test`\n\nThe local `protocPluginExecutable` override is an ARM workstation workaround for the downloaded `protoc-gen-grpc-java` artifact.\n\nCloses #3714 from sunchao/CELEBORN-2346-request-slots-failure-metrics.\n\nAuthored-by: Chao Sun \u003cchao@openai.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "221129bb858f14743412a1d7a25886540333d94e",
      "tree": "0b6de78cd91c785d74e9ec84077961c04b88cc87",
      "parents": [
        "c3bee9c5aaf0d05cc9f775401bf731f0d3ad72bb"
      ],
      "author": {
        "name": "Sanskar Modi",
        "email": "sanskarmodi97@gmail.com",
        "time": "Wed Jun 03 10:48:22 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Wed Jun 03 10:48:22 2026 +0800"
      },
      "message": "[CELEBORN-2343] Fix timer leak in handlePushData\n\n### What changes were proposed in this pull request?\n\nFix the timer leak in handlePushData\n\n### Why are the changes needed?\n\nTo avoid timer leak and publish correct `PRIMARY_PUSH_DATA_TIME` and `REPLICA_PUSH_DATA_TIME` metrics\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nNA\n\nCloses #3709 from s0nskar/fix_callback.\n\nAuthored-by: Sanskar Modi \u003csanskarmodi97@gmail.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "c3bee9c5aaf0d05cc9f775401bf731f0d3ad72bb",
      "tree": "61275f16b88af26547dcbb846d3c006e1c5e3f32",
      "parents": [
        "c79937379383b305d43d275eb1e2b9e73d9ab439"
      ],
      "author": {
        "name": "Sanskar Modi",
        "email": "sanskarmodi97@gmail.com",
        "time": "Tue Jun 02 19:57:39 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Tue Jun 02 19:57:39 2026 +0800"
      },
      "message": "[CELEBORN-2310][FOLLOWUP] Account actualUsableSpace while excluding workers\n\n### What changes were proposed in this pull request?\n\nAccount actualUsableSpace while excluding workers. We should use `DeviceInfo.isHealthy()` which actually account for available actualUsableSpace.\n\n### Why are the changes needed?\n\nCurrently we are just checking the DiskStatus, which could not be set if the device monitor is not enabled.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nMinor Change.\n\nCloses #3715 from s0nskar/disk_health.\n\nAuthored-by: Sanskar Modi \u003csanskarmodi97@gmail.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "c79937379383b305d43d275eb1e2b9e73d9ab439",
      "tree": "8d750a203eaebdbfc3e4cdef9cad7b7ce58cbbe7",
      "parents": [
        "2497739705629efd455a2e90f73f4e3a93ca0871"
      ],
      "author": {
        "name": "afterincomparableyum",
        "email": "224495379+afterincomparableyum@users.noreply.github.com",
        "time": "Tue Jun 02 19:47:04 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Tue Jun 02 19:47:04 2026 +0800"
      },
      "message": "[CELEBORN-2275][CIP-14] Add C++ merge-write and Java-read hybrid integration test\n\n### What changes were proposed in this pull request?\n\nAdd a new C++ test client that for the mergeData/pushMergedData write path and validates data integrity by reading back from the Java ShuffleClient. This complements the existing pushData based hybrid test by covering the merge write path.\n\n  - Add DataSumWithMergeWriterClient.cpp and its CMake build target\n  - Add CppMergeWriteJavaReadTest entry points for NONE, LZ4, and ZSTD compression codecs\n  - Add runCppMergeWriteJavaRead to JavaCppHybridReadWriteTestBase\n  - Update cpp_integration CI workflow to run the new tests\n\n### Why are the changes needed?\n\nThis is to add integration tests for https://github.com/apache/celeborn/pull/3611.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nTested through running unit tests and compiling locally.\n\nCloses #3619 from afterincomparableyum/cpp-client/celeborn-2275.\n\nLead-authored-by: afterincomparableyum \u003c224495379+afterincomparableyum@users.noreply.github.com\u003e\nCo-authored-by: afterincomparableyum \u003cafterincomparableyum\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "2497739705629efd455a2e90f73f4e3a93ca0871",
      "tree": "067670cec321bb36fffd1c53da6ab1c620ea772a",
      "parents": [
        "c546a21fbbddd559f28966cf0b82d48116b984f5"
      ],
      "author": {
        "name": "James Xu",
        "email": "xumingming@dewu.com",
        "time": "Mon Jun 01 15:00:30 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Mon Jun 01 15:00:30 2026 +0800"
      },
      "message": "[CELEBORN-2342] Fix object aliasing in LegacySkewHandlingPartitionValidator corrupting sub-range metadata\n\n### What changes were proposed in this pull request?\n\nWhen an AQE-skewed partition is split into N sub-ranges, the first sub-range\u0027s CommitMetadata object was stored by reference in both subRangeToCommitMetadataMap and currentCommitMetadataForReducer. Each subsequent sibling RPC mutated that object in-place via addCommitData(), silently inflating the TreeMap entry from bytes(A) to bytes(A)+bytes(siblings). Any task retry then sent the correct bytes(A) but found the inflated value, causing a permanent CelebornIOException mismatch and job abort.\n\n### Why are the changes needed?\n\nThis is a bug of E2E Integrity Check.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nAdded Unit Test.\n\nCloses #3708 from xumingming/fix/legacy-skew-validator-object-aliasing.\n\nAuthored-by: James Xu \u003cxumingming@dewu.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "c546a21fbbddd559f28966cf0b82d48116b984f5",
      "tree": "752190003f57a1e71b08db366f740e656a561c01",
      "parents": [
        "2d5098b5ef9f864a0b3aafcd8e06be3b4faf42b0"
      ],
      "author": {
        "name": "Sanskar Modi",
        "email": "sanskarmodi97@gmail.com",
        "time": "Mon Jun 01 14:55:05 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Mon Jun 01 14:55:05 2026 +0800"
      },
      "message": "[CELEBORN-2345] Fix allocation for rpcAskTimeout\n\n### What changes were proposed in this pull request?\n\n- Reuse the `rpcAskTimeout` variable in LifecycleManager\n- Make  `rpcAskTimeout` and `rpcRetryWait` lazy in `RpcEndpointRef`. Since many places are using `RpcEndpointRef` for just endpoint name. Examples are all the callers using `Dispatcher.postMessage`\n\n### Why are the changes needed?\n\n`rpcAskTimeout` and `rpcRetryWait` are causing 4% of total allocations.\n\n\u003cimg width\u003d\"644\" height\u003d\"379\" alt\u003d\"Screenshot 2026-05-29 at 3 23 53 PM\" src\u003d\"https://github.com/user-attachments/assets/d9d3d858-b56a-42c7-bdfb-cbf93cb18d07\" /\u003e\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nMinor change\n\nCloses #3712 from s0nskar/fix_conf_allocation.\n\nAuthored-by: Sanskar Modi \u003csanskarmodi97@gmail.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "2d5098b5ef9f864a0b3aafcd8e06be3b4faf42b0",
      "tree": "d62b3938aacc6c2cda96a29237174eb7d2358164",
      "parents": [
        "cf8d472718a597fa254bc4db5c38ec65abbdeaaf"
      ],
      "author": {
        "name": "Cheng Pan",
        "email": "chengpan@apache.org",
        "time": "Wed May 27 12:58:21 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Wed May 27 12:58:21 2026 +0800"
      },
      "message": "[CELEBORN-2339] Add tools.jar into classpath only for Java 8\n\n### What changes were proposed in this pull request?\n\nThis is an enhancement of CELEBORN-1682, limit the `tools.jar` injection only for Java 8.\n\n### Why are the changes needed?\n\nhttps://docs.oracle.com/en/java/javase/17/migrate/migrating-jdk-8-later-jdk-releases.html\n\n\u003e Class and resource files previously stored in lib/rt.jar, lib/tools.jar, lib/dt.jar and various other internal JAR files are stored in a more efficient format in implementation-specific files in the lib directory.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nEnsure the warning has gone on JDK 17.\n\n```\n\"WARNING: cannot locate tools.jar. Expected to find it in either /opt/openjdk-17/lib/tools.jar or /opt/openjdk-17/../lib/tools.jar\"\n```\n\nCloses #3703 from pan3793/CELEBORN-2339.\n\nAuthored-by: Cheng Pan \u003cchengpan@apache.org\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "cf8d472718a597fa254bc4db5c38ec65abbdeaaf",
      "tree": "42f72619479af923854dce8165666aebe39920c4",
      "parents": [
        "dfe7def07fabc4fca6fc786ab45edb02076929eb"
      ],
      "author": {
        "name": "Cheng Pan",
        "email": "chengpan@apache.org",
        "time": "Tue May 26 20:17:38 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Wed May 27 10:11:15 2026 +0800"
      },
      "message": "[CELEBORN-2338] Remove hardcoded CELEBORN_PRINT_LAUNCH_COMMAND\u003d0\n\n### What changes were proposed in this pull request?\n\nRemove hardcoded `CELEBORN_PRINT_LAUNCH_COMMAND\u003d0` in `bin/celeborn-class`\n\n### Why are the changes needed?\n\nIt should be picked from the environment variable.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\n```\n$ CELEBORN_PRINT_LAUNCH_COMMAND\u003d1 sbin/celeborn-cli --version\n...\nStart to launch /opt/java/openjdk/bin/java -XX:+IgnoreUnrecognizedVMOptions -cp /opt/celeborn/conf::/opt/celeborn/cli-jars/*: org.apache.celeborn.cli.CelebornCli --version\n...\n```\n\nCloses #3702 from pan3793/CELEBORN-2338.\n\nAuthored-by: Cheng Pan \u003cchengpan@apache.org\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "dfe7def07fabc4fca6fc786ab45edb02076929eb",
      "tree": "3d45fae17a9e967316594acb00cb17edb2335e4f",
      "parents": [
        "759e7b547641139576f21ddecbf1db5b42f83b01"
      ],
      "author": {
        "name": "Cheng Pan",
        "email": "chengpan@apache.org",
        "time": "Tue May 26 20:14:54 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Tue May 26 20:14:54 2026 +0800"
      },
      "message": "[CELEBORN-2337] Celeborn OpenAPI client should not shade slf4j-api\n\n### What changes were proposed in this pull request?\n\nAs the title, it\u0027s a packaging change.\n\n### Why are the changes needed?\n\nI found that `celeborn-cli` always prints such warnings, but actually the slf4j-api and log4j2 jars are correctly present in classpath.\n\n```\nSLF4J: Failed to load class \"org.slf4j.impl.StaticLoggerBinder\".\nSLF4J: Defaulting to no-operation (NOP) logger implementation\nSLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.\n```\n\nafter some investigation, I found that `celeborn-openapi-client-*.jar` bundles shaded slf4j classes, which causes the issue.\n\n```\n$ jar tf $CELEBORN_HOME/cli-jars/celeborn-openapi-client-*.jar | grep slf4j\n...\norg/apache/celeborn/shaded/org/slf4j/\norg/apache/celeborn/shaded/org/slf4j/ILoggerFactory.class\norg/apache/celeborn/shaded/org/slf4j/IMarkerFactory.class\norg/apache/celeborn/shaded/org/slf4j/Logger.class\n...\n```\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nTested with `celeborn-cli`, `SLF4J` binding warnings have gone.\n\nCloses #3701 from pan3793/CELEBORN-2337.\n\nAuthored-by: Cheng Pan \u003cchengpan@apache.org\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "759e7b547641139576f21ddecbf1db5b42f83b01",
      "tree": "0050140045d14ca915fc0640af8f38da85672746",
      "parents": [
        "180db931beb410bcfab80d9abb3be9de9aaf2553"
      ],
      "author": {
        "name": "Chao Sun",
        "email": "chao@openai.com",
        "time": "Tue May 26 10:21:44 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Tue May 26 10:21:44 2026 +0800"
      },
      "message": "[CELEBORN-2331] Parallelize batch open stream client creation\n\n## Why are the changes needed?\n\n`CelebornShuffleReader` batches stream-open requests by worker, but it previously created the data client for each worker serially before sending those already-parallel batch requests. When a reducer reads from multiple workers, connection setup for a slow or unavailable worker can delay useful work against the remaining healthy workers.\n\nParallelizing this setup removes the worker-by-worker wait from the normal path. Because this changes task-side connection scheduling, the optimization also needs an operational fallback that restores the prior behavior without requiring a code rollback.\n\n## What changes were proposed in this PR?\n\nThe reader now first gathers pending stream-open locations by worker address, then creates one data client per distinct worker concurrently using the existing stream-creator pool. Once client setup completes, it sends the existing `BATCH_OPEN_STREAM` requests only for workers with an available client, allowing healthy workers to proceed even if another worker fails during setup.\n\nThe client-creation phase preserves the prior retry behavior for later locations on the same worker when an earlier client attempt fails. It also handles task cancellation explicitly: if the waiting Spark task is interrupted, it restores the interrupt status and cancels unfinished setup work; worker-side interruption is propagated rather than treated as an ordinary retryable failure.\n\nThis optimization is controlled by `celeborn.client.spark.batch.openStream.parallelClientCreation.enabled`, which defaults to `true`. Setting it to `false` selects the original serial client-creation and request-building flow, giving deployments a targeted rollback switch if parallel connection setup causes unexpected operational behavior.\n\n## How was this PR tested?\n\n- Unit tests for parallel client setup, failure/retry handling, cancellation on interruption, and the new configuration default and override.\n- Configuration documentation generation validation for the new client setting.\n- Spotless formatting validation.\n\nCloses #3692 from sunchao/dev/chao/codex/port-pr72-to-oss-main.\n\nAuthored-by: Chao Sun \u003cchao@openai.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "180db931beb410bcfab80d9abb3be9de9aaf2553",
      "tree": "c7cbb71f43b70597816692edd30d6d93f4eec33d",
      "parents": [
        "1c807d99a16e59fca0022b1524ec158bfa869514"
      ],
      "author": {
        "name": "Yuriy Malygin",
        "email": "yuriy@malyg.in",
        "time": "Tue May 26 09:58:05 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Tue May 26 09:58:05 2026 +0800"
      },
      "message": "[CELEBORN-2326] Bump log4j-core version up to 2.25.4\n\n### What changes were proposed in this pull request?\nBump Log4j2 version from 2.24.3 to 2.25.4\n\n### Why are the changes needed?\nApache Celeborn currently depends on Apache Log4j Core versions affected by CVE-2026-34480.\n\n### Does this PR resolve a correctness bug?\nNo.\n\n### Does this PR introduce any user-facing change?\nNo.\n\n### How was this patch tested?\nCI.\n\nCloses #3684 from yuriymalygin/patch-1.\n\nAuthored-by: Yuriy Malygin \u003cyuriy@malyg.in\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "1c807d99a16e59fca0022b1524ec158bfa869514",
      "tree": "bf41091c8233c969fcff354a9ca6e2f54a2700dd",
      "parents": [
        "91175cecbdbdc5e5b209ac6622f6fe081bc7957a"
      ],
      "author": {
        "name": "Chao Sun",
        "email": "chao@openai.com",
        "time": "Mon May 25 13:59:36 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Mon May 25 13:59:36 2026 +0800"
      },
      "message": "[CELEBORN-2330] Fix HA master bootstrap redirect handling\n\n## Why are the changes needed?\n\nIn HA mode, clients can hit a failover window where the master they contact is no longer the leader and returns a `MasterNotLeaderException` with a suggested leader address. The client-facing symptom is an RPC failure surfaced as:\n\n`CelebornException: Exception thrown in awaitResult`\n\nThe redirect signal is still present underneath that wrapper, but the existing bootstrap and retry logic does not consistently preserve and follow it. In particular:\n\n- bootstrap-time redirects can be treated like generic connection failures instead of explicit leader hints\n- suggested leaders can themselves redirect again or fail setup\n- after such failures, the client may not continue cleanly to the remaining configured masters\n\nThe consequence is that a client can fail to establish a master connection during HA leader transitions even when a reachable leader or another configured master is available. That turns a recoverable redirect/failover event into an avoidable client-visible failure and makes rolling upgrades or leader changes noisier than necessary.\n\n## What changes were proposed in this PR?\n\nThis port brings the HA redirect handling fix from `openai/celeborn#70` onto upstream `main`.\n\nWhen a master tells the client which leader to use, the client now keeps that redirect information even if it is wrapped inside another exception, and it actively follows the suggested leader during bootstrap and failover. If that suggested leader points to another leader, the client can continue along that redirect chain instead of giving up too early.\n\nIf the redirect path is no longer useful, for example because the suggested leader cannot be reached, no leader is currently presented, or redirects start looping, the client falls back to the remaining configured masters and keeps searching for a usable endpoint. The retry scan also advances correctly after endpoint setup fails, so one bad redirect does not prevent the client from trying the next viable master.\n\nThe PR also adds focused HA tests that cover the bootstrap redirect cases, chained redirects, redirect cycles, and fallback to configured masters.\n\n## How was this PR tested?\n\n- `build/mvn test -pl common -am -Dtest\u003dMasterClientSuiteJ -DwildcardSuites\u003dorg.apache.celeborn.common.client.__NoSuchSuite__`\n\nCloses #3691 from sunchao/dev/chao/codex/port-pr70-to-oss-main.\n\nAuthored-by: Chao Sun \u003cchao@openai.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "91175cecbdbdc5e5b209ac6622f6fe081bc7957a",
      "tree": "2502139503eb11aeae043448805f223e1d684fa4",
      "parents": [
        "671ef2566196ad090a64eebda563f2efc75fb666"
      ],
      "author": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Mon May 25 12:55:40 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Mon May 25 12:55:40 2026 +0800"
      },
      "message": "[CELEBORN-2335] Bump Spark from 4.1.1 to 4.1.2\n\n### What changes were proposed in this pull request?\n\nBump Spark from 4.1.1 to 4.1.2.\n\n### Why are the changes needed?\n\nSpark 4.1.2 has been announced to release: [Spark 4.1.2 released](https://spark.apache.org/news/spark-4-1-2-released.html). The profile spark-4.1 could bump Spark from 4.1.1 to 4.1.2.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nCI.\n\nCloses #3700 from SteNicholas/CELEBORN-2335.\n\nAuthored-by: SteNicholas \u003cprogramgeek@163.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "671ef2566196ad090a64eebda563f2efc75fb666",
      "tree": "849fa3c499e247100d27fd6d4f92782e64e4342a",
      "parents": [
        "56444d2ea58ed98b0e6bc8612e8c816fb2406e63"
      ],
      "author": {
        "name": "Fei Wang",
        "email": "fwang12@ebay.com",
        "time": "Tue May 19 11:48:00 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Tue May 19 11:48:00 2026 +0800"
      },
      "message": "[CELEBORN-2328] Auto-apply correctness label from PR template checkbox\n\n### What changes were proposed in this pull request?\n\nReplace the free-text `Yes/No` comment under \"Does this PR resolve a correctness bug?\" with a single checkbox in the PR template. Add a GitHub Actions workflow (`correctness-label.yml`) that automatically adds or removes the `correctness` label based on whether the box is checked, triggered on every PR open/edit.\n\n### Why are the changes needed?\n\nPreviously the note said \"committer will add `correctness` label\" — a manual step that was easy to miss. This automates it: checking the box applies the label immediately, and unchecking it removes the label, with no committer action required.\n\nTo track all correctness PR: https://github.com/apache/celeborn/issues?q\u003dlabel%3Acorrectness\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nCI workflow logic verified by code review. End-to-end behavior can be confirmed by opening a test PR against the repo and toggling the checkbox.\n\n\u003cimg width\u003d\"1796\" height\u003d\"144\" alt\u003d\"image\" src\u003d\"https://github.com/user-attachments/assets/ccd25ac5-1b24-4d8d-ab10-e6df406d2843\" /\u003e\n\nCloses #3688 from turboFei/correctness.\n\nAuthored-by: Fei Wang \u003cfwang12@ebay.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "56444d2ea58ed98b0e6bc8612e8c816fb2406e63",
      "tree": "e273d501f3bd4416bc2c42b0f85a21ef7dc6e698",
      "parents": [
        "8b36b3d25106c35d09c52a0e5e1b6e97c0ebd292"
      ],
      "author": {
        "name": "Chao Sun",
        "email": "chao@openai.com",
        "time": "Tue May 19 02:41:10 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Tue May 19 02:41:10 2026 +0800"
      },
      "message": "[CELEBORN-2327] Add active-slot weight to load-aware placement\n\n## Why are the changes needed?\n\nCeleborn load-aware slot placement currently orders candidate disks using flush and fetch timing only. That can still keep assigning new partitions onto disks that already carry a large amount of reserved active-slot pressure, which makes placement skew worse under overlapping shuffle-heavy workloads. CELEBORN-2327 tracks this gap.\n\n## What changes were proposed in this PR?\n\n- Add an optional, default-off `celeborn.master.slot.assign.loadAware.activeSlotsWeight` config.\n- Include `activeSlots * activeSlotsWeight` in load-aware disk ordering.\n- Thread the new config through the master allocation path.\n- Document the new tuning knob and update the slot-allocation developer docs.\n- Add a regression test showing that, when configured, lower-active-slot disks are preferred over otherwise equivalent disks.\n\n## How was this PR tested?\n\n- `UPDATE\u003d1 build/mvn clean test -pl common -am -Dtest\u003dnone -DwildcardSuites\u003dorg.apache.celeborn.ConfigurationSuite`\n- `./build/mvn -pl master -am -Dtest\u003dSlotsAllocatorSuiteJ -DwildcardSuites\u003dorg.apache.celeborn.NoSuchSuite -DfailIfNoTests\u003dfalse test`\n- `./build/mvn -pl master -am -DskipTests test-compile`\n\nCloses #3685 from sunchao/dev/chao/codex/active-slots-placement-oss.\n\nAuthored-by: Chao Sun \u003cchao@openai.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "8b36b3d25106c35d09c52a0e5e1b6e97c0ebd292",
      "tree": "1f253db3e16b6b5d1319872b8daf930767c09a1b",
      "parents": [
        "f83350f49b9c2e1456f196736da8275367dcc6ee"
      ],
      "author": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Mon May 18 14:58:54 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Mon May 18 14:58:54 2026 +0800"
      },
      "message": "[CELEBORN-2333] Bump Flink from 2.2.0 to 2.2.1\n\n### What changes were proposed in this pull request?\n\nBump Flink from 2.2.0 to 2.2.1\n\n### Why are the changes needed?\n\nFlink 2.2.1 has been announced to release: [Apache Flink 2.2.1 Release Announcement](https://flink.apache.org/2026/05/15/apache-flink-2.2.1-release-announcement/). The profile flink-2.2 could bump Flink from 2.2.0 to 2.2.1.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nCI.\n\nCloses #3694 from SteNicholas/CELEBORN-2333.\n\nAuthored-by: SteNicholas \u003cprogramgeek@163.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "f83350f49b9c2e1456f196736da8275367dcc6ee",
      "tree": "00d55e86641aee3809d07c45df2cdc7608f094fc",
      "parents": [
        "ee7529f83a1994c2d8f92a5d12166939b7cce895"
      ],
      "author": {
        "name": "The Apache Software Foundation",
        "email": "root-asf-gitbox-commits@apache.org",
        "time": "Mon May 18 11:20:46 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Mon May 18 11:20:46 2026 +0800"
      },
      "message": "[INFRA] Set up default rulesets for default and release branches\n\nThis Pull Request enables the repository to conform with the \"sane default security settings\" of the Apache Software Foundation by configuring a default branch ruleset that protects the default branch and any release branches.\n\nNote that `~DEFAULT_BRANCH` is a GitHub symbolic link to the current default branch (HEAD) of the repository and does not need changing.\nIf the managing project does not wish to set up these defaults, please close this Pull Request. Alternatively, the project may merge this Pull Request to apply the changes immediately.\n\nIf no action is taken, this Pull Request will be automatically merged by the Apache Infrastructure team on **2026-06-14** (30 days from now).\n\nFor any further information, please reach us on Slack or at: usersinfra.apache.org\n\nCloses #3690 from asf-gitbox-commits/infrastructure-ruleset-bot/default-branch-protection.\n\nAuthored-by: The Apache Software Foundation \u003croot-asf-gitbox-commits@apache.org\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "ee7529f83a1994c2d8f92a5d12166939b7cce895",
      "tree": "640acd9d0a9b008eb5c29837315e5ac64bc425af",
      "parents": [
        "71a7d0afa21b807f360a9c5e3d21e45258a6c441"
      ],
      "author": {
        "name": "Saurabh Dubey",
        "email": "saurabhd336@uber.com",
        "time": "Mon May 18 10:53:33 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Mon May 18 10:53:33 2026 +0800"
      },
      "message": "[CELEBORN-2310] Reject RESERVE_SLOTS when disks are full\n\n### What changes were proposed in this pull request?\n\nDisk full only lead to HARD_SPLITs as a response to writes. However, doesn\u0027t lead to reserve slot rejections. This means too many write retries (due to HARD_SPLITs on each write attempt) leads to wasted network I/O. We can reject RESERVE_SLOT during disk full to avoid the wasted data write network IO.\n\n### Why are the changes needed?\n\nReject reserve slots during disk full, avoid unnecessary network IO.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nAdded UTs, CI.\n\nCloses #3666 from saurabhd336/diskFullReserveSlotsRejection.\n\nAuthored-by: Saurabh Dubey \u003csaurabhd336@uber.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "71a7d0afa21b807f360a9c5e3d21e45258a6c441",
      "tree": "895974f951981830714017b60dd738fd27662c4e",
      "parents": [
        "9ebbc6b36ea94b1b665954d699d76e4711c3dd94"
      ],
      "author": {
        "name": "Filip Darmanovic",
        "email": "dzeri96@proton.me",
        "time": "Thu May 14 09:56:37 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Thu May 14 09:56:37 2026 +0800"
      },
      "message": "[CELEBORN-2257] Add reporting of remote disks during registration\n\n### What changes were proposed in this pull request?\n1. Disks reported to the master on registration now include remote disks (HDFS, S3, OSS)\n2. Refactored method names to clarify difference between local and remote disks.\n3. Embedded disk type information into the enum.\n4. Refactored unnecessarily complicated code in the slot assignment and worker registration path.\n\n### Why are the changes needed?\n1. Before the first heartbeat, the master won\u0027t be able to assign slots from the remote disks on the worker.\n2. All other changes are in preparation for better support of remote disks.\n\n### Does this PR resolve a correctness bug?\nNot a correctness bug\n\n### Does this PR introduce _any_ user-facing change?\nNo\n\n### How was this patch tested?\n**Important**: I want help from the community on how to write tests for this.\n\nCloses #3597 from Dzeri96/CELEBORN-2257.\n\nAuthored-by: Filip Darmanovic \u003cdzeri96@proton.me\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "9ebbc6b36ea94b1b665954d699d76e4711c3dd94",
      "tree": "92e33cb96c1f01129ef8d86970e1b403978c6e9a",
      "parents": [
        "50323e1f323c9432692fcc65bc703be107395288"
      ],
      "author": {
        "name": "pithecuse527",
        "email": "gihong96@gmail.com",
        "time": "Wed May 13 15:35:43 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Wed May 13 15:35:43 2026 +0800"
      },
      "message": "[CELEBORN-2324] Fix JVMQuake threshold and JVMStat timer unit conversion\n\n### What changes were proposed in this pull request?\n\nThis PR fixes JVMQuake time accounting by preserving threshold config values as milliseconds and converting JVMStat GC timer tick deltas to nanoseconds before updating the token bucket.\n\n### Why are the changes needed?\n\nJVMQuake thresholds were parsed as milliseconds but wrapped as microseconds, making values such as 60s behave like 60ms. JVMStat GC timer metrics are reported in ticks, so using them directly can misaccount GC time.\n\n### Does this PR resolve a correctness bug?\n\nYes\n\n### Does this PR introduce _any_ user-facing change?\nYes\n\n### How was this patch tested?\n\n1. UT - Added unit coverage for JVMQuake threshold parsing and JVMStat tick-to-nanosecond conversion.\n2. E2E -  Verified the patched image in a Kubernetes spark namespace with JVMQuake enabled using `dump.threshold\u003d30s`, `kill.threshold\u003d60s`, and `runtimeWeight\u003d0`. Under repeated GC, the worker stayed Ready with restart count 0, while the original image terminated early because the configured `60s` threshold was effectively interpreted as `60ms`.\n  Verified the kill path with a low-threshold configuration: `dump.threshold\u003d30ms`, `kill.threshold\u003d60ms`, `runtimeWeight\u003d0`, and `check.interval\u003d100ms`. Under GC activity, the worker logged bucket: `62995087` and killThreshold: `60000000`, exited via JVMQuake, and Kubernetes restarted the pod, increasing the restart count from 0 -\u003e 1.\n\nCloses #3682 from pithecuse527/CELEBORN-2324.\n\nAuthored-by: pithecuse527 \u003cgihong96@gmail.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "50323e1f323c9432692fcc65bc703be107395288",
      "tree": "6274e395fe2655a7f615551013cc296cb80c93c6",
      "parents": [
        "e329b16ff70b13419e862acf2dcd7bc05d829ab6"
      ],
      "author": {
        "name": "AmandeepSingh285",
        "email": "mailto.amandeep.singh.28@gmail.com",
        "time": "Wed May 13 14:47:38 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Wed May 13 14:47:38 2026 +0800"
      },
      "message": "[CELEBORN-2316] Introduce metadata operation metrics\n\n## What changes were proposed in this pull request?\n\nIntroduce metrics around RocksDB operations. Metrics to have success and failure count for metadata operations for RocksDB observability.\n\n### Why are the changes needed?\n\nIntroduce metrics around RocksDB metadata operations. Current implementation, metadata operations do not have any observability added. RocksDB goes into a read only mode when any critical errors are encountered which results in all write operations failing. Observability around metadata operations is critical and failures could result in metadata entering an inconsistent state.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nTested in local staging setup -\n\n\u003cimg width\u003d\"2166\" height\u003d\"314\" alt\u003d\"Screenshot 2026-05-11 at 3 06 05 PM\" src\u003d\"https://github.com/user-attachments/assets/f6f75515-f937-436d-9ee3-de6ca83bfcdf\" /\u003e\n\n\u003cimg width\u003d\"1113\" height\u003d\"362\" alt\u003d\"image\" src\u003d\"https://github.com/user-attachments/assets/89cc0b40-e96c-420e-a5aa-36849fb15fd4\" /\u003e\n\nCloses #3673 from AmandeepSingh285/adding-metadata-metrics.\n\nLead-authored-by: AmandeepSingh285 \u003cmailto.amandeep.singh.28@gmail.com\u003e\nCo-authored-by: amandeeps.28 \u003camandeeps.28@uber.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "e329b16ff70b13419e862acf2dcd7bc05d829ab6",
      "tree": "78a1b72959d3f8a5fb6a7fab2b27437ae4969f32",
      "parents": [
        "a70c8fddc456aeedcf7a5bb94d91fd9e23be278e"
      ],
      "author": {
        "name": "1fanwang",
        "email": "1fannnw@gmail.com",
        "time": "Wed May 13 14:45:53 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Wed May 13 14:45:53 2026 +0800"
      },
      "message": "[CELEBORN-2253] Fix IndexOutOfBoundsException reading shuffle data from HDFS\n\n### What changes were proposed in this pull request?\n\n`HdfsFlushTask.writeAndRecordMetrics` calls `hdfsStream.write(bytes)`, which writes the full `bytes.length`. When the provider passes a reusable `copyBytes` buffer (whose length is `\u003e\u003d size`), this leaks trailing bytes from previous flushes into the current partition file. Pass the actual readable size to write only `size` bytes.\n\n### Why are the changes needed?\n\nThe S3 and OSS flush paths had the same bug and were fixed in #3600 for CELEBORN-2263; the HDFS path was missed. Without the fix, shuffle data flushed to HDFS can be corrupted when `copyBytes` is reused across flushes, and readers later fail with `IndexOutOfBoundsException` in `CelebornInputStream.fillBuffer`, for example:\n\n```\nIndexOutOfBoundsException: readerIndex(4154253) + length(808530018)\n  exceeds writerIndex(12457470)\n```\n\n### Does this PR resolve a correctness bug?\n\nYes.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nNew unit test in `FlushTaskSuite` mirrors the S3/OSS coverage added in #3600. It drives `HdfsFlushTask.flush` with `copyBytes` arrays of three sizes (equal, larger, smaller than the buffer payload), captures the `FSDataOutputStream.write` arguments via Mockito\u0027s `ArgumentCaptor`, and asserts the offset/length pair matches the buffer content. The test fails on master with `ArgumentsAreDifferent` at `FlushTask.scala:128` and passes with the fix.\n\nCloses #3683 from 1fanwang/CELEBORN-2253-fix-hdfs-flush-trailing-bytes.\n\nAuthored-by: 1fanwang \u003c1fannnw@gmail.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "a70c8fddc456aeedcf7a5bb94d91fd9e23be278e",
      "tree": "6f815899a0f88fc3d2b0ff82bbf18a14a4eb3cb0",
      "parents": [
        "886e359d5e5a12698c1ed63f181f1f8a086c754b"
      ],
      "author": {
        "name": "Kartikay Bhutani",
        "email": "kbhutani0001@gmail.com",
        "time": "Tue May 12 10:19:34 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Tue May 12 10:19:34 2026 +0800"
      },
      "message": "[CELEBORN-2306] Master adds shutdown hook for ratis stepdown\n\n### What changes were proposed in this pull request?\n\n- Adds master shutdown hook\n- Updates RAFT stepdown to return bool\n- Calls RAFT stepdown on manager shutdown to do graceful stepdown\n\n### Why are the changes needed?\n\n- Manager.stop() isnt being called from anywhere, it emmits some logs as well for \"Stopping manager\" but since there is no shutdown hook defined, none of them are logged or the function is called at all\n- We faced a certain issue where the leader got removed from service mesh before shutdown and followers redirected to it because it was still running (able to send requests but not receive) for a brief period. This method adds a graceful shutdown option to do a RATIS stepdown before shutting down.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes, adds an additional config.\n\n### How was this patch tested?\n\nAdded a tests and validated that.\n\nCloses #3659 from kaybhutani/kartikay/graceful-master-shutdown.\n\nLead-authored-by: Kartikay Bhutani \u003ckbhutani0001@gmail.com\u003e\nCo-authored-by: Zaynt \u003cshuaizhentao@gmail.com\u003e\nCo-authored-by: kartikay \u003ckbhutani0001@gmail.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "886e359d5e5a12698c1ed63f181f1f8a086c754b",
      "tree": "cbcb40d79ba9a2a954fe614b2f6369d29b1f73ac",
      "parents": [
        "7340428ba0849fed00e0575efd5fc3c69e79badf"
      ],
      "author": {
        "name": "afterincomparableyum",
        "email": "224495379+afterincomparableyum@users.noreply.github.com",
        "time": "Mon May 11 17:48:20 2026 +0800"
      },
      "committer": {
        "name": "子懿",
        "email": "programgeek@163.com",
        "time": "Mon May 11 17:48:20 2026 +0800"
      },
      "message": "[CELEBORN-2314] Optimize the performance of DataBatches.requireBatches\n\n### What changes were proposed in this pull request?\n\n`requireBatches(int requestSize)` currently calls `batches.remove(0)` per iteration, which shifts all remaining elements each time, overall O(kn). Replacing with a two pass approach (find split point, then subList(0, count).clear()) reduces this to O(n).\n\n### Why are the changes needed?\n\nThis is a minor performance optimization.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo, this is just a performance improvement.\n\n### How was this patch tested?\n\nCI Unit/Integration tests.\n\nCloses #3671 from afterincomparableyum/celeborn-2314.\n\nAuthored-by: afterincomparableyum \u003c224495379+afterincomparableyum@users.noreply.github.com\u003e\nSigned-off-by: 子懿 \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "7340428ba0849fed00e0575efd5fc3c69e79badf",
      "tree": "1c8ed4ff7b2ea707f342e6b1f9d2ba90e95de4e3",
      "parents": [
        "70ff956d049ab071d6ff93ff55acc4fea0a46635"
      ],
      "author": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Mon May 11 09:41:50 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Mon May 11 09:41:50 2026 +0800"
      },
      "message": "[CELEBORN-2322] Upgrade version of docker/login-action for Login to Docker Hub\n\n### What changes were proposed in this pull request?\n\nUpgrade version of docker/login-action for Login to Docker Hub to `docker/login-action4907a6ddec9925e35a0a9e82d7399ccc52663121`.\n\n### Why are the changes needed?\n\nThere is error of dockerhub login in https://github.com/apache/celeborn/actions/runs/24821432736, which is as follows:\n\n```\nThe action docker/login-actionv3 is not allowed in apache/celeborn because all actions must be from a repository owned by your enterprise, created by GitHub, or match one of the patterns: 1Password/load-secrets-action13f58eec611f8e5db52ec16247f58c508398f3e6, 1Password/load-secrets-action8d0d610af187e78a2772c2d18d627f4c52d3fbfb, 1Password/load-secrets-action92467eb28f72e8255933372f1e0707c567ce2259, 1Password/load-secrets-actiondafbe7cb03502b260e2b2893c753c352eee545bf, AdoptOpenJDK/install-jdk*, BobAnkh/auto-generate-changelog*, DavidAnson/markdownlint-cli2-action07035fd053f7be764496c0f8d8f9f41f98305101, DavidAnson/markdownlint-cli2-actionce4853d43830c74c1753b39f3cf40f71c2031eb9, EnricoMi/publish-unit-test-result-action*, JamesIves/github-pages-deploy-action4a3abc783e1a24aeb44c16e869ad83caf6b4cc23, JamesIves/github-pages-deploy-actiond92aa235d04922e8f08b40ce78cc5442fcfbfa2f, JetBrains/qodana-action89eb4357efd2b52e639f3216e63edaf33b82622b, Jimver/cuda-toolkit3d45d157f327c...\n```\n\n[INFRA-27901](https://issues.apache.org/jira/projects/INFRA/issues/INFRA-27901) gives the following suggestion:\n\n\u003e Following the Trivy compromise, more controls have been put in place regarding use of third party actions.\n\u003e\n\u003e The only allowed versions of this action are those in the repo:\n\u003e\n\u003e https://github.com/apache/infrastructure-actions\n\u003e\n\u003e In: https://raw.githubusercontent.com/apache/infrastructure-actions/refs/heads/main/actions.yml at the moment you can use :\n\u003e\n\u003e - docker/login-actionc94ce9fb468520275223c153574b00df6fe4bcc9\n\u003e - docker/login-actionb45d80f862d83dbcd57f89517bcf500b2ab88fb2\n\u003e - docker/login-action4907a6ddec9925e35a0a9e82d7399ccc52663121\n\u003e\n\u003e which correspond to these tagged versions:\n\u003e\n\u003e docker/login-action:\n\u003e   c94ce9fb468520275223c153574b00df6fe4bcc9:\n\u003e     tag: v3.7.0\n\u003e     expires_at: 2026-06-14\n\u003e   b45d80f862d83dbcd57f89517bcf500b2ab88fb2:\n\u003e     tag: v4.0.0\n\u003e     expires_at: 2026-07-05\n\u003e   4907a6ddec9925e35a0a9e82d7399ccc52663121:\n\u003e     tag: v4.1.0\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nNo.\n\nCloses #3681 from SteNicholas/CELEBORN-2322.\n\nAuthored-by: SteNicholas \u003cprogramgeek@163.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "70ff956d049ab071d6ff93ff55acc4fea0a46635",
      "tree": "4ec39101e0bff5d72cf9181672fc6a778a04dfac",
      "parents": [
        "8d473c5af5a754a5b0329a805298ce9e8f0d27e7"
      ],
      "author": {
        "name": "Chao Sun",
        "email": "chao@openai.com",
        "time": "Sun May 10 12:35:50 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Sun May 10 12:35:50 2026 +0800"
      },
      "message": "[CELEBORN-2321] Avoid locking disk writers during memory split checks\n\n### Why are the changes needed?\n\n`needHardSplitForMemoryShuffleStorage()` runs on the push path. Disk-backed writers can never require this memory-only split check, but the method currently acquires the writer lock before returning `false`. For the common disk-backed case, that adds avoidable contention with writes and evictions on a hot path.\n\n### What changes were proposed in this PR?\n\nThis PR adds an unlocked fast path for non-memory writers so they return immediately without taking the `PartitionDataWriter` monitor. For memory-backed writers, it rechecks `currentTierWriter` after entering the synchronized block before evaluating the existing hard-split conditions, which preserves the original behavior if the writer tier changes concurrently.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\n- Attempted `build/mvn -pl worker -am -DskipTests compile` on current `main`.\n- The Maven reactor fails before reaching `worker` because `celeborn-master_2.12` cannot resolve snapshot test-jar artifacts for `celeborn-common_2.12` and `celeborn-service_2.12`; that failure is unrelated to this change.\n\nCloses #3680 from sunchao/dev/chao/codex/celeborn-fast-memory-split-check-oss-main.\n\nAuthored-by: Chao Sun \u003cchao@openai.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "8d473c5af5a754a5b0329a805298ce9e8f0d27e7",
      "tree": "9e99c3d43bb897777dd9e364020dddd682aff94b",
      "parents": [
        "fc087567442427cb1b3b161d65a7d3eb7080c345"
      ],
      "author": {
        "name": "Kartikay Bhutani",
        "email": "kbhutani0001@gmail.com",
        "time": "Fri May 08 15:42:51 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Fri May 08 15:42:51 2026 +0800"
      },
      "message": "[CELEBORN-2318] Miss increment to WRITE_DATA_HARD_SPLIT_COUNT on returning HARD_SPLIT in handlePushData\n\n### What changes were proposed in this pull request?\nMissing increment to `WRITE_DATA_HARD_SPLIT_COUNT` on returning HARD_SPLIT\n\n### Why are the changes needed?\n- The post-restart detection branch in `handlePushData` (Case2: shuffleKey in storageManager but not in shuffleMapperAttempts) returns HARD_SPLIT without incrementing `WRITE_DATA_HARD_SPLIT_COUNT`\n- The sibling Case1 branch (line 398) and all other HARD_SPLIT return paths already increment it\n- This makes Case2 invisible to monitoring during rolling restarts\n\n### Does this PR resolve a correctness bug?\nNo\n\n### Does this PR introduce _any_ user-facing change?\nNo\n\n### How was this patch tested?\nExisting UTs\n\nCloses #3676 from kaybhutani/kartikay/missing-hard-split-metric.\n\nAuthored-by: Kartikay Bhutani \u003ckbhutani0001@gmail.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "fc087567442427cb1b3b161d65a7d3eb7080c345",
      "tree": "4d19d6137a32b397790c76cfe251c0053b3e2175",
      "parents": [
        "69df893b4133d8594680a78396cc21171fe29a14"
      ],
      "author": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Fri May 08 15:41:12 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Fri May 08 15:41:12 2026 +0800"
      },
      "message": "[CELEBORN-2320] Grafana dashboard linter should build from a checkout instead\n\n### What changes were proposed in this pull request?\n\nGrafana dashboard linter should build from a checkout instead.\n\nBackport https://github.com/grafana/dashboard-linter/pull/252.\n\n### Why are the changes needed?\n\nThis PR introduces [GoReleaser](https://github.com/goreleaser/goreleaser-action) to help create GitHub releases and compiled binaries.\n\nThe new release workflow will trigger on git tags starting with v.\n\nIn particular, providing pre-built release binaries addresses the following problem currently seen in main:\n\n```\ngo: downloading github.com/grafana/dashboard-linter v0.1.0\ngo: github.com/grafana/dashboard-linterlatest (in github.com/grafana/dashboard-linterv0.1.0):\n\tThe go.mod file for the module providing named packages contains one or\n\tmore replace directives. It must not contain directives that would cause\n\tit to be interpreted differently than if it were the main module.\n```\n\n`go install github.com/grafana/dashboard-linter\u003cversion\u003e` does not currently work because `go.mod` contains a `replace` directive — Go refuses to install a module with replaces. Build from a checkout instead:\n\n```\n$ git clone https://github.com/grafana/dashboard-linter.git\n$ cd dashboard-linter\n$ go build -o dashboard-linter .\n$ ./dashboard-linter lint dashboard.json\n```\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\n`Grafana Dashboard CI / lint (pull_request)`.\n\nCloses #3679 from SteNicholas/CELEBORN-2320.\n\nAuthored-by: SteNicholas \u003cprogramgeek@163.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "69df893b4133d8594680a78396cc21171fe29a14",
      "tree": "36b50e89756ac8b7609e0ffcb55afe2c39d9ff88",
      "parents": [
        "59fd7a8402364a1b82a8480dc33e7a5722ec6d2f"
      ],
      "author": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Wed Apr 22 14:19:35 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Wed Apr 22 14:19:35 2026 +0800"
      },
      "message": "[CELEBORN-2258][FOLLOWUP] Use IoHandlerFactories for EventLoopGroups to replace deprecated transport-specific event loop groups\n\n### What changes were proposed in this pull request?\n\nUse IoHandlerFactories for EventLoopGroups to replace deprecated transport-specific event loop groups.\n\nBackport: https://github.com/apache/spark/pull/52719.\n\n### Why are the changes needed?\n\nNetty 4.2 introduces some new APIs, and deprecates some old APIs. As part of your migration to Netty 4.2, we encourage you to look through your code base for opportunities to clean up any use of deprecated APIs.\n\n- **IoHandlerFactories for EventLoopGroups**\n\nAll transport-specific event loop groups, such as `NioEventLoopGroup`, have been deprecated. Integrators should now instead pass a transport-specific `IoHandlerFactory` to a `MultiThreadedEventLoopGroup` constructor.\n\nTherefore, Netty 4.2 upgrade could follow the best practices from https://netty.io/wiki/netty-4.2-migration-guide.html#new-best-practices.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nCI.\n\nCloses #3669 from SteNicholas/CELEBORN-2258.\n\nAuthored-by: SteNicholas \u003cprogramgeek@163.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "59fd7a8402364a1b82a8480dc33e7a5722ec6d2f",
      "tree": "ded487669ea9ff9de82d9b10512a6a5c3f1fc6f5",
      "parents": [
        "a56f69ae0abdf4e375b442aebfc6843fc8520bc9"
      ],
      "author": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Wed Apr 22 10:44:24 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Wed Apr 22 10:44:24 2026 +0800"
      },
      "message": "[CELEBORN-2309] Introduce JavaDeserializerFilter to support deserialization filter\n\n### What changes were proposed in this pull request?\n\nIntroduce `JavaDeserializerFilter` — an allowlist-only deserialization filter that prevents CWE-502 (Deserialization of Untrusted Data) attacks on Celeborn\u0027s internal RPC channel.\n\nKey design points:\n- **Dual-layer defense:** On JDK 9+, both a `resolveClass`-based class allowlist and JVM-level `ObjectInputFilter` (resource limits: maxdepth, maxarray, maxrefs, maxbytes) are enforced. On JDK 8, only the `resolveClass` allowlist is active.\n- **Reflection-based JDK compatibility:** Uses reflection to access `java.io.ObjectInputFilter` APIs, gracefully degrading on JDK 8 where the API does not exist.\n- **Minimal overhead:** Filter pattern and logging proxy are built once at construction time; hot-path `isClassAllowed` is a simple `String.startsWith` loop over a `String[]` — no streams, no allocation.\n- **Configurable via CelebornConf:** Enabled by default with sensible defaults; operators can customize allowed packages and resource limits without code changes.\n\nDefault allowed package prefixes: `java.`, `scala.`, `org.apache.celeborn.`, `com.google.protobuf.`, `[` (arrays).\n\n### Why are the changes needed?\n\nCeleborn\u0027s internal RPC between Master, Workers, and clients uses Java serialization (`JavaSerializer`). Without a deserialization filter, an attacker with network access to RPC ports can craft a malicious serialized payload containing gadget-chain classes (e.g., from commons-collections, Spring, etc.) to achieve Remote Code Execution.\n\nThis is a well-known attack vector (CWE-502 / OWASP A8:2017). Adding an allowlist-only filter ensures that only classes from trusted packages can be deserialized, blocking arbitrary gadget-chain exploitation regardless of which libraries are on the classpath.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce any user-facing change?\n\nNo. The filter is transparent to existing functionality — all legitimate Celeborn RPC classes are under `org.apache.celeborn.` which is allowed by default. Operators gain new configuration knobs if they need to extend the allowlist:\n\n| Config Key | Default |\n|-----------|---------|\n| `celeborn.serializer.deserialization.filter.enabled` | `true` |\n| `celeborn.serializer.deserialization.filter.allowedPackages` | `java.,jdk.,sun.,scala.,org.apache.celeborn.,com.google.protobuf.,[` |\n| `celeborn.serializer.deserialization.filter.maxDepth` | `100` |\n| `celeborn.serializer.deserialization.filter.maxArrayLength` | `10000` |\n| `celeborn.serializer.deserialization.filter.maxReferences` | `100000` |\n| `celeborn.serializer.deserialization.filter.maxStreamBytes` | `100000000` |\n\n### How was this patch tested?\n\n- `JavaDeserializerFilterSuiteJ`: unit tests covering default allowlist, custom allowlist, valid deserialization round-trip, and rejection of blocked classes via crafted serialization payload.\n- Manual integration testing on JDK 8 (graceful degradation) and JDK 11/17 (full ObjectInputFilter enforcement).\n\nCloses #3664 from SteNicholas/CELEBORN-2309.\n\nAuthored-by: SteNicholas \u003cprogramgeek@163.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "a56f69ae0abdf4e375b442aebfc6843fc8520bc9",
      "tree": "9b677ce255803c2d8976c913fd2b86247576d966",
      "parents": [
        "95419e14a0d1986a4582a6364057193dce296992"
      ],
      "author": {
        "name": "Shuang",
        "email": "lvshuang.xjs@alibaba-inc.com",
        "time": "Thu Apr 16 13:52:22 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Thu Apr 16 13:52:22 2026 +0800"
      },
      "message": "[MINOR] Update DingTalk Contact Info\n\n### What changes were proposed in this pull request?\n\nUpdate DingTalk contact information.\n\n### Why are the changes needed?\n\nDingTalk contact information has been expired.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nNo.\n\nCloses #3665 from RexXiong/UPDATE_DINGTALK_INFO.\n\nAuthored-by: Shuang \u003clvshuang.xjs@alibaba-inc.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "95419e14a0d1986a4582a6364057193dce296992",
      "tree": "50733b57b8b1f97dc32ec2fcde5d606baac8bcc6",
      "parents": [
        "149f3b98b8500627c2fd6432d5fb0cabf314fd9b"
      ],
      "author": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Wed Apr 15 19:10:11 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Wed Apr 15 19:10:11 2026 +0800"
      },
      "message": "[CELEBORN-2301] MessageEncoder enables zero-copy sendfile for FileRegion in Netty native transports\n\n### What changes were proposed in this pull request?\n\n`MessageEncoder` enables zero-copy sendfile for `FileRegion` in Netty native transports, which emits the header `ByteBuf` and `FileRegion` as **separate objects** in the outbound message list when the body is a `FileRegion` backed by `FileSegmentManagedBuffer`, instead of wrapping them together in a `MessageWithHeader`.\n\nPreviously, all messages with a body were unconditionally wrapped in `MessageWithHeader`. This caused native transports (EPOLL, KQUEUE) to fall into a generic `FileRegion.transferTo()` fallback path that copies data through user-space, bypassing the optimized `sendfile()` / `splice()` zero-copy path that Netty\u0027s native transports provide.\n\nThe split is only applied when the `ManagedBuffer` is a `FileSegmentManagedBuffer`, whose `release()` is a no-op, making it safe to emit the `FileRegion` independently of write lifecycle management. Other `ManagedBuffer` types (e.g., `BlockManagerManagedBuffer`) still use the `MessageWithHeader` wrapper because they perform resource cleanup in `release()` that must be tied to `MessageWithHeader.deallocate()`.\n\nBackport: https://github.com/apache/spark/pull/55087.\n\n### Why are the changes needed?\n\nWhen using native transports (AUTO/EPOLL on Linux), file-backed shuffle fetch performance was severely degraded compared to NIO mode. The root cause lies in how Netty\u0027s native transports dispatch `FileRegion` writes.\n\nIn `AbstractEpollStreamChannel.doWriteSingle()` (and the analogous KQueue path), Netty uses an `instanceof` check to choose between two write strategies:\n\nhttps://github.com/netty/netty/blob/eeb5674526f0b49a142580686a5a9a7147ddadec/transport-classes-epoll/src/main/java/io/netty/channel/epoll/AbstractEpollStreamChannel.java#L474-L493\n\n```java\n} else if (msg instanceof DefaultFileRegion) {\n    return writeDefaultFileRegion(in, (DefaultFileRegion) msg);  // → socket.sendFile() (zero-copy)\n} else if (msg instanceof FileRegion) {\n    return writeFileRegion(in, (FileRegion) msg);                // → region.transferTo() (user-space copy)\n}\n```\n\n- **`writeDefaultFileRegion()`** calls `socket.sendFile()`, which maps directly to the Linux `sendfile()` syscall — a true zero-copy path where data is transferred from the file page cache to the socket buffer entirely within the kernel, with no user-space copy.\n\nhttps://github.com/netty/netty/blob/eeb5674526f0b49a142580686a5a9a7147ddadec/transport-classes-epoll/src/main/java/io/netty/channel/epoll/AbstractEpollStreamChannel.java#L367-L386\n\n```java\n    private int writeDefaultFileRegion(ChannelOutboundBuffer in, DefaultFileRegion region) throws Exception {\n        final long offset \u003d region.transferred();\n        final long regionCount \u003d region.count();\n        if (offset \u003e\u003d regionCount) {\n            in.remove();\n            return 0;\n        }\n\n        final long flushedAmount \u003d socket.sendFile(region, region.position(), offset, regionCount - offset);\n        if (flushedAmount \u003e 0) {\n            in.progress(flushedAmount);\n            if (region.transferred() \u003e\u003d regionCount) {\n                in.remove();\n            }\n            return 1;\n        } else if (flushedAmount \u003d\u003d 0) {\n            validateFileRegion(region, offset);\n        }\n        return WRITE_STATUS_SNDBUF_FULL;\n    }\n```\n\n- **`writeFileRegion()`** falls back to `region.transferTo(WritableByteChannel)`, which writes data through a `SocketWritableByteChannel` wrapper — effectively a user-space copy path.\n\nhttps://github.com/netty/netty/blob/eeb5674526f0b49a142580686a5a9a7147ddadec/transport-classes-epoll/src/main/java/io/netty/channel/epoll/AbstractEpollStreamChannel.java#L402-L420\n\n```java\nprivate int writeFileRegion(ChannelOutboundBuffer in, FileRegion region) throws Exception {\n        if (region.transferred() \u003e\u003d region.count()) {\n            in.remove();\n            return 0;\n        }\n\n        if (byteChannel \u003d\u003d null) {\n            byteChannel \u003d new EpollSocketWritableByteChannel();\n        }\n        final long flushedAmount \u003d region.transferTo(byteChannel, region.transferred());\n        if (flushedAmount \u003e 0) {\n            in.progress(flushedAmount);\n            if (region.transferred() \u003e\u003d region.count()) {\n                in.remove();\n            }\n            return 1;\n        }\n        return WRITE_STATUS_SNDBUF_FULL;\n    }\n```\n\nSpark\u0027s `MessageWithHeader extends AbstractFileRegion` (not `DefaultFileRegion`). When `MessageEncoder` wraps a `DefaultFileRegion` body inside `MessageWithHeader`, the resulting object is a generic `FileRegion` from Netty\u0027s perspective. This means Netty dispatches it to the `writeFileRegion()` fallback, which calls `MessageWithHeader.transferTo()`:\n\n```java\n// MessageWithHeader.java, line 121\nif (body instanceof FileRegion fileRegion) {\n    writtenBody \u003d fileRegion.transferTo(target, totalBytesTransferred - headerLength);\n}\n```\n\nHere, even though the inner body is a `DefaultFileRegion`, its `transferTo()` is invoked with a `WritableByteChannel` (not a file descriptor), so the data is read from the file into a user-space buffer and then written to the socket — **the zero-copy opportunity is lost**.\n\nBy emitting the `DefaultFileRegion` directly into Netty\u0027s outbound buffer (instead of wrapping it in `MessageWithHeader`), Netty\u0027s native transport recognizes it via `instanceof DefaultFileRegion` and routes it to `socket.sendFile()`, restoring the zero-copy `sendfile()` path.\n\n**Benchmark results (File-Backed Shuffle Fetch) show dramatic improvement:**\n\n| Scenario | Before (ms) | After (ms) | Improvement |\n|---|---|---|---|\n| EPOLL, sequential fetch (JDK8) | 524 | 134 | **~3.9x faster** |\n| EPOLL, parallel fetch (JDK8) | 191 | 55 | **~3.4x faster** |\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo. This is an internal optimization to the Netty transport layer. Users benefit from improved shuffle fetch performance when using native transports (the default on Linux) without any configuration changes.\n\n### How was this patch tested?\n\n- Re-ran `NettyTransportBenchmark` with JDK8 to confirm the performance improvement. Updated benchmark result files accordingly.\n\nCloses #3649 from SteNicholas/CELEBORN-2301.\n\nLead-authored-by: SteNicholas \u003cprogramgeek@163.com\u003e\nCo-authored-by: Cheng Pan \u003cchengpan@apache.org\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "149f3b98b8500627c2fd6432d5fb0cabf314fd9b",
      "tree": "276c06da6a28836edc202846d30e9ffb497bd48b",
      "parents": [
        "c456df3ec6a4611895dcc7dfe4167ef1e261560c"
      ],
      "author": {
        "name": "Sanskar Modi",
        "email": "sanskarmodi97@gmail.com",
        "time": "Wed Apr 15 15:07:35 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Wed Apr 15 15:07:35 2026 +0800"
      },
      "message": "[CELEBORN-1577][BUG] Quota cancel shuffle should use app shuffle id\n\n### What changes were proposed in this pull request?\n\n- Added a new mapping for celebornShuffleId -\u003e appShuffleId\n- cancelAllActiveStages should passing appShuffleId not celebornShuffleId\n\n### Why are the changes needed?\n\n`shuffleAllocatedWorkers` worker contains celebornShuffleId, we need to use `appShuffleId` because DAGScheduler only understand app shuffle id.\n\n### Does this PR resolve a correctness bug?\n\nNo\n\n### Does this PR introduce _any_ user-facing change?\n\nNo\n\n### How was this patch tested?\n\nNA\n\nCloses #3662 from s0nskar/fix_quota_shuffle_id.\n\nAuthored-by: Sanskar Modi \u003csanskarmodi97@gmail.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "c456df3ec6a4611895dcc7dfe4167ef1e261560c",
      "tree": "222b50934eec1eb54bd9e91a7fdf21dd03abfc02",
      "parents": [
        "913d027efd1e56d080ec8080818e05ccf8eaa025"
      ],
      "author": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Mon Apr 13 10:26:56 2026 +0700"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Mon Apr 13 10:26:56 2026 +0700"
      },
      "message": "[CELEBORN-2298] Introduce NettyTransportBenchmark for Netty transport layer performance evaluation\n\n### What changes were proposed in this pull request?\n\nIntroduce `NettyTransportBenchmark` for Netty transport layer performance evaluation.\n\nAll suites measure performance through the actual Celeborn transport pipeline\n(`TransportServer` + `TransportClientFactory` + `TransportContext`).\n\nSuite overview:\n1. RPC Latency            - server-client RPC overhead at different payload sizes\n2. Concurrent Throughput  - multi-client pressure on the transport layer\n3. IOMode Comparison      - NIO vs native transport (Automatically selects EPOLL/KQUEUE)\n4. Server Thread Scaling  - validates MAX_DEFAULT_NETTY_THREADS\u003d8 cap\n5. Multi-Connection       - numConnectionsPerPeer\u003d1 vs 2 vs 4\n6. Async Write Pressure   - fire-and-forget RPCs to saturate the write path\n7. Large Block Transfer   - shuffle-like 16MB block transfers (in-memory payload)\n8. File-Backed Shuffle    - ChunkFetch from disk, NIO vs native transport (EPOLL sendfile bypass detection)\n\nBackport: https://github.com/apache/spark/pull/55061.\n\n### Why are the changes needed?\n\nNetty is a crucial third-party component for Celeborn. Introduce a micro-benchmark test facilitates:\n\n- Verify performance during subsequent Netty upgrades;\n- Validate performance after changes to relevant code in Celeborn.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nGenerated `NettyTransportBenchmark-results.txt`.\n\nCloses #3647 from SteNicholas/CELEBORN-2298.\n\nLead-authored-by: SteNicholas \u003cprogramgeek@163.com\u003e\nCo-authored-by: Cheng Pan \u003cchengpan@apache.org\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "913d027efd1e56d080ec8080818e05ccf8eaa025",
      "tree": "9a4cb97d0b6087452de4264ca509e447f806deca",
      "parents": [
        "163bcb36edc11d1b999827a4e06bfcdd3bc7b3ea"
      ],
      "author": {
        "name": "zhengtao",
        "email": "shuaizhentao.szt@alibaba-inc.com",
        "time": "Sat Apr 11 11:44:19 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Sat Apr 11 11:44:19 2026 +0800"
      },
      "message": "[CELEBORN-2287] Split mode should be HARD_SPLIT when disk is full\n\n### What changes were proposed in this pull request?\n\nChange the split mode to `HARD_SPLIT` when disk is full.\n\n### Why are the changes needed?\n\nWhen the disk is already in a full state, continuous writing data in `SOFT_SPLIT` mode may cause the reserved space to be filled up as well.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nManual test and UT.\n\nCloses #3653 from zaynt4606/clb2287.\n\nAuthored-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "163bcb36edc11d1b999827a4e06bfcdd3bc7b3ea",
      "tree": "61a031efca759baf3b9a72d7e137dd8b6ff3c7e2",
      "parents": [
        "a30166df08f884869ef3c0ef03829cd21bed8814"
      ],
      "author": {
        "name": "sychen",
        "email": "sychen@ctrip.com",
        "time": "Fri Apr 10 14:41:33 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Fri Apr 10 14:41:33 2026 +0800"
      },
      "message": "[CELEBORN-2297] Update workflow to manually install Helm and chart-testing\n\n### What changes were proposed in this pull request?\n\n### Why are the changes needed?\n```\nThe actions azure/setup-helmv4.2.0, docker/setup-buildx-actionv1, and docker/build-push-actionv2 are not allowed in apache/celeborn\n```\nhttps://github.com/apache/celeborn/actions/runs/23730849688\n\n```yml\n      - name: Setup chart-testing\n        uses: ./.github/actions/chart-testing-action\n```\n\n```\nThe action sigstore/cosign-installer11086d25041f77fe8fe7b9ea4e48e3b9192b8f19 is not allowed in apache/celebor\n```\n### Does this PR resolve a correctness bug?\nNo\n\n### Does this PR introduce _any_ user-facing change?\nNo\n\n### How was this patch tested?\nGHA\nhttps://github.com/apache/celeborn/actions/runs/23884713736/job/69645124360?pr\u003d3639\n\nCloses #3639 from cxzl25/fix_it_test.\n\nAuthored-by: sychen \u003csychen@ctrip.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "a30166df08f884869ef3c0ef03829cd21bed8814",
      "tree": "973a7f4d526ed2802804873e13bf46f90542f60e",
      "parents": [
        "c688c76d3c8de471afbf8c5fd1857b0a5926a721"
      ],
      "author": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Fri Apr 10 14:39:51 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Fri Apr 10 14:39:51 2026 +0800"
      },
      "message": "[CELEBORN-2305] Bump Ratis version from 3.2.1 to 3.2.2\n\n### What changes were proposed in this pull request?\n\nBump Ratis version from 3.2.1 to 3.2.2.\n\n### Why are the changes needed?\n\nBump Ratis version from 3.2.1 to 3.2.2. Ratis has released v3.2.2 of which release note refers to [3.2.2](https://ratis.apache.org/post/3.2.2.html). The 3.2.2 version is a maintenance release with multiple improvements and bugfixes.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nCI.\n\nCloses #3658 from SteNicholas/CELEBORN-2305.\n\nAuthored-by: SteNicholas \u003cprogramgeek@163.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "c688c76d3c8de471afbf8c5fd1857b0a5926a721",
      "tree": "6f36aa996d8ad62c0644e384f0492f4b05c96a04",
      "parents": [
        "c246031889495c45c3e96cf682f2378382411e0e"
      ],
      "author": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Thu Apr 09 20:35:35 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Thu Apr 09 20:35:35 2026 +0800"
      },
      "message": "[CELEBORN-2063][FOLLOWUP] Fix timeout unit for parallel creation of partition writer\n\n### What changes were proposed in this pull request?\n\nFix timeout unit for parallel creation of partition writer in `Utils#tryFuturesWithTimeout`.\n\nFollow up #3387, #3656.\n\n### Why are the changes needed?\n\n`Utils#tryFuturesWithTimeout` uses wrong timeout unit which does not match the config option `celeborn.worker.writer.create.parallel.timeout` as follows:\n\n```\nval WORKER_WRITER_CREATE_PARALLEL_TIMEOUT: ConfigEntry[Long] \u003d\n    buildConf(\"celeborn.worker.writer.create.parallel.timeout\")\n      .categories(\"worker\")\n      .version(\"0.6.3\")\n      .doc(\"Timeout for a worker to create a file writer in parallel.\")\n      .timeConf(TimeUnit.MILLISECONDS)\n      .createWithDefaultString(\"120s\")\n```\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nCI.\n\nCloses #3657 from SteNicholas/CELEBORN-2063.\n\nAuthored-by: SteNicholas \u003cprogramgeek@163.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "c246031889495c45c3e96cf682f2378382411e0e",
      "tree": "e424bf054b33dd198e3b8dfb00c8835e387d792e",
      "parents": [
        "84830d9fcfe1067c050774d8f21a121b4b22911a"
      ],
      "author": {
        "name": "Xianming Lei",
        "email": "xianming.lei@shopee.com",
        "time": "Thu Apr 09 14:49:01 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Thu Apr 09 14:49:01 2026 +0800"
      },
      "message": "[CELEBORN-2294][FOLLOWUP] Fix flaky test SparkUtilsSuite\n\n### What changes were proposed in this pull request?\nRemove racy assertions in the \"check if fetch failure task another attempt is running or successful\" test in SparkUtilsSuite.\n\n### Why are the changes needed?\n\nAfter CELEBORN-2294 added a zombie TaskSetManager check in shouldReportShuffleFetchFailure, the test became flaky due to a race condition. The test calls shouldReportShuffleFetchFailure a second time from the test thread, but by that point the FetchFailed has already been processed by Spark\u0027s DAGScheduler — **the TaskSetManager is either marked as zombie** or the task has been removed from taskIdToTaskSetManager. This causes the second call to return false, failing the assertion.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nThe affected UT: SparkUtilsSuite - check if fetch failure task another attempt is running or successful.\n\nCloses #3655 from leixm/FOLLOW-CELEBORN-2294.\n\nAuthored-by: Xianming Lei \u003cxianming.lei@shopee.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "84830d9fcfe1067c050774d8f21a121b4b22911a",
      "tree": "dc44136c990291c6f63c631bd1d609e8aac4f62a",
      "parents": [
        "7f1bac3443f191d2bc5dcebebb88cd1744b53cd7"
      ],
      "author": {
        "name": "sychen",
        "email": "sychen@ctrip.com",
        "time": "Thu Apr 09 08:24:21 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Thu Apr 09 08:24:21 2026 +0800"
      },
      "message": "[CELEBORN-2304] Fix timeout unit mismatch in disk monitor check\n\n### What changes were proposed in this pull request?\nChanged tryWithTimeoutAndCallback and tryFutureWithTimeoutAndCallback in Utils.scala to accept timeout in milliseconds instead of seconds.\n\n### Why are the changes needed?\nWORKER_DEVICE_STATUS_CHECK_TIMEOUT is configured in milliseconds (e.g. 30s → 30000), but was passed directly to Future.get(..., TimeUnit.SECONDS)\n\nhttps://github.com/apache/celeborn/blob/7f1bac3443f191d2bc5dcebebb88cd1744b53cd7/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala#L4100-L4107\n\nhttps://github.com/apache/celeborn/blob/7f1bac3443f191d2bc5dcebebb88cd1744b53cd7/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/DeviceMonitor.scala#L287-L291\n\n### Does this PR resolve a correctness bug?\nNo\n\n### Does this PR introduce _any_ user-facing change?\nNo\n\n### How was this patch tested?\nGHA\n\nCloses #3656 from cxzl25/CELEBORN-2304.\n\nAuthored-by: sychen \u003csychen@ctrip.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "7f1bac3443f191d2bc5dcebebb88cd1744b53cd7",
      "tree": "ee96a224a2ce2d92e94077cdb95006889bf12149",
      "parents": [
        "234ff2d705bd5a6c4a6603d3976af8a3615bcbf9"
      ],
      "author": {
        "name": "sychen",
        "email": "sychen@ctrip.com",
        "time": "Wed Apr 08 11:06:35 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Wed Apr 08 11:06:35 2026 +0800"
      },
      "message": "[CELEBORN-2302] Fix NPE in MemoryManager.close() when readBufferDispatcher is not initialized\n\n### What changes were proposed in this pull request?\nAdd a null check for readBufferDispatcher before calling close() in MemoryManager.close().\n\n### Why are the changes needed?\nreadBufferDispatcher is only initialized when readBufferThreshold \u003e 0.\n\n### Does this PR resolve a correctness bug?\nNo\n\n### Does this PR introduce _any_ user-facing change?\nNo\n\n### How was this patch tested?\nGHA\n\nCloses #3654 from cxzl25/CELEBORN-2302.\n\nAuthored-by: sychen \u003csychen@ctrip.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "234ff2d705bd5a6c4a6603d3976af8a3615bcbf9",
      "tree": "bd828e62cf9201f8e95a60c352375cb4c1d7d2ac",
      "parents": [
        "ca8533c8390bfcdcb418144e9fc95c3561fe641a"
      ],
      "author": {
        "name": "afterincomparableyum",
        "email": "224495379+afterincomparableyum@users.noreply.github.com",
        "time": "Wed Apr 08 11:04:32 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Wed Apr 08 11:04:32 2026 +0800"
      },
      "message": "[CELEBORN-2219][CIP-14] Support PushMergedData in CppClient\n\n### What changes were proposed in this pull request?\n\nImplement PushMergedData functionality in the C++ client, enabling batch merging and pushing of shuffle data grouped by worker address.\n\nKey changes:\n  - Add mergeData() and pushMergedData() to ShuffleClient, which accumulateper partition data batches and push them as merged payloads when the buffer threshold is exceeded or at mapper end.\n  - Introduce DataBatches class to manage batch accumulation thread-safe add/take operations and size-bounded requireBatches().\n  - Add PushMergedDataCallback to handle success responses (split handling, congestion control, MAP_ENDED) and failure responses with revive-based retry via submitRetryPushMergedData().\n  - Add PushMergedData network message type with encoding for partitionUniqueIds and batchOffsets arrays.\n  - Extend Encoders with encode/decode support for vector\u003cstring\u003e and vector\u003cint32_t\u003e.\n  - Add pushMergedDataAsync() to TransportClient.\n  - Add unit tests for DataBatches, PushMergedData message encoding, and array encoders.\n\n### Why are the changes needed?\n\nThis is needed to extend the functionality of C++ Client and there is a Bolt dependency on it with https://github.com/bytedance/bolt/issues/370\n\n### Does this PR resolve a correctness bug?\n\nNo\n\n### Does this PR introduce _any_ user-facing change?\n\nYes because it is a new functionality in the c++ client\n\n### How was this patch tested?\n\nTested through running unit tests and compiling locally.\n\nCloses #3611 from afterincomparableyum/cpp-client/celeborn-2219.\n\nAuthored-by: afterincomparableyum \u003c224495379+afterincomparableyum@users.noreply.github.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "ca8533c8390bfcdcb418144e9fc95c3561fe641a",
      "tree": "05ca0cca878a6e1c7208e0dcee9be35ab8c365c5",
      "parents": [
        "37aed7ea5421e0de9bfdfbd452746103a579232e"
      ],
      "author": {
        "name": "sychen",
        "email": "sychen@ctrip.com",
        "time": "Wed Apr 08 10:55:07 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Wed Apr 08 10:55:07 2026 +0800"
      },
      "message": "[CELEBORN-2300] Change the default value of celeborn.port.maxRetries from 1 to 16\n\n### What changes were proposed in this pull request?\n\n### Why are the changes needed?\n\nAlign with spark.port.maxRetries, default to 16 retries.\n\n```java\nERROR Executor: Exception in task 1153.0 in stage 11763.0 (TID 516674)\njava.lang.RuntimeException: java.net.BindException: Address already in use: Service \u0027ShuffleClient\u0027 failed after 1 retries (on a random free port)! Consider explicitly setting the appropriate binding address for the service \u0027ShuffleClient\u0027 to the correct binding address.\n\tat org.apache.spark.shuffle.celeborn.SparkShuffleManager.getWriter(SparkShuffleManager.java:310)\n```\n\n### Does this PR resolve a correctness bug?\nNo\n\n### Does this PR introduce _any_ user-facing change?\nYes\n\n### How was this patch tested?\nGHA\n\nCloses #3648 from cxzl25/CELEBORN-2300.\n\nAuthored-by: sychen \u003csychen@ctrip.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "37aed7ea5421e0de9bfdfbd452746103a579232e",
      "tree": "fb701374002a3929cc7008693d851ed9e2d8d546",
      "parents": [
        "37a27bc4cb3c993289e5b94439db224572cd3e66"
      ],
      "author": {
        "name": "Xianming Lei",
        "email": "jerrylei@apache.org",
        "time": "Fri Apr 03 18:00:01 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Fri Apr 03 18:00:01 2026 +0800"
      },
      "message": "[CELEBORN-2295] CommitHandler should support retry interval\n\n### What changes were proposed in this pull request?\n\n`CommitHandler` should support retry interval for retry of committing file.\n\n### Why are the changes needed?\n\nWhen commitFiles RPC fails, the current implementation retries immediately without any backoff. If the worker is experiencing transient network issues, immediate retries are likely to fail again. Adding a configurable retry interval (`celeborn.client.requestCommitFiles.retryInterval`, default 10s) gives the worker time to recover before the next attempt, significantly improving the success rate of retries. A dedicated `ScheduledExecutorService` is used to avoid blocking threads in the shared RPC pool during the wait.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nCI.\n\nCloses #3642 from leixm/CELEBORN-2295.\n\nLead-authored-by: Xianming Lei \u003cjerrylei@apache.org\u003e\nCo-authored-by: Xianming Lei \u003cxianming.lei@shopee.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "37a27bc4cb3c993289e5b94439db224572cd3e66",
      "tree": "727807a0fb7165ebcdf3eb56dd1873bac849ba37",
      "parents": [
        "097f1df172e7ed8f2d32c3c5a8c04b22a595fe25"
      ],
      "author": {
        "name": "sychen",
        "email": "sychen@ctrip.com",
        "time": "Thu Apr 02 10:56:26 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Thu Apr 02 10:56:26 2026 +0800"
      },
      "message": "[CELEBORN-2296] Fix race condition in MemoryManager singleton initialization\n\n### What changes were proposed in this pull request?\nApply double-checked locking to MemoryManager.initialize() and add synchronized to MemoryManager.reset() to make the singleton lifecycle thread-safe.\n\n### Why are the changes needed?\n\n```java\n26/04/01 10:32:20,050 ERROR [worker 3 starter thread] WordCountTestWithAuthentication: create worker failed, detail:\njava.lang.NullPointerException\n        at org.apache.celeborn.service.deploy.worker.memory.ChannelsLimiter.\u003cinit\u003e(ChannelsLimiter.java:52)\n        at org.apache.celeborn.service.deploy.worker.Worker.\u003cinit\u003e(Worker.scala:240)\n        at org.apache.celeborn.service.deploy.MiniClusterFeature.createWorker(MiniClusterFeature.scala:172)\n        at org.apache.celeborn.service.deploy.MiniClusterFeature.createWorker$(MiniClusterFeature.scala:153)\n        at org.apache.celeborn.tests.flink.WordCountTestBase.createWorker(WordCountTest.scala:44)\n        at org.apache.celeborn.service.deploy.MiniClusterFeature.createWorker(MiniClusterFeature.scala:150)\n        at org.apache.celeborn.service.deploy.MiniClusterFeature.createWorker$(MiniClusterFeature.scala:149)\n        at org.apache.celeborn.tests.flink.WordCountTestBase.createWorker(WordCountTest.scala:44)\n        at org.apache.celeborn.service.deploy.MiniClusterFeature.$anonfun$setUpWorkers$2(MiniClusterFeature.scala:221)\n        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)\n        at org.apache.celeborn.service.deploy.MiniClusterFeature$RunnerWrap.$anonfun$run$1(MiniClusterFeature.scala:50)\n        at org.apache.celeborn.common.util.Utils$.tryLogNonFatalError(Utils.scala:234)\n        at org.apache.celeborn.service.deploy.MiniClusterFeature$RunnerWrap.run(MiniClusterFeature.scala:50)\n```\n\n### Does this PR resolve a correctness bug?\nNo\n\n### Does this PR introduce _any_ user-facing change?\nNo\n\n### How was this patch tested?\nGHA\n\nCloses #3643 from cxzl25/CELEBORN-2296.\n\nAuthored-by: sychen \u003csychen@ctrip.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "097f1df172e7ed8f2d32c3c5a8c04b22a595fe25",
      "tree": "f6dc3c19226cdf0132931530f905a7dcb0236f6b",
      "parents": [
        "c3adbca3cd02f5f34ba120f86bffbee71fe78e03"
      ],
      "author": {
        "name": "Xianming Lei",
        "email": "jerrylei@apache.org",
        "time": "Thu Apr 02 10:52:28 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Thu Apr 02 10:52:28 2026 +0800"
      },
      "message": "[CELEBORN-2294] The shuffle fetch failed report from the zombie stage should be ignored\n\n### What changes were proposed in this pull request?\nThe shuffle fetch failed report from the zombie stage should be ignored\n\n### Why are the changes needed?\nWithout this PR, if a stage attempt has already triggered FetchFailed, there will still be running tasks reporting fetch failed to LifeCycleManager, which will cause the current Stage Attempt to mistakenly trigger a stage rerun.\n\nSpark also ignores FetchFailed from the previous stage attempt, and Celeborn should keep the same logic.\n\u003cimg width\u003d\"1760\" height\u003d\"664\" alt\u003d\"image\" src\u003d\"https://github.com/user-attachments/assets/201cd849-475b-442f-bec1-0b8ef1048036\" /\u003e\n\n### Does this PR resolve a correctness bug?\nNo.\n\n### Does this PR introduce _any_ user-facing change?\nNo.\n\n### How was this patch tested?\nExisting UTs.\n\nCloses #3640 from leixm/main.\n\nAuthored-by: Xianming Lei \u003cjerrylei@apache.org\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "c3adbca3cd02f5f34ba120f86bffbee71fe78e03",
      "tree": "379b2a992c9f7abe7206a3f38ffccaa972aead0d",
      "parents": [
        "d3b75132daeaa0484ae2562fe2a9c9fadfa70c92"
      ],
      "author": {
        "name": "Kartikay Bhutani",
        "email": "kbhutani0001@gmail.com",
        "time": "Thu Apr 02 10:50:09 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Thu Apr 02 10:50:09 2026 +0800"
      },
      "message": "[CELEBORN-2293] Fix ConcurrentModificationException in WorkerStatusTracker.shuttingWorkers\n\n### What changes were proposed in this pull request?\n\nReplace `HashSet` with `ConcurrentHashMap.newKeySet()` for `shuttingWorkers` in `WorkerStatusTracker`.\n\n### Why are the changes needed?\n\nWhen multiple shuffles hit a shutting-down worker simultaneously, one thread iterates `shuttingWorkers` in `currentFailedWorkers()` (for logging) while another modifies it in `recordWorkerFailure()`.\n\n### Does this PR resolve a correctness bug?\n\nYes.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nAdded a unit test in `WorkerStatusTrackerSuite` and ran the same. Was able to reproduce only once (without the fix) out of multiple runs.\n\nCloses #3638 from kaybhutani/fix-concurrent-shuttingworkers.\n\nLead-authored-by: Kartikay Bhutani \u003ckbhutani0001@gmail.com\u003e\nCo-authored-by: kartikay \u003ckbhutani0001@gmail.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "d3b75132daeaa0484ae2562fe2a9c9fadfa70c92",
      "tree": "f3c10bf4c3e9baa4af5cde522c24052379c37a6f",
      "parents": [
        "42f1a08c1bcf0ba117d05d91a766d1214e5b881f"
      ],
      "author": {
        "name": "luogen.lg",
        "email": "luogen.lg@alibaba-inc.com",
        "time": "Wed Apr 01 15:48:42 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Wed Apr 01 15:48:42 2026 +0800"
      },
      "message": "[CELEBORN-2292] Fix ArithmeticException when PUSH_DATA_HAND_SHAKE fails before any data written\n\n### What changes were proposed in this pull request?\n\nNOTE: This is the same patch with #3637 pushing to main branch. Because some code has been refactored, the original patch can not be simply cherry-picked.\n\nHandle the case where numSubpartitions is zero in MapPartitionDataReader.open(). When the partition is empty, treat it as a normal empty partition and notify consumers accordingly.\n\n### Why are the changes needed?\n\nWhen the first PUSH_DATA_HAND_SHAKE request fails (e.g., timeout), client triggers revive with reason HARD_SPLIT. Manager adds the failed partition to partition locations, but numSubpartitions remains uninitialized (zero). Reading such partition causes ArithmeticException: / by zero.\nSince this is caused by client-side behavior, we handle it on worker side first for cross-version compatibility. The issue that flink shuffle client revives with fixed reason HARD_SPLIT can be addressed in later PRs.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce any user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nManually tested with a hacked version that throws exception on the first handshake invocation. But the test code is too hacky to included into this PR. Advices are welcomed on how to add a proper unit test for this scenario without introducing too much complexity.\n\nCloses #3641 from pltbkd/CELEBORN-2292-on-main.\n\nAuthored-by: luogen.lg \u003cluogen.lg@alibaba-inc.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "42f1a08c1bcf0ba117d05d91a766d1214e5b881f",
      "tree": "a0a969901d16522734ca7c9e80e34e302a9f1166",
      "parents": [
        "235f07de49339d4cc2d5ac94deb55106a79ca7b4"
      ],
      "author": {
        "name": "Aravind Patnam",
        "email": "akpatnam25@gmail.com",
        "time": "Mon Mar 30 14:20:21 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Mon Mar 30 14:20:21 2026 +0800"
      },
      "message": "[CELEBORN-2284] Fix TLS Memory Leak\n\n### What changes were proposed in this pull request?\nWhile running jobs with TLS enabled, we encountered memory leaks which cause worker OOMs.\n```\n26/02/13 21:02:52,779 ERROR [push-server-9-9] ResourceLeakDetector: LEAK: ByteBuf.release() was not called before it\u0027s garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.\nRecent access records:\nCreated at:\n\tio.netty.buffer.AbstractByteBufAllocator.compositeDirectBuffer(AbstractByteBufAllocator.java:224)\n\tio.netty.buffer.AbstractByteBufAllocator.compositeBuffer(AbstractByteBufAllocator.java:202)\n\torg.apache.celeborn.common.network.util.TransportFrameDecoder.decodeNext(TransportFrameDecoder.java:143)\n\torg.apache.celeborn.common.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:66)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tio.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tio.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475)\n\tio.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338)\n\tio.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387)\n\tio.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)\n\tio.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)\n\tio.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tio.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tio.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tio.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tio.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tio.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tio.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tio.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tio.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tjava.base/java.lang.Thread.run(Thread.java:840)\n\n```\n\nWhen a Celeborn worker receives a PushData or PushMergedData message, it replicates that frame to a secondary worker for fault tolerance. On an SSL-enabled cluster this replication goes through SslMessageEncoder.encode(). Here is the flow of what happens inside SslMessageEncoder.encode():\n\n- The encoder asks the message body for an SSL-friendly copy by calling convertToNettyForSsl(). For shuffle data, the body is a NettyManagedBuffer — data already loaded in off-heap memory. This call runs buf.duplicate().retain(), which creates a second reference to the same memory and increments the reference count from 1 to 2.\n\n- The encoder places this second reference inside a composite buffer and hands it to Netty for writing.\n\n- Netty writes the composite to the network, then releases it — decrementing the count from 2 to 1.\n\n- Nothing releases the original NettyManagedBuffer\u0027s hold on the data, so the count stays at 1 forever.\n\n- This results in every replicated PushData frame leaking a chunk of off-heap memory, eventually causing OOM and worker crash.\n\nThe fix for this issue is to release the original message body, so that the net reference count is preserved. The second reference — now living inside the composite buffer in out — keeps the memory alive while Netty writes it to the network. When Netty finishes and releases the composite, the count reaches 0 and the memory is freed cleanly.\n\nThis is exactly what the non-SSL MessageEncoder already does via MessageWithHeader.deallocate() — the SSL path simply needed to replicate that behavior explicitly.\n\n### Why are the changes needed?\nfix memory leak\n\n### Does this PR resolve a correctness bug?\n\n### Does this PR introduce _any_ user-facing change?\nno\n\n### How was this patch tested?\nalready internally in production and tested.\nAlso added unit tests\n\nCloses #3630 from akpatnam25/CELEBORN-2284.\n\nAuthored-by: Aravind Patnam \u003cakpatnam25@gmail.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "235f07de49339d4cc2d5ac94deb55106a79ca7b4",
      "tree": "5070577cb04f73b7a0887f4809a90f912104a227",
      "parents": [
        "6fc5565319c6622aef8637a5cf5fa348fc763745"
      ],
      "author": {
        "name": "sychen",
        "email": "sychen@ctrip.com",
        "time": "Mon Mar 30 10:37:43 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Mon Mar 30 10:37:43 2026 +0800"
      },
      "message": "[CELEBORN-2277] Replace synchronized in Flusher.getWorkerIndex with AtomicInteger\n\n### What changes were proposed in this pull request?\n\nReplace the synchronized block in getWorkerIndex with an AtomicInteger.updateAndGet call using a CAS-based atomic operation.\n\n### Why are the changes needed?\n\nThe synchronized keyword locks the entire object and causes thread contention under high concurrency. Using AtomicInteger reduces lock scope to a single variable and avoids blocking overhead for this lightweight index increment operation.\n\n### Does this PR resolve a correctness bug?\nNo.\n\n### Does this PR introduce _any_ user-facing change?\nNo.\n\n### How was this patch tested?\n\nGHA.\n\nCloses #3621 from cxzl25/CELEBORN-2277.\n\nLead-authored-by: sychen \u003csychen@ctrip.com\u003e\nCo-authored-by: cxzl25 \u003c3898450+cxzl25@users.noreply.github.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "6fc5565319c6622aef8637a5cf5fa348fc763745",
      "tree": "9f614a4b7cfcddecbe4e95e2687f0ce6eb1b65d2",
      "parents": [
        "3773c6568a3041bab2294e4a2e31ce6a42a05ab6"
      ],
      "author": {
        "name": "Kartikay Bhutani",
        "email": "kbhutani0001@gmail.com",
        "time": "Fri Mar 27 10:35:55 2026 +0800"
      },
      "committer": {
        "name": "子懿",
        "email": "ziyi.jxf@antgroup.com",
        "time": "Fri Mar 27 10:35:55 2026 +0800"
      },
      "message": "[CELEBORN-2291] Support fsync on commit to ensure shuffle data durability\n\n### What changes were proposed in this pull request?\n  Add a new configuration `celeborn.worker.commitFiles.fsync` (default `false`) that calls `FileChannel.force(false)` (fdatasync) before closing the channel in\n   `LocalTierWriter.closeStreams()`.\n\n  ### Why are the changes needed?\n\n  Without this, committed shuffle data can sit in the OS page cache before the kernel flushes it to disk. A hard crash in that window loses data even though Celeborn considers it committed. This option lets operators opt into stronger durability guarantees.\n\n  ### Does this PR resolve a correctness bug?\n\n  No. It adds an optional durability enhancement.\n\n  ### Does this PR introduce _any_ user-facing change?\n\n  Yes. New configuration key `celeborn.worker.commitFiles.fsync` (boolean, default `false`).\n\n  ### How was this patch tested?\n\n  Existing unit tests. Configuration verified via `ConfigurationSuite` and for LocalTierWriter added a new test with fsync enabled and ran `TierWriterSuite`.\n\nAdditional context: [slack](https://apachecelebor-kw08030.slack.com/archives/C04B1FYS6SY/p1774259245973229)\n\nCloses #3635 from kaybhutani/kartikay/fsync-on-commit.\n\nAuthored-by: Kartikay Bhutani \u003ckbhutani0001@gmail.com\u003e\nSigned-off-by: 子懿 \u003cziyi.jxf@antgroup.com\u003e\n"
    },
    {
      "commit": "3773c6568a3041bab2294e4a2e31ce6a42a05ab6",
      "tree": "e026d2ebb409123e181430dafaa2507b7548a211",
      "parents": [
        "28a0733bb2441f98531f531b26800c0d3ea06e99"
      ],
      "author": {
        "name": "sychen",
        "email": "sychen@ctrip.com",
        "time": "Mon Mar 23 14:02:54 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Mon Mar 23 14:02:54 2026 +0800"
      },
      "message": "[CELEBORN-2285] Bump maven 3.9.14\n\n### What changes were proposed in this pull request?\n\n### Why are the changes needed?\nhttps://maven.apache.org/docs/3.9.14/release-notes.html\n\n### Does this PR resolve a correctness bug?\nNo\n\n### Does this PR introduce _any_ user-facing change?\nNo\n\n### How was this patch tested?\nGHA\n\nCloses #3634 from cxzl25/CELEBORN-2285.\n\nAuthored-by: sychen \u003csychen@ctrip.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "28a0733bb2441f98531f531b26800c0d3ea06e99",
      "tree": "397acb36495a4f97d5b4677df79c952f49b7f82d",
      "parents": [
        "5fc0f199a4080c407bcb2bd5225b6e1edafaadf8"
      ],
      "author": {
        "name": "sychen",
        "email": "sychen@ctrip.com",
        "time": "Thu Mar 19 11:12:12 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Thu Mar 19 11:12:12 2026 +0800"
      },
      "message": "[CELEBORN-2276] Fix race condition in MemoryManager.releaseSortMemory\n\n### What changes were proposed in this pull request?\nUse `updateAndGet`\n\n### Why are the changes needed?\n`reserveSortMemory` calls `sortMemoryCounter.addAndGet` without any synchronized, so the lock in `releaseSortMemory` doesn\u0027t actually protect against races with `reserveSortMemory` anyway.\n\n### Does this PR resolve a correctness bug?\nNo\n\n### Does this PR introduce _any_ user-facing change?\nNo\n\n### How was this patch tested?\nGHA\n\nCloses #3620 from cxzl25/CELEBORN-2276.\n\nAuthored-by: sychen \u003csychen@ctrip.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "5fc0f199a4080c407bcb2bd5225b6e1edafaadf8",
      "tree": "79b2c6b2efc645803668eaca45a08812b2f97b0b",
      "parents": [
        "b4cb5a0b1ac097d33baf8dded1b5be2afd0578a4"
      ],
      "author": {
        "name": "yew1eb",
        "email": "yew1eb@gmail.com",
        "time": "Wed Mar 18 22:22:55 2026 +0800"
      },
      "committer": {
        "name": "Shuang",
        "email": "lvshuang.xjs@alibaba-inc.com",
        "time": "Wed Mar 18 22:22:55 2026 +0800"
      },
      "message": "[CELEBORN-2282] Eliminate redundant HashMap lookups in CelebornInputStream#fillBuffer\n\n### What changes were proposed in this pull request?\n\nReplace three redundant HashMap operations (`containsKey` + `put` + `get`) with a single `computeIfAbsent` call in `CelebornInputStream#fillBuffer`.\n\n### Why are the changes needed?\n\n`fillBuffer` is on the hot shuffle-read path and called for every batch read. The original code performs up to three HashMap lookups per batch instead of one, causing unnecessary CPU overhead at scale.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nExisting unit tests.\n\nCloses #3627 from yew1eb/CELEBORN-2282.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: Shuang \u003clvshuang.xjs@alibaba-inc.com\u003e\n"
    },
    {
      "commit": "b4cb5a0b1ac097d33baf8dded1b5be2afd0578a4",
      "tree": "d24110ceb4550b5f3dbb920f48673316e9f11f79",
      "parents": [
        "7b25797fb8cf08c3495419b1d872663b08f829fb"
      ],
      "author": {
        "name": "yew1eb",
        "email": "yew1eb@gmail.com",
        "time": "Wed Mar 18 22:20:47 2026 +0800"
      },
      "committer": {
        "name": "Shuang",
        "email": "lvshuang.xjs@alibaba-inc.com",
        "time": "Wed Mar 18 22:20:47 2026 +0800"
      },
      "message": "[CELEBORN-2283][BUG] Fix missing return in Master.handleRequestSlots when all workers are excluded\n\n### What changes were proposed in this pull request?\n  Add a missing `return` statement after `context.reply()` in `Master#handleRequestSlots`\n  when `numAvailableWorkers \u003d\u003d 0`.\n\n### Why are the changes needed?\n  When all workers are excluded, the code replies with `WORKER_EXCLUDED` but continues\n  executing to `Random.nextInt(numAvailableWorkers)` (i.e. `Random.nextInt(0)`), which\n  throws `IllegalArgumentException`. This results in a duplicate response being sent to\n  the client and misleading error logs on the Master side.\n\n### Does this PR resolve a correctness bug?\n No.\n\n### Does this PR introduce _any_ user-facing change?\n No.\n\n### How was this patch tested?\nExisting unit tests.\n\nCloses #3628 from yew1eb/CELEBORN-2283.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: Shuang \u003clvshuang.xjs@alibaba-inc.com\u003e\n"
    },
    {
      "commit": "7b25797fb8cf08c3495419b1d872663b08f829fb",
      "tree": "f698d78b036c5534c79d58601d729d6c99d3440a",
      "parents": [
        "15cba469467b6038c5c40ee50c6420726a7c1a2e"
      ],
      "author": {
        "name": "Shuai Lu",
        "email": "lushuainada@gmail.com",
        "time": "Wed Mar 18 22:18:21 2026 +0800"
      },
      "committer": {
        "name": "Shuang",
        "email": "lvshuang.xjs@alibaba-inc.com",
        "time": "Wed Mar 18 22:18:21 2026 +0800"
      },
      "message": "[CELEBORN-2274] Fix replicate channels not resumed when transitioning from PUSH_AND_REPLICATE_PAUSED to PUSH_PAUSED\n\n### What changes were proposed in this pull request?\n\nFix a bug in `MemoryManager.switchServingState()` where replicate channels permanently lose `autoRead\u003dtrue` after a memory pressure event.\n\nWhen the serving state transitions from `PUSH_AND_REPLICATE_PAUSED` to `PUSH_PAUSED`, `resumeReplicate()` was only called inside the `!tryResumeByPinnedMemory()` guard. If `tryResumeByPinnedMemory()` returned `true`, the entire block was skipped and replicate channels were never resumed.\n\nThe fix moves `resumeReplicate()` outside the `tryResumeByPinnedMemory()` guard so it is always called when stepping down from `PUSH_AND_REPLICATE_PAUSED` to `PUSH_PAUSED`. This is a state machine invariant: `PUSH_PAUSED` means only push is paused; replicate must always be resumed.\n\n### Why are the changes needed?\n\nOnce replicate channels are stuck with `autoRead\u003dfalse`, Netty I/O threads stop reading from all replicate connections. Remote workers writing to the affected worker see their TCP send buffers fill up (zero window), causing pending writes to accumulate in `ChannelOutboundBuffer`. Each pending write holds a reference to a direct memory `ByteBuf`, causing direct memory to grow indefinitely on the remote workers.\n\nThe failure sequence:\n1. Worker hits memory pressure → state \u003d `PUSH_AND_REPLICATE_PAUSED` → all channels paused\n2. Pinned memory is low → `tryResumeByPinnedMemory()` returns `true` → `resumeByPinnedMemory(PUSH_PAUSED)` resumes push only, replicate not resumed\n3. Memory drops to push-only range → state \u003d `PUSH_PAUSED`, but `resumeReplicate()` is never called\n4. Replicate channels permanently stuck with `autoRead\u003dfalse`, causing unbounded direct memory growth on remote workers\n\n### Does this PR resolve a correctness bug?\n\nYes.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nAdded a new unit test `Test MemoryManager resume replicate by pinned memory` in `MemoryManagerSuite` that reproduces the exact failure scenario:\n1. Enter `PUSH_AND_REPLICATE_PAUSED` with low pinned memory (channels resumed by pinned memory path)\n2. Raise pinned memory so both push and replicate get paused\n3. Drop memory to `PUSH_PAUSED` range with low pinned memory\n4. Assert replicate listener is resumed — this assertion fails without the fix\n\nCloses #3616 from sl3635/CELEBORN-2274.\n\nAuthored-by: Shuai Lu \u003clushuainada@gmail.com\u003e\nSigned-off-by: Shuang \u003clvshuang.xjs@alibaba-inc.com\u003e\n"
    },
    {
      "commit": "15cba469467b6038c5c40ee50c6420726a7c1a2e",
      "tree": "f60554c3990fbe0d826ae6a149306847fa6d3e50",
      "parents": [
        "af0ba1a5ec0e1faf3d4a0d189058c755aeb6b18c"
      ],
      "author": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Wed Mar 18 20:41:01 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Wed Mar 18 20:41:01 2026 +0800"
      },
      "message": "[MINOR] Update repository references from incubator-gluten to gluten for TLP graduation of gluten\n\n### What changes were proposed in this pull request?\n\nUpdate repository references from incubator-gluten to gluten for TLP graduation of gluten.\n\n- Backport https://github.com/apache/gluten/pull/11735.\n- Close #3631.\n\n### Why are the changes needed?\n\nApache Gluten has already TLP graduated, which should update repository references.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nNo.\n\nCloses #3633 from SteNicholas/gluten-graduation.\n\nAuthored-by: SteNicholas \u003cprogramgeek@163.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "af0ba1a5ec0e1faf3d4a0d189058c755aeb6b18c",
      "tree": "c991f7586da7fe2cded1fb9dd0b498deffd69afd",
      "parents": [
        "400e9518d7f19cacf8797971e169afa6bdea5be6"
      ],
      "author": {
        "name": "sychen",
        "email": "sychen@ctrip.com",
        "time": "Wed Mar 18 15:55:32 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Wed Mar 18 15:55:32 2026 +0800"
      },
      "message": "[CELEBORN-2281] Improve error logging and null checks in CreditStreamManager\n\n### What changes were proposed in this pull request?\n\n- Initialize `AtomicReference\u003cIOException\u003e` with proper syntax.\n- Add exception to `logger.error` for better error context.\n- Simplify and improve null checks and logging in `addCredit` and `cleanResource` methods.\n\n### Why are the changes needed?\n\nnit.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nGHA.\n\nCloses #3626 from cxzl25/CELEBORN-2281.\n\nLead-authored-by: sychen \u003csychen@ctrip.com\u003e\nCo-authored-by: cxzl25 \u003c3898450+cxzl25@users.noreply.github.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "400e9518d7f19cacf8797971e169afa6bdea5be6",
      "tree": "4cf6016fecd2cbcc7f539a8b3a796b71234acc8f",
      "parents": [
        "8c2c9523d07a38a6c750834e8d9598d1e8571157"
      ],
      "author": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Sat Mar 14 17:27:00 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Sat Mar 14 17:27:00 2026 +0800"
      },
      "message": "[CELEBORN-2280] Support celeborn.network.memory.allocator.type to specify netty memory allocator\n\n### What changes were proposed in this pull request?\n\nSupport `celeborn.network.memory.allocator.type` to specify netty memory allocator including `AdaptiveByteBufAllocator `.\n\n### Why are the changes needed?\n\nNetty 4.2 introduces `AdaptiveByteBufAllocator` an auto-tuning pooling `ByteBufAllocator` which uses `AdaptivePoolingAllocator` added in https://github.com/netty/netty/pull/13075.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nIntroduce `celeborn.network.memory.allocator.type` to specify netty memory allocator.\n\n### How was this patch tested?\n\nCI.\n\nCloses #3625 from SteNicholas/CELEBORN-2280.\n\nAuthored-by: SteNicholas \u003cprogramgeek@163.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "8c2c9523d07a38a6c750834e8d9598d1e8571157",
      "tree": "1662f58628e70605270ca37536b07a14d9e53023",
      "parents": [
        "4b157c68a5f7e82f4a50b8a2c8ae8989aff843c1"
      ],
      "author": {
        "name": "sychen",
        "email": "sychen@ctrip.com",
        "time": "Tue Mar 10 17:22:51 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Wed Mar 11 19:43:16 2026 +0800"
      },
      "message": "[CELEBORN-2279] Update log level from `INFO` to `ERROR` for console output in spark-it tests\n\n### What changes were proposed in this pull request?\n\nUpdate log level from `INFO` to `ERROR` for console output in spark-it tests.\n\n### Why are the changes needed?\n\n`spark-it` outputs too many INFO level logs to stdout.\n\n\u003cimg width\u003d\"1017\" height\u003d\"143\" alt\u003d\"image\" src\u003d\"https://github.com/user-attachments/assets/41667a37-050b-4174-afe6-6e4afcda8fcc\" /\u003e\n\n\u003cimg width\u003d\"783\" height\u003d\"188\" alt\u003d\"image\" src\u003d\"https://github.com/user-attachments/assets/6bb4f3e3-223d-4f8c-bea6-4c8d451bf4b9\" /\u003e\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nGHA.\n\nCloses #3623 from cxzl25/CELEBORN-2279.\n\nAuthored-by: sychen \u003csychen@ctrip.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "4b157c68a5f7e82f4a50b8a2c8ae8989aff843c1",
      "tree": "630c63b237000990298bee7089d64a1a7e789784",
      "parents": [
        "b78177f3ac7adceb1f0510d2111943702e726eba"
      ],
      "author": {
        "name": "Aravind Patnam",
        "email": "akpatnam25@gmail.com",
        "time": "Tue Mar 10 10:36:54 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Tue Mar 10 10:36:54 2026 +0800"
      },
      "message": "[CELEBORN-2278] Make HTTP auth bypass API paths configurable\n\n### What changes were proposed in this pull request?\n\nAllow http paths that should be bypassed from auth to be configured. This is particularly useful when one of the read endpoints is used for health checks, and should not require auth each time for a high frequency operation.\n\n### Why are the changes needed?\n\nSee above.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nTrivial change, already added in our cluster for certain endpoints.\n\nCloses #3622 from akpatnam25/CELEBORN-2278.\n\nAuthored-by: Aravind Patnam \u003cakpatnam25@gmail.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    },
    {
      "commit": "b78177f3ac7adceb1f0510d2111943702e726eba",
      "tree": "37c12a2779ffe43aca621e06fcc2d74b01350d08",
      "parents": [
        "391ef4bfc42b4c121d6a029d60689af15ab16b5b"
      ],
      "author": {
        "name": "Shuai Lu",
        "email": "lushuainada@gmail.com",
        "time": "Mon Mar 09 10:34:10 2026 +0800"
      },
      "committer": {
        "name": "SteNicholas",
        "email": "programgeek@163.com",
        "time": "Mon Mar 09 10:34:10 2026 +0800"
      },
      "message": "[CELEBORN-2238] Fix RuntimeException during stream cleanup preventing peer failover\n\n### What changes were proposed in this pull request?\n\nFix a bug in `CelebornInputStream` where a `RuntimeException` thrown during best-effort stream cleanup prevents peer failover when a primary worker becomes unregistered.\n\nIn `createReaderWithRetry`, when reader creation fails on the primary, the code tries to close the existing stream by calling `clientFactory.createClient()` before switching to the peer. This cleanup was wrapped in `catch (InterruptedException | IOException ex)`. When SASL authentication is configured, `SaslClientBootstrap` wraps `IOException` in `RuntimeException`, so the cleanup call also throws `RuntimeException`. This uncaught exception escapes the retry loop entirely, bypassing `location \u003d location.getPeer()` and causing the executor to exhaust retries on the same failed primary worker.\n\nThe fix adds `RuntimeException` to the cleanup catch clause — `catch (InterruptedException | IOException | RuntimeException ex)` — so that any exception during best-effort cleanup is logged and swallowed, allowing the peer switch to proceed.\n\n### Why are the changes needed?\n\nWithout this fix, when a worker pod is rotated or becomes unregistered and SASL authentication is enabled, the replica retry mechanism silently fails. The executor retries multiple times on the same dead primary worker and eventually fails the task, even though a healthy replica exists.\n\n### Does this PR resolve a correctness bug?\n\nYes.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nAdded `CelebornInputStreamPeerFailoverTest` with three unit tests:\n- `testPeerFailoverWithRuntimeExceptionDuringCleanup`: primary fails, cleanup throws `RuntimeException` (simulates SASL wrapping), replica succeeds — verifies the fix\n- `testPeerFailoverWithIOExceptionDuringCleanup`: same scenario with plain `IOException` during cleanup — verifies existing behavior is preserved\n- `testFailureWithoutPeer`: no replica configured, verifies retries are exhausted and `CelebornIOException` is thrown\n\nCloses #3617 from sl3635/CELEBORN-2238.\n\nAuthored-by: Shuai Lu \u003clushuainada@gmail.com\u003e\nSigned-off-by: SteNicholas \u003cprogramgeek@163.com\u003e\n"
    }
  ],
  "next": "391ef4bfc42b4c121d6a029d60689af15ab16b5b"
}