)]}' { "log": [ { "commit": "fca9da23687c4d484f449abce9f87ed66eaf2ae1", "tree": "8b71c966d83e742239011adaabecd7ce70b9311d", "parents": [ "69200e209a21168d29faa17be03dadfb5a6f259f" ], "author": { "name": "afterincomparableyum", "email": "224495379+afterincomparableyum@users.noreply.github.com", "time": "Tue Oct 20 11:32:35 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Tue Oct 20 11:32:35 2026 +0800" }, "message": "[CELEBORN-2482] Report pushes failed by push-timeout channel close as push timeout instead of connection exception\n\n### What changes were proposed in this pull request?\n\nWhen a push request times out and its timeout handler closes the channel (CELEBORN-2458), `TransportResponseHandler` now records the timeout status (`PUSH_DATA_TIMEOUT_PRIMARY`/`REPLICA`) before the close. The other pushes on that channel that fail because of the close are reported with this status instead of as connection exceptions. That covers both `channelInactive` and write failures on the closed channel.\n\n### Why are the changes needed?\n\nSince CELEBORN-2458, a push timeout closes the channel. The other pushes still outstanding on it are then failed with `Connection from ... closed` (or `Failed to send request ...`), which `ExceptionUtils.connectFail` maps to `PUSH_DATA_CONNECTION_EXCEPTION_PRIMARY`/`REPLICA`. These collateral failures overwrite the worker\u0027s `PUSH_DATA_TIMEOUT_*` entry in the excluded workers, so a worker that stopped responding gets misreported as a connection failure. This also makes `PushDataTimeoutTest` flaky:\n\n```\n- celeborn spark integration test - pushdata timeout will add to pushExcludedWorkers *** FAILED ***\n PUSH_DATA_CONNECTION_EXCEPTION_PRIMARY did not equal PUSH_DATA_TIMEOUT_PRIMARY, and PUSH_DATA_CONNECTION_EXCEPTION_PRIMARY did not equal PUSH_DATA_TIMEOUT_REPLICA (PushDataTimeoutTest.scala:157)\n```\n\nIn the [failed run](https://github.com/apache/celeborn/actions/runs/38396707096/job/115293643837), a push to a worker timed out (`PUSH_DATA_TIMEOUT_PRIMARY`). In the same millisecond, `Still have 1 requests outstanding when connection from ... is closed` failed a second push on that channel, and the worker was then reported as `PUSH_DATA_CONNECTION_EXCEPTION_PRIMARY`.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\n- New tests in `TransportResponseHandlerSuiteJ`. The new push-timeout test fails without this change.\n - A push timeout that closes the channel fails the other outstanding pushes, and later write failures, with the timeout status.\n - A channel close without a push timeout still fails pushes with the connection exception.\n- `TransportResponseHandlerSuiteJ` (10 tests) and `TransportClientFactorySuiteJ` (11 tests): passed.\n- `PushDataTimeoutTest` with Spark 3.5: 5 tests passed.\n\nCloses #3876 from afterincomparableyum/push-timeout-close-status.\n\nAuthored-by: afterincomparableyum \u003c224495379+afterincomparableyum@users.noreply.github.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/42\nAI-Contributed/UT: 0/100\n" }, { "commit": "69200e209a21168d29faa17be03dadfb5a6f259f", "tree": "aefc0917caaa3a71f5786f2482db3a1ebd268d21", "parents": [ "60a90e578a0cf0c439adf3dd70061fe22fa099cc" ], "author": { "name": "afterincomparableyum", "email": "224495379+afterincomparableyum@users.noreply.github.com", "time": "Tue Oct 20 11:23:09 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Tue Oct 20 11:23:09 2026 +0800" }, "message": "[CELEBORN-2220][CIP-14] Support Slow Start Push in C++ Client\n\n### What changes were proposed in this pull request?\n\nAdd `SlowStartPushStrategy` to the C++ client, a port of the Java client\u0027s TCP-like push congestion control:\n\n- Keeps per-worker congestion state: the in-flight request limit grows by one per successful push until it reaches `celeborn.client.push.maxReqsInFlight.perWorker`, then grows linearly. The limit halves when a worker responds with `PUSH_DATA_SUCCESS_{PRIMARY,REPLICA}_CONGESTED`, and pushes sleep while the limit is below the maximum.\n- Selected with `celeborn.client.push.limit.strategy\u003dSLOWSTART` (case-insensitive, as in Java). Before this change, the C++ client rejected `SLOWSTART`.\n- Adds `celeborn.client.push.slowStart.initialSleepTime` (default `500ms`) and `celeborn.client.push.slowStart.maxSleepTime` (default `2s`), with the same keys and defaults as Java.\n\nThe C++ push callbacks and `PushState` already call `onSuccess`, `onCongestControl`, `limitPushSpeed` and `getCurrentMaxReqsInFlight`, so this PR only adds the strategy itself.\n\n#3730 is closed as stale. That PR\u0027s review comments are addressed here.\n\n### Why are the changes needed?\n\nTo close a feature gap between the C++ and Java clients (CIP-14).\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes\n\nC++ client users can set `celeborn.client.push.limit.strategy\u003dSLOWSTART` and the two `celeborn.client.push.slowStart.*` configs. The default strategy is still `SIMPLE`. Like the Java client, SLOWSTART only backs off when workers have `celeborn.worker.congestionControl.enabled\u003dtrue`.\n\n### How was this patch tested?\n\n- New `PushStrategyTest`: strategy selection, sleep-time calculation, slow start and congestion avoidance, and isolation between hosts. Its assertions mirror Java\u0027s `SlowStartPushStrategyTest`.\n- `CelebornConfTest`: defaults and overrides for the new configs.\n\nCloses #3878 from afterincomparableyum/CELEBORN-2220.\n\nAuthored-by: afterincomparableyum \u003c224495379+afterincomparableyum@users.noreply.github.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/194\nAI-Contributed/UT: 0/175\n" }, { "commit": "60a90e578a0cf0c439adf3dd70061fe22fa099cc", "tree": "cbacb586bd32d5abfe2c280182ecd5f822e9ae7a", "parents": [ "d7d6f7c5db6a60e786c4c50b137226fb501294c4" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Tue Oct 20 11:03:02 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Tue Oct 20 11:03:02 2026 +0800" }, "message": "[CELEBORN-2488] Cache Maven distribution in CI workflows\n\n### What changes were proposed in this pull request?\n\nEach job in `maven.yml` re-downloads the Maven distribution from `archive.apache.org` via `build/mvn` (~25s typical, 1m49s observed on a slow network), even though `~/.m2` is already cached by `setup-java`. This PR adds an `actions/cachev4` step for the extracted Maven install under `build/`, so `build/mvn` reuses the cached binary — its `install_app` already skips the download when `apache-maven-\u003cversion\u003e/bin/mvn` exists.\n\nThis mirrors the sbt launcher caching added for `sbt.yml` in CELEBORN-2432.\n\n### Why are the changes needed?\n\nSave ~25s–2min per job; ~10–40 minutes of total machine time per CI run.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] No\n\n### How was this patch tested?\n\nVerified locally that `build/mvn` skips the download when `build/apache-maven-\u003cversion\u003e/bin/mvn` is present. CI on this branch shows the cache step restoring and no `curl` download of the Maven tarball.\n\nCloses #3880 from yew1eb/CELEBORN-2488.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/36\nAI-Contributed/UT: 0/0\n" }, { "commit": "d7d6f7c5db6a60e786c4c50b137226fb501294c4", "tree": "59bffb4c8010f1c409101e186ce242e2f4503c29", "parents": [ "eca67458ed3a3b9a92ba24a675890a78628b7024" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Tue Oct 20 10:25:42 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Tue Oct 20 10:25:42 2026 +0800" }, "message": "[CELEBORN-2487] Fix leaked LifecycleManager in spark-it client suites\n\n### What changes were proposed in this pull request?\n\nStop the LifecycleManager instances leaked by spark-it client suites:\n\n- `LifecycleManagerSuite` \"CELEBORN-1258\" test: add `lifecycleManager.stop()` at the end of the test.\n- `ChangePartitionManagerUpdateWorkersSuite` \"verify partition location storage info\" test: add `lifecycleManager.stop()` at the end of the test.\n- `LifecycleManagerReserveSlotsSuite` (its only test): add `lifecycleManager.stop()` at the end of the test.\n- `ChangePartitionManagerUpdateWorkersSuite.afterEach()` now calls `super.afterEach()` before shutting down the mini cluster, so the managers created by `WithShuffleClientSuite`\u0027s base tests (via `prepareService()`) are stopped.\n\n### Why are the changes needed?\n\nSeveral spark-it suites create a `LifecycleManager` but never `stop()` it. Each leaked manager\u0027s `ApplicationHeartbeater` keeps heartbeating the already-stopped master for the rest of the test run, producing ~1200 ERROR stack traces — about a third of the CI test log — which drowns out real failures.\n\nThe `afterEach()` case is the largest source: `ChangePartitionManagerUpdateWorkersSuite` overrides `afterEach()` to shut down the mini cluster without calling `super`, so every base test\u0027s manager leaks and errors every ~20s until JVM exit.\n\nThe cleanup follows the pattern already used by the other client suites (e.g. `LifecycleManagerCommitFilesSuite`), which stop the manager at the end of each test.\n\u003cimg width\u003d\"1287\" height\u003d\"615\" alt\u003d\"image\" src\u003d\"https://github.com/user-attachments/assets/927603cd-2e2a-4062-8561-6050712dcccd\" /\u003e\n\nJIRA: [CELEBORN-2487](https://issues.apache.org/jira/browse/CELEBORN-2487)\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nRan the three affected suites with Spark 4.2:\n\n```\nbuild/mvn -Pspark-4.2,google-mirror -pl tests/spark-it test \\\n -DwildcardSuites\u003dorg.apache.celeborn.tests.client.LifecycleManagerSuite,org.apache.celeborn.tests.client.LifecycleManagerReserveSlotsSuite,org.apache.celeborn.tests.client.ChangePartitionManagerUpdateWorkersSuite\n```\n\n17 tests pass, and the `ApplicationHeartbeater: AskSync HeartbeatFromApplication failed` / `MasterClient: Send rpc with failure` ERROR stacks are gone (before the fix they were reproduced every ~20s until JVM exit).\n\nCloses #3882 from yew1eb/CELEBORN-2487.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/0\nAI-Contributed/UT: 0/10\n" }, { "commit": "eca67458ed3a3b9a92ba24a675890a78628b7024", "tree": "06956a6520020dd7666d993d631ca0bbb1eb978f", "parents": [ "bc06a1830148672ef50edb3c946d9a9b349ef962" ], "author": { "name": "afterincomparableyum", "email": "224495379+afterincomparableyum@users.noreply.github.com", "time": "Mon Oct 19 17:15:04 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Mon Oct 19 17:15:04 2026 +0800" }, "message": "[CELEBORN-2484][BUILD] Make sbt downloads resilient to transient Maven Central failures\n\n### What changes were proposed in this pull request?\n\n- `build/sbt-launch-lib.bash`: the sbt launcher download now retries. It adds `--retry-all-errors` when curl is 7.71 or newer, because plain `--retry` skips DNS, connection and 403 errors. It prints the download error, and if repo1 still fails it falls back to Google\u0027s Maven Central mirror. That fallback is skipped when `DEFAULT_ARTIFACT_REPOSITORY` is set.\n- `build/sbt-config/repositories`: point `gcs-maven-central-mirror` at the live `/maven2/` path. The old `/repos/central/data/` path returns 404. `repositories-asia.template` already uses `/maven2/`.\n\n### Why are the changes needed?\n\nSBT CI jobs fail before any test runs whenever Maven Central has a brief error:\n- The launcher was fetched with a single silent `curl` and no retry. The job died in under a second with only `Our attempt to download sbt locally ... failed` in the log ([job 1](https://github.com/apache/celeborn/actions/runs/38999585335/job/117299481319?pr\u003d3875), [job 2](https://github.com/apache/celeborn/actions/runs/38999734574/job/117299925875?pr\u003d3876)). The Actions cache is at its 10GB limit, so the cached launcher is usually evicted and every matrix job downloads it.\n- The GCS mirror path was dead, so every dependency came from repo1 alone. A single connection reset failed the build ([job 3](https://github.com/apache/celeborn/actions/runs/38376216131/job/115225130981): `could not retrieve sbt 1.9.4`).\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\n- Ran the launcher download locally three ways: normal, with repo1 returning 404, and with repo1 failing DNS. Each case retried as expected, fell back to the mirror and produced a valid jar.\n- `curl` against the GCS mirror: `/maven2/` returns 200 and `/repos/central/data/` returns 404.\n- CI on this PR covers the full dependency resolution through the corrected mirror.\n\nCloses #3877 from afterincomparableyum/sbt-launcher-download-retry.\n\nAuthored-by: afterincomparableyum \u003c224495379+afterincomparableyum@users.noreply.github.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n" }, { "commit": "bc06a1830148672ef50edb3c946d9a9b349ef962", "tree": "482cb12634cdfc7de3f61b81dd5208f4d89f75cd", "parents": [ "f4d6851663d865907070059077eff260d638246f" ], "author": { "name": "Rudy Attias", "email": "rattias@zencore.dev", "time": "Mon Oct 19 12:05:21 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Mon Oct 19 12:05:21 2026 +0800" }, "message": "[CELEBORN-2481] Fix NPE when persisting MapFileMeta without segment granularity\n\n### What changes were proposed in this pull request?\n\n`PbSerDeUtils.toPbFileInfo` now adds the MAP partition\u0027s writing-segment map and segment indexes only when they are non-null.\n\n`MapFileMeta` creates both lazily, on the first `addPartitionSegmentId` / `addSegmentIdAndFirstBufferIndex` call. The worker builds MAP file metas with `new MapFileMeta()`, so for a MAP partition written without segment granularity both fields stay `null`, and `putAllPartitionWritingSegment(null)` threw:\n\n```\njava.lang.NullPointerException: Cannot invoke \"java.util.Map.keySet()\" because \"m\" is null\n at com.google.protobuf.MapField$MutabilityAwareMap.putAll(MapField.java:312)\n at org.apache.celeborn.common.protocol.PbFileInfo$Builder.putAllPartitionWritingSegment(PbFileInfo.java:1674)\n at org.apache.celeborn.common.util.PbSerDeUtils$.toPbFileInfo(PbSerDeUtils.scala:164)\n```\n\nOn restore, `fromPbFileInfo` restores both fields as `null` when they are empty, matching a live meta that never recorded a segment, so `MapFileMeta#hasPartitionSegmentIds()` gives the same answer before and after a restart. Restored metas are already marked writer-closed, so nothing writes to them. Partitions with segment granularity serialize and restore exactly as before.\n\n### Why are the changes needed?\n\nWhen graceful shutdown is enabled, the worker persists committed file infos through `toPbFileInfoMap` → `toPbFileInfo` in two places, and the NPE breaks both:\n\n- **The periodic saver** (`worker-storage-manager-committed-fileinfo-saver`). It runs with `scheduleWithFixedDelay` and does not catch the NPE, so the first failure cancels it and nothing is saved afterwards.\n- **`saveAllCommittedFileInfosToDB` at shutdown.** It logs the error and skips any shuffle that contains such a MAP partition.\n\nAs a result, after a graceful restart the worker cannot recover MAP partitions written without segment granularity, such as Flink shuffle partitions. Readers fail to find them, and the producing tasks have to be re-run.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\nShuffle data is never incorrect. Partitions are lost on graceful restart instead of being recovered.\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes\n\nWith graceful shutdown enabled, a worker restart now recovers committed MAP partitions without segment granularity. No configuration or protocol change.\n\n### How was this patch tested?\n\nAdded two tests to `PbSerDeUtilsTest`:\n\n- `fromAndToPbFileInfo with MapFileMeta without segment granularity` builds the meta the way the worker does (no-arg constructor plus buffer size and subpartition count). It round-trips it through `toPbFileInfo`/`fromPbFileInfo` and through `toPbFileInfoMap`/`fromPbFileInfoMap`, checking the buffer size, subpartition count, segment-granularity flag, writer-closed flag, that `hasPartitionSegmentIds()` is false and the segment indexes are null, a segment lookup, path and length. On `main` it fails with the NPE above.\n- `fromAndToPbFileInfo with MapFileMeta with segment granularity` is a regression check. It confirms that `hasPartitionSegmentIds()` stays true and that recorded segment IDs and segment indexes still round-trip.\n\n```\nbuild/mvn -pl common -am -Dtest\u003dnone -DwildcardSuites\u003dorg.apache.celeborn.common.util.PbSerDeUtilsTest test\nTests: succeeded 36, failed 0\n```\n\nCloses #3874 from attiasr/fix-map-fileinfo-serde.\n\nAuthored-by: Rudy Attias \u003crattias@zencore.dev\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/19\nAI-Contributed/UT: 0/60\n" }, { "commit": "f4d6851663d865907070059077eff260d638246f", "tree": "7fcc0c1f4fe564baa8e41915884a9d6ab279bab4", "parents": [ "f93797717f5e7541667ee3f23de62be86797fff2" ], "author": { "name": "afterincomparableyum", "email": "224495379+afterincomparableyum@users.noreply.github.com", "time": "Mon Oct 19 11:57:28 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Mon Oct 19 11:57:28 2026 +0800" }, "message": "[CELEBORN-2483] Fix flaky RemoteShuffleMasterSuiteJ shuffleCount assertions racing with the initial app heartbeat\n\n### What changes were proposed in this pull request?\n\n- Add the internal conf `celeborn.test.client.application.heartbeat.enabled` (default `true`). When it is `false`, `ApplicationHeartbeater` does not schedule the app heartbeat.\n- Disable the app heartbeat in `RemoteShuffleMasterSuiteJ`, replacing the `heartbeatInterval\u003d1h` workaround from CELEBORN-2431.\n\n### Why are the changes needed?\n\n`RemoteShuffleMasterSuiteJ.testRegisterPartitionWithProducer` is still flaky after CELEBORN-2431:\n\n```\nRemoteShuffleMasterSuiteJ.testRegisterPartitionWithProducer:127 expected:\u003c1\u003e but was:\u003c0\u003e\nRemoteShuffleMasterSuiteJ.testRegisterPartitionWithProducer:158 expected:\u003c3\u003e but was:\u003c1\u003e\n```\n\n`ApplicationHeartbeater` schedules its first heartbeat with an initial delay of 0. Each heartbeat drains `LifecycleManager#shuffleCount` and the fallback counts via `sumThenReset`. The `LifecycleManager` is created in `registerJob`, so the first heartbeat runs at the same time as the test\u0027s `registerPartitionWithProducer` calls and can reset the counter between an increment and its assertion. Raising the interval to 1h only removed the periodic heartbeats, not the initial one. The suite has no master, so no test in it needs the heartbeat. The production initial delay is unchanged.\n\nExample failures: [flink1 (11, 1.18)](https://github.com/apache/celeborn/actions/runs/38396707127/job/115293644538), [flink2 (17, 2.2)](https://github.com/apache/celeborn/actions/runs/38396707127/job/115293645021).\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\n`RemoteShuffleMasterSuiteJV120`: 7 tests passed. The `AskSync HeartbeatFromApplication failed` errors, which appeared once per test in CI, should no longer appear.\n\nCloses #3875 from afterincomparableyum/flaky-remote-shuffle-master-heartbeat.\n\nAuthored-by: afterincomparableyum \u003c224495379+afterincomparableyum@users.noreply.github.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/20\nAI-Contributed/UT: 0/6\n" }, { "commit": "f93797717f5e7541667ee3f23de62be86797fff2", "tree": "441c26b6b7ee116debdaf148c005264179641363", "parents": [ "cbcbeeb8ed4a4ccf770028cd912ac4ed90cb73f6" ], "author": { "name": "SparksFyz", "email": "sparksvann@gmail.com", "time": "Tue Oct 13 20:21:26 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Oct 13 20:21:26 2026 +0800" }, "message": "[CELEBORN-2476] Fix initial storage fallback for MEMORY locations\n\n### What changes were proposed in this pull request?\n\n - Use the selected backend type when a MEMORY location falls back to non-memory storage.\n - Start from the first policy entry if the worker\u0027s creation policy excludes MEMORY, avoiding an index of `-1`.\n - Keep existing behavior for non-memory locations and eviction.\n\n ### Why are the changes needed?\n\n When creating a file for a new MEMORY location (`evict\u003dfalse`), the worker skips memory if its memory file storage quota is exhausted. However, it still passes MEMORY to `createDiskFile()`, so creation on an allowed remote backend\n fails.\n\n For example, a MEMORY location allowing `MEMORY,S3` cannot fall back to S3 when memory is unavailable, even though S3 is working. With local storage disallowed, the worker returns `RESERVE_SLOTS_FAILED`.\n\n The fix allows the worker to create the file on the selected fallback backend. Replication is not required to trigger the issue.\n\n ### Does this PR resolve a correctness bug?\n\n - [ ] Yes\n\n This fixes partition reservation failures, not incorrect shuffle data results.\n\n ### Does this PR introduce _any_ user-facing change?\n\n - [x] Yes\n\n Affected reservations can now succeed using an allowed fallback backend. No new configuration is introduced.\n\n ### How was this patch tested?\n\nCloses #3868 from SparksFyz/fyz/celeborn-2476.\n\nAuthored-by: SparksFyz \u003csparksvann@gmail.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "cbcbeeb8ed4a4ccf770028cd912ac4ed90cb73f6", "tree": "769d09f5332145be624cb38b8b8eac777a5bf918", "parents": [ "626a07cfc055927f6436cba1d97bd4436e779c22" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Tue Oct 13 18:10:40 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Tue Oct 13 18:10:40 2026 +0800" }, "message": "[CELEBORN-2403] Add worker metrics for disk status and unhealthy disk count\n\n### What changes were proposed in this pull request?\n\nAdd two gauges on the worker to expose disk health status, registered in `LocalDeviceMonitor.init()` and evaluated lazily from the live `DiskInfo.status` at scrape time:\n\n1. `UnhealthyDiskCount` — the number of local disks whose status is not `HEALTHY`, intended for alerting (e.g. `\u003e 0`).\n2. `DiskStatus` with a `mountpoint` label — the current `DiskStatus` value (0: HEALTHY, 1: READ_OR_WRITE_FAILURE, 2: IO_HANG, 3: HIGH_DISK_USAGE, 4: CRITICAL_ERROR) of each disk, for locating the exact unhealthy disk.\n\nAlso included: two corresponding panels in the Grafana dashboard (`assets/grafana/celeborn-dashboard.json`) and the new metrics in `docs/monitoring.md`.\n\n### Why are the changes needed?\n\n`DeviceMonitor` already maintains `DiskInfo.status` for every local disk, but none of the existing metrics expose it — `DeviceOSFreeBytes`/`DeviceCelebornFreeBytes` report capacity, and `Device_\u003cname\u003e_\u003cStatus\u003e_Count` only counts non-critical error events. To notice a broken disk, operators have to grep worker logs, which makes alerting on disk failures impossible.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes\n\nTwo new worker metrics are exposed: `UnhealthyDiskCount` and `DiskStatus` (with a `mountpoint` label).\n\n### How was this patch tested?\n\n- Added a new test case `monitor disk status metrics` in `DeviceMonitorSuite` covering the multi-disk scenario: all disks healthy initially, then one disk marked `READ_OR_WRITE_FAILURE` (aggregate count becomes 1 and only that disk\u0027s gauge changes), then recovered to `HEALTHY`.\n- `./build/mvn test -pl worker` passes.\n- `python3 dev/lint_grafana.py assets/grafana/celeborn-dashboard.json` passes.\n- `./build/mvn spotless:check -pl worker` passes.\n\nCloses #3782 from yew1eb/worker-disk-status-metrics.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/213\nAI-Contributed/UT: 0/45\n" }, { "commit": "626a07cfc055927f6436cba1d97bd4436e779c22", "tree": "a74a0206cbdb1b9692f45f84141ab68ff7c93018", "parents": [ "1fe959a3226e53602bebf05a39124875fc0d864f" ], "author": { "name": "Venkata Krishnan Sowrirajan", "email": "vsowrirajan+microsoft@microsoft.com", "time": "Tue Oct 13 14:47:54 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Tue Oct 13 14:52:59 2026 +0800" }, "message": "[CELEBORN-2458] Invalidate transport client after push timeout\n\n## Problem\n\nA push request timeout leaves the pooled `TransportClient` eligible for reuse while its Netty channel still reports active. After a Worker restart or Pod IP replacement, long-running applications can therefore repeatedly send replica pushes over a stale connection and receive `PUSH_DATA_TIMEOUT_REPLICA`.\n\nJIRA: [CELEBORN-2458](https://issues.apache.org/jira/browse/CELEBORN-2458)\n\n## Fix\n\n- Invalidate and asynchronously close the transport client when a push request expires.\n- Ensure `TransportClientFactory` creates a fresh connection for the next request.\n- Temporarily add a timed-out replica peer to `unavailablePeers`, reusing the existing replica fast-fail duration.\n- Preserve higher-level retry/revive semantics rather than replaying the payload in the transport layer.\n- Keep connection invalidation private to the transport implementation.\n\n## Verification\n\n- Added a pooled-client test with multiple outstanding pushes, exactly-once callbacks, connection replacement, and a successful push over the new channel.\n- Added Spark integration assertions for normal and merged replica timeouts populating `unavailablePeers`.\n- Added Worker coverage for fast-fail expiration and subsequent peer recovery.\n- `TransportResponseHandlerSuiteJ` and `TransportClientFactorySuiteJ`: passed.\n- `WorkerSuite`: 7 passed.\n- `PushDataTimeoutTest` with Spark 3.5: 5 passed.\n- Spark 3.5 `common`, `worker`, and `spark-it` package build: passed.\n\nCloses #3842 from venkata91/vsowrirajan/invalidate-push-timeout-client.\n\nLead-authored-by: Venkata Krishnan Sowrirajan \u003cvsowrirajan+microsoft@microsoft.com\u003e\nCo-authored-by: Venkata krishnan Sowrirajan \u003cvsowrirajan@linkedin.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/38\nAI-Contributed/UT: 0/147\n\n(cherry picked from commit c32ccce1f8a2640b8b12002b2ff749a1973df623)\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n" }, { "commit": "1fe959a3226e53602bebf05a39124875fc0d864f", "tree": "eea1ad75bbca95f0ef01c470057ef74b64b6c2fc", "parents": [ "931e730cde445cfd1a1bc185e360dd50f8f32453" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Tue Oct 13 12:34:00 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Tue Oct 13 12:34:34 2026 +0800" }, "message": "[CELEBORN-2462][WORKER] Reduce misleading WARN/ERROR logs during decommission shutdown\n\n### What changes were proposed in this pull request?\n\n1. `Worker.decommissionWorker()`: skip the state transition when the worker is already in `InDecommission` state.\n2. `ReadBufferDispatcher`: catch `InterruptedException` separately in the dispatcher loop — log as INFO when shutting down (`stopFlag\u003dtrue`), WARN otherwise.\n\n### Why are the changes needed?\n\nWhen a worker exits after decommission is triggered, the shutdown hook prints misleading logs that look like real errors:\n\n\u003cimg width\u003d\"1836\" height\u003d\"1626\" alt\u003d\"image\" src\u003d\"https://github.com/user-attachments/assets/ec6801ed-dbb1-4b72-b729-0b84fe3af941\" /\u003e\n\nBoth are harmless shutdown noise:\n\n1. The worker is already in `InDecommission` state (set by the decommission event path via heartbeat response or REST `/exit?type\u003dDECOMMISSION`) when the shutdown hook calls `decommissionWorker()`, and the state machine has no self-loop, so the repeated transition is rejected with a WARN.\n2. `ReadBufferDispatcher.close()` sets `stopFlag\u003dtrue` and interrupts the dispatcher thread blocked in `requests.poll()`; the `InterruptedException` falls into the generic `catch (Throwable)` and is logged as ERROR with a `null` message (`InterruptedException.getMessage()` returns null).\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nCompiled the worker module: `build/mvn -pl worker -am -DskipTests compile`. Logging-only change, no behavior change.\n\nCloses #3847 from yew1eb/CELEBORN-2462.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\nCo-Authored-By: Codex \u003cnoreply@openai.com\u003e\nAI-Model: gpt-6\nAI-Contributed/Feature: 0/10\nAI-Contributed/UT: 0/0\n" }, { "commit": "931e730cde445cfd1a1bc185e360dd50f8f32453", "tree": "bdf4212cc9a248ebecc31d6c22f98c761d761863", "parents": [ "e107e2d0c70e59890b0e428f197947a35cc0b359" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Tue Oct 13 12:23:36 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Tue Oct 13 12:23:36 2026 +0800" }, "message": "[CELEBORN-2463] Add ManuallyExcludedWorkerCount metric to distinguish manually excluded workers\n\n### What changes were proposed in this pull request?\n\nAdd a new master gauge metric `ManuallyExcludedWorkerCount` that reports the count of workers in the manually excluded list (`statusSystem.manuallyExcludedWorkers.size`).\n\nThe existing `ExcludedWorkerCount` metric is kept unchanged (still the sum of both excluded and manually excluded workers) for backward compatibility.\n\n### Why are the changes needed?\n\n`ExcludedWorkerCount` counts both workers excluded due to failure reports / unhealthy heartbeats, and workers excluded manually via the `/exclude` API during maintenance. As a result, alerts based on `ExcludedWorkerCount` fire during routine maintenance even though the cluster is healthy.\n\nThe two sets are already tracked separately internally (`excludedWorkers` vs `manuallyExcludedWorkers`) and exposed separately in the `/workers` API.\n\nWith this PR, users can alert on `ExcludedWorkerCount - ManuallyExcludedWorkerCount` to monitor only failure-driven exclusions.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes\n\nAdd a new master metric `ManuallyExcludedWorkerCount`.\n\n### How was this patch tested?\n\nCompiled the master module locally. The new gauge follows the exact same pattern as the existing `ExcludedWorkerCount` gauge.\n\nCloses #3848 from yew1eb/CELEBORN-2463.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/6\nAI-Contributed/UT: 0/0\n" }, { "commit": "e107e2d0c70e59890b0e428f197947a35cc0b359", "tree": "1664cabb18c4e3b9b7212fdfc90bccd747c50462", "parents": [ "b34cd83a58d0dc0d7f579ea5476b568aeb56d5fb" ], "author": { "name": "strelok89", "email": "alexgilko@outlook.co.il", "time": "Tue Oct 13 11:41:24 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Tue Oct 13 11:42:15 2026 +0800" }, "message": "[CELEBORN-2466] Support per-zone worker StatefulSets and autoscaling in the Helm chart\n\n### What changes were proposed in this pull request?\n\nThree values for `charts/celeborn`, all disabled by default.\n\n**`worker.zoneAwareReplication`** renders one worker StatefulSet per zone instead of a single\none for the whole cluster. Each carries a `celeborn.apache.org/zone` label as part of its\nselector, so a zone\u0027s worker count is declared rather than inferred from scheduling.\n`worker.replicas` becomes the total across zones, split `ceil(replicas / zones)`, with a\nper-zone override; `zones[].nodeSelector` merges over `worker.nodeSelector` and\n`zones[].affinity` replaces `worker.affinity`. All zones share the existing headless service,\nso the service and pod monitor keep selecting every zone.\n\n**`worker.zoneAwareReplication.metricsLabel`** passes the zone into\n`celeborn.metrics.extraLabels` so the worker stamps a `zone` label on everything it emits. It\nis injected as a system property, since master and worker share one config map:\n`Utils.loadDefaultCelebornProperties` applies the file with `setIfMissing`, so a system\nproperty takes precedence, and `start-worker.sh` builds `CELEBORN_JAVA_OPTS` from\n`CELEBORN_WORKER_JAVA_OPTS`, so it composes with whatever `worker.env` sets.\n\n**`worker.drain`** adds the `preStop` hook described below, independent of autoscaling, since\nany rollout removes workers too.\n\n**`worker.autoscaling`** creates a KEDA `ScaledObject` per worker StatefulSet — one per zone\nwith the above enabled. Two triggers ship, on the two resources whose exhaustion takes a\nworker out of service, both scoped by the `zone` label and both excluding a decommissioning\nworker:\n\n- `diskUsage`, on `DeviceCelebornFreeBytes / DeviceCelebornTotalBytes`, default 0.70.\n- `memoryUsage`, on `DirectMemoryUsageRatio`, default 0.70 — below\n `celeborn.worker.directMemoryRatioToPauseReceive` (0.85) so capacity arrives before pushes\n pause.\n\nBoth use `metricType: Value`, since a ratio must not be divided across replicas.\n`worker.autoscaling.triggers` appends user triggers, rendered through `tpl` against the zone\u0027s\ncontext.\n\n`worker.drain` adds a `preStop` hook calling `POST /api/v1/workers/exit` with\n`DECOMMISSION`, because Kubernetes scale-in only sends `SIGTERM`, which never enters the\ndecommission path — the worker would leave with its shuffle data. The hook must not fire on a\nrolling update or node drain, where waiting for shuffle data to expire would block every pod\nreplacement for hours, so it distinguishes them by ordinal: the StatefulSet controller lowers\n`spec.replicas` before deleting pods on a scale-in, so an ordinal at or above the desired count\nis being removed for good, while anything else shuts down gracefully and returns. A desired\ncount that cannot be read falls back to a graceful shutdown, so a failed lookup cannot stall a\nrollout. The hook reads the count with the pod\u0027s own service account, so the role gains `get`\non `statefulsets/scale` when the drain is enabled.\n\n### Why are the changes needed?\n\nWorkers all live in one StatefulSet at a fixed replica count, which leaves two gaps.\n\nTheir spread across availability zones is uneven. The scheduler decides where a worker lands,\nso pod-level spreading (`worker.affinity`, or `worker.topologySpreadConstraints` from\nCELEBORN-2446) balances it only on a best-effort basis: the balance is re-derived on every\nscheduling decision, holds while the constraint is satisfiable, and degrades quietly when it is\nnot — a zone out of capacity for the workers\u0027 instance type, or several nodes replaced at once.\n\nAnd a zone cannot be scaled at all. Load per zone is genuinely uneven when applications select\nworkers by tag (`celeborn.tags.tagsExpr`), so the fleet has to be sized for the busiest zone\nand left that size. A single StatefulSet also cannot be rolled or paused one zone at a time.\n\nNote that this controls where worker pods run and how many there are. It is not zone-aware data\nplacement: which workers a client may use is decided in the master by worker tags, not by pod\ntopology.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes\n\nNew chart values: `worker.zoneAwareReplication` and `worker.autoscaling`. Both default to\ndisabled and the rendered output with default values is unchanged, so existing releases are\nunaffected until they opt in. `charts/celeborn/README.md` documents both, including that\nenabling zone-aware replication renames the worker StatefulSet and so recreates every worker —\na maintenance-window change rather than a rolling update.\n\n### How was this patch tested?\n\n- `helm unittest charts/celeborn --file \"tests/**/*_test.yaml\" --strict` — 121 tests pass, 36\n of them new, covering per-zone StatefulSets and selectors, the replica split and per-zone\n override, the node selector merge, the zone metric label and its composition with existing\n extra labels, the per-zone ScaledObjects and their default triggers, the `preStop` hook with\n its environment and scoped RBAC rule, and each `fail` guard.\n- `helm lint`, with default values and with `charts/celeborn/ci/values.yaml`.\n- `helm template` with default values, diffed against `main`: byte-identical.\n- `helm template` with three zones and autoscaling enabled: three StatefulSets with distinct\n names and selectors, integer replica counts, one shared `serviceName`, unchanged service and\n pod monitor selectors, and one ScaledObject per zone targeting its own StatefulSet.\n- The trigger queries were run against a live Celeborn deployment\u0027s Prometheus to confirm the\n metric names exist and each query returns the single element KEDA requires.\n- `files/worker-drain.sh` was run directly over scale-in, rolling update, always-decommission,\n unreadable replica count, missing pod name, IPv6 bracketing, and an unreachable exit\n endpoint.\n\nCloses #3855 from strelok89/helm-worker-zone-aware-statefulsets.\n\nAuthored-by: strelok89 \u003calexgilko@outlook.co.il\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\nCo-Authored-By: Claude Opus 5.5 \u003cnoreply@anthropic.com\u003e\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\nCo-Authored-By: Codex \u003cnoreply@openai.com\u003e\nAI-Model: gpt-6\nAI-Contributed/Feature: 0/860\nAI-Contributed/UT: 0/852\n" }, { "commit": "b34cd83a58d0dc0d7f579ea5476b568aeb56d5fb", "tree": "40082bb435cdc375682b3f9a91daad87e340312d", "parents": [ "a649f32db5c6c655e3dbde5414e4e9471ab445ce" ], "author": { "name": "Venkata Krishnan Sowrirajan", "email": "venkat.sowrirajan@gmail.com", "time": "Tue Oct 13 11:34:28 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Tue Oct 13 11:35:45 2026 +0800" }, "message": "[CELEBORN-2470] Add backoff with jitter to push retries and log hard split once per file\n\nJIRA: [CELEBORN-2470](https://issues.apache.org/jira/browse/CELEBORN-2470)\n\n## Summary\n\nPush retries currently run immediately and can synchronize across many tasks, increasing connection and DNS load when a worker is unhealthy. Hard splits can also emit the same INFO log repeatedly while pushes continue targeting the old file.\n\nThis PR:\n\n- adds exponential backoff with jitter to push retries\n- logs the hard-split cause once per `PartitionDataWriter`, for both disk and memory files\n- moves the repeated merged-push hard-split message to DEBUG\n\n## Configuration\n\n| Config | Default |\n|---|---|\n| `celeborn.client.push.retry.backoff.initial` | `500ms` |\n| `celeborn.client.push.retry.backoff.max` | `10s` |\n| `celeborn.client.push.retry.backoff.jitter` | `0.5` |\n\nSetting `celeborn.client.push.retry.backoff.initial` to `0` disables backoff. `celeborn.client.push.retry.backoff.max` caps the final delay, including jitter, so with the defaults a retry at the cap waits between 5s and 10s.\n\n## Tests\n\n- added coverage for exponential growth, maximum delay, jitter bounds (including at the cap), disabled backoff, and large retry counts\n- added callback-level coverage for scheduled single and merged `HARD_SPLIT` retries and revive completion during backoff\n- added concurrent coverage verifying the hard-split log latch triggers once per writer\n- added coverage that repeated memory-shuffle `HARD_SPLIT` checks log only once\n\nCloses #3856 from venkata91/celeborn-push-retry-backoff.\n\nAuthored-by: Venkata Krishnan Sowrirajan \u003cvenkat.sowrirajan@gmail.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/143\nAI-Contributed/UT: 0/223\n\nCo-authored-by: Copilot \u003c223556219+Copilot@users.noreply.github.com\u003e\nCo-authored-by: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n" }, { "commit": "a649f32db5c6c655e3dbde5414e4e9471ab445ce", "tree": "2676a971828fa9e36616a5faffedca434e0ec7b2", "parents": [ "9a088699ba1689136c4247947176de29fb6b74fb" ], "author": { "name": "senthh", "email": "senthil.kumar@acceldata.io", "time": "Tue Oct 13 10:05:40 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Tue Oct 13 10:06:09 2026 +0800" }, "message": "[CELEBORN-2472] Unregister Celeborn application from MR AppMaster on shutdown\n\n### What changes were proposed in this pull request?\n\nStore the `LifecycleManager` as a field in `MRAppMasterWithCeleborn` and stop it from `serviceStop()` (rather than `stop()`), so the application is unregistered from the Celeborn master on MR AppMaster shutdown. `LifecycleManager.stop() -\u003e heartbeater.stop() -\u003e unregisterApplication()` sends `ApplicationLost` (gated by `celeborn.client.application.unregister.enabled`), releasing worker slots and shuffle files immediately.\n\nOnly `client-mr/mr/src/main/java/org/apache/celeborn/mapreduce/v2/app/MRAppMasterWithCeleborn.java` is changed.\n\n### Why are the changes needed?\n\n`MRAppMasterWithCeleborn` creates a `LifecycleManager` and registers the application with the Celeborn master, but never stops it or unregisters on shutdown. The master therefore only frees the application (and the worker\u0027s shuffle files / slots) via heartbeat expiry (`appHeartbeatTimeoutMs`, default 120s), leaking worker disk after every MR job. Stopping from `serviceStop()` runs exactly once even though both `MRAppMaster#shutDownJob()` and the JVM shutdown hook can reach `stop()`. This mirrors the Tez client `CelebornDagAppMaster#serviceStop()` and matches every other engine client (Spark, Flink), which already stop the `LifecycleManager` on exit.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\n- Compiles clean; the change follows the identical shutdown pattern already used by the other engine clients (Spark, Flink, Tez).\n\nCloses #3864 from senthh/CELEBORN-2472.\n\nAuthored-by: senthh \u003csenthil.kumar@acceldata.io\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nCo-Authored-By: Codex \u003cnoreply@openai.com\u003e\nAI-Model: gpt-6\nAI-Contributed/Feature: 0/28\nAI-Contributed/UT: 0/0\n" }, { "commit": "9a088699ba1689136c4247947176de29fb6b74fb", "tree": "a7e397ed9fe69f3e66ffd4a115375afeda5651cb", "parents": [ "054af9d9990a280199b5532b988829389a0aed16" ], "author": { "name": "senthh", "email": "senthil.kumar@acceldata.io", "time": "Mon Oct 12 22:23:00 2026 +0800" }, "committer": { "name": "Shuang", "email": "lvshuang.xjs@alibaba-inc.com", "time": "Mon Oct 12 22:23:00 2026 +0800" }, "message": "[CELEBORN-2474] Send WWW-Authenticate challenge when no auth header present\n\n### What changes were proposed in this pull request?\n\nWhen `AuthenticationFilter.doFilter` receives a request with no `Authorization` header (the first request in a SPNEGO/Kerberos handshake), respond `401` with `WWW-Authenticate` headers listing all registered auth schemes, so a SPNEGO client (e.g. `curl --negotiate`) can receive the challenge and respond with its Kerberos token. If an `Authorization` header is present but matches no handler, the original error message is preserved.\n\nAlso fix the scheme name in the `supportedSchemes` documentation (`celeborn.master.http.auth.supportedSchemes` / `celeborn.worker.http.auth.supportedSchemes`): the Kerberos scheme was listed as `SPNEGO` but the enumeration value is `NEGOTIATE`. Updated `CelebornConf.scala`, `docs/configuration/master.md` and `docs/configuration/worker.md` for consistency with the `ConfigurationSuite` golden-file check.\n\nFiles:\n- `service/src/main/scala/org/apache/celeborn/server/common/http/authentication/AuthenticationFilter.scala`\n- `common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala`\n- `docs/configuration/master.md`\n- `docs/configuration/worker.md`\n\n### Why are the changes needed?\n\nPreviously the filter returned a bare `401` with no `WWW-Authenticate` header, so the SPNEGO handshake was broken at the very first step (error: \"No auth scheme matched for null\"). A SPNEGO client never receives the challenge and cannot send its Kerberos token.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nVerified with `curl --negotiate` against a Celeborn server configured with:\n- `celeborn.master.http.auth.supportedSchemes NEGOTIATE`\n- `celeborn.master.http.spnego.keytab /path/to/keytab`\n- `celeborn.master.http.spnego.principal HTTP/hostREALM`\n\nBefore: `401` without `WWW-Authenticate` header, curl never sends the token.\nAfter: `401` with `WWW-Authenticate: NEGOTIATE`, curl completes the SPNEGO handshake and returns `200`.\n\nCloses #3863 from senthh/CELEBORN-9999.\n\nAuthored-by: senthh \u003csenthil.kumar@acceldata.io\u003e\nSigned-off-by: Shuang \u003clvshuang.xjs@alibaba-inc.com\u003e\n" }, { "commit": "054af9d9990a280199b5532b988829389a0aed16", "tree": "dade9096f3c56b033c698a919d674eba46134da6", "parents": [ "4d7fb8e4c88e8b63cffe518e59b325f5eaf47ef3" ], "author": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Mon Oct 12 22:16:17 2026 +0800" }, "committer": { "name": "Shuang", "email": "lvshuang.xjs@alibaba-inc.com", "time": "Mon Oct 12 22:16:17 2026 +0800" }, "message": "[CELEBORN-2475] Fix stale HA configuration documentation that breaks CI\n\n### What changes were proposed in this pull request?\n\nAlign the generated HA configuration documentation with `ConfigurationSuite` output by restoring the missing trailing space after the final table delimiter for `celeborn.master.ha.ratis.unexpected.close.action`.\n\n### Why are the changes needed?\n\n`ConfigurationSuite` compares generated configuration rows with `docs/configuration/ha.md` exactly. The checked-in row differs by one trailing space, causing six service jobs in the [main SBT CI run](https://github.com/apache/celeborn/actions/runs/38038252682) to fail across Scala 2.12 and 2.13. The change only fixes the documentation golden file.\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\nNo runtime behavior or rendered documentation changes.\n\n### How was this patch tested?\n\n- Verified that the complete file differs only by the trailing space required by `ConfigurationSuite`\u0027s table-row generator.\n- Attempted `build/mvn test -pl common -am -Dtest\u003dnone -DwildcardSuites\u003dorg.apache.celeborn.ConfigurationSuite`; it could not reach the tests because this local environment does not have `javac`. The PR CI will run the suite.\n\nCloses #3866 from zaynt4606/CELEBORN-2475-fix-ha-config-doc.\n\nAuthored-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\nSigned-off-by: Shuang \u003clvshuang.xjs@alibaba-inc.com\u003e\n" }, { "commit": "4d7fb8e4c88e8b63cffe518e59b325f5eaf47ef3", "tree": "b4bd152eab5eb0fe6958123e69466c053b946795", "parents": [ "ec098ee8db8843721ef1e6e7c09aee2cbccd2e42" ], "author": { "name": "wang-haihua", "email": "wang-haihua@foxmail.com", "time": "Mon Oct 12 21:18:36 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Mon Oct 12 21:20:25 2026 +0800" }, "message": "[CELEBORN-2452] Prevent device error broadcast from deleting committed shuffle files\n\n### What changes were proposed in this pull request?\n\nAdd a `committed` flag on `TierWriterBase`, set in `close()` right before `notifyFileCommitted()`. `PartitionDataWriter.notifyError` then skips `destroy()` when the writer is already committed, so a disk `CRITICAL_ERROR` broadcast no longer deletes committed files.\n\nThe rollback path (`Controller.destroyWriters`) and shuffle-expiry cleanup (`StorageManager.cleanFileInternal`) call `destroy()` directly rather than through `notifyError`, so they are unaffected and still remove committed files when the shuffle actually ends.\n\n### Why are the changes needed?\n\n`DeviceMonitor` broadcasts `CRITICAL_ERROR` to every writer registered on a disk:\n\n```\nObservedDevice.notifyObserversOnError(CRITICAL_ERROR)\n -\u003e PartitionDataWriter.notifyError -\u003e destroy(IOException)\n -\u003e TierWriterBase.destroy -\u003e cleanLocalOrDfsFiles()\n -\u003e DiskFileInfo.deleteAllFiles(null) // deletes data + index + sorted files\n```\n\nA `LocalTierWriter` registers as an observer at file creation and, before this fix, is removed from the observer set only on the `destroy` path — never when the file commits. So a committed writer stays registered for the entire remaining lifetime of the shuffle. If the disk crosses the `CRITICAL_ERROR` threshold during that window, the broadcast destroys the committed writer and deletes its data/index/sorted files out from under in-flight reducers, causing `FileNotFoundException` on fetch and genuine shuffle data loss. The broadcast hits every observer on the disk, so a committed file is deleted merely for sharing a disk with unrelated failing writes.\n\nSee [CELEBORN-2452](https://issues.apache.org/jira/browse/CELEBORN-2452) for full analysis.\n\n### Does this PR resolve a correctness bug?\n\n- [x] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\n- Existing `DiskReducePartitionDataWriterSuiteJ` write/close tests pass, confirming the added `committed` flag in `close()` doesn\u0027t regress the normal commit path.\n- The `notifyError` guard itself is verified by tracing all `destroy()` callers (device-error must not delete committed files; rollback/expiry still must) and by a successful `worker` test-compile.\n\nCloses #3837 from wang-haihua/CELEBORN-2452-committed-file-deletion.\n\nAuthored-by: wang-haihua \u003cwang-haihua@foxmail.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nCo-Authored-By: Codex \u003cnoreply@openai.com\u003e\nAI-Model: gpt-6\nAI-Contributed/Feature: 0/9\nAI-Contributed/UT: 0/0\n" }, { "commit": "ec098ee8db8843721ef1e6e7c09aee2cbccd2e42", "tree": "2e6db6f831be0597cfa6d0707885d2d8afe61c19", "parents": [ "63381a8593b6402965ffbd9dde7058bfc04e027b" ], "author": { "name": "strelok89", "email": "alexgilko@outlook.co.il", "time": "Mon Oct 12 19:36:28 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Mon Oct 12 19:37:17 2026 +0800" }, "message": "[CELEBORN-2446] Support topologySpreadConstraints for master and worker pods in Helm chart\n\n### What changes were proposed in this pull request?\n\nAdd support for `topologySpreadConstraints` on Celeborn master and worker pods in the Helm chart:\n\n- `charts/celeborn/templates/master/statefulset.yaml` and `charts/celeborn/templates/worker/statefulset.yaml` — render `topologySpreadConstraints` from the corresponding value, following the existing `{{- with }}` idiom already used for `nodeSelector`, `affinity` and `tolerations`.\n- `charts/celeborn/values.yaml` — declare `topologySpreadConstraints: []` for both roles.\n- `charts/celeborn/tests/{master,worker}/statefulset_test.yaml` — unit tests covering both the unset (field absent) and set (field rendered) cases.\n\nExample usage:\n\n```yaml\nworker:\n topologySpreadConstraints:\n - maxSkew: 1\n topologyKey: topology.kubernetes.io/zone\n whenUnsatisfiable: DoNotSchedule\n labelSelector:\n matchLabels:\n app.kubernetes.io/name: celeborn\n app.kubernetes.io/role: worker\n```\n\n### Why are the changes needed?\n\nThe chart already exposes `nodeSelector`, `affinity`, `tolerations` and `priorityClass` for both master and worker pods, but not `topologySpreadConstraints`. Setting the value today is silently ignored — Helm accepts `worker.topologySpreadConstraints` without error and nothing is rendered on the StatefulSet, so the constraint simply does not exist.\n\n`podAntiAffinity` — which the chart defaults to, on `kubernetes.io/hostname` — cannot substitute for a multi-AZ deployment. Pod anti-affinity is one-or-none per topology domain: it can express \"at most one pod per zone\", but not \"N pods per zone\" or \"spread evenly across zones within a bounded skew\". That is what `topologySpreadConstraints` is for.\n\nThis also lines up with item 8, \"Enhanced K8S support\", on the improvements schedule in `CONTRIBUTING.md`.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes\n\nTwo new chart values, `master.topologySpreadConstraints` and `worker.topologySpreadConstraints`, both defaulting to `[]`. When they are unset the rendered StatefulSet is unchanged, so this is purely additive.\n\n### How was this patch tested?\n\n- `helm unittest charts/celeborn --file \"tests/**/*_test.yaml\" --strict` — all chart unit tests pass, including the four new cases. Reverting only the template hunks makes exactly those new assertions fail, confirming they exercise the change.\n- `helm lint charts/celeborn` — passes.\n- `helm template` with a zone-spread constraint set renders `topologySpreadConstraints` on the worker StatefulSet; `helm template` with default values renders no `topologySpreadConstraints` key anywhere.\n\nCloses #3831 from strelok89/helm-topology-spread-constraints.\n\nAuthored-by: strelok89 \u003calexgilko@outlook.co.il\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\nCo-Authored-By: Codex \u003cnoreply@openai.com\u003e\nAI-Model: gpt-6\nAI-Contributed/Feature: 0/14\nAI-Contributed/UT: 0/60\n" }, { "commit": "63381a8593b6402965ffbd9dde7058bfc04e027b", "tree": "8f037a9dd307279ad91c1fce096251b8db44ef06", "parents": [ "71724db23980cc80cb920fee2a057b96f26893f7" ], "author": { "name": "afterincomparableyum", "email": "224495379+afterincomparableyum@users.noreply.github.com", "time": "Mon Oct 12 19:02:05 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Mon Oct 12 19:02:36 2026 +0800" }, "message": "[CELEBORN-2359] Deduplicate # HELP/# TYPE lines in PrometheusServlet output\n\n### What changes were proposed in this pull request?\n\nAbstractSource emits a `# HELP` and `# TYPE` line for every metric sample. When a metric family has multiple label sets — like ActiveConnectionCount (per application), RequestSlotsFailedCount (per status code), per-user/app resource consumption, or FlushWorkingQueueSize (per mountpoint) — the same family is exported with repeated `# HELP` and `# TYPE` lines. This violates the Prometheus exposition format.\n\nThe fix is to override PrometheusServlet.getMetricsSnapshot to keep only the first `# HELP` and `# TYPE` line per metric family while preserving all sample lines, producing valid Prometheus output.\n\nI also added a PrometheusServletSuite covering a multi-label-set counter and a distinct gauge.\n\n### Why are the changes needed?\n\nTo dedup the help and type lines to no longer violate prometheus exposition format.\n\n### Does this PR resolve a correctness bug?\n\n- [x] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nCI/CD\n\nCloses #3852 from afterincomparableyum/CELEBORN-2359.\n\nAuthored-by: afterincomparableyum \u003c224495379+afterincomparableyum@users.noreply.github.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\nCo-Authored-By: Codex \u003cnoreply@openai.com\u003e\nAI-Model: gpt-6\nAI-Contributed/Feature: 0/46\nAI-Contributed/UT: 0/106\n" }, { "commit": "71724db23980cc80cb920fee2a057b96f26893f7", "tree": "046566b0f6612e1c3250414be0c4212da1dc2de6", "parents": [ "29f3bae144de507e23fcb1ae92470e9a0dad5c50" ], "author": { "name": "strelok89", "email": "alexgilko@outlook.co.il", "time": "Mon Oct 12 17:47:47 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Mon Oct 12 17:50:24 2026 +0800" }, "message": "[CELEBORN-2361] Unblock stage-end when CommitFilesResponse carries an empty map-id bitmap\n\nThis revives #3736 by afterincomparableyum, which was closed by the stale bot rather than\nrejected. Rebased onto current `main`, credit preserved via `Co-Authored-By`.\n\nA worker serializes an empty `RoaringBitmap` as an empty protobuf bytestring, and\n`Utils.byteStringToRoaringBitmap` deserializes empty bytes back to `null`. That `null` was\nstored in `CommitFilesResponse.committedMapIdBitMap` (a `HashMap`, which allows null values)\nand then merged into the `ConcurrentHashMap` `ShuffleCommittedInfo.committedMapIdBitmap` via\n`putAll`, which rejects null values and throws NPE in `CommitHandler.processResponse`.\n\nSince `ReducePartitionCommitHandler.tryFinalCommit` had no `try/finally`, the shuffle was\nleft in `inProcessStageEndShuffleSet` and `setStageEnd` was never called, so every\n`GetReducerFileGroup` request parked until the executor RPC timeout, hanging the Spark\nclient process.\n\nFixes:\n\n- `ControlMessages`: skip null bitmaps when deserializing `CommitFilesResponse` so `null`\n never reaches `ConcurrentHashMap.putAll`.\n- `ReducePartitionCommitHandler`: wrap the commit in `try/finally` so the in-process\n stage-end marker is always cleared, allowing a retry.\n\nThe original revision of this PR also made `CelebornInputStream.skipLocation` treat a null\nmapId bitmap as \"do not skip\". That has since landed on `main` via CELEBORN-2442\n(`adadbd924`), which moved the logic into `CelebornInputStream.shouldSkipLocation` with the\nsame null-safe behavior, so it is dropped here.\n\nThe null-means-unknown semantics are kept deliberately. The wire format collapses null and\nempty to `ByteString.EMPTY`, so making `byteStringToRoaringBitmap` return an empty bitmap\nwould turn \"unknown\" into \"no map ids here\" at the `PbSerDeUtils` call sites and make\n`shouldSkipLocation` skip locations that must be read.\n\nWithout this, Spark clients can hang for a long time and emit repeated NPEs.\n\n- [x] Yes\n\n- [ ] Yes\n\nAdded a `UtilsSuite` regression test asserting a `CommitFilesResponse` with an empty bitmap\nsurvives the transport round-trip without null map values. Verified it fails without the\n`ControlMessages` fix and passes with it. `celeborn-client/Test/compile` and\n`spotless:check -pl common,client` pass locally; the rest is left to CI.\n\n🤖 Generated with [Claude Code](https://claude.com/claude-code)\n\nCloses #3833 from strelok89/CELEBORN-2361.\n\nAuthored-by: strelok89 \u003calexgilko@outlook.co.il\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nCo-Authored-By: afterincomparableyum \u003c224495379+afterincomparableyum@users.noreply.github.com\u003e\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\nCo-Authored-By: Codex \u003cnoreply@openai.com\u003e\nAI-Model: gpt-6\nAI-Contributed/Feature: 0/44\nAI-Contributed/UT: 0/30\n" }, { "commit": "29f3bae144de507e23fcb1ae92470e9a0dad5c50", "tree": "e7a10da75cb8c160060ff89b54d26887ec10f70d", "parents": [ "1256618a2b4d0119deb365990cb01dcf30ffabaf" ], "author": { "name": "afterincomparableyum", "email": "224495379+afterincomparableyum@users.noreply.github.com", "time": "Mon Oct 12 17:02:32 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Mon Oct 12 17:02:32 2026 +0800" }, "message": "[CELEBORN-2356] Fix NPE before null-check in handleChunkFetchRequest\n\n### What changes were proposed in this pull request?\n\ngetStreamState returns null for an unknown id. The streamState.buffers pattern match dereferences it before the null guard runs, throwing NullPointerException. The intended ChunkFetchFailure reply is unreachable. Here we should reorder null-check first, then compute storageMetrics.\n\n### Why are the changes needed?\n\nTo prevent potential NPEs.\n\n### Does this PR resolve a correctness bug?\n\n- [x] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nCI\n\nCloses #3850 from afterincomparableyum/CELEBORN-2356.\n\nAuthored-by: afterincomparableyum \u003c224495379+afterincomparableyum@users.noreply.github.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/16\nAI-Contributed/UT: 0/33\n" }, { "commit": "1256618a2b4d0119deb365990cb01dcf30ffabaf", "tree": "bdf058f13d7b0f509fcc9b48f6bac4f1c716b53d", "parents": [ "ad5b99f24808e89f9273e33ce79f7b9b8df30d32" ], "author": { "name": "afterincomparableyum", "email": "224495379+afterincomparableyum@users.noreply.github.com", "time": "Mon Oct 12 14:14:48 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Mon Oct 12 14:14:48 2026 +0800" }, "message": "[CELEBORN-2357] Add Metrics Prefix\n\n### What changes were proposed in this pull request?\n\nEnable users the option to prefix their metrics with something unique that will allow them to easily find any celeborn related metircs in observability platforms.\n\n### Why are the changes needed?\n\nUsers will be able to easily identify celeborn metrics on observability tools with this prefix.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nCI/CD\n\nCloses #3851 from afterincomparableyum/celeborn-2357.\n\nAuthored-by: afterincomparableyum \u003c224495379+afterincomparableyum@users.noreply.github.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/18\nAI-Contributed/UT: 0/31\n" }, { "commit": "ad5b99f24808e89f9273e33ce79f7b9b8df30d32", "tree": "d56d2d914dca9f8be73a00b4e261d6c7acca5b56", "parents": [ "8c6e0c03a78ebab6836c9b06958a79c893dc0c26" ], "author": { "name": "senthh", "email": "senthil.kumar@acceldata.io", "time": "Mon Oct 12 11:34:11 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Mon Oct 12 11:34:11 2026 +0800" }, "message": "[CELEBORN-2473] Do not relocate zstd-jni in the Tez shaded jar\n\n## [CELEBORN-2473] Do not relocate zstd-jni in the Tez shaded jar\n\n**Problem:** the Maven shade plugin in `client-tez/tez-shaded/pom.xml` and the sbt assembly rules in `project/CelebornBuild.scala` relocate `com.github.luben` (zstd-jni) to the shading prefix. zstd-jni is a JNI library whose bundled native `.so`/`.dll` exports symbols using the original package names (e.g. `Java_com_github_luben_zstd_Zstd_compressBound`). After relocation the JVM binds those symbols from the remapped class names, so the native load fails with `UnsatisfiedLinkError`/`NoClassDefFoundError` and ZSTD compression in the Tez shuffle breaks.\n\n**Fix:** keep zstd-jni bundled (it stays in the `artifactSet` includes) but **stop relocating** `com.github.luben`. This matches `client-mr/mr-shaded`, which already bundles zstd-jni without remapping it. (Spark/Flink shaded clients don\u0027t bundle zstd-jni at all — those engines ship their own — so `mr-shaded` is the direct precedent.)\n\n**Files:**\n- `client-tez/tez-shaded/pom.xml` (−4)\n- `project/CelebornBuild.scala` (−1)\n\n**Validation:** pom remains well-formed XML; zstd-jni still present in the artifact set. cf. HADOOP-17891 / HBASE-23007: never relocate JNI libraries.\n\nCloses #3865 from senthh/CELEBORN-2473.\n\nAuthored-by: senthh \u003csenthil.kumar@acceldata.io\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/5\nAI-Contributed/UT: 0/0\n" }, { "commit": "8c6e0c03a78ebab6836c9b06958a79c893dc0c26", "tree": "bb6d3aa5aaa0ba224c2e2c2c77251990eddecc18", "parents": [ "c090945514d39f888ac27a706e4807df7dd815b8" ], "author": { "name": "sychen", "email": "sychen@ctrip.com", "time": "Mon Oct 12 10:12:43 2026 +0800" }, "committer": { "name": "Shuang", "email": "lvshuang.xjs@alibaba-inc.com", "time": "Mon Oct 12 10:12:43 2026 +0800" }, "message": "[MINOR] Update pnpm/action-setup to specific version v6.1.0 in web_lint.yml\n\n### What changes were proposed in this pull request?\n\nhttps://github.com/apache/infrastructure-actions/blob/43e849c8f402e1b92de378a1a7e603e5aad0e492/actions.yml#L907-L912\n\n### Why are the changes needed?\n\nhttps://github.com/apache/celeborn/actions/runs/37881121516\n\n\u003e The action pnpm/action-setupv4 is not allowed\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nCloses #3862 from cxzl25/web_lint_pnpm_version.\n\nAuthored-by: sychen \u003csychen@ctrip.com\u003e\nSigned-off-by: Shuang \u003clvshuang.xjs@alibaba-inc.com\u003e\n" }, { "commit": "c090945514d39f888ac27a706e4807df7dd815b8", "tree": "d74f0711af2b507838779d611e12f35bf28cc231", "parents": [ "abdfd05d804e8a128682d1d87c540501ed5f8be2" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Sat Oct 10 17:56:56 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Sat Oct 10 17:56:56 2026 +0800" }, "message": "[CELEBORN-2434] Clamp endMapIndex in sliceSortedBufferByMapRange\n\n### What changes were proposed in this pull request?\n\nAdd the missing `endMapIndex \u003d\u003d Integer.MAX_VALUE` clamp in `ShuffleBlockInfoUtils.sliceSortedBufferByMapRange`, mirroring the existing guard in the sibling method `getChunkOffsetsFromShuffleBlockInfos`. Also add a unit test covering the full re-read (`Integer.MAX_VALUE`) case.\n\n### Why are the changes needed?\n\nWhen a memory-stored shuffle partition is read once with a map range (e.g. AQE skew join) and then re-read in full (`endIndex \u003d Int.MaxValue`, e.g. Spark ReusedExchange), `FetchHandler.handleReduceOpenStreamInternal` passes the raw `Int.MaxValue` into `getSortedFileInfo` because `FileInfo.addStream` returns false for the already-sorted file. The memory branch then loops `startMapIndex..Integer.MAX_VALUE` in `sliceSortedBufferByMapRange` — billions of `TreeMap.get` calls on the Netty fetch event loop, stalling all fetch/openStream traffic on that event loop for minutes to hours.\n\nThe disk path is not affected (it goes through the already-clamped `getChunkOffsetsFromShuffleBlockInfos`).\n\nSymptoms / evidence:\n\n• MemoryStorageReusedExchangeSuite (a single tiny test with 1000 rows) takes 28–54 minutes instead of ~1 minute, both in CI and locally.\n• Client side repeatedly logs BatchOpenStream for N cost 240006ms (exactly the 240s fetch timeout), followed by retries that stall again.\n• Client logs Connection ... has been quiet for 4.0 m while there are outstanding requests against both workers\u0027 fetch ports.\n• A thread dump taken while hung shows the fetch event loop spinning:\n\n```\n \"fetch-server-34-2\" RUNNABLE\n at java.util.TreeMap.getEntry(TreeMap.java:359)\n at java.util.TreeMap.get(TreeMap.java:278)\n at ShuffleBlockInfoUtils.sliceSortedBufferByMapRange(ShuffleBlockInfoUtils.java:140)\n at PartitionFilesSorter.getSortedFileInfo(PartitionFilesSorter.java:224)\n at FetchHandler.handleReduceOpenStreamInternal(FetchHandler.scala:277)\n```\n\nIn production this means: memory storage + a range read followed by a full re-read of the same partition can wedge a worker\u0027s fetch event loop for an\nunbounded time (the loop count scales with Int.MaxValue, multiplied by the number of files in the batch request).\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\n- Added `testSliceSortedBufferByMapRangeWithMaxEndIndex` to `ShuffleBlockInfoUtilsTest`, asserting a full re-read with `Integer.MAX_VALUE` produces exactly the same result as an explicit full-range read.\n- `ShuffleBlockInfoUtilsTest`: all 6 tests pass.\n\nCloses #3815 from yew1eb/CELEBORN-2434.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/7\nAI-Contributed/UT: 0/60\n" }, { "commit": "abdfd05d804e8a128682d1d87c540501ed5f8be2", "tree": "e117852639bd4571e462c1015f6a661ed65d10d9", "parents": [ "cfaac4e076e1d8eb23b147a485c0171ea1efb837" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Sat Oct 10 16:32:34 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Sat Oct 10 16:32:34 2026 +0800" }, "message": "[CELEBORN-2453] Master should fail fast when the Ratis server is closed unexpectedly\n\n### What changes were proposed in this pull request?\nWhen the embedded Ratis server is closed unexpectedly (e.g. by `JvmPauseMonitor` after a JVM pause longer than `raft.server.close.threshold`), the master process keeps running as a zombie: its RPC/HTTP ports stay alive but it has silently left the Raft group.\nThis PR makes the master fail fast in this situation:\n1. Add a new config `celeborn.master.ha.ratis.unexpected.close.action` (`exit` / `none`, default `exit`). When the periodic role check detects the local `RaftServer` has reached a terminal lifecycle state (`CLOSED` / `EXCEPTION`), the master logs the root cause and exits, so the HA group keeps a healthy quorum and the node can be restarted by external supervision (systemd/k8s).\n2. `HARaftServer.updateServerRole()` short-circuits when the Raft server is in a terminal state (`CLOSED` / `EXCEPTION`), avoiding repeated ERROR log flooding from subsequent role checks and metrics scrapes, while leaving transitional states (`STARTING` / `CLOSING`) on the existing code path.\n\n### Why are the changes needed?\n In our production, one master of a 3-node HA cluster got stuck for ~78s by `jmap -F -histo` (non-GC pause). Ratis `JvmPauseMonitor` then closed the local RaftServer:\n\u003cimg width\u003d\"2314\" height\u003d\"1124\" alt\u003d\"image\" src\u003d\"https://github.com/user-attachments/assets/1ed38a5b-e707-4a80-83fe-4aa9f802700d\" /\u003e\n\nAfter that the master process was still alive but out of the Raft group, role became `UNRECOGNIZED`, and every metrics scrape / role check printed a full ERROR stack (`ServerNotReadyException`, `SegmentedRaftLog is expected to be opened but it is CLOSED`). Nothing recovers the node except a manual restart — failing fast makes the failure explicit and lets supervision restart it.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n- New unit test `RatisMasterStatusSystemSuiteJ#testUnexpectedRaftServerClose`.\n- Existing master module tests pass (`./build/mvn test -pl master`).\n\nThis patch had conflicts when merged, resolved by\nCommitter: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nCloses #3838 from yew1eb/CELEBORN-2453.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/76\nAI-Contributed/UT: 0/58\n" }, { "commit": "cfaac4e076e1d8eb23b147a485c0171ea1efb837", "tree": "19534e45b505446aae275b456534b4f62d4234ae", "parents": [ "9dfb74bdaaa3af217b3b80e0fc4e183a3ba74f90" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Sat Oct 10 16:29:22 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Sat Oct 10 16:29:22 2026 +0800" }, "message": "[CELEBORN-2454] Add config celeborn.master.ha.ratis.raft.server.close.threshold and docs for Ratis JvmPauseMonitor\n\n### What changes were proposed in this pull request?\n\nAdd a config `celeborn.master.ha.ratis.raft.server.close.threshold` (default 60s) for the Ratis `raft.server.close.threshold`, and document the JvmPauseMonitor behavior in the deploy guide.\n\nThe dedicated config takes precedence over the `celeborn.ratis.raft.server.close.threshold` passthrough when both are set; the passthrough alone still applies for backward compatibility.\n\n### Why are the changes needed?\nRelated PR: https://github.com/apache/celeborn/pull/3838\n\nRatis `JvmPauseMonitor` closes the local RaftServer when a JVM pause exceeds `raft.server.close.threshold` (default 60s), e.g. caused by `jmap -F`, long GC, or disk I/O hang. Today this threshold can only be set via the generic `celeborn.ratis.*` custom config passthrough, which is undocumented and easy to miss — operators hit the unexpected master shutdown without knowing why or how to tune it.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes - adds a new configuration `celeborn.master.ha.ratis.raft.server.close.threshold` and documents the Ratis JvmPauseMonitor behavior (including low-intrusion GC diagnostics guidance) in the deployment guide.\n\n### How was this patch tested?\nExisting master HA tests pass.\n\nCloses #3839 from yew1eb/CELEBORN-2454.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/39\nAI-Contributed/UT: 0/0\n" }, { "commit": "9dfb74bdaaa3af217b3b80e0fc4e183a3ba74f90", "tree": "9d57365f08034ead060cf1394a1351fcfd9c493a", "parents": [ "43b6e657ebcd5e47ebad6f3e285c277b32a730ce" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Sat Oct 10 15:10:34 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Sat Oct 10 15:10:34 2026 +0800" }, "message": "[CELEBORN-2456] Fix wrong isAssignableFrom direction in Basic/Bearer authenticationSupported\n\n### What changes were proposed in this pull request?\n\n Fix the reversed `isAssignableFrom` direction in `BasicAuthenticationHandler` and `BearerAuthenticationHandler` (its result was also discarded, making the check a no-op), and fail fast in `AuthenticationFilter.init` when schemes are configured but no handler is registered.\n\n### Why are the changes needed?\n\nThe broken check lets invalid providers pass startup validation; if the provider class cannot be loaded and it is the only configured scheme, all requests are silently served without authentication.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\nAdded `AuthenticationHandlerSuite`\n\nCloses #3841 from yew1eb/CELEBORN-2456.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/11\nAI-Contributed/UT: 0/52\n" }, { "commit": "43b6e657ebcd5e47ebad6f3e285c277b32a730ce", "tree": "f7d98f71036d8ebc34764d58c8d0315f7e94410f", "parents": [ "5c116df97be78538f8c98140f8b833ca72efda3f" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Sat Oct 10 14:16:35 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Sat Oct 10 14:16:35 2026 +0800" }, "message": "[CELEBORN-2455] Remove redundant synchronized in Dispatcher.postMessage\n\n### What changes were proposed in this pull request?\n\nRemove the redundant `synchronized` block around the first `endpoints.get(endpointName)` read in `Dispatcher.postMessage`.\n\n### Why are the changes needed?\n\n `endpoints` is a `ConcurrentHashMap`, so the read is already thread-safe.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\nExisting UTs.\n\nCloses #3840 from yew1eb/CELEBORN-2455.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/4\nAI-Contributed/UT: 0/0\n" }, { "commit": "5c116df97be78538f8c98140f8b833ca72efda3f", "tree": "80f87bf610a81b9f52e53d5e000cf980c4b79053", "parents": [ "f43f424c04c4259a18251987965ddb25745e28e4" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Fri Oct 09 22:06:17 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Fri Oct 09 22:06:17 2026 +0800" }, "message": "[CELEBORN-2415] Add Celeborn Spark UI extension\n\n### What changes were proposed in this pull request?\n\nAdd a new module `celeborn-client-spark-3-ui` — a Spark UI extension that renders a **\"Celeborn\" tab** in both the live Spark UI and the History Server (enabled via `spark.plugins\u003dorg.apache.spark.shuffle.celeborn.CelebornPlugin`, supports Spark 3.x and 4.x).\n\nThis is an **initial version** that lays the foundation; it intentionally starts minimal. The page shows:\n\n1. **Summary** — shuffle write/read bytes, time, speed, and shuffle-vs-task duration ratio, aggregated from Spark\u0027s built-in `TaskMetrics`\n2. **Celeborn Properties** — a collapsible table of `spark.celeborn.*` configs\n\nThe skeleton is: `CelebornPlugin` (live UI) / `CelebornHistoryServerPlugin` (SHS, via `AppHistoryServerPlugin` SPI) → listener aggregates into Spark `KVStore` → `CelebornStatusStore` → `CelebornUITab` renders; a `javax`/`jakarta` servlet bridge covers Spark 3.x and 4.x. This plugin/store/render pipeline is designed so that fine-grained Celeborn client metrics can be added as collapsible sections in follow-up PRs:\n\n- **Shuffle Write Times** — serialize / copy / queue wait / compress / queue stall / inflight wait / drain wait / max push RTT / slow push\n- **Shuffle Read Times** — chunk wait / decompress / retry wait / max chunk RTT / slow chunk\n- **Shuffle Write Servers** — per-worker push bytes, push count, RTT, soft/hard splits, congested counts, last failure reason\n- **Shuffle Read Servers** — per-worker chunk count, read bytes, total/max RTT\n- **Shuffle Assignments** — partition-to-worker slot assignments\n\n### Why are the changes needed?\n\nSpark UI already shows stage/task-level shuffle metrics, but nothing `Celeborn-specific`: which workers served the data, push/chunk latencies, congestion and split events are invisible there, while Celeborn\u0027s own metrics sit in cluster-level Grafana dashboards that cannot be attributed back to a specific application. This tab closes that gap by putting per-application Celeborn shuffle information inside the Spark UI and History Server, alongside the job\u0027s stages and tasks.\n\n### Does this PR introduce any user-facing change?\n\nYes.\nSetting `spark.plugins\u003dorg.apache.spark.shuffle.celeborn.CelebornPlugin` (with the extension jar on the classpath) adds a \"Celeborn\" tab to the Spark UI and History Server. Disabled by default; no effect otherwise.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### How was this patch tested?\n\nManual end-to-end verification:\n\n\u003cimg width\u003d\"2584\" height\u003d\"1352\" alt\u003d\"image\" src\u003d\"https://github.com/user-attachments/assets/202eb4b7-389c-4dab-a2e4-799971594b92\" /\u003e\n\nCloses #3793 from yew1eb/celeborn-ui-mvp.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/716\nAI-Contributed/UT: 0/170\n" }, { "commit": "f43f424c04c4259a18251987965ddb25745e28e4", "tree": "e02cb6ba87719a4e9f71d47504e278d61c1c63fc", "parents": [ "adadbd924823416147680fae5131390a8f3833fd" ], "author": { "name": "senthh", "email": "senthil.kumar@acceldata.io", "time": "Fri Oct 09 10:13:32 2026 +0800" }, "committer": { "name": "Shuang", "email": "lvshuang.xjs@alibaba-inc.com", "time": "Fri Oct 09 10:13:32 2026 +0800" }, "message": "[CELEBORN-2471] Fix Flink uncompressed shuffle failing to load codec NONE\n\n### What changes were proposed in this pull request?\n\nCeleborn lets you turn off shuffle compression for Flink by setting the codec to `NONE` (uncompressed shuffle).\n\n**Bug:** when the codec is `NONE`, the read side of the shuffle passes the literal string `\"NONE\"` to CelebornFlinkShim.createBufferDecompressor, which builds a Flink `BufferDecompressor`. Flink resolves that value as a class name (`Class.forName(\"NONE\")`) and throws `ClassNotFoundException: NONE`, so the Flink job fails to start. Uncompressed shuffle was unusable.\n\n**Fix:** mirror the write-side NONE guard into the read path — when the codec is `NONE`, pass a null decompressor instead of constructing one. This is safe because `RemoteShuffleInputGateDelegation.decompressBufferIfNeeded` only dereferences the decompressor inside `if (buffer.isCompressed())`, and NONE writes never produce compressed buffers.\n\n### Why are the changes needed?\n\nThe write and read paths handled the `NONE` sentinel asymmetrically:\n\n- Write side (`RemoteShuffleResultPartitionFactory.getBufferCompressor`): correctly guards `NONE` and returns a null compressor (no compression).\n- Read side (`RemoteShuffleInputGateFactory.create` / `createInputGate`): did not guard `NONE`, so it passed the literal string to Flink\u0027s class resolution and crashed.\n\nA simple mismatch — one half of the code knew what `NONE` meant, the other half didn\u0027t.\n\n### Does this PR resolve a correctness bug?\n\n- [x] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes\n\n### How was this patch tested?\n\n- Compiled the affected module: `mvn -Pflink-1.18 compile -pl client-flink/common -am` → BUILD SUCCESS (covers the shared factory used by all Flink modules).\n- Full end-to-end repro (`TestCelebornFlinkShuffle \u003e Uncompressed shuffle produces the correct result`) to be run on a cluster once the shaded Flink jar is rebuilt and redeployed.\n\nFixes CELEBORN-2471.\n\nCloses #3861 from senthh/CELEBORN-2471.\n\nAuthored-by: senthh \u003csenthil.kumar@acceldata.io\u003e\nSigned-off-by: Shuang \u003clvshuang.xjs@alibaba-inc.com\u003e\n" }, { "commit": "adadbd924823416147680fae5131390a8f3833fd", "tree": "b7c410798221534c9f22dd1e09db273d5befb5c3", "parents": [ "f28246dbd89074b58221ef4b2e2c542605f325f2" ], "author": { "name": "fengmingxiao", "email": "fengmingxiao@bytedance.com", "time": "Tue Sep 15 14:18:26 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Tue Sep 15 14:18:26 2026 +0800" }, "message": "[CELEBORN-2442] Optimize meta memory for very large partitions\n\n### What changes were proposed in this pull request?\n\n This PR reduces the memory footprint of `PartitionLocation` metadata while preserving the existing wire protocol and public getter behavior.\n\n The main changes are:\n\n - Introduce an immutable, weakly interned `WorkerEndpoint` to share worker host and port information across partition locations.\n - Move the cached `host:pushPort` and `host:fetchPort` strings into the shared endpoint.\n - Lazily initialize `StorageInfo` and `RoaringBitmap` using thread-safe initialization.\n - Keep `getMapIdBitMap()` non-null for compatibility, and add a non-materializing accessor for serialization, HTTP API, and read-side inspection.\n - Avoid allocating empty bitmaps when locations are created by `LifecycleManager` and `SlotsAllocator`.\n - Preserve the current `storageTypes.head` selection behavior in `LifecycleManager`.\n - Keep string keys in `WorkerPartitionLocationInfo` to avoid parsing and boxing overhead on the Worker hot path.\n - Add allocation-free helpers for converting partition location IDs into packed `long` values.\n - Avoid materializing optional metadata from logging, protobuf serialization, HTTP rendering, and range-read paths.\n - Re-intern `WorkerEndpoint` instances after same-version Java deserialization.\n - Fix peer bitmap serialization so that the peer protobuf contains the peer\u0027s bitmap instead of the primary location\u0027s bitmap.\n - Make range reads fall back to the peer bitmap and fail open when bitmap metadata is unavailable.\n - Add JOL-based manual memory benchmarks and focused unit tests.\n\n No protobuf schema is changed by this PR.\n\n ### Why are the changes needed?\n\n A shuffle may contain millions of `PartitionLocation` instances. Previously, every instance independently retained:\n\n - Worker host and port fields.\n - Cached host/port strings.\n - An empty `StorageInfo`.\n - An empty `RoaringBitmap`.\n\n Most of this metadata is identical across locations or unused for the majority of their lifetime. The eager allocations therefore contribute significant heap usage and GC pressure on the Master, Worker, and client.\n\n The JOL benchmark includes both the live location graph and the weak interner overhead:\n\n| Scenario | Before | After, including interner | Reduction |\n|---|---:|---:|---:|\n| Allocator, 10,000 pairs | 4,383,504 bytes | 2,148,056 bytes | 51.00% |\n| Allocator, 1,000,000 pairs | 424,143,504 bytes | 184,308,056 bytes | 56.55% |\n| Packed decoded pair shape, 10,000 pairs | 3,752,304 bytes | 2,148,056 bytes | 42.75% |\n| Packed decoded pair shape, 1,000,000 pairs | 375,200,304 bytes | 184,308,056 bytes | 50.88% |\n\n The additional serialization and range-read fixes are needed to ensure that lazy bitmap allocation does not introduce data loss, incorrect peer metadata, or compatibility regressions.\n\n ### Does this PR resolve a correctness bug?\n\n - [x] Yes\n\n It fixes peer bitmap serialization and ensures that missing bitmap metadata does not cause an NPE or silently exclude a partition location during range reads.\n\n ### Does this PR introduce _any_ user-facing change?\n\n - [ ] Yes\n\n There are no configuration, wire-protocol, or intended behavioral changes for users.\n\n ### How was this patch tested?\n\n The following verification was completed:\n\n - Ran the `common`, `client`, and `master` module regression suites:\n - 366 JUnit tests, with 1 ignored.\n - 325 ScalaTest tests.\n - 691 tests in total, with no failures or errors.\n - Ran the final `PartitionLocationSuiteJ`:\n - 20 tests passed.\n - Covers lazy initialization, endpoint interning, endpoint setters, ID parsing, peer references, compatibility getters, and same-version Java serialization.\n - Ran Worker partition and storage regression suites:\n - 27 Java tests passed.\n - 9 Scala tests passed.\n - Ran the Spark 3.5 `LifecycleManagerReserveSlotsSuite` integration test:\n - 1 test passed.\n - All 11 reactor modules completed successfully.\n - Compiled the affected modules through SBT:\n - `celeborn-common/Test/compile`\n - `celeborn-service/Compile/compile`\n - Ran `spotless:check` and `git diff --check`.\n - Manually ran `PartitionLocationMemorySuiteJ` with JOL for both the 10,000-pair and 1,000,000-pair scenarios.\n\nCloses #3825 from FMX/optimize-meta-memory.\n\nAuthored-by: fengmingxiao \u003cfengmingxiao@bytedance.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/406\nAI-Contributed/UT: 0/547\n" }, { "commit": "f28246dbd89074b58221ef4b2e2c542605f325f2", "tree": "6c3b82a67a8e69f0f5c5794773c5932c6bee8459", "parents": [ "fd453c074ad0f997e98d7c4546a7412f6e0133d8" ], "author": { "name": "Kalvin2077", "email": "wk.huang2077@outlook.com", "time": "Tue Sep 15 13:55:28 2026 +0800" }, "committer": { "name": "Shuang", "email": "lvshuang.xjs@alibaba-inc.com", "time": "Tue Sep 15 13:55:28 2026 +0800" }, "message": "[CELEBORN-2430] Expose unexpected errors in client background threads\n\n### What changes were proposed in this pull request?\n\nCatch and log unexpected `Throwable`s in client background tasks. Partition readers also propagate background failures through their existing exception channel. No recovery behavior is added.\n\n### Why are the changes needed?\n\n`ExecutorService.submit` stores task failures in the returned `Future`. If that `Future` is not inspected, the failures can remain invisible. Catching only `Exception` also misses `Error`s.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce *any* user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nAdded unit tests verifying that DFS and local partition readers propagate background errors, and that change- and release-partition tasks log them. The focused tests passed, including red/green verification for the manager tasks. Formatting checks also passed.\n\nCloses #3829 from Kalvin2077/fix/CELEBORN-2430.\n\nAuthored-by: Kalvin2077 \u003cwk.huang2077@outlook.com\u003e\nSigned-off-by: Shuang \u003clvshuang.xjs@alibaba-inc.com\u003e\n" }, { "commit": "fd453c074ad0f997e98d7c4546a7412f6e0133d8", "tree": "de100c4802ced1a159779ef39d8b035617883c85", "parents": [ "6b6efe2c2e7a0efef26b12419c7f2f50152767f0" ], "author": { "name": "Kalvin2077", "email": "wk.huang2077@outlook.com", "time": "Tue Sep 15 11:21:12 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Tue Sep 15 11:21:12 2026 +0800" }, "message": "[CELEBORN-2428] Fix TierWriter buffer accounting when CompositeByteBuf insertion fails\n\n### What changes were proposed in this pull request?\n\nFix `TierWriter` buffer accounting and reference-count handling when `CompositeByteBuf.addComponent` fails before insertion or during consolidation. Add regression tests and include the shuffle key in write-error logs.\n\n### Why are the changes needed?\n\n`addComponent` can fail before or after taking ownership of the buffer. Treating both cases identically can cause incorrect storage counters or buffer reference-count leaks.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\n- Added tests for pre-insertion OOM, capacity overflow, and post-insertion consolidation OOM.\n- Ran `TierWriterSuite`: 11 tests passed.\n- Ran `./build/mvn spotless:check -pl worker`.\n\nCloses #3809 from Kalvin2077/fix/CELEBORN-2428.\n\nAuthored-by: Kalvin2077 \u003cwk.huang2077@outlook.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/69\nAI-Contributed/UT: 0/255\n" }, { "commit": "6b6efe2c2e7a0efef26b12419c7f2f50152767f0", "tree": "fb2db3aaf43df065cdb21b3050edee073f36f216", "parents": [ "0d63a20e4c08b713e1d0cf69a9d03c67cb6db6af" ], "author": { "name": "Kalvin2077", "email": "wk.huang2077@outlook.com", "time": "Tue Sep 15 10:46:52 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Tue Sep 15 10:46:52 2026 +0800" }, "message": "[CELEBORN-2398] Allow ChangePartitionManager to refresh candidate workers without allocating slots\n\n### What changes were proposed in this pull request?\n\nThis PR introduces a Protobuf-based `RequestWorkers`/`RequestWorkersResponse` RPC for read-only worker discovery.\n\nOn the Master side, the new handler selects currently available workers after applying client exclusions and tag filters. It limits the response using the smaller of `celeborn.client.slot.assign.maxWorkers` and `celeborn.master.splitSlot.assign.maxWorkers`, while preserving the minimum worker count required for replication. When authentication is enabled, application metadata is pushed to the selected workers.\n\nOn the client side, LifecycleManager periodically requests workers from the Master, creates endpoints for newly selected workers, records connection failures, and maintains an endpoint-ready worker pool. ChangePartitionManager uses that pool for change-partition requests and falls back to the shuffle\u0027s existing worker snapshots when no refreshed candidates are available.\n\nThe PR also makes the following configuration changes:\n\n- removes `celeborn.client.shuffle.dynamicResourceFactor`;\n- adds `celeborn.client.shuffle.dynamicResource.updateTime`, defaulting to `30s`;\n- adds `celeborn.master.splitSlot.assign.maxWorkers`, defaulting to `500`;\n- documents the breaking client configuration change in the migration guide.\n\n### Why are the changes needed?\n\nThe existing factor-based logic refreshes candidates only after enough workers from the shuffle\u0027s original allocation become unavailable. It therefore cannot use newly added workers during normal workers scale-out while the original workers remain healthy.\n\nRequesting slots merely to discover workers also mixes worker discovery with slot allocation and mutates Master shuffle state. A dedicated read-only RPC allows ChangePartitionManager to use the current cluster membership without those side effects, while rate limiting and worker-count caps prevent excessive RPC connections and oversized responses.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes\n\nUsers of `celeborn.client.shuffle.dynamicResourceFactor` must migrate to `celeborn.client.shuffle.dynamicResource.updateTime`.\n\n### How was this patch tested?\n\n- Added unit tests for Worker selection, storage eligibility, dynamic Worker merging, deduplication, and excluded-Worker filtering.\n- Built with JDK 8 and deployed to a four-Worker Celeborn cluster.\n- Ran an E2E shuffle that started with one Worker, added three Workers, and paused the original Worker. The client detected the failure, revived partitions on new Workers, and successfully validated 8 million rows with fallback disabled.\n- Restored all Workers and reran the standard Spark/YARN smoke test successfully.\n\nCloses #3775 from Kalvin2077/feat/dynamic-resource.\n\nAuthored-by: Kalvin2077 \u003cwk.huang2077@outlook.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/471\nAI-Contributed/UT: 0/431\n" }, { "commit": "0d63a20e4c08b713e1d0cf69a9d03c67cb6db6af", "tree": "896043535c5b6cd6f1a78791616011e0c53d91a5", "parents": [ "231ffee85fb30c7c8b7db0dabf814b00a8cf6104" ], "author": { "name": "strelok89", "email": "alexgilko@outlook.co.il", "time": "Mon Sep 14 19:10:22 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Mon Sep 14 19:10:22 2026 +0800" }, "message": "[CELEBORN-2447] Add a /healthz endpoint to master and worker\n\n### What changes were proposed in this pull request?\n\nAdd a `GET /healthz` endpoint served by both master and worker, whose HTTP status code reflects whether the service is able to serve: `200` when healthy and `503` when not.\n\n- `HttpService#healthCheck` defaults to a shallow check that reports healthy once the HTTP service is available. This is what the master uses.\n- `Worker` overrides it to report healthy only when the worker is registered with the master and its state is `Normal`.\n- `/healthz` is added to the default HTTP authentication bypass paths.\n\nThe endpoint is mounted at the root rather than under `/api/v1` so that a probe is not coupled to the API version, consistent with the other operational paths (`/ping`, `/metrics/prometheus`). The checked-in OpenAPI spec covers only `/api/v1`, so the generated client is unaffected.\n\nExample:\n\n```\n$ curl -s -o /dev/null -w \u0027%{http_code}\\n\u0027 http://worker:9096/healthz\n200\n\n$ curl -s http://worker:9096/healthz\n{\"service\":\"worker\",\"healthy\":false,\"reason\":\"worker is not registered with master\"}\n```\n\n### Why are the changes needed?\n\nCeleborn currently exposes no endpoint whose status code reflects whether the process can serve, so a meaningful Kubernetes probe cannot be written. `/ping` always returns `200` regardless of state, and `/api/v1/workers` returns `200` even when `isRegistered` is false, so operators have to fall back to an `exec` probe that shells out and greps the JSON body.\n\nThe practical cost is rolling updates. Without a readiness signal, a StatefulSet marks a worker pod Ready as soon as its container process starts and immediately proceeds to the next ordinal, before the restarted worker has re-registered with the master. On a large cluster this can remove a meaningful fraction of workers from service before any of them rejoin.\n\nTwo design points worth calling out for review:\n\n**The worker check mirrors the master\u0027s own definition.** `AbstractMetaManager#isWorkerAvailable` accepts only `Normal`, so a worker that is idle, decommissioning or exiting is already excluded from `availableWorkers` and never selected in `offerSlots`. Reporting such a worker as healthy would put `/healthz` at odds with the master. Whether `Idle` specifically should fail readiness is discussed on CELEBORN-2447 — it is reachable only through an explicit `DecommissionThenIdle` event, so the alternative (200 for `Idle`, 503 only for the decommission/exit states) is a reasonable variation if reviewers prefer it.\n\n**The master check is deliberately shallow.** Masters are fronted by a headless Service that does not set `publishNotReadyAddresses`. A quorum-aware or leader-aware check would report every master as unhealthy during a cold start, withholding their DNS records and preventing them from discovering each other to form quorum. A follower is also a healthy replica. Quorum and leadership remain observable through `/api/v1/ratis` and `/api/v1/masters`.\n\n`/healthz` bypasses authentication by default because a kubelet cannot present credentials, and `celeborn.http.auth.bypass.api.paths` defaults to empty (CELEBORN-2278).\n\nWiring probes into the Helm chart, which currently defines no `livenessProbe`, `readinessProbe` or `startupProbe` for either role, is intended as a follow-up so the two changes can be reviewed independently.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes\n\nA new `/healthz` endpoint on master and worker, documented in `docs/restapi.md`.\n\n### How was this patch tested?\n\nNew tests:\n\n- `ApiBaseResourceSuite`: `/healthz` returns `200` and reports the correct service name, for both master and worker.\n- `ApiBaseResourceAuthenticationSuite`: `/healthz` is reachable without credentials.\n- `ApiWorkerResourceSuite`: `/healthz` returns `503` when the worker is not registered.\n\nVerified locally on JDK 11: `celeborn-service/Test/compile` and `celeborn-worker/Test/compile` pass, `ApiMasterResourceSuite` (19/19) and `ApiMasterResourceAuthenticationSuite` (9/9) pass, and `spotless:check` passes for the `service` and `worker` modules.\n\nThe worker-side suites were not run locally: they were developed on Windows, where any `MiniClusterFeature` test fails during setup because `CelebornConf#workerBaseDirs` splits each storage dir on `:` and rejects `C:\\...` paths. That is pre-existing and unrelated to this change, and those suites are covered by CI.\n\nCloses #3832 from strelok89/CELEBORN-2447.\n\nAuthored-by: strelok89 \u003calexgilko@outlook.co.il\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "231ffee85fb30c7c8b7db0dabf814b00a8cf6104", "tree": "56ab949a1fa41eac3164ce3630e7300c094e2955", "parents": [ "749e45522a1726d25f5949d6ae82f1e30ac9f47e" ], "author": { "name": "Cheng Pan", "email": "chengpan@apache.org", "time": "Mon Sep 14 16:31:33 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Mon Sep 14 16:31:33 2026 +0800" }, "message": "[CELEBORN-2440][INFRA] Post a merge summary comment on the PR in merge_pr.py\n\n### What changes were proposed in this pull request?\n\nPort a few `merge_pr.py` improvements from Spark\u0027s `merge_spark_pr.py`:\n\n1. Post a \"Merge Summary\" comment on the PR after the merge, recording every branch the\n change landed on and a link to the resulting commit. `cherry_pick` now returns the\n `(ref, hash)` pair it pushed so backports are included, and the comment is posted from\n a `finally` block, so a cancelled cherry-pick does not drop what already landed. The\n backport path (PR already merged) posts a summary too.\n\n2. Close pull requests that GitHub does not auto-close. `Closes #N` closes the PR only\n when the commit lands on the default branch, so one merged into `branch-x.y` stays\n open. Merges into `main` are left to GitHub, whose close event carries the commit that\n the backport path below relies on.\n\n3. Recognize pull requests merged into `branch-x.y`. For the same reason, GitHub links the\n merge commit to the `closed` event only on the default branch, so those pull requests\n were not recognized as merged and re-running the script on one attempted a fresh merge\n instead of offering a backport. `find_merge_commit` now falls back to `referenced`\n events, confirmed against the `Closes #N from \u003cref\u003e` merge footer.\n\n4. Polish the interactive prompts: capitalize `N` in the `(y/n)` prompts, since all of\n them treat any input other than `y` as no, and add the missing space after `\":\"` in\n the JIRA assignee prompt.\n\nThe merge has already been pushed by the time the comment and the close run, so neither\naborts the rest of the bookkeeping: failures are reported rather than raised, and the\nclose is skipped when `GITHUB_OAUTH_KEY` is not configured.\n\nFor example, #3788 landed on `main` and `branch-0.7`, and would have got:\n\n\u003e **Merge Summary:**\n\u003e - merged into main https://github.com/apache/celeborn/commit/8cbf3fa4\n\u003e - merged into branch-0.7 https://github.com/apache/celeborn/commit/f67883a4\n\u003e\n\u003e *Posted by `merge_pr.py`*\n\n### Why are the changes needed?\n\nA merged PR currently records nothing about where the change landed: the commit is\nreachable only by searching for the `Closes #N` footer, and backport targets are not\nvisible from the PR page at all. Pull requests merged into `branch-x.y` also stay open\nuntil someone closes them by hand, and the script does not recognize them as merged\nafterwards.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nManually: `python3 -m py_compile dev/merge_pr.py` and the module doctests, which now cover\n`has_merge_footer`, pass. `find_merge_commit` was checked against the real GitHub events of\n#3697 (merged into `branch-0.6`, previously undetected, now resolves `0bf9c5a58`), #3788 and\n#3768 (both merged into `main`, still resolved from the `closed` event). The GitHub write\ncalls will be exercised by the next real merge.\n\nCloses #3823 from pan3793/CELEBORN-2440.\n\nAuthored-by: Cheng Pan \u003cchengpan@apache.org\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "749e45522a1726d25f5949d6ae82f1e30ac9f47e", "tree": "00475a6a9b49900866317b27db5cdbc9d9db7f2e", "parents": [ "4e20a611c9953bc817b83712caaee233a7b1db10" ], "author": { "name": "Kalvin2077", "email": "wk.huang2077@outlook.com", "time": "Tue Aug 25 00:07:24 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Aug 25 00:07:24 2026 +0800" }, "message": "[CELEBORN-2060][FOLLOWUP] Fix rack-aware replica storage selection\n\n### What changes were proposed in this pull request?\n\nThis follow-up to [PR #3781](https://github.com/apache/celeborn/pull/3781):\n\n- Validates that rack-aware replica candidates support the requested storage type.\n- Builds replica `StorageInfo` from the selected replica worker.\n- Adds regression tests for diskless replicas and different mount points.\n\n### Why are the changes needed?\n\nDuring best-effort fallback, the allocator could select a diskless worker for a local-disk replica or incorrectly reuse the primary worker\u0027s mount point.\n\n### Does this PR resolve a correctness bug?\n\n- [x] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\n`build/mvn -pl master -am -Dtest\u003dSlotsAllocatorRackAwareSuiteJ test`\n\nCloses #3803 from Kalvin2077/fix/CELEBORN-2423.\n\nAuthored-by: Kalvin2077 \u003cwk.huang2077@outlook.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "4e20a611c9953bc817b83712caaee233a7b1db10", "tree": "1d58e5e68feb1d210c051a1155121aed7640b49e", "parents": [ "cb086f2ca023f2882a712944b9535a1f9b17a034" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Mon Aug 24 16:50:48 2026 +0800" }, "committer": { "name": "Cheng Pan", "email": "chengpan@apache.org", "time": "Mon Aug 24 16:50:48 2026 +0800" }, "message": "[CELEBORN-2390] Add docker-compose for local cluster\n\n### What changes were proposed in this pull request?\nAdd a `docker-compose.yaml` under `docker/` to run a minimal local Celeborn cluster (1 master + 3 workers) for development and debugging, with a `build-image.sh` wrapper and a `conf/`. The existing `docker/Dockerfile` is unchanged.\n\n### Why are the changes needed?\nThere is no lightweight local deployment path today — only the K8s Helm chart or running master/worker manually on the host.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nRan the cluster locally with `docker compose up -d`:\n\u003e $ docker compose -f docker/docker-compose.yaml up -d\n\u003e $ docker compose -f docker/docker-compose.yaml ps\n\u003cimg width\u003d\"1354\" height\u003d\"132\" alt\u003d\"image\" src\u003d\"https://github.com/user-attachments/assets/86b8e10b-9058-498f-b532-15ab2ce5f3fc\" /\u003e\n\n\u003cimg width\u003d\"1510\" height\u003d\"119\" alt\u003d\"image\" src\u003d\"https://github.com/user-attachments/assets/96cd905d-be6a-4c46-9c62-70f77927a3d1\" /\u003e\n\nCloses #3768 from yew1eb/add-docker-compose.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: Cheng Pan \u003cchengpan@apache.org\u003e\n" }, { "commit": "cb086f2ca023f2882a712944b9535a1f9b17a034", "tree": "a514ed57393e92fe85d76787e9e0ec26aa572b41", "parents": [ "b356ccec00dc63f9b6efaac88c26517eae61e042" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Mon Aug 24 16:44:26 2026 +0800" }, "committer": { "name": "Cheng Pan", "email": "chengpan@apache.org", "time": "Mon Aug 24 16:44:26 2026 +0800" }, "message": "[CELEBORN-2432] Cache sbt launcher and dependencies in SBT CI workflow\n\n### What changes were proposed in this pull request?\nAdd an actions/cache step to every job in sbt.yml, caching:\n- build/*.jar (the sbt launcher downloaded by build/sbt)\n- ~/.sbt\n- ~/.cache/coursier (dependency repository)\n\nKeys are derived from hashFiles of the poms and build scripts, so downloads only happen when the build files change. This follows the pattern used by [Spark\u0027s `build_and_test.yml`](https://github.com/apache/spark/blob/52eb078867cf809a92fd6ebdc96c9d72bce63417/.github/workflows/build_and_test.yml#L720).\n\n### Why are the changes needed?\n\nThe sbt CI jobs currently configure no caching at all: every run of every matrix job (~40 jobs) re-downloads the sbt launcher and the entire dependency set, which exposes CI to network flakes. Recent examples on PR CI and on main:\n\n- \"Our attempt to download sbt locally to build/sbt-launch-1.9.4.jar failed\" (job fails in ~15s)\n- scala compile failing with MissingResourceException:\n org.scalactic.ScalacticBundle caused by java.io.IOException: Stream closed (corrupted jar download)\n\u003cimg width\u003d\"2110\" height\u003d\"444\" alt\u003d\"image\" src\u003d\"https://github.com/user-attachments/assets/9d24b07d-3910-400e-b8ee-af550ca444e2\" /\u003e\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Ye\n\n### How was this patch tested?\n\nWorkflow-only change; the PR\u0027s own CI run exercises the new cache steps (first run populates the cache, subsequent runs hit it).\n\nCloses #3812 from yew1eb/CELEBORN-2432.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: Cheng Pan \u003cchengpan@apache.org\u003e\n" }, { "commit": "b356ccec00dc63f9b6efaac88c26517eae61e042", "tree": "fdfce55aa3a8d96808ca5f1ac95272b1524dc6f6", "parents": [ "7b050089b837055a3813b26f05b1fce88aa32e0b" ], "author": { "name": "Cheng Pan", "email": "chengpan@apache.org", "time": "Mon Aug 24 00:03:11 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Mon Aug 24 00:03:11 2026 +0800" }, "message": "[CELEBORN-2437][INFRA] Fix JIRA fix version suggestion in merge_pr.py\n\n### What changes were proposed in this pull request?\n\nWhen resolving a JIRA after merging a PR, the suggested fix version for `main` was the first unreleased version returned by the JIRA API, which is ordered by creation time rather than version number. As a result, `main` could suggest an old unreleased version (e.g. `0.3.3`) instead of the next main release (e.g. `1.0.0`).\n\nThis PR follows the version inference approach of SPARK-56865 in `dev/merge_spark_pr.py`: suggested fix versions are picked by semantic version, with `main` contributing the greatest unreleased `N.0.0` and `branch-M.N` contributing the greatest unreleased `M.N.x`. It also requires JIRA version names to be exactly `x.y.z`, excluding malformed entries such as the current `0.5.2, 0.4.3, 0.6.0`.\n\n### Why are the changes needed?\n\nThe default suggestion was wrong for PRs merged to `main`, requiring committers to correct it manually.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\n`python3 -m py_compile dev/merge_pr.py`, plus manual checks of `compute_default_fix_versions` against the current unreleased CELEBORN versions: `main + branch-0.7` now suggests `1.0.0,0.7.1` instead of `0.3.3,0.7.1`.\n\nAssisted-by: deepseek-v4-pro\n\nCloses #3818 from pan3793/CELEBORN-2437.\n\nAuthored-by: Cheng Pan \u003cchengpan@apache.org\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "7b050089b837055a3813b26f05b1fce88aa32e0b", "tree": "6120e0a5d3ba52d85df3af7c29ba08df8caaeede", "parents": [ "83be003260455cbb1b905b4884b23b1d92796e74" ], "author": { "name": "Kalvin2077", "email": "wk.huang2077@outlook.com", "time": "Thu Aug 20 23:15:54 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Thu Aug 20 23:15:54 2026 +0800" }, "message": "[CELEBORN-2060] Refactor master slot allocation with strategy abstraction\n\n### What changes were proposed in this pull request?\n\nThis PR refactors master-side slot allocation so that strategy-specific slot-budget computation is\nseparated from the shared allocation workflow. The existing round-robin and load-aware algorithms\nare moved behind `SlotsAssignStrategy`, while `SlotsAllocator` retains the common progressive\nallocation and best-effort fallback behavior.\n\nThe change is intentionally structural. Existing names and allocation behavior are preserved where\npossible so that the old implementation can be compared directly with the refactored code.\n\nThe refactor also exposed several potential correctness issues that are intentionally left for\nseparate follow-up changes:\n\n- With replication enabled, remote-only storage and rack awareness disabled, primary and replica\n locations may be assigned to the same worker.\n- During budget-free rack-aware fallback, a replica may reuse the primary worker\u0027s `StorageInfo`,\n which can produce an incorrect mount-point hint and slot accounting when workers have different\n mount points.\n- During budget-free rack-aware fallback with local storage enabled, replica selection does not\n verify that the selected worker has a local disk, so allocation may be deferred to a worker that\n cannot satisfy the requested local placement.\n\n### Why are the changes needed?\n\n`SlotsAllocator` previously mixed strategy-specific budget calculation with worker selection,\nstorage selection, rack awareness, interruption awareness, and fallback handling. Separating the\nstrategy boundary makes the allocation stages easier to follow and allows strategies to evolve\nwithout duplicating the common allocation workflow.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\n- `build/sbt \"celeborn-master/testOnly org.apache.celeborn.service.deploy.master.slotsalloc.*\"`\n\nCloses #3781 from Kalvin2077/refactor/slots-alloc-cleanup.\n\nAuthored-by: Kalvin2077 \u003cwk.huang2077@outlook.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "83be003260455cbb1b905b4884b23b1d92796e74", "tree": "b856e4405c1b11f398971c3a2330d2c9839e6066", "parents": [ "9c7ea13c3b08db32cb0895bde6c1ac27c6a7a7ec" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Thu Aug 20 23:05:15 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Thu Aug 20 23:05:15 2026 +0800" }, "message": "[CELEBORN-2431] Fix flaky RemoteShuffleMasterSuiteJ shuffleCount assertions racing with app heartbeat counter reset\n\n### What changes were proposed in this pull request?\n\nIncrease `celeborn.client.application.heartbeatInterval` from `30s` to `1h` in `RemoteShuffleMasterSuiteJ.setUp()`, which effectively pauses the LifecycleManager app heartbeater for the duration of this test suite. All version-specific subclasses (`RemoteShuffleMasterSuiteJV118` ~ `V23`) inherit from this base suite, so a single change covers all of them.\n\n### Why are the changes needed?\n\n`RemoteShuffleMasterSuiteJ.testRegisterPartitionWithProducer` (and the other `shuffleCount` assertions in the same suite) is flaky:\n\n```\njava.lang.AssertionError: expected:\u003c1\u003e but was:\u003c0\u003e\n at RemoteShuffleMasterSuiteJ.testRegisterPartitionWithProducer(RemoteShuffleMasterSuiteJ.java:124)\n```\n\nRoot cause is a race on `LifecycleManager#shuffleCount`:\n\n- `RemoteShuffleMaster.registerPartitionWithProducer` increments the counter (`RemoteShuffleMaster.java:161`)\n- the LifecycleManager app-heartbeat thread periodically ships the counter to the master via `shuffleCount.sumThenReset()` (`LifecycleManager.scala:234`)\n\nThe suite configures `celeborn.client.application.heartbeatInterval\u003d30s`. On slow CI runners a test can run past the 30s mark, so a heartbeat fires between the increment and the assertion and resets the counter to 0. `shuffleFallbackCounts` assertions are affected by the same race (`resetFallbackCounts`). No test in this suite depends on an app heartbeat actually being sent, so enlarging the interval has no side effects.\n\nExample failure (flink1 (11, 1.18)): https://github.com/apache/celeborn/actions/runs/32273603672/job/96135975529\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\n`./build/mvn -Pflink-1.18 -pl client-flink/flink-1.18 test -Dtest\u003dRemoteShuffleMasterSuiteJV118` — all 7 tests passed (previously `shuffleCount` assertions could intermittently fail on slow runners).\n\nCloses #3811 from yew1eb/CELEBORN-2431.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "9c7ea13c3b08db32cb0895bde6c1ac27c6a7a7ec", "tree": "a3329199d4ea929a3770398ecb2dbfb04b620471", "parents": [ "02daddede2be4f0cd66a5392aa2de66d2475262e" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Thu Aug 20 22:57:50 2026 +0800" }, "committer": { "name": "Cheng Pan", "email": "chengpan@apache.org", "time": "Thu Aug 20 22:57:50 2026 +0800" }, "message": "[CELEBORN-2410] Add missing module coverage to labeler.yml\n\n### What changes were proposed in this pull request?\n\n`labeler.yml` did not match changes under `lifecycle-manager/`, `multipart-uploader/`, and `rust/` — PRs touching only these paths received no module label.\n\n- `lifecycle-manager/**` → `module:service` (daemon wrapper around `client.LifecycleManager`, package `org.apache.celeborn.server.lifecyclemanager`)\n- `multipart-uploader/**` → `module:service` (S3/OSS impls of `server.common.service.mpu.MultipartUploadHandler`, defined in the `service` module)\n- `rust/**` → new `module:rust` (independent Cargo-built Rust client)\n\n### Why are the changes needed?\n\nThese three directories have real source but no labeler entry, so the labeler action silently skips them. Changes to them never get triaged into the correct module label.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nYAML syntax validated. Existing module globs unchanged.\n\nCloses #3789 from yew1eb/CELEBORN-2410-labeler-modules.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: Cheng Pan \u003cchengpan@apache.org\u003e\n" }, { "commit": "02daddede2be4f0cd66a5392aa2de66d2475262e", "tree": "c12b4c2c2952266eaf01eccdae110ed33ff83935", "parents": [ "8cbf3fa4dd142ce04e7ed88d342eb64051fb90dc" ], "author": { "name": "r7raul1984", "email": "8199300+r7raul1984@users.noreply.github.com", "time": "Thu Aug 20 22:56:03 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Thu Aug 20 22:56:03 2026 +0800" }, "message": "[CELEBORN-2421] Avoid redundant ByteBuffer allocation in LocalPartitionDataReader\n\n### What changes were proposed in this pull request?\n\nReplace the temporary heap `ByteBuffer` allocation in `readBufferIntoReadBuffer` with a direct write loop using `ByteBuf.writeBytes(FileChannel, pos, len)`, which writes from the channel straight into the output `ByteBuf` with no intermediate allocation. The channel position is advanced manually since the absolute-read API does not move the cursor.\n\nAdditionally, fix two correctness issues in the read loop:\n- Throw `EOFException` when `writeBytes` returns `-1` (premature EOF due to file truncation) instead of silently breaking and returning a partially filled `ByteBuf` to the caller.\n- Throw `IOException` when `writeBytes` returns `0` to prevent the loop from spinning indefinitely.\n\n### Why are the changes needed?\n\nPreviously, `readBufferIntoReadBuffer` allocated a temporary heap `ByteBuffer` on every call, performed a `FileChannel` read into it, then copied the data into the output `ByteBuf` — two copies and one allocation per read. The new approach eliminates the intermediate allocation and one copy.\n\nThe original loop also had two correctness issues: `dataFileSize` is cached at construction time, so a subsequent file truncation could cause `writeBytes` to return `-1` after the integrity check passes, resulting in the caller receiving partial data without any error. A return value of `0` would cause an infinite spin.\n\n### Does this PR resolve a correctness bug?\n\n- [x] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nBenchmarks at 1 MB buffer size over 500 iterations show ~2x throughput improvement and ~40% fewer GC collections compared to the previous implementation.\n\nCloses #3800 from r7raul1984/jj_fix.\n\nAuthored-by: r7raul1984 \u003c8199300+r7raul1984@users.noreply.github.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "8cbf3fa4dd142ce04e7ed88d342eb64051fb90dc", "tree": "8f2743b240af9c1c7140d3de36c8e6d9210ce983", "parents": [ "52fa5ffb85221c9b8d2907352620860f83b1dd5e" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Thu Aug 20 22:54:24 2026 +0800" }, "committer": { "name": "Cheng Pan", "email": "chengpan@apache.org", "time": "Thu Aug 20 22:54:24 2026 +0800" }, "message": "[CELEBORN-2409] Fix outdated Spark version descriptions in docs\n\n### What changes were proposed in this pull request?\n\nFix outdated Spark version descriptions in docs:\n\n- `README.md`: add Spark 4.1/4.2 to the supported Spark versions list\n- `docs/README.md`, `docs/deploy.md`: \"clients for Spark 2.x and Spark 3.x\" -\u003e \"clients for Spark 2.x, Spark 3.x and Spark 4.x\"\n- `docs/developers/sbt.md`: update the shaded client publish example from `-Pspark-4.0` to `-Pspark-4.2`, aligning with `build/release/release.sh`\n\n### Why are the changes needed?\n\nSpark 4.1/4.2 support was added in CELEBORN-2239 and CELEBORN-2383, but these doc descriptions were not updated accordingly.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nDoc-only changes, no code changes.\n\nCloses #3788 from yew1eb/spark-4.2-docs.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: Cheng Pan \u003cchengpan@apache.org\u003e\n" }, { "commit": "52fa5ffb85221c9b8d2907352620860f83b1dd5e", "tree": "00b8d937512515f4abec3139571173e8ac1d2a2a", "parents": [ "ba4fa59f0279bee81e7cfa838e083a603d331ff2" ], "author": { "name": "Cheng Pan", "email": "chengpan@apache.org", "time": "Thu Aug 20 22:44:24 2026 +0800" }, "committer": { "name": "Cheng Pan", "email": "chengpan@apache.org", "time": "Thu Aug 20 22:44:24 2026 +0800" }, "message": "[CELEBORN-2438][INFRA] Remove empty protected_branches in .asf.yaml\n\n### What changes were proposed in this pull request?\n\nRemove the empty protected_branches key from `.asf.yaml`.\n\n### Why are the changes needed?\n\nASF infra validation rejects .asf.yaml because github.protected_branches\nexpects a mapping, and the empty value fails schema parsing. Branch\nprotection is already handled by github.rulesets, so the key is redundant.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nParsed .asf.yaml with a YAML parser to confirm it loads.\n\nAssisted-by: DeepSeek V4 Pro\n\nCloses #3819 from pan3793/CELEBORN-2438.\n\nAuthored-by: Cheng Pan \u003cchengpan@apache.org\u003e\nSigned-off-by: Cheng Pan \u003cchengpan@apache.org\u003e\n" }, { "commit": "ba4fa59f0279bee81e7cfa838e083a603d331ff2", "tree": "d59802860340aa2c75773b9599e671b96e172eca", "parents": [ "8b7b73b6d4093cb12cbfca318f8e91a574259c3e" ], "author": { "name": "yangjie01", "email": "yangjie01@baidu.com", "time": "Thu Aug 20 22:20:19 2026 +0800" }, "committer": { "name": "Cheng Pan", "email": "chengpan@apache.org", "time": "Thu Aug 20 22:20:19 2026 +0800" }, "message": "[CELEBORN-2433] Fix Utils.tryWithResources evaluating by-name resource expression twice\n\n### What changes were proposed in this pull request?\n\n`Utils.tryWithResources` declared its resource parameter by-name (`f: \u003d\u003e R`) but referenced it twice: `val res \u003d f` evaluated the caller\u0027s expression once, then `func(f)` evaluated it a *second* time. The instance handed to `func` was therefore never the one closed in `finally`. This PR passes the already-evaluated `res` to `func` (one line), and adds a guard test to `UtilsSuite` covering both halves of the helper\u0027s contract: the expression evaluates exactly once and the instance received by the function is the one closed; when `func` throws, `finally` still closes that instance.\n\n### Why are the changes needed?\n\nEvery call with a non-idempotent resource expression opened the resource twice and leaked one handle. Current callers affected (CLI): `CliConfigManager` opens its config file twice per read and leaks one `FileInputStream` (which also blocks file deletion on Windows); `CliVersionProvider` leaks one `BufferedSource` per lookup. Concise repro against current main:\n\n```scala\nval evaluations \u003d new AtomicInteger(0)\nval received \u003d Utils.tryWithResources {\n evaluations.incrementAndGet()\n new Closeable { var closed \u003d false; def close() \u003d closed \u003d true }\n}(res \u003d\u003e res)\n// main: evaluations \u003d\u003d 2, received.closed \u003d\u003d false (leaked)\n```\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\nNo: internal utility fix; the only observable effect is that the CLI no longer opens files twice / leaks handles.\n\n### How was this patch tested?\n\n- Guard test verified red/green: against the unfixed tree the assertions fail (`2 did not equal 1`, `received.closed \u003d\u003d false`); they pass with the fix.\n\nCloses #3814 from LuciferYang/fix-trywithresources-double-eval.\n\nAuthored-by: yangjie01 \u003cyangjie01@baidu.com\u003e\nSigned-off-by: Cheng Pan \u003cchengpan@apache.org\u003e\n" }, { "commit": "8b7b73b6d4093cb12cbfca318f8e91a574259c3e", "tree": "e180f17b18e57aa29a1002ec309e2b6140dbcaf6", "parents": [ "77f8c4b7d0f8fb8ed78eda4f3224b1e64cc254bc" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Thu Aug 20 21:03:02 2026 +0800" }, "committer": { "name": "Cheng Pan", "email": "chengpan@apache.org", "time": "Thu Aug 20 21:03:02 2026 +0800" }, "message": "[CELEBORN-2408] Exclude unused Netty QUIC/HTTP3 modules from shaded clients\n\n### What changes were proposed in this pull request?\n\nExclude `netty-codec-http3`, `netty-codec-classes-quic`, and `netty-codec-native-quic` from the `netty-all` dependency in the root pom\u0027s dependencyManagement.\n\n### Why are the changes needed?\n\nSince the Netty `4.1.118` → `4.2.10.Final` upgrade, `netty-all` pulls in the QUIC/HTTP3 stack that 4.2 merged into the mainline:\n\n```\nnetty-all\n └─ netty-codec-http3 (compile)\n └─ netty-codec-classes-quic (compile)\n └─ netty-codec-native-quic (runtime, 5 classifiers)\n```\n\n`netty-codec-native-quic` ships a `libnetty_quiche42_*` native library per platform classifier (`linux-x86_64`, `linux-aarch_64`, `osx-x86_64`, `osx-aarch_64`, `windows-x86_64`) — **~29MB uncompressed / ~15MB compressed** in total, plus 156 QUIC classes + 112 HTTP3 classes.\n\nBecause every `*-shaded` module uses `maven-shade-plugin` with `\u003cinclude\u003eio.netty:*\u003c/include\u003e`, the wildcard drags all of the above into every shaded client jar. As a result `celeborn-client-spark-3-shaded` grew from **15MB** (0.6.3, Netty 4.1.118) to **27MB** (1.0.0-SNAPSHOT, Netty 4.2.10). The same bloat applies to all 11 shaded modules (spark-3/4, flink 1.18~2.3, mr, tez) and to the runtime classpath of common/client/master/service/worker.\n\nCeleborn only uses Netty TCP RPC; a full-repo grep confirms **zero** references to `codec.quic` / `codec.http3` / `quiche` in source. These are pure dead weight.\n\nExcluding them at the `netty-all` declaration (inherited by all modules via dependencyManagement) drops the shaded jars back to ~15MB with no functional impact.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\n- Built `celeborn-client-spark-3-shaded` (spark-3.5 profile) and `celeborn-client-flink-1.20-shaded` (flink-1.20 profile). Each jar shrank from 27MB to 15MB, with zero `*quiche*` native libs and zero `handler/codec/quic/**` / `handler/codec/http3/**` classes.\n- Verified epoll/kqueue native libs (renamed by antrun) remain present.\n- `mvn dependency:tree` on `common`: quic dependency nodes dropped from 7 → 0 after the exclusion, confirming it takes effect at the dependency resolution layer for all inheriting modules.\n\nCloses #3787 from yew1eb/CELEBORN-2408.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: Cheng Pan \u003cchengpan@apache.org\u003e\n" }, { "commit": "77f8c4b7d0f8fb8ed78eda4f3224b1e64cc254bc", "tree": "1be8ad5cf6f6b862f92202a96dce15370f89b8e6", "parents": [ "994106443354f6c5e8d4f03f36437d6d82a0f1d4" ], "author": { "name": "yangjie01", "email": "yangjie01@baidu.com", "time": "Thu Aug 20 20:58:47 2026 +0800" }, "committer": { "name": "Cheng Pan", "email": "chengpan@apache.org", "time": "Thu Aug 20 20:58:47 2026 +0800" }, "message": "[CELEBORN-2435] Make CelebornConfSuite transport io.mode expectation platform-aware\n\n### What changes were proposed in this pull request?\n\n`CelebornConfSuite`\u0027s `transportTestNetworkIoMode` was hardcoded to `IOMode.EPOLL.name()`. This PR derives it from `CelebornConf.networkIoMode()` so the expected value follows the platform (EPOLL on Linux, KQUEUE on macOS, NIO when no native transport is available), and drops the now-unused `IOMode` import.\n\n### Why are the changes needed?\n\nTwo transport-conf tests fail on macOS because the accessor under test, `CelebornConf#networkIoMode(module)`, falls back to the platform default (KQUEUE on macOS) while the expectation is hardcoded EPOLL:\n\n```\n\"[EPOLL]\" -\u003e \"[KQUEUE]\" (expected, actual)\n```\n\nOn Linux the assertion passes only because the hardcoded EPOLL happens to equal the Linux platform default. Deriving the expectation from the same platform-aware helper keeps the suite meaningful on every platform.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\nNo: test-only change.\n\n### How was this patch tested?\n\nGitHub Actions (Linux) keeps exercising the EPOLL path. Since the failing scenario is macOS-only and CI has no macOS runner, verified locally on macOS (aarch64): before the change the two tests fail with `\"[EPOLL]\" -\u003e \"[KQUEUE]\"`; after the change `mvn -pl common -am test` passes 234/234 in `celeborn-common`.\n\nCloses #3816 from LuciferYang/test-celebornconf-suite-platform-io-mode.\n\nAuthored-by: yangjie01 \u003cyangjie01@baidu.com\u003e\nSigned-off-by: Cheng Pan \u003cchengpan@apache.org\u003e\n" }, { "commit": "994106443354f6c5e8d4f03f36437d6d82a0f1d4", "tree": "169bf55ccf08e5e04817fbb4d736dd78cd662f2c", "parents": [ "51357a4c67eb1526796a2ebc4a3ce9a65e2c9486" ], "author": { "name": "SparksFyz", "email": "sparksvann@gmail.com", "time": "Wed Aug 19 19:50:27 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Wed Aug 19 19:50:27 2026 +0800" }, "message": "[CELEBORN-2411][CIP22] Preserve Spark IO cipher transformation\n\n### What changes were proposed in this pull request?\n\n Preserve Spark\u0027s effective IO cipher transformation when `SparkCryptoHandler`\n builds its minimized `SparkConf`.\n\n Add bidirectional interoperability tests between `SparkCryptoHandler` and\n Spark\u0027s native `CryptoStreamUtils` using `AES/CBC/PKCS5Padding`.\n\n ### Why are the changes needed?\n\n `CryptoStreamUtils.toCryptoConf()` only extracts\n `spark.io.encryption.commons.config.*`, while Spark reads\n `IO_CRYPTO_CIPHER_TRANSFORMATION` separately from `SparkConf`.\n\n As a result, `SparkCryptoHandler` previously dropped a non-default\n transformation and silently fell back to Spark\u0027s default\n `AES/CTR/NoPadding`.\n\n Existing self-round-trip tests did not detect this because both encryption\n and decryption used the same minimized configuration.\n\n ### Does this PR resolve a correctness bug?\n\n - [ ] Yes\n\n ### Does this PR introduce _any_ user-facing change?\n\n - [x] Yes\n\n A non-default Spark IO cipher transformation is now preserved by Celeborn.\n The default `AES/CTR/NoPadding` behavior remains unchanged.\n\n ### How was this patch tested?\n\n - Added Celeborn encryption -\u003e Spark `CryptoStreamUtils` decryption coverage.\n - Added Spark `CryptoStreamUtils` encryption -\u003e Celeborn decryption coverage.\n - Spark 3.5 `SparkCryptoHandlerSuiteJ`: 10 tests passed.\n - Spark 3.0 / Scala 2.12.10 / JDK 11 compilation passed.\n - Spark 4.0 / Scala 2.13.16 / JDK 17 compilation passed.\n - Spark 3.5 Spotless check passed.\n\nCloses #3790 from SparksFyz/CELEBORN-2411-cipher-transformation.\n\nAuthored-by: SparksFyz \u003csparksvann@gmail.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n" }, { "commit": "51357a4c67eb1526796a2ebc4a3ce9a65e2c9486", "tree": "fa3083aadffed9354ad326d52c595a5807b382b6", "parents": [ "08a2da48972b2983590ddcddeea67a0948759921" ], "author": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Aug 18 19:12:39 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Aug 18 19:12:39 2026 +0800" }, "message": "[CELEBORN-2425] Bump Ratis version from 3.2.2 to 3.3.0\n\n### What changes were proposed in this pull request?\n\nBump Ratis version from 3.2.2 to 3.3.0.\n\n### Why are the changes needed?\n\nBump Ratis version from 3.2.2 to 3.3.0. Ratis has released v3.3.0 of which release note refers to [3.3.0](https://ratis.apache.org/post/3.3.0.html). The 3.3.0 version is a maintenance release with multiple improvements and bugfixes.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nCI.\n\nCloses #3806 from SteNicholas/CELEBORN-2425.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "08a2da48972b2983590ddcddeea67a0948759921", "tree": "1eb477e357c64b31ec93212a4129fe139361c18b", "parents": [ "9d3a5b50bd8a7dce4f3445b0a0d54d1187f9f5b1" ], "author": { "name": "Rajeev Kumar", "email": "rajeevks21@iitk.ac.in", "time": "Tue Aug 18 17:52:34 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Aug 18 17:52:34 2026 +0800" }, "message": "[CELEBORN-2426] Correct the calculation of the count of unreleased partitions at shutdown\n\n### What changes were proposed in this pull request?\nThis PR corrects the calculation of the number of unreleased partitions at shutdown.\n\n### Why are the changes needed?\n\n#### Bug\nThe current calculation of the number of unreleased partitions at shutdown is incorrect ([source](https://github.com/rjvkr2021/celeborn/blob/main/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/Worker.scala#L488-L489)).\n```\npartitionLocationInfo.primaryPartitionLocations.size() +\n        partitionLocationInfo.replicaPartitionLocations.size()\n```\n\n#### Why so?\n\n_primaryPartitionLocations__ and _replicaPartitionLocations__ are maps shaped as:\n```\nshuffleKey -\u003e (uniquePartitionLocationId -\u003e PartitionLocation)\n```\nTherefore, the current calculation counts the number of shuffles containing unreleased partitions at shutdown, not the number of unreleased partitions itself.\n\n#### Fix\n\nIt should sum the sizes of the inner maps.\n```\npartitionLocationInfo.primaryPartitionLocations.values().asScala.map(_.size()).sum +\n partitionLocationInfo.replicaPartitionLocations.values().asScala.map(_.size()).sum\n```\n\n### Does this PR resolve a correctness bug?\n\n- [x] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\nNo\n\n### How was this patch tested?\n\nNA\n\nCloses #3807 from rjvkr2021/rajeevkumar/fix_unreleased_partition_count_calculation.\n\nAuthored-by: Rajeev Kumar \u003crajeevks21@iitk.ac.in\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "9d3a5b50bd8a7dce4f3445b0a0d54d1187f9f5b1", "tree": "ac5c1bb8fbe1de13a24ead3161212510dd0d6b5c", "parents": [ "c951e98041dab1d4302b2dd0b016ff52307cbc2d" ], "author": { "name": "Yu Gan", "email": "zhongheng.gy@alibaba-inc.com", "time": "Tue Aug 18 17:08:05 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Aug 18 17:08:05 2026 +0800" }, "message": "[CELEBORN-2413] Fail transport requests retriably when the C++ connection is closed or a write fails\n\n### What changes were proposed in this pull request?\n\n`MessageDispatcher` guards both send paths with a hard `CELEBORN_CHECK(!closed_)` and drops the future returned by the write, so sending on a closed connection raises a **non-retriable** error and a failed write is never reported at all — the request\u0027s promise stays pending until its timeout:\n\n```cpp\nfolly::Future\u003cstd::unique_ptr\u003cMessage\u003e\u003e MessageDispatcher::operator()(...) {\n CELEBORN_CHECK(!closed_); // throws a non-retriable CelebornRuntimeError\n ...\n this-\u003epipeline_-\u003ewrite(std::move(toSendMsg)); // future discarded\n CELEBORN_CHECK(!closed_); // throws a non-retriable CelebornRuntimeError\n return f;\n}\n```\n\nThis PR:\n\n- Merges the two request registries and the closed flag into one `ConnectionState` under a single mutex, so a request is either registered before the connection is retired — and the retirement then fails it — or refused outright with a ready retriable future. Both `CELEBORN_CHECK`s go away.\n- Routes every connection-killing event through `ConnectionState::retire(reason)`, the analogue of Java\u0027s `failOutstandingRequests`: it marks the connection unavailable, so `TransportClient::active()` reports false and the client pool stops handing it out, then fails everything outstanding retriably. It is idempotent, and replaces `cleanup()` on the close path with the per-request messages unchanged.\n- Observes the write future on all three send paths, which is where a socket write failure surfaces (`wangle::AsyncSocketHandler::write` fails it either immediately or later from `writeErr`, without necessarily flipping the closed flag first). Like Java\u0027s `StdChannelListener`, the connection is retired before the failure is reported: `retire()` flips the flag and drains the registries in one critical section, then fails the promises — which matters because `setException` runs the caller\u0027s continuation inline, and it may ask the pool for a client straight away.\n- Handles the other way a write can fail: by throwing. Messages are serialized on the way down the pipeline and `wangle::Pipeline::write` has no try/catch, so that failure never reaches the write future and left the request registered forever. It is now unregistered, and the exception keeps propagating — a broken message is not a broken connection. The fire-and-forget path logs it, since `~WorkerPartitionReader` sends `BufferStreamEnd` there.\n- Keeps the continuations independent of the dispatcher\u0027s lifetime: `ConnectionState` is refcounted separately and every continuation and interrupt handler holds a `weak_ptr` to it. A write failure is reported from the socket\u0027s write callback, so it can arrive during teardown, where capturing `this` is a deterministic segfault. Member reordering can\u0027t fix that — `~ClientDispatcherBase` does `pipeline_-\u003eremove(this).finalize()`, so the dispatcher must be destroyed while its pipeline is alive. For the same reason `close()` now closes the pipeline before retiring, and `~MessageDispatcher` retires instead of leaving `BrokenPromise` to the caller.\n- Preserves `isRetriable` through `TransportClient`, which previously re-threw via `CELEBORN_FAIL` or flattened the cause into a plain `std::runtime_error`. Non-retriable causes keep byte-identical messages, so `getPushDataFailCause` string matching is unaffected.\n- Makes `TransportClientFactory::createClient` throw a retriable error on a failed connect.\n- Reports a duplicate registration through `CELEBORN_CHECK` instead of `registries[key]` handing back the existing holder, where `setInterruptHandler` threw a bare `std::logic_error` from inside the critical section.\n\nThe retirement reason is phrased `\"Failed to send request {}, errorMsg: {}\"` to match the Java listener, so `connectFail()` classifies it — and the requests the dead connection takes down with it — as `PUSH_DATA_CONNECTION_EXCEPTION_PRIMARY` rather than as the non-critical default.\n\n### Why are the changes needed?\n\nA closed connection or a failed write is a normal recoverable condition — a worker restart, an idle timeout — not an invariant violation. Java fails the affected requests (`TransportResponseHandler#channelInactive` → `failOutstandingRequests`, `StdChannelListener#operationComplete` → `handleFailure`) and `CelebornInputStream#createReaderWithRetry` retries or fails over to the replica; the C++ client hard-failed instead, or stalled until the request timeout. Leaving the connection available afterwards also meant every retry got the same dead connection back from the pool.\n\nNo C++ path branches on `isRetriable()` yet, but until now the classification was not observable at the public API at all.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\nShuffle output is unaffected: this is a failure-handling fix.\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\nNo config or API change. A connection-closed or failed-write condition that previously failed non-retriably, or hung until timeout, now fails the affected request retriably.\n\n### How was this patch tested?\n\nNew tests in `MessageDispatcherTest.cpp`:\n\n- `sendRpcRequestAfterCloseFailsRetriably`, `sendFetchChunkRequestAfterCloseFailsRetriably`, `closeFailsInFlightRequestsRetriably` — sends after `close()`, and requests in flight when it happens.\n- `sendRpcRequestFailedWriteRetiresConnection`, `sendFetchChunkRequestFailedWriteRetiresConnection` — a `MockHandler` failing from the second write on, so an earlier request is still in flight on the connection the failing write kills. Each asserts both requests fail retriably, `isAvailable()` is false, and a further send fails fast.\n- `failedWriteWithoutResponseRetiresConnection`, `sendWithoutResponseAfterCloseIsSkipped` — the fire-and-forget path, previously uncovered, where the old `CELEBORN_CHECK` threw out of a destructor.\n- `throwingWriteUnregistersTheRequest`, `throwingFetchWriteUnregistersTheRequest`, `throwingWriteWithoutResponseIsReported`, `nonStdThrowingWriteWithoutResponseIsReported` — the mock rejects one message by throwing, with and without deriving from `std::exception`; the connection stays available and the rejected request is no longer registered.\n- `failedWriteMakesTransportClientInactive` — `TransportClient::active()` is false after a failed write, which is what the factory gates connection reuse on.\n- `failedWriteAfterDispatcherDestroyedIsIgnored`, `failedFetchWriteAfterDispatcherDestroyedIsIgnored` — the mock hands its write promise back to the test, which destroys the dispatcher before failing the write, as `AsyncSocket` does during teardown. Both segfault against a continuation capturing `this`.\n\nNew tests in `TransportClientTest.cpp` asserting the classification survives to the public API: `sendRpcRequestSyncPreservesRetriableFailure`, `pushDataAsyncPreservesRetriableFailure`, `fetchChunkAsyncPreservesRetriableFailure`.\n\nNot covered: the factory dropping a cached client and connecting again end to end, which needs a listening socket that no C++ unit test brings up today.\n\nThe C++ suite passes locally on macOS/arm64, and is covered by the `Celeborn Cpp Integration Test` workflow.\n\nCloses #3801 from yugan95/CELEBORN-2413.\n\nAuthored-by: Yu Gan \u003czhongheng.gy@alibaba-inc.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "c951e98041dab1d4302b2dd0b016ff52307cbc2d", "tree": "bf63a3299cc66a29d2a700d1ce24274b7f3e6185", "parents": [ "14b2cf597cc4f50a94aec4ad78657bad05559557" ], "author": { "name": "Fei Wang", "email": "fwang12@ebay.com", "time": "Tue Aug 18 17:06:11 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Aug 18 17:06:11 2026 +0800" }, "message": "[CELEBORN-2427] Upgrade Netty to 4.2.17.Final\n\n### What changes were proposed in this pull request?\nUpgrade Netty to 4.2.17.Final\n\n### Why are the changes needed?\nTo bring the latest bug fixes.\n\nhttps://github.com/netty/netty/releases/tag/netty-4.2.17.Final (2026-08-04)\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\nGA.\n\nCloses #3808 from turboFei/netty.\n\nAuthored-by: Fei Wang \u003cfwang12@ebay.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "14b2cf597cc4f50a94aec4ad78657bad05559557", "tree": "75cea5aa816e24958fccd3fd4aead699ba327687", "parents": [ "9c16805fc18fe94bca3b6a55cb75960250a5d24b" ], "author": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Mon Aug 17 19:00:50 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Mon Aug 17 19:00:50 2026 +0800" }, "message": "[CELEBORN-2322][FOLLOWUP] Upgrade docker/login-action for release workflow\n\n### What changes were proposed in this pull request?\n\nUpgrade docker/login-action to the ASF-allowlisted v4.6.0 commit.\n\n### Why are the changes needed?\n\nPublishing v0.7.0 triggered https://github.com/apache/celeborn/actions/runs/31992095865, but the workflow failed at startup because the pinned action SHA is no longer allowed by the ASF GitHub Actions policy. After this change is merged, the workflow can be dispatched from main with celeborn_version\u003d0.7.0.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\n- Verified the new SHA is listed in apache/infrastructure-actions as docker/login-action v4.6.0.\n- Verified the SHA matches the upstream v4.6.0 tag.\n- Ran git diff --check.\n\nCloses #3804 from zaynt4606/codex/fix-docker-login-action.\n\nAuthored-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nCo-Authored-By: Codex \u003cnoreply@openai.com\u003e\nAI-Model: gpt-5\nAI-Contributed/Feature: 2/2\nAI-Contributed/UT: 0/0\n" }, { "commit": "9c16805fc18fe94bca3b6a55cb75960250a5d24b", "tree": "beb6310989e891a96af4cbefb4c002133dff456e", "parents": [ "cfd01ef3426e77c5dc60414504839c41559cace4" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Fri Aug 14 14:15:37 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Fri Aug 14 14:15:37 2026 +0800" }, "message": "[CELEBORN-2420] Bump scala version to 2.12.18\n\n### What changes were proposed in this pull request?\n\n1. Unify `scala.version` to 2.12.18 across all spark-3.x profiles.\n2. Remove `maven.plugin.silencer.version` override (1.6.0) from spark-3.0 and spark-3.1 profiles, so they use the default 1.7.19 which supports scala 2.12.18.\n\n### Why are the changes needed?\nscala-maven-plugin 4.9.x auto-generates `-release 8` from `maven.compiler.target\u003d8` and passes it to scalac. Scala 2.12.10–2.12.17 all have issues with `-release` on JDK 8:\n\n- **2.12.15/2.12.17**: unconditionally rejects `-release` on JDK 8\n- **2.12.10**: predates the `-release` option, would fail with \"bad option\"\n\n\u003cimg width\u003d\"1346\" height\u003d\"372\" alt\u003d\"image\" src\u003d\"https://github.com/user-attachments/assets/8cd6588b-ad24-4a8b-8eba-b786288a6035\" /\u003e\n\nScala 2.12.18 allows `-release 8` on JDK 8.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n`./build/make-distribution.sh -Pspark-3.0` on JDK 8.\n`./build/make-distribution.sh -Pspark-3.3` on JDK 8.\n\n\u003cimg width\u003d\"735\" height\u003d\"152\" alt\u003d\"image\" src\u003d\"https://github.com/user-attachments/assets/24b21ce0-8504-4a32-9f2c-6d234cc8110e\" /\u003e\n\nCloses #3799 from yew1eb/fix/java8-release-flag.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "cfd01ef3426e77c5dc60414504839c41559cace4", "tree": "d3a84183c55546b748a92d158ec33f336ddecc35", "parents": [ "87aae3fa7a5ef8a2e31ebeb673c18a3a61e4a964" ], "author": { "name": "Yu Gan", "email": "zhongheng.gy@alibaba-inc.com", "time": "Wed Aug 12 15:19:14 2026 +0800" }, "committer": { "name": "子懿", "email": "programgeek@163.com", "time": "Wed Aug 12 15:19:14 2026 +0800" }, "message": "[CELEBORN-2412] Propagate the worker error message through cpp MessageDispatcher failures\n\n### What changes were proposed in this pull request?\n\n`MessageDispatcher::read` decodes the error string a worker sends back on `RPC_FAILURE` / `CHUNK_FETCH_FAILURE` and then throws it away — both paths fulfil the pending promise with a blank `std::exception()`:\n\n```cpp\nholder.msgPromise.setException(folly::exception_wrapper(std::exception()));\n```\n\nThis PR builds the exception from the message instead:\n\n- `RPC_FAILURE` → `folly::make_exception_wrapper\u003cstd::runtime_error\u003e(failure-\u003eerrorMsg())`\n- `CHUNK_FETCH_FAILURE` → same, using the already-formatted `errorMsg` that includes the `streamChunkSlice` context.\n\n`MessageDispatcherTest` is strengthened accordingly: a shared `takeExceptionMessage()` helper is added, the two failure tests now assert on the message content rather than merely `hasException()`, and `sendPushDataAndReceiveFailure` uses a real StatusCode name (`PUSH_DATA_FAIL_PARTITION_NOT_FOUND`) so it exercises the classification path it is meant to protect.\n\n### Why are the changes needed?\n\n`ShuffleClientImpl::getPushDataFailCause` classifies a push failure by matching StatusCode names as substrings of the error message, and `PushDataCallback::onFailure` / `PushMergedDataCallback` feed it `exception-\u003ewhat()`. With a blank `std::exception`, `what()` is a fixed runtime string that matches none of the 13 candidate causes, so classification always falls through to `PUSH_DATA_FAIL_NON_CRITICAL_CAUSE`. Two consequences:\n\n1. `excludeWorkerByCause` only acts on the connection-fail / timeout causes and hits its `default:` branch every time, so `celeborn.client.excludeWorker.enabled` never actually excludes a worker on the C++ push path.\n2. When revive attempts are exhausted, `classifyPushFailure` raises `\"PUSH_DATA_FAIL_NON_CRITICAL_CAUSE: \u003cblank\u003e\"` — the worker\u0027s real reason (partition not found, worker excluded, replica write failure, …) is gone from the task-failure message, which makes these failures very hard to diagnose from client logs alone.\n\nChunk fetch failures lose the worker\u0027s diagnostics the same way.\n\nThis restores parity with the Java client, where `TransportResponseHandler` surfaces the worker\u0027s error text and `ShuffleClientImpl#getPushDataFailCause` can act on it.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\nShuffle output is unaffected, so this is not a result-correctness bug. It is a failure-classification bug: worker exclusion never engages on the C++ push path, and the worker\u0027s reason is missing from the task-failure message.\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\nNo config, API or behaviour change. Push and fetch failure messages now carry the worker\u0027s reason instead of a placeholder.\n\n### How was this patch tested?\n\nUnit tests in `cpp/celeborn/network/tests/MessageDispatcherTest.cpp`:\n\n- `sendRpcRequestAndReceiveFailure` — asserts the RPC failure message reaches the caller.\n- `sendPushDataAndReceiveFailure` — asserts a StatusCode name (`PUSH_DATA_FAIL_PARTITION_NOT_FOUND`) survives on the exception, which is exactly what `getPushDataFailCause` needs to see.\n\nCovered by the `Celeborn Cpp Integration Test` workflow (`Run Unittests of Celeborn Cpp`).\n\nCloses #3791 from yugan95/CELEBORN-2412.\n\nAuthored-by: Yu Gan \u003czhongheng.gy@alibaba-inc.com\u003e\nSigned-off-by: 子懿 \u003cprogramgeek@163.com\u003e\n" }, { "commit": "87aae3fa7a5ef8a2e31ebeb673c18a3a61e4a964", "tree": "56bf879f01343cccce1cfc9b22b3e1860da06de5", "parents": [ "1f15bdcda40bcc9107d3f0b8ba64ad66c785cd0b" ], "author": { "name": "lijianfu03", "email": "lijianfu03@meituan.com", "time": "Wed Aug 12 15:15:49 2026 +0800" }, "committer": { "name": "子懿", "email": "programgeek@163.com", "time": "Wed Aug 12 15:15:49 2026 +0800" }, "message": "[CELEBORN-2032][FOLLOWUP] Disable replica preference for skewed partitions without map range\n\n### What changes were proposed in this pull request?\nCELEBORN-2032 introduced attempt-based primary/replica switching (`preferReplicaRead \u003d context.attemptNumber % 2 \u003d\u003d 1`) in `CelebornShuffleReader` so that odd-numbered task attempts prefer reading the replica `PartitionLocation` instead of the primary, improving fault tolerance across retries/speculative execution.\n\nThis PR disables that replica preference specifically when `celeborn.client.adaptive.optimizeSkewedPartitionRead.enabled` is on and the partition is being read as a skewed partition without map range (`splitSkewPartitionWithoutMapRange`). In that mode, all attempts for the same skewed partition will now consistently read the primary (or previously-resolved) locations instead of alternating between primary and replica.\n\nA unit test (`CelebornPartitionUtilSuiteJ#testSkewPartitionSplitDiffersBetweenPrimaryAndReplicaChunkOffsets`) is added to demonstrate the root cause directly against `CelebornPartitionUtil#splitSkewedPartitionLocations`.\n\n### Why are the changes needed?\n\nWhen `celeborn.client.adaptive.optimizeSkewedPartitionRead.enabled\u003dtrue`, a skewed reduce partition is not read by map-id range. Instead, Celeborn treats all `PartitionLocation`s of that partition as one logical byte stream and splits it into `subPartitionSize` sub-partitions purely by byte offset (`CelebornPartitionUtil#splitSkewedPartitionLocations`). For a given `subPartitionIndex`, this method computes a `chunkRange` (physical chunk index interval) by walking the `chunkOffsets` of whichever `PartitionLocation` objects are passed in.\n\nThe primary and its replica are flushed independently by two different Workers. Even though they hold logically identical data and share the same `uniqueId`, their physical `chunkOffsets` (the byte positions at which each flush produced a new chunk) are **not guaranteed to be identical**.\n\nBecause of CELEBORN-2032, whether a task attempt reads the primary or the replica depends on `attemptNumber % 2`. So:\n- Attempt 0 (first run) reads the primary and resolves `chunkRange` from the primary\u0027s chunk offsets.\n- Attempt 1 (retry / speculative execution) reads the replica and resolves `chunkRange` from the replica\u0027s (possibly different) chunk offsets.\n\nFor the exact same logical `subPartitionIndex`, this can produce two **different physical byte ranges**, e.g. primary resolves to chunk range `[2, 3]` while replica resolves to `[3, 3]` (dropping chunk 2 entirely). The two attempts then read different bytes for what should be the identical logical sub-partition, so their computed byte-count/CRC diverge and fail `SkewHandlingWithoutMapRangeValidator`, surfacing as:org.apache.celeborn.common.exception.CelebornIOException: AQE Partition \u003cn\u003e failed validation check ... Mismatch in metadata for the same chunk range on retry\n\n\u003cimg width\u003d\"1711\" height\u003d\"605\" alt\u003d\"image\" src\u003d\"https://github.com/user-attachments/assets/b4bff79e-7287-4bb5-8ba1-f2bf7cf21f20\" /\u003e\n\n### Does this PR resolve a correctness bug?\n\n- [x] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n- Added `CelebornPartitionUtilSuiteJ#testSkewPartitionSplitDiffersBetweenPrimaryAndReplicaChunkOffsets`, which constructs a primary and a replica `PartitionLocation` sharing the same `uniqueId` but with different `chunkOffsets` (simulating independent flush behavior), and asserts that `CelebornPartitionUtil#splitSkewedPartitionLocations` resolves different chunk ranges (`[2, 3]` vs `[3, 3]`) for the identical `subPartitionIndex`, proving the root cause of the non-idempotent read.\n- Ran the full `CelebornPartitionUtilSuiteJ` (5 tests) and `CelebornShuffleReaderSuite` (10 tests) in `client-spark/spark-3` — all passed.\n\nCloses #3798 from buska88/celeborn-2032-fix.\n\nAuthored-by: lijianfu03 \u003clijianfu03@meituan.com\u003e\nSigned-off-by: 子懿 \u003cprogramgeek@163.com\u003e\n" }, { "commit": "1f15bdcda40bcc9107d3f0b8ba64ad66c785cd0b", "tree": "30006016fa8d7cd467510e7c236762f03dd37844", "parents": [ "f7b878e137c582718094f44c7f8494cc8bd662f3" ], "author": { "name": "Yu Gan", "email": "zhongheng.gy@alibaba-inc.com", "time": "Tue Aug 11 20:19:33 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Aug 11 20:19:33 2026 +0800" }, "message": "[CELEBORN-2416] Export the C ABI from libceleborn_client.so by dropping --exclude-libs,ALL\n\n### What changes were proposed in this pull request?\n\nDrop `-Wl,--exclude-libs,ALL` from the `celeborn_client` link options on ELF platforms, leaving the version script as the sole symbol-visibility mechanism, and record at the call site why the flag must not come back.\n\n### Why are the changes needed?\n\nOn ELF platforms the shared library is currently linked with both a version script and `--exclude-libs,ALL`:\n\n```cmake\ntarget_link_options(celeborn_client PRIVATE\n \"-Wl,--version-script\u003d${CMAKE_CURRENT_SOURCE_DIR}/exports.map\"\n \"-Wl,--exclude-libs,ALL\")\n```\n\n`--exclude-libs,ALL` forces `STV_HIDDEN` on every symbol that comes out of a static archive. The `celeborn_ffi_*` entry points live in `libceleborn_ffi.a`, which is pulled into the shared library with `--whole-archive` a few lines above, so they are exactly the symbols it hides. Hidden visibility takes precedence over the version script, so the `global:` entries in `exports.map` have no effect:\n\n```\n{\n global:\n extern \"C++\" { celeborn::*; };\n celeborn_*;\n celeborn_ffi_*;\n local:\n *;\n};\n```\n\nThe result is that the C ABI never reaches the dynamic symbol table of `libceleborn_client.so`. A downstream binding that links the library by name cannot resolve `celeborn_ffi_create_client` and friends — for the Rust crate under `rust/`, which links it as `dylib\u003dceleborn_client`, the link step fails outright. The FFI shim is unreachable on Linux.\n\nmacOS is unaffected: it uses `-Wl,-exported_symbols_list` with `exports.txt`, which has no equivalent flag.\n\nDropping the flag loses nothing, because the version script already does what it was there for: the `local: *;` rule localizes every symbol that is not `celeborn::*` or `celeborn_*`, so third-party symbols (folly, protobuf, glog, abseil) stay hidden and cannot collide with the host process\u0027s own runtimes.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\nThis is a build/packaging bug. It does not affect shuffle results; it makes the C ABI unusable from a Linux build.\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\nNo config, API or behaviour change. The exported symbol set of `libceleborn_client.so` gains the `celeborn_ffi_*` entry points it was always meant to expose; the set of hidden third-party symbols is unchanged.\n\n### How was this patch tested?\n\nManual check on a Linux build — before the change `celeborn_ffi_create_client` is absent from the dynamic symbol table, after it is present and `T`:\n\n```\n$ nm -D --defined-only build/libceleborn_client.so | grep celeborn_ffi_create_client\n```\n\nThere is no CI step that inspects the exported symbols of the produced library, so this is not covered by an automated check. If reviewers would like one, I am happy to add an assertion to `.github/workflows/cpp_integration.yml` in this PR or a follow-up.\n\nThe existing `Celeborn Cpp Integration Test` workflow still exercises the build and the C++ unit tests.\n\nCloses #3794 from yugan95/CELEBORN-2416.\n\nAuthored-by: Yu Gan \u003czhongheng.gy@alibaba-inc.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "f7b878e137c582718094f44c7f8494cc8bd662f3", "tree": "3d781dabf3d088d7f3d827a110adaa2a6b13835d", "parents": [ "3e2820c66a3f1da036d3f51ca5b72ce50417707c" ], "author": { "name": "Yu Gan", "email": "zhongheng.gy@alibaba-inc.com", "time": "Tue Aug 11 20:11:09 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Aug 11 20:11:09 2026 +0800" }, "message": "[CELEBORN-2417] Pass CelebornConf properties through the C++ FFI\n\n### What changes were proposed in this pull request?\n\nReplace the per-option parameters of `celeborn_ffi_create_client` with a generic `(keys, values, num_props)` property array, and update the Rust wrapper and examples accordingly.\n\nC ABI:\n\n```c\n-celeborn_ffi_handle* celeborn_ffi_create_client(\n- const char* app_id, size_t app_id_len,\n- int32_t push_buffer_max_size,\n- const char* codec, size_t codec_len,\n- char** err_out);\n+celeborn_ffi_handle* celeborn_ffi_create_client(\n+ const char* app_id, size_t app_id_len,\n+ const char* const* keys,\n+ const char* const* values,\n+ size_t num_props,\n+ char** err_out);\n```\n\nEach pair is fed to `CelebornConf::registerProperty`, which already validates key names and value formats. Null `keys`/`values` and null entries at any index are rejected at the boundary with a descriptive error rather than segfaulting.\n\nOn the Rust side, `Config::push_buffer_max_size` and `Config::shuffle_compression_codec` are replaced by `properties: Vec\u003c(String, String)\u003e` with a `set_property()` builder applied in insertion order:\n\n```rust\nlet mut config \u003d Config::new(app_id);\nconfig.set_property(\"celeborn.client.shuffle.compression.codec\", \"ZSTD\");\n```\n\n### Why are the changes needed?\n\nOnly `celeborn.client.push.buffer.max.size` and `celeborn.client.shuffle.compression.codec` were reachable through the FFI. Every further setting a binding wants to expose costs a C ABI break plus a matching change in each downstream language wrapper, so in practice the rest of `CelebornConf` is unavailable to non-Java clients. Passing properties through verbatim means a binding picks up new configuration keys with no ABI change and no code change at all.\n\nIt also removes a duplicated source of truth. The Rust wrapper carried its own codec allow-list (`NONE` / `LZ4` / `ZSTD`) that the C++ side already owns, so the two could drift; validation is now left to `registerProperty`. The Rust-side checks that remain are the ones that are genuinely client-side: empty `app_id`, non-positive port, empty property key, and interior NUL bytes in a property string.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes\n\nThis is a deliberate break of the `celeborn_ffi_create_client` signature and of the `Config` struct in the `celeborn-client` crate. Neither has a released consumer outside this repository — the C ABI and the Rust crate are both new and unpublished — and the two in-tree examples under `rust/examples/` are updated in the same change. No Celeborn configuration key, Java API or wire protocol is affected.\n\n### How was this patch tested?\n\nUnit tests in `rust/celeborn-client/src/lib.rs`. `validate_rejects_unknown_codec` is dropped along with the allow-list it covered; the following are added or reworked:\n\n- `validate_accepts_valid_args`\n- `validate_rejects_empty_property_key`\n- `set_property_accumulates_native_keys` — properties are recorded in insertion order\n- `to_c_strings_rejects_interior_nul`\n\n`cargo test -p celeborn-client` passes 6/6. `cargo clippy` reports one `too_many_arguments` warning on `push_data`, which is pre-existing on `main` and untouched here; `cargo fmt --check` reports two diffs in `shutdown`/`Drop`, also pre-existing and untouched.\n\nNote that no CI workflow currently builds or tests the `rust/` crates, so these tests do not run in CI today — the `Celeborn Cpp Integration Test` workflow compiles the C++ side including `CelebornFfi.cc`, but nothing exercises the binding. If reviewers want Rust coverage wired into CI, I am happy to do it in a follow-up.\n\nCloses #3795 from yugan95/CELEBORN-2417.\n\nAuthored-by: Yu Gan \u003czhongheng.gy@alibaba-inc.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "3e2820c66a3f1da036d3f51ca5b72ce50417707c", "tree": "a5ced6a96d53f186db1788cb211be819824565e7", "parents": [ "a0444ea4c5e3c44efbe073b16b8ebab32c543d93" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Tue Aug 11 19:33:11 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Aug 11 19:33:11 2026 +0800" }, "message": "[CELEBORN-2419] Fix the warning: deprecated systemProperties parameter in maven-surefire-plugin\n\n### What changes were proposed in this pull request?\n\nReplace deprecated `\u003csystemProperties\u003e` with `\u003csystemPropertyVariables\u003e` in maven-surefire-plugin configuration in root `pom.xml`.\n\n### Why are the changes needed?\n\nMaven Surefire Plugin 3.0.0-M5+ deprecated the `\u003csystemProperties\u003e` parameter. Using it produces a warning on every build:\n```\n[INFO] --- surefire:3.0.0-M7:test (default-test) celeborn-client_2.12 ---\n[WARNING] Parameter \u0027systemProperties\u0027 is deprecated: Use systemPropertyVariables instead.\n[INFO] Tests are skipped.\n[INFO]\n[INFO] --- surefire:3.0.0-M7:test (test) celeborn-client_2.12 ---\n[WARNING] Parameter \u0027systemProperties\u0027 is deprecated: Use systemPropertyVariables instead.\n[INFO] Tests are skipped.\n\n```\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nCloses #3797 from yew1eb/CELEBORN-2419.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "a0444ea4c5e3c44efbe073b16b8ebab32c543d93", "tree": "89d8b2b041bebe465db7b6aa6885f3b8f38f433e", "parents": [ "00113daf3a5d2476aaa04a8b576152893d377fda" ], "author": { "name": "SparksFyz", "email": "sparksvann@gmail.com", "time": "Tue Aug 11 19:10:05 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Aug 11 19:10:05 2026 +0800" }, "message": "[CELEBORN-2418] Fix flaky Flink WordCountTest file length check\n\n### What changes were proposed in this pull request?\n\n Wait for asynchronous disk flushing to complete before checking the file length in Flink `WordCountTest`.\n\n This applies the same bounded `eventually` check already used by `HybridShuffleWordCountTest`.\n\n ### Why are the changes needed?\n\n `DiskFileInfo` records the logical file length before `LocalFlusher` finishes writing the physical file.\n\n The test may check the file too early and observe a temporary length of zero, causing an intermittent failure.\n\n ### Does this PR resolve a correctness bug?\n\n - [ ] Yes\n\n ### Does this PR introduce _any_ user-facing change?\n\n - [ ] Yes\n\n ### How was this patch tested?\n\n - Verified the change with `git diff --check`.\n - The fix follows the existing handling in `HybridShuffleWordCountTest`.\n\nCloses #3796 from SparksFyz/fyz/fix-flink-word-count-flaky-ut.\n\nAuthored-by: SparksFyz \u003csparksvann@gmail.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "00113daf3a5d2476aaa04a8b576152893d377fda", "tree": "95293b42199fce58297d493674353721c00c8be4", "parents": [ "90954f7236c8f7a44ae9a583af12ea3c76d49616" ], "author": { "name": "wang-haihua", "email": "wang-haihua@foxmail.com", "time": "Mon Aug 10 17:24:35 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Mon Aug 10 17:24:35 2026 +0800" }, "message": "[CELEBORN-2401] Change default of celeborn.client.spark.fetch.cleanFailedShuffle to true\n\n### What changes were proposed in this pull request?\n\nFlip the default of `celeborn.client.spark.fetch.cleanFailedShuffle` from `false` to `true`.\n\nOnly two lines change:\n\n- `common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala` — `createWithDefault(false)` → `createWithDefault(true)` on `CLIENT_FETCH_CLEAN_FAILED_SHUFFLE`.\n- `docs/configuration/client.md` — the golden config table is updated in the same commit so `ConfigurationSuite` stays green.\n\n### Why are the changes needed?\n\n`celeborn.client.spark.fetch.cleanFailedShuffle` was introduced in 0.6.0 by [CELEBORN-1896](https://issues.apache.org/jira/browse/CELEBORN-1896) (PR #3109) and hardened in 0.7.0 by [CELEBORN-2088](https://issues.apache.org/jira/browse/CELEBORN-2088) (PR #3401). When enabled:\n\n1. The driver records shuffleIds that hit `FetchFailed`.\n2. When the retried stage registers a new shuffleId, the previously failed shuffleId is enqueued for cleanup by `FailedShuffleCleaner`.\n3. `FailedShuffleCleaner` reuses the existing `UnregisterShuffle` path to reclaim the stale shuffle data on workers.\n\nSince 0.4.0 `celeborn.client.spark.stageRerun.enabled` has defaulted to `true`, so `FetchFailed` stage rerun is already the out-of-the-box behavior. However `cleanFailedShuffle` still defaults to `false`, so every retried stage silently leaves its old shuffle data on the workers until the app exits or the periodic expired-shuffle cleanup catches it. The two defaults contradict each other.\n\nA recent production incident on our cluster:\n\n- A single stage retried three times.\n- Retry 1: wrote ~143 TiB of shuffle data before failing.\n- Retry 2: wrote another ~143 TiB before failing.\n- Retry 3: was still running, ~61 TiB written so far.\n- **300+ TiB of stale shuffle bytes were sitting on the workers simultaneously**, none of which the job would ever fetch again.\n\nOn a shared cluster this drives workers into disk-full / high-usage exclusion, cascading into more `FetchFailed` events for unrelated applications and destabilizing the cluster until the offending app exits. Flipping the default removes this failure mode without adding any new deletion path.\n\nThe proposal was raised on dev and supported there; see the [discussion thread](https://lists.apache.org/list.html?devceleborn.apache.org) for context.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes — behavior change with an explicit escape hatch.\n\n- **Before**: after a `FetchFailed`-driven stage rerun, the failed shuffleId\u0027s data lingered on the workers until the application exited (or `celeborn.worker.shuffle.expired.timeout` caught it).\n- **After**: the failed shuffleId\u0027s data is reclaimed as soon as the retried stage registers, via the existing `UnregisterShuffle` path.\n- **Rollback**: users who need the previous behavior can set `celeborn.client.spark.fetch.cleanFailedShuffle\u003dfalse` in their Spark configuration.\n\nThe release notes should call out both the new default and the rollback flag.\n\n### How was this patch tested?\n\n- `ConfigurationSuite` — golden `docs/configuration/client.md` regenerated to keep the doc test green.\n- `CelebornFetchFailureDiskCleanSuite` — the end-to-end integration test for this path already sets `spark.celeborn.client.spark.fetch.cleanFailedShuffle\u003dtrue` explicitly and continues to pass; it exercises the exact code path affected by this default flip.\n- No new deletion route or code path is introduced; the mechanism is the existing `FailedShuffleCleaner` → `UnregisterShuffle` flow that has shipped since 0.6.0.\n\nRelated JIRA: [CELEBORN-2401](https://issues.apache.org/jira/browse/CELEBORN-2401).\n\nCloses #3779 from wang-haihua/flip-cleanfailedshuffle-default.\n\nAuthored-by: wang-haihua \u003cwang-haihua@foxmail.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/6\nAI-Contributed/UT: 0/2\n" }, { "commit": "90954f7236c8f7a44ae9a583af12ea3c76d49616", "tree": "e078471e7fdf2a79ddcc07172c0a63b94cb511c9", "parents": [ "d1d69cb0aead510f88b145f5a042e0841a9f89ce" ], "author": { "name": "ewoodbury", "email": "ewoodbury@x.ai", "time": "Mon Aug 10 16:53:25 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Mon Aug 10 16:53:25 2026 +0800" }, "message": "[CELEBORN-2399] Fix HA worker heartbeat using stateStartTime instead of actual time\n\n### What changes were proposed in this pull request?\n\nIn HA `MetaHandler` for `WorkerHeartbeat`, pass `request.getWorkerHeartbeatRequest().getTime()` into `updateWorkerHeartbeatMeta` instead of `getWorkerStatus().getStateStartTime()`.\n\n### Why are the changes needed?\n\n`HAMasterMetaManager` correctly sets `.setTime(...)` when submitting the Ratis request. After CELEBORN-1984, the apply path reads the wrong field.\n\n`stateStartTime` is fixed when the worker entered Normal, so `lastHeartbeat` never advances. Once `now \u003e stateStartTime + workerHeartbeatTimeout`, the HA master treats workers as lost despite regular heartbeats.\n\nNon-HA (`SingleMasterMetaManager`) is unaffected. `branch-0.6` still uses `getTime()` and is unaffected. Affects `main` / `branch-0.7` (including v0.7.0-rc0).\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nRoot-cause analysis of submit path (`HAMasterMetaManager` sets `.setTime`) vs apply path (`MetaHandler` was reading `stateStartTime`). Verified the same code path on `branch-0.6` already uses `getTime()`. Observed in HA testing on v0.7 where workers were repeatedly marked lost after the heartbeat timeout.\n\n(Should be cherry-picked to `branch-0.7` for 0.7.0 release)\n\nCloses #3777 from ewoodbury/fix-ha-worker-heartbeat-time.\n\nAuthored-by: ewoodbury \u003cewoodbury@x.ai\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/2\nAI-Contributed/UT: 0/0\n" }, { "commit": "d1d69cb0aead510f88b145f5a042e0841a9f89ce", "tree": "5b83dc73b30b086cd898566034ae6de7a6129589", "parents": [ "e8464d6461a4eb37f17790c39f2fdbf4132ad941" ], "author": { "name": "Venkata Krishnan Sowrirajan", "email": "vsowrirajan@microsoft.com", "time": "Mon Aug 10 15:07:42 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Mon Aug 10 15:07:42 2026 +0800" }, "message": "[CELEBORN-2406] Avoid blocking GetReducerFileGroup RPC under ConcurrentHashMap bin lock in updateFileGroup\n\n### What changes were proposed in this pull request?\n\n`ShuffleClientImpl.updateFileGroup` loaded the reducer file group via `reduceFileGroupsMap.compute(shuffleId, ...)`. `ConcurrentHashMap.compute` holds the per-key bin lock for the entire remapping function, so the blocking `GetReducerFileGroup` RPC to the `LifecycleManager` ran while holding that lock.\n\nAt high reduce parallelism the first task of a shuffle takes the bin lock and issues the RPC; every other reduce task for the same `shuffleId` then blocks on that bin lock, even cache hits. On a large shuffle the stage sits idle for minutes with reduce threads BLOCKED in `updateFileGroup` and no fetch activity, i.e. a load convoy.\n\nThis PR moves the RPC out from under the bin lock:\n- Lock-free `get()` fast path, so cache hits never take a lock.\n- Cold loads serialize on a dedicated per-shuffle monitor (`fileGroupLoadLocks`, cleared in `cleanupShuffle`) with a double-check, so only one RPC is issued per shuffle.\n- Caching semantics unchanged: a cached tuple whose `_1()` is null still reloads.\n\n`FlinkShuffleClientImpl` overrides `updateFileGroup` and is unaffected; the Spark plugin uses the base `ShuffleClientImpl` and was exposed.\n\n### Why are the changes needed?\n\nObserved in production (tens of thousands of reduce partitions, ~900 executors): a stage stalled over 20 minutes with reduce threads blocked in `updateFileGroup` -\u003e `ConcurrentHashMap.compute`. A slow RPC under the bin lock turns a transient lookup into a multi-minute wedge.\n\n### Does this PR introduce any user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nNew `ShuffleClientSuiteJ#testUpdateReducerFileGroupConcurrentLoadIssuesSingleRpc`: 16 threads concurrently load the same shuffle against a mock RPC that blocks 500ms; asserts all callers finish and exactly one RPC is issued.\n\nCloses #3776 from venkata91/celeborn-updatefilegroup-convoy-fix.\n\nLead-authored-by: Venkata Krishnan Sowrirajan \u003cvsowrirajan@microsoft.com\u003e\nCo-authored-by: Venkata krishnan Sowrirajan \u003cvsowrirajan@linkedin.com\u003e\nSigned-off-by: zhengtao \u003cshuaizhentao.szt@alibaba-inc.com\u003e\n\nAI-Contributed/Feature: 0/28\nAI-Contributed/UT: 0/54\n" }, { "commit": "e8464d6461a4eb37f17790c39f2fdbf4132ad941", "tree": "ab070f131245cd4251838223b6df02198e604b85", "parents": [ "c0cad1926d4b3ef6d56b277b01dd6351cc5e894f" ], "author": { "name": "r7raul1984", "email": "8199300+r7raul1984@users.noreply.github.com", "time": "Thu Aug 06 13:32:08 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Thu Aug 06 13:32:08 2026 +0800" }, "message": "[CELEBORN-2402] Log the hostname for celeborn shuffle fetch failures\n\n### What changes were proposed in this pull request?\n\nAdded `workerHost` parameter to the `ExceptionMaker.makeFetchFailureException()` interface and all its implementations (spark-3). `CelebornInputStream` now resolves the hostname from `currentReader.getLocation()` (falling back to `\"unknown\"` if unavailable) and embeds it in the `FetchFailedException` message.\n\nNew exception message format:\n```\nCeleborn FetchFailure with appShuffleId/shuffleId: \u003cid\u003e/\u003cid\u003e from worker \u003chost\u003e\n```\n\n### Why are the changes needed?\n\nPreviously, the `FetchFailedException` message contained no information about which worker was involved in the failure. This made it difficult to identify the problematic worker during incident investigation. Embedding the hostname improves observability at no cost.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nAdded unit test in `CelebornShuffleReaderSuite` (spark-3) to verify that the worker hostname appears in the `FetchFailedException` message.\n\nCloses #3780 from r7raul1984/CELEBORN-2402.\n\nAuthored-by: r7raul1984 \u003c8199300+r7raul1984@users.noreply.github.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "c0cad1926d4b3ef6d56b277b01dd6351cc5e894f", "tree": "156aafd5f0525a4de916beacbc2f5d6046509d71", "parents": [ "65132bbe849a5187df0faefc85c89d0c650feb58" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Thu Aug 06 13:29:09 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Thu Aug 06 13:29:09 2026 +0800" }, "message": "[CELEBORN-2404] Fix required field typo in SendWorkerEventRequest schema\n\n### What changes were proposed in this pull request?\n\nFix a typo in `master_rest_v1.yaml`: the `SendWorkerEventRequest` schema declared `worker` in its `required` list, but the actual property name is `workers`.\n\n### Why are the changes needed?\n\nThe openapi-generator silently ignores the non-existent `worker` field, so the generated client compiles fine, but the spec declares a required field that does not exist and Swagger UI misleadingly shows `worker` as required. The server side validates `request.getWorkers.isEmpty`, consistent with the `workers` property.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\n- Regenerated the openapi client (`./build/mvn clean generate-sources -pl openapi/openapi-client`): no diff in generated code, as the generator does not emit anything for `required`.\n- The module builds successfully, which also validates the YAML syntax.\n\nCloses #3783 from yew1eb/fix-send-worker-event-request-required.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "65132bbe849a5187df0faefc85c89d0c650feb58", "tree": "63cd18de71bee3e3df5920a6fa2121ffb087faa7", "parents": [ "fd6aaa59ba4ed796db9e4373fc31fde5dfb0b3f3" ], "author": { "name": "Kalvin2077", "email": "wk.huang2077@outlook.com", "time": "Mon Aug 03 14:10:18 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Mon Aug 03 14:10:18 2026 +0800" }, "message": "[CELEBORN-2394] Add REST v1 API to unregister shuffles\n\n### What changes were proposed in this pull request?\n\n- Add an OpenAPI-based `POST /api/v1/shuffles/unregister` endpoint that unregisters multiple shuffles for an application in one request.\n- Generate Java client support for `ShuffleApi.unregisterShuffles` and `UnregisterShufflesRequest` from the updated OpenAPI specification.\n- Use the existing batch unregister RPC when handling the REST request.\n- Add `--unregister-shuffles` support to Celeborn Shell, reusing the existing `--apps` and `--shuffleIds` options.\n- Fix the legacy application resource test to deserialize `ApplicationsHeartbeatResponse` instead of `ApplicationsResponse`.\n\n### Why are the changes needed?\n\nShuffle unregistration was previously available only through Celeborn\u0027s internal RPC path. REST API, generated Java client, and Celeborn Shell users had no corresponding administrative operation.\n\nSupporting multiple shuffle IDs in one request also avoids requiring callers to issue one request per shuffle and aligns the public interfaces with Celeborn\u0027s existing batch unregister RPC.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes\n\nUsers can unregister multiple shuffles through the REST v1 API, the generated OpenAPI Java client, or Celeborn Shell.\n\nREST request example:\n\n```http\nPOST /api/v1/shuffles/unregister\nContent-Type: application/json\n\n{\n \"appId\": \"app1\",\n \"shuffleIds\": [1, 2, 3]\n}\n```\n\nCeleborn Shell example:\n\n```shell\nceleborn-cli master --unregister-shuffles --apps app1 --shuffleIds 1,2,3\n```\n\n### How was this patch tested?\n\n- Added two tests in `ApiV1MasterResourceSuite` covering batch unregistration, preservation of unrelated shuffles, idempotency, and invalid requests.\n- Extended `ApiV1OpenapiClientSuite` to cover batch unregistration through the generated Java client.\n- Added Celeborn Shell integration coverage for successful batch unregistration, missing options, multiple application IDs, malformed shuffle IDs, and negative shuffle IDs.\n- Verified the generated Java sources with:\n\n ```shell\n mvn -pl openapi/openapi-client -Pgenerate generate-resources -DskipTests\n ```\n\n- Ran the affected test suites with Maven. The Master REST suite passed 9 tests, the OpenAPI client suite passed 11 tests, and the CLI suite passed 43 tests with one pre-existing HA test canceled.\n- Verified formatting with:\n\n ```shell\n mvn spotless:check -pl cli -am -DskipTests\n ```\n\nCloses #3770 from Kalvin2077/feat/unregister-shuffle.\n\nAuthored-by: Kalvin2077 \u003cwk.huang2077@outlook.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "fd6aaa59ba4ed796db9e4373fc31fde5dfb0b3f3", "tree": "12801c84557c9249328eed4d12b65b26e26ec6e5", "parents": [ "07dde50e4043e096e385197c53208e21e8d316ce" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Mon Aug 03 10:37:52 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Mon Aug 03 10:37:52 2026 +0800" }, "message": "[CELEBORN-2396][CLI] Fix CelebornCli.main discarding the configured CommandLine instance\n\n### What changes were proposed in this pull request?\n\nCall `execute()` on the configured `CommandLine` instance instead of creating and executing a new one.\n\n### Why are the changes needed?\n\n`CelebornCli.main` creates a `CommandLine`, sets case-sensitivity options on it, but then executes a brand-new instance, so the configuration is silently discarded and the two setter calls are dead code.\n\n### Does this PR introduce any user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nCloses #3773 from yew1eb/CELEBORN-2396.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "07dde50e4043e096e385197c53208e21e8d316ce", "tree": "df9efcd5136c74e892135bcb65b6b8ee1d8e0798", "parents": [ "5a59fb764de99a531c1a58a41064243cc6fabb2e" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Mon Aug 03 10:34:01 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Mon Aug 03 10:34:01 2026 +0800" }, "message": "[CELEBORN-2395] Reduce allocations in Utils.split* methods\n\n## What changes were proposed in this pull request?\n\nRewrite the three `String.split(\"-\")`-based key splitters in `Utils` to locate the `-` separator once and slice with `substring`, instead of regex-splitting and rejoining:\n\n- `splitShuffleKey` → `lastIndexOf(\u0027-\u0027)` + two `substring` calls\n- `splitPartitionLocationUniqueId` → `lastIndexOf(\u0027-\u0027)` + two `substring` calls\n- `splitAttemptKey` → `indexOf(\u0027-\u0027)` + two `substring` calls\n\nThis removes the regex compilation, the intermediate `String[]`, the `dropRight(1)` array copy, and the `StringBuilder` + new `String` from `mkString`.\n\n## Why are the changes needed?\n\n`splitShuffleKey` is on the per-push-message hot path, called from both `recordAppActiveConnection` (default-on per-app metrics) and `checkAuth`. A production async-profiler CPU flame graph on `celeborn-worker` (80,611 samples; true stack-reconstruction, self sum \u003d 100%) shows the frame\u0027s self cost is near zero, but its inclusive cost is ~4.23%, of which ~2.95% comes through the default-on `recordAppActiveConnection` path and ~1.27% through the always-on `checkAuth` path.\n\n\u003cimg width\u003d\"1619\" height\u003d\"260\" alt\u003d\"image\" src\u003d\"https://github.com/user-attachments/assets/6a807653-fcdd-4037-8d16-117ebe70046e\" /\u003e\n\n\u003cimg width\u003d\"1686\" height\u003d\"244\" alt\u003d\"image\" src\u003d\"https://github.com/user-attachments/assets/240be695-4985-4f8a-abc1-e32c21c6d0ee\" /\u003e\n\nThe allocation reduction also lowers young-GC pressure under high push QPS.\n\n## Does this PR resolve a correctness bug?\n\n No\n\n## Does this PR introduce any user-facing change?\n\n No\n\n## How was this patch tested?\n\n- Added unit tests in `UtilsSuite` covering `splitShuffleKey` (including `applicationId` containing `-`, plus a `makeShuffleKey` round-trip), `splitPartitionLocationUniqueId`, and `splitAttemptKey`.\n- `./build/mvn -pl common -Dtest\u003dUtilsSuite test` — all `split*` tests pass. (Two unrelated `CelebornConfSuite` failures — `Fallback to parent module\u0027s config...` and `rpc_service and rpc_client...` — are pre-existing on a clean `upstream/main`; verified by stashing this change and re-running.)\n- `./build/mvn -pl common spotless:check` — clean.\n\nCloses #3771 from yew1eb/CELEBORN-2395.\n\nAuthored-by: yew1eb \u003cyew1eb@gmail.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "5a59fb764de99a531c1a58a41064243cc6fabb2e", "tree": "9084a1f0f4f39ee76809b5824cc9d906348e5147", "parents": [ "2fcd8a1ead664936267afb1a97702b5d57fb87a3" ], "author": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Sun Aug 02 13:16:37 2026 +0800" }, "committer": { "name": "zhengtao", "email": "shuaizhentao.szt@alibaba-inc.com", "time": "Sun Aug 02 13:16:37 2026 +0800" }, "message": "Bump 1.0.0-SNAPSHOT\n\nAI-Contributed/Feature: 0/12\nAI-Contributed/UT: 0/0\n" }, { "commit": "2fcd8a1ead664936267afb1a97702b5d57fb87a3", "tree": "b75a74c3ab26f516f5d93960416d170e9e971b66", "parents": [ "2b5dc8d5f14ed575be8d4a3f4d72f718fd945701" ], "author": { "name": "yew1eb", "email": "yew1eb@gmail.com", "time": "Fri Jul 31 15:56:17 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 31 15:56:17 2026 +0800" }, "message": "[CELEBORN-2397][CLI] Expose logger level get/set endpoints in celeborn-cli (#3774)\n\n* [CELEBORN-2397][CLI] Expose logger level get/set endpoints in celeborn-cli\n\n* [CELEBORN-2397][CLI] Trigger CI\n\n* [CELEBORN-2397][CLI] Trigger CI" }, { "commit": "2b5dc8d5f14ed575be8d4a3f4d72f718fd945701", "tree": "089c81207fddaec40eb51424bd617a6db906bccb", "parents": [ "f0c8139621f3da73ccf30566fb344ffb2ff47cf9" ], "author": { "name": "sychen", "email": "sychen@ctrip.com", "time": "Tue Jul 28 20:27:45 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Jul 28 20:27:45 2026 +0800" }, "message": "[CELEBORN-2308] Bump commons-lang3 version from 3.17.0 to 3.20.0\n\n### What changes were proposed in this pull request?\n\nhttps://commons.apache.org/proper/commons-lang/changes.html#a3.20.0\n\nhttps://issues.apache.org/jira/browse/LANG-1786\n\n### Why are the changes needed?\nJDK25\n```\nWARNING: Use of the three-letter time zone ID \"IET\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"JST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"IST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"BET\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"ACT\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"HST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"AET\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"AGT\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"VST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"CNT\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"EST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"NET\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"PLT\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"CST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"SST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"CTT\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"PNT\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"BST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"MIT\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"ART\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"AST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"PRT\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"PST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"NST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"EAT\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"MST\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"ECT\" is deprecated and it will be removed in a future release\nWARNING: Use of the three-letter time zone ID \"CAT\" is deprecated and it will be removed in a future release\n```\n\n```java\n\tat java.base/java.util.TimeZone.getTimeZone(TimeZone.java:600)\n\tat java.base/java.util.TimeZone.getTimeZone(TimeZone.java:549)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateParser$TimeZoneStrategy.\u003cinit\u003e(FastDateParser.java:920)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateParser.getLocaleSpecificStrategy(FastDateParser.java:661)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateParser.getStrategy(FastDateParser.java:630)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateParser.access$200(FastDateParser.java:75)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateParser$StrategyParser.letterPattern(FastDateParser.java:234)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateParser$StrategyParser.getNextStrategy(FastDateParser.java:220)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateParser.init(FastDateParser.java:163)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateParser.\u003cinit\u003e(FastDateParser.java:149)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateFormat.\u003cinit\u003e(FastDateFormat.java:399)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateFormat.\u003cinit\u003e(FastDateFormat.java:383)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:105)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:102)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FormatCache.getInstance(FormatCache.java:79)\n\tat org.apache.celeborn.shaded.org.apache.commons.lang3.time.FastDateFormat.getInstance(FastDateFormat.java:162)\n\tat org.apache.celeborn.common.util.Utils$.\u003cinit\u003e(Utils.scala:1238)\n\tat org.apache.celeborn.common.util.Utils$.\u003cclinit\u003e(Utils.scala)\n```\n\nhttps://bugs.openjdk.org/browse/JDK-8342550\n\n### Does this PR resolve a correctness bug?\nNo\n\n### Does this PR introduce _any_ user-facing change?\nNo\n\n### How was this patch tested?\nGHA\n\nCloses #3663 from cxzl25/CELEBORN-2308.\n\nAuthored-by: sychen \u003csychen@ctrip.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "f0c8139621f3da73ccf30566fb344ffb2ff47cf9", "tree": "93e6454121150c93d542589c91a313fddf9c767b", "parents": [ "76ce74504ba66e022e4b7950db6af75e5e0089e2" ], "author": { "name": "Chao Sun", "email": "chao@openai.com", "time": "Fri Jul 24 10:14:00 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Fri Jul 24 10:14:00 2026 +0800" }, "message": "[CELEBORN-2371] Bound Spark batch-open client creation retries and stop them on interruption\n\n## Why are the changes needed?\n\n[CELEBORN-2371](https://issues.apache.org/jira/browse/CELEBORN-2371) follows up on the parallel Spark batch-open client creation added by [#3692](https://github.com/apache/celeborn/pull/3692).\n\nBatch-open locations are grouped by `host:fetchPort`, but each worker task previously walked multiple `PartitionLocation` entries. Since every `createClient` invocation already receives `TransportClientFactory`\u0027s full retry budget, that outer loop could multiply connection latency without targeting a different endpoint.\n\nCancellation also needs to terminate consistently. A direct or wrapped `InterruptedException` must stop reader setup and fetch retry flow before it is recorded as a worker failure, added to shared exclusion state, retried against a peer, or reported as a shuffle fetch failure. The unlimited-timeout transport branch also needed explicit cleanup on cancelled or failed connection futures.\n\n## What changes were proposed in this PR?\n\n### Keep one retry budget per worker endpoint\n\nEach grouped `host:fetchPort` now uses one representative location for client creation. `TransportClientFactory` remains the owner of connection retries, controlled by the existing `celeborn.data.io.maxRetries`; there is no second outer retry setting or multiplicative retry budget.\n\n### Propagate cancellation through reader paths\n\n`CelebornShuffleReader` now uses a shared interrupt-aware client-creation helper in both sequential and parallel batch-open paths. It checks a pre-existing interrupt flag, restores the flag when `createClient` throws `InterruptedException`, and exits without invoking the ordinary failure callback.\n\n`CelebornInputStream` now detects interruption throughout reader creation, failed-stream cleanup, reconnect, and buffer-fill paths. It exits before exclusion, retry, peer failover, or shuffle-fetch-failure reporting. If reader cleanup itself fails while propagating cancellation, the cleanup failure is retained as a suppressed exception without masking the original interruption.\n\n### Preserve interruption and cleanup in transport bootstrap\n\n`TransportClientFactory` uses Guava\u0027s `Throwables.getCausalChain()` to detect wrapped `InterruptedException`, restores the interrupt flag, and stops retrying immediately. TCP-connect and TLS-handshake waits close the in-progress channel before propagating interruption. The unlimited-timeout connection path now also closes cancelled and failed channel futures explicitly.\n\n## How was this PR tested?\n\nFormatting was applied with:\n\n```text\n./build/mvn --no-transfer-progress -DskipTests spotless:apply\n```\n\nThe focused transport tests passed (11 tests):\n\n```text\n./build/mvn --no-transfer-progress -pl common -am \\\n -Dtest\u003dTransportClientFactorySuiteJ,TransportClientFactoryInterruptSuiteJ \\\n -DwildcardSuites\u003dnone clean test\n```\n\nThe focused input-stream peer-failover and interruption tests passed (6 tests):\n\n```text\n./build/mvn --no-transfer-progress -pl client -am \\\n -Dtest\u003dCelebornInputStreamPeerFailoverTest \\\n -DwildcardSuites\u003dnone clean test\n```\n\nThe Spark 3.5 reader suite passed (7 tests):\n\n```text\n./build/mvn --no-transfer-progress -Pspark-3.5 -pl client-spark/spark-3 -am \\\n -Dtest\u003dnone \\\n -DwildcardSuites\u003dorg.apache.spark.shuffle.celeborn.CelebornShuffleReaderSuite \\\n clean test\n```\n\nSpark 4.0 / Scala 2.13 production and test compilation also passed:\n\n```text\n./build/mvn --no-transfer-progress -Pspark-4.0 -pl client-spark/spark-3 -am \\\n -DskipTests clean test\n```\n\nCloses #3746 from sunchao/dev/chao/codex/bound-batch-open-client-retries-oss.\n\nAuthored-by: Chao Sun \u003cchao@openai.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "76ce74504ba66e022e4b7950db6af75e5e0089e2", "tree": "7735e851225e90c3b78c07dc61759f7ec8e62f4b", "parents": [ "59413d040f7e9a6c6145ce13ced3e1707058682f" ], "author": { "name": "senthh", "email": "senthil.kumar@acceldata.io", "time": "Fri Jul 24 10:06:42 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Fri Jul 24 10:06:42 2026 +0800" }, "message": "[CELEBORN-2380] Support JmxSink for exposing metrics as JMX MBeans\n\n### What changes were proposed in this pull request?\n\nThis PR adds a new `JmxSink` metrics sink that exposes Celeborn metrics as JMX MBeans.\n\n- Add `JmxSink` (`common/src/main/scala/org/apache/celeborn/common/metrics/sink/JmxSink.scala`), backed by Dropwizard Metrics\u0027 `JmxReporter`. It follows the existing `Sink` contract and is loaded reflectively by `MetricsSystem` via the standard `(Properties, MetricRegistry)` constructor, so no changes to `MetricsSystem` are required.\n- Add the `io.dropwizard.metrics:metrics-jmx` dependency (which contains `JmxReporter` in Dropwizard Metrics 4.x) to both the Maven build (`pom.xml` dependency management + `common/pom.xml`) and the SBT build (`project/CelebornBuild.scala`), pinned to the existing `${codahale.metrics.version}` (4.2.25).\n- Add a commented-out example to `conf/metrics.properties.template` and `charts/celeborn/files/conf/metrics.properties` showing how to enable the sink.\n- Document `JmxSink` in `docs/monitoring.md`.\n- Register `metrics-jmx-4.2.25.jar` in all `dev/deps/dependencies-*` manifests (server + client profiles) and add `io.dropwizard.metrics:metrics-jmx` to `LICENSE-binary`, since the artifact is now bundled in the distributions.\n\nEnabling the sink is opt-in:\n\n```properties\n*.sink.jmx.class\u003dorg.apache.celeborn.common.metrics.sink.JmxSink\n\nTest Output:\n\n\u003cimg width\u003d\"1315\" height\u003d\"800\" alt\u003d\"Celeborn_jmx\" src\u003d\"https://github.com/user-attachments/assets/7af729f8-7eb9-4749-af9c-e2af64d48fb8\" /\u003e\n\nCloses #3758 from senthh/CELEBORN-2380.\n\nLead-authored-by: senthh \u003csenthil.kumar@acceldata.io\u003e\nCo-authored-by: senthh \u003csenthh@gmail.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "59413d040f7e9a6c6145ce13ced3e1707058682f", "tree": "675b7043fa198479b2e89dec91fadb2e87e3a8c1", "parents": [ "17f7d4f9f46fb23fd668f3b57f6053d651123693" ], "author": { "name": "sychen", "email": "sychen@ctrip.com", "time": "Thu Jul 23 10:12:19 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Thu Jul 23 10:12:19 2026 +0800" }, "message": "[CELEBORN-2385] Guard ByteBufAllocator cast in ChannelsLimiter trim path\n\n### What changes were proposed in this pull request?\n\n`ChannelsLimiter#userEventTriggered` unconditionally casts `ctx.alloc()` to\n`PooledByteBufAllocator` when handling a `TrimCache` event:\n\n```java\n((PooledByteBufAllocator) ctx.alloc()).trimCurrentThreadCache();\n```\n\nThe concrete type of ctx.alloc() is determined by\nceleborn.network.memory.allocator.pooled. When it is set to false (the\ndocumented escape hatch for workers whose direct memory stays high even after\ntrimming), the allocator is an UnpooledByteBufAllocator, so this cast throws a\nClassCastException on the Netty EventLoop thread the next time memory pressure\ntriggers MemoryManager#trimAllListeners → ChannelsLimiter#onTrim →\ntrimCache().\n\n### Why are the changes needed?\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nCloses #3763 from cxzl25/CELEBORN-2385.\n\nAuthored-by: sychen \u003csychen@ctrip.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "17f7d4f9f46fb23fd668f3b57f6053d651123693", "tree": "32db933aaaeba7fc34133a298b9a8cac46b28aca", "parents": [ "d77543ef4bbd3a62aa8f247b6434e7c438d3850a" ], "author": { "name": "sychen", "email": "sychen@ctrip.com", "time": "Thu Jul 16 13:58:25 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Thu Jul 16 13:58:25 2026 +0800" }, "message": "[CELEBORN-2379] Keep sbt and maven dependency lists in sync and cross-check both in CI\n\n### What changes were proposed in this pull request?\n\nKeep the sbt and maven client dependency lists in sync and cross-check both build systems in CI, and fix the `paranamer` dependency convergence for the Spark 4.x clients.\n\n- `dev/dependencies.sh`: drop the process-wide `export LC_ALL\u003dC` and apply `LC_ALL\u003dC sort` locally to the two sort calls, so the sort order stays machine-independent without forcing the locale globally.\n- `.github/workflows/deps.yml`: merge the separate `sbt` and `maven-jdk{8,11,17}` jobs into `jdk{8,11,17}` jobs that fan out over a `tool: [maven, sbt]` matrix (with the `sbt` + `tez` combination excluded, since there is no sbt tez project), so every module is cross-checked by both maven and sbt on its matching JDK.\n- `project/CelebornBuild.scala`: gate the `jdkTools` (`maven-jdk-tools-wrapper`) dependency to JDK 8 only, matching the `JDKTools` plugin and the maven `jdk-8` profile; the wrapper is a no-op on JDK 9+ where the tools API ships with the JDK, and including it unconditionally desynced the sbt/maven classpaths.\n- Fix the `paranamer` version conflict — `spark-sql` declares `2.8.3` while the transitive `jackson-module-scala 2.15.3` pulls `2.8` — for the Spark 4.x clients, keeping maven and sbt in sync:\n - spark-4.0: `spark-sql 4.0.3` itself declares `2.8`, so the manifest is deduped to `paranamer 2.8` with no override.\n - spark-4.1: pin `paranamer` to `2.8.3` via the `spark-4.1` maven profile `dependencyManagement` and the `Spark41` sbt `dependencyOverrides`.\n - spark-4.2: same conflict (`spark-sql 4.2.0` → `2.8.3`); pin to `2.8.3` via the `spark-4.2` maven profile `dependencyManagement` and the `Spark42` sbt `dependencyOverrides`, and drop the stray `paranamer 2.8` line from the manifest.\n\n### Why are the changes needed?\n\nThe sbt and maven dependency manifests could silently drift apart; CI previously validated only one toolchain per module, so divergence went unnoticed. The `paranamer` conflict left two versions on the runtime classpath and two entries in the manifest, breaking dependency convergence. Pinning the conflict and cross-checking both builds in CI keeps the two dependency lists identical. The unconditional `jdkTools` dependency likewise left a stale `tools.jar` entry on JDK 9+ sbt classpaths that had no maven counterpart.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\nGHA `deps` workflow (maven + sbt on JDK 8/11/17); `./dev/dependencies.sh --module spark-4.2 --check` passes locally.\n\nCloses #3757 from cxzl25/CELEBORN-2379.\n\nAuthored-by: sychen \u003csychen@ctrip.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "d77543ef4bbd3a62aa8f247b6434e7c438d3850a", "tree": "9845953e44e39988c28f9d20b55c808832e7e710", "parents": [ "3aa25932fad50a2193a5f1958f49a82a8da22598" ], "author": { "name": "Kalvin2077", "email": "wk.huang2077@outlook.com", "time": "Wed Jul 15 17:56:32 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Wed Jul 15 17:56:32 2026 +0800" }, "message": "[CELEBORN-2382] Authorize `PbReviseLostShuffles` to prevent cross-application metadata modification\n\n### What changes were proposed in this pull request?\n\nCall `checkAuth(context, pb.getAppId)` before serving `PbReviseLostShuffles` in the Master, consistent with the other application-scoped handlers.\n\n### Why are the changes needed?\n\nThe Master handled `PbReviseLostShuffles` without an authorization check. With authentication enabled, an application could specify another application\u0027s ID and add shuffle IDs to its `registeredAppAndShuffles` metadata.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\n- New case in `MasterApplicationMetaAuthSuite` verifying same-application requests succeed and cross-application requests are rejected with authentication enabled\n- New case in `MasterSuite` verifying same-application requests succeed, cross-application requests are rejected, and authentication-disabled behavior is preserved.\n\nCloses #3761 from Kalvin2077/fix/pbrevise-checkauth.\n\nAuthored-by: Kalvin2077 \u003cwk.huang2077@outlook.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "3aa25932fad50a2193a5f1958f49a82a8da22598", "tree": "2a3c55d1337ff868c920f8bb0433609736e32c0e", "parents": [ "50c18eb6b66c1926d037441d954270b60f6f638f" ], "author": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Wed Jul 15 13:37:16 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Wed Jul 15 13:37:16 2026 +0800" }, "message": "[CELEBORN-2383] Support Spark 4.2\n\n### What changes were proposed in this pull request?\n\nSupport Spark 4.2.\n\nAdd a Spark 4.2 build target for the Celeborn Spark client, pinning Spark 4.2.0 on Scala 2.13.18:\n\n- Add the `spark-4.2` Maven profile (`pom.xml`, `tests/spark-it/pom.xml`) and the `Spark42` sbt project (`project/CelebornBuild.scala`), and register Scala 2.13.18 in `ALL_SCALA_VERSIONS`.\n- Add the `dev/deps/dependencies-client-spark-4.2` dependency manifest.\n- Wire `spark-4.2` into `build/make-distribution.sh`, `dev/reformat`, and the deps/maven/sbt CI matrices, and bump the release deploy profile from `spark-4.1` to `spark-4.2`.\n- Document Spark 4.2 compatibility in `README.md` and `docs/developers/sbt.md`.\n\n### Why are the changes needed?\n\nSpark 4.2.0 has already released, which release notes refer to [Spark Release 4.2.0](https://spark.apache.org/releases/spark-release-4-2-0.html).\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nCI.\n\nCloses #3760 from SteNicholas/CELEBORN-2383.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "50c18eb6b66c1926d037441d954270b60f6f638f", "tree": "78552548688fabb25e20c70609728712b1da743c", "parents": [ "da8c1cec328110452458ee4174f03570ba8dddea" ], "author": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Jul 14 21:58:10 2026 +0800" }, "committer": { "name": "Shuang", "email": "lvshuang.xjs@alibaba-inc.com", "time": "Tue Jul 14 21:58:10 2026 +0800" }, "message": "[CELEBORN-2381] Authorize PbApplicationMetaRequest to prevent secret disclosure\n\n### What changes were proposed in this pull request?\n\nCall `checkAuth(context, appId)` before serving `PbApplicationMetaRequest` in the Master, consistent with the other application-scoped handlers (`RequestSlots`, `UnregisterShuffle`, `ApplicationLost`).\n\nWorkers fetch application meta over the internal channel, where the connection has no per-application client id, so the check is a no-op for them; it only rejects an external application that asks for another application\u0027s secret.\n\n### Why are the changes needed?\n\nThe Master served `PbApplicationMetaRequest` by returning the requested application\u0027s SASL secret without an authorization check. With authentication enabled, a caller on the external application port could read another application\u0027s secret and impersonate it.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\n- New `MasterApplicationMetaAuthSuite` verifying `PbApplicationMetaRequest` is authorized against the registered application.\n- New case in `MasterSuite` verifying a caller requesting another application\u0027s secret is rejected.\n\nCloses #3759 from SteNicholas/CELEBORN-2381.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Shuang \u003clvshuang.xjs@alibaba-inc.com\u003e\n" }, { "commit": "da8c1cec328110452458ee4174f03570ba8dddea", "tree": "d586366364cbb7a355bc3c02c54a4b7cbdee214a", "parents": [ "b2f1da858fa374b6755b7e50b9fd875b517962a5" ], "author": { "name": "afterincomparableyum", "email": "224495379+afterincomparableyum@users.noreply.github.com", "time": "Tue Jul 14 15:36:53 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Jul 14 15:36:53 2026 +0800" }, "message": "[CELEBORN-2362] Fix Flaky CI/CD\n\n### What changes were proposed in this pull request?\n\n Address several independent root causes of CI/CD flakiness, spanning the\n build tooling, JaCoCo instrumentation, the test mini-cluster, and the\n process-wide shuffle client.\n\nCI infrastructure\n\n - build/mvn — resilient Maven bootstrap. Default to archive.apache.org instead of the\n closer.lua mirror redirector, and validate that each download is a real gzip tarball before\n extracting (closer.lua intermittently returns an HTML mirror-chooser page with HTTP 200).\n Retry up to 3 times, backing off only between attempts. A valid\n pre-staged/previously-downloaded tarball is reused (keeps offline/air-gapped builds working),\n and curl falls through to wget when curl is absent or produced no valid file (proxy/TLS/CA\n failures wget would survive).\n - pom.xml — exclude io/netty/** from JaCoCo. JaCoCo\u0027s agent rejects Netty 4.2\u0027s\n already-enhanced JFR event classes, tearing down channels mid-write and flaking the Flink\n integration tests.\n\n Process-wide ShuffleClient made per-application\n\n The static single-slot ShuffleClient singleton couldn\u0027t represent the multiple applications a\n reused spark-it JVM runs with overlapping lifecycles: on an app switch it evicted the\n previous app\u0027s client without teardown, and a lock-free get() fast path could hand a live app\n the wrong app\u0027s client — surfacing as celebornShuffleId 0 corruption\n (ArrayIndexOutOfBoundsException / CommitMetadata CRC mismatch).\n\n - Replaced the single slot with a per-appUniqueId registry (ConcurrentHashMap\u003cString,\n ShuffleClient\u003e). Each application gets its own isolated client; only fully-initialized\n instances are ever published, closing the torn read.\n - Added ShuffleClient.removeInstance(...), and SparkShuffleManager.stop() now removes+shuts\n down only its own app\u0027s client instead of calling the global reset() (which would tear down\n other live applications\u0027 clients in a multi-app JVM). This also eliminates the\n orphaned-client leak (RpcEnv, Netty factory, retry pool, reviveManager) that previously\n accumulated across app switches.\n\n Spark-it lifecycle isolation\n\n - SparkTestBase — stop any SparkSession/SparkContext still alive in the JVM in afterAll\n (triggering SparkShuffleManager.stop()) and reset the client, so a straggler task from a\n leaked context can\u0027t bind to a later suite\u0027s LifecycleManager through the shared client.\n Reduced worker count from 5 to 3 (the MiniCluster default): the suites run serially in one\n JVM, so each surplus worker multiplies the long-lived thread/CPU footprint and, under CI\n contention, starves RPC/fetch handlers past the 240s network timeout.\n\n Cluster-setup test robustness\n\n - MiniClusterFeature — retry worker startup with exponential backoff, recreating the worker\n each attempt. Record a worker in the returned set only once it has registered, so a torn-down\n failed worker (distinct identity — Worker has no equals/hashCode) doesn\u0027t leak into the\n returned set and get double-stopped. Tear failed workers down locally (stop +\n rpcEnv.shutdown) instead of exitImmediately(), which issues a blocking WorkerLost master RPC\n and pollutes the master\u0027s excluded list inside the retry loop.\n - Random port selection — draw ports below the ephemeral floor (Utils.selectRandomPort() /\n MAX_SELECTABLE_PORT) to avoid a TOCTOU with OS-assigned/TIME_WAIT ports that look free at\n selection time but fail to bind. Consolidated the duplicated picker into a shared\n RandomPortSupport test trait.\n - RatisMasterStatusSystemSuiteJ — re-point each server to a fresh storage directory on every\n start attempt; Ratis releases the directory lock asynchronously, so a retry would otherwise\n hit \"directory is already locked\".\n - JVMQuake / JVMQuakeSuite — extract checkAndDump(...) so the threshold/heap-dump logic can\n be exercised deterministically without inducing real GC pressure.\n - LifecycleManagerUnregisterShuffleSuite — shorten the expired-check interval to 5s so the\n shuffle unregister runs within the eventually() window with retry margin.\n\n### Why are the changes needed?\n\nCI/CD is always failing. With this, CI/CD rarely fails.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nCI/CD\n\nCloses #3737 from afterincomparableyum/flaky-cicd.\n\nAuthored-by: afterincomparableyum \u003c224495379+afterincomparableyum@users.noreply.github.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "b2f1da858fa374b6755b7e50b9fd875b517962a5", "tree": "75d0b9fcfa805dc1c49bb3959e0cfa7385d67ded", "parents": [ "1e715b2cf2936a3704c6a44da463628f7e737f35" ], "author": { "name": "yz", "email": "yz@U-YVQK6W4G-2059.local", "time": "Thu Jul 09 19:59:16 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Thu Jul 09 19:59:16 2026 +0800" }, "message": "[CELEBORN-2377] Support Heartbeat message decoding in C++ client\n\n### What changes were proposed in this pull request?\nThis PR adds Heartbeat message decoding support to the C++ client network.\n\n### Why are the changes needed?\n\nThe HEARTBEAT type (id 22) is already declared in the Message::Type enum and recognized by decodeType, but Message::decodeFrom had no case for it, so any heartbeat frame received by the C++ client hit the unsupported Message decode type failure path. As the protocol/server side begins to send heartbeat messages, the C++ client must be able to decode them instead of throwing.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce any user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nAdded MessageTest.decodeHeartbeat, which constructs a heartbeat wire frame and asserts that Message::decodeFrom returns a non-null message with type HEARTBEAT. Built and ran the test with -- gtest_filter\u003d\"MessageTest.decodeHeartbeat\" inside the jraaaay/celeborn-cpp-dev:0.4 container; it passed.\n\nCloses #3755 from Kalvin2077/main.\n\nAuthored-by: yz \u003cyz@U-YVQK6W4G-2059.local\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "1e715b2cf2936a3704c6a44da463628f7e737f35", "tree": "a2f314d78596321978cf46eab535fe96b0dfc4ed", "parents": [ "a9055aeadc7725e7e14b2de4a5befd043c647365" ], "author": { "name": "Aravind Patnam", "email": "akpatnam25@gmail.com", "time": "Wed Jul 08 13:13:12 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Wed Jul 08 13:13:12 2026 +0800" }, "message": "[CELEBORN-2329][CIP22] Encryption at Rest Spark Impl\n\n### What changes were proposed in this pull request?\n\nAdds EAR support for Spark side.\nSee more details in [doc](https://docs.google.com/document/d/1xBrLtpb8bk8CdJENiM3aLJCbFThRKLor8uAjSsfXoxE/edit?usp\u003dsharing).\n\n### Why are the changes needed?\n\nSee more details in [doc](https://docs.google.com/document/d/1xBrLtpb8bk8CdJENiM3aLJCbFThRKLor8uAjSsfXoxE/edit?usp\u003dsharing).\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes.\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes.\n\n### How was this patch tested?\n\nUnit tests and tested in production internally.\n\nCloses #3689 from akpatnam25/ear-spark-impl.\n\nLead-authored-by: Aravind Patnam \u003cakpatnam25@gmail.com\u003e\nCo-authored-by: Aravind Patnam \u003capatnam@linkedin.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "a9055aeadc7725e7e14b2de4a5befd043c647365", "tree": "687fc9eb2ca03467ef937b535dbfd178a3dd2d0b", "parents": [ "17159eb34ac8132e53e8f19b2c8d38d63d7853cb" ], "author": { "name": "sychen", "email": "sychen@ctrip.com", "time": "Thu Jul 02 16:27:08 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Thu Jul 02 16:27:08 2026 +0800" }, "message": "[CELEBORN-2374] Bump maven version from 3.9.14 to 3.9.16\n\n### What changes were proposed in this pull request?\nhttps://maven.apache.org/docs/3.9.16/release-notes.html\n\n### Why are the changes needed?\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\nGHA\n\nCloses #3752 from cxzl25/CELEBORN-2374.\n\nAuthored-by: sychen \u003csychen@ctrip.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "17159eb34ac8132e53e8f19b2c8d38d63d7853cb", "tree": "dfbb44b130d5ae319c61f8b0d9bf1621e18f1607", "parents": [ "63e28aedd7f0703ffeba438d49ea6f3809160892" ], "author": { "name": "Chao Sun", "email": "chao@openai.com", "time": "Tue Jun 30 11:23:05 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Jun 30 11:23:05 2026 +0800" }, "message": "[CELEBORN-2369] Retry stopped master Outbox failures\n\n## Why are the changes needed?\n\nWhen a Celeborn client loses its cached master connection, the associated Outbox can be stopped before queued or late master RPC messages are sent. Those requests fail with `CelebornException: Message is dropped because Outbox is stopped`.\n\n`MasterClient` currently retries I/O and RPC timeout failures, but it does not recognize this stopped-Outbox failure when it is wrapped by the normal `awaitResult` path. In HA mode, a request can therefore fail instead of clearing the stale `RpcEndpointRef` and reconnecting to another available master.\n\nThe same legacy message is also used when the local `RpcEnv` shuts down permanently. Treating every occurrence as retryable without distinguishing shutdown would cause futile reconnect attempts through an already-stopped environment.\n\nJIRA: https://issues.apache.org/jira/browse/CELEBORN-2369\n\n## What changes were proposed in this PR?\n\n- Recognize the exact legacy stopped-Outbox failure through its cause chain and reset the cached master endpoint so the existing HA retry path can reconnect.\n- Use `RpcEnvStoppedException` for terminal local shutdown so shutdown failures remain non-retryable.\n- Preserve each Outbox\u0027s original stop cause for queued and late messages.\n- Fail messages submitted after `RpcEnv` shutdown immediately with the terminal cause.\n- Add focused coverage for HA reconnection, terminal no-reconnect behavior, and transient versus terminal Outbox causes.\n\n### Does this PR resolve a correctness bug?\n\n- [x] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes. Master RPCs can now fail over after a transient stopped-Outbox failure instead of failing immediately. No public API or configuration changes are introduced.\n\n## How was this PR tested?\n\nUsing JDK 17:\n\n- `build/mvn --no-transfer-progress -DskipTests spotless:apply`\n- `build/mvn --no-transfer-progress -pl common -DskipTests test-compile`\n- `build/mvn --no-transfer-progress -pl common -DargLine\u003d -Dtest\u003dorg.apache.celeborn.common.client.MasterClientSuiteJ surefire:test` — 17 tests passed.\n- `build/mvn --no-transfer-progress -pl common -DargLine\u003d -DwildcardSuites\u003dorg.apache.celeborn.common.rpc.netty.OutboxSuite scalatest:test` — 3 tests passed.\n- `build/mvn --no-transfer-progress -pl common -DargLine\u003d -DwildcardSuites\u003dorg.apache.celeborn.common.rpc.netty.NettyRpcEnvSuite scalatest:test` — 33 tests passed.\n- `build/mvn --no-transfer-progress -DskipTests spotless:check` — all 10 reactor modules passed.\n\nCloses #3744 from sunchao/codex/retry-stopped-master-outbox-oss.\n\nAuthored-by: Chao Sun \u003cchao@openai.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "63e28aedd7f0703ffeba438d49ea6f3809160892", "tree": "dfd450abc3aba76a96bd1d83ff796d24d8a531be", "parents": [ "38f823141cbb794d754185d9d24623535fd63302" ], "author": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Jun 30 10:35:14 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Jun 30 10:35:14 2026 +0800" }, "message": "[CELEBORN-2373] Support Flink 2.3\n\n### What changes were proposed in this pull request?\n\nThis PR adds an Apache Flink 2.3 client for Celeborn:\n\n- Add the `client-flink/flink-2.3` and `client-flink/flink-2.3-shaded` modules.\n- Introduce `CelebornFlinkShimV23` and `tiered/CelebornTierFactory` for Flink 2.3, registered via the `org.apache.celeborn.plugin.flink.CelebornFlinkShim` service file.\n- Adapt the common `RemoteShuffleInputGate` to the Flink 2.3 `IndexedInputGate`/`InputGate` API surface: add `getBufferFilteringCompleteFuture()` and the `setCheckpointingDuringRecoveryEnabled`/`isCheckpointingDuringRecoveryEnabled` hooks, and pass the extra recovered-state argument (`new ArrayDeque\u003c\u003e()`) introduced by the recovery-checkpointing work in Flink 2.3.\n- Port the existing Flink client UT suites to the new module (`RemoteShuffleMasterSuiteJV23`, `RemoteShuffleResultPartitionSuiteJV23`, `RemoteShuffleResultPartitionFactorySuiteJV23`, `RemoteShuffleServiceFactorySuiteJV23`, `ShuffleResourceTrackerSuiteJV23`, `tiered/CelebornTierMasterAgentSuiteJV23`).\n- Wire the new module into the build/release/CI: `pom.xml`, `project/CelebornBuild.scala`, `build/make-distribution.sh`, `build/release/release.sh`, `dev/dependencies.sh`, `dev/reformat`, the `deps`/`license`/`style` workflows, and add the `dev/deps/dependencies-client-flink-2.3` dependency list plus the shaded `LICENSE`/`NOTICE` files.\n- Update docs (`README.md`, `docs/README.md`, `docs/deploy.md`, `docs/developers/overview.md`, `docs/developers/sbt.md`) to include Flink 2.3.\n\n### Why are the changes needed?\n\n[Apache Flink 2.3.0](https://flink.apache.org/2026/06/25/apache-flink-2.3.0-release-announcement/) was released on June 25, 2026. Among other things it adds support for checkpointing during unaligned-checkpoint recovery, which changed the `InputGate` API that the Celeborn Flink client integrates against:\n\n- [FLINK-38543] [apache/flink#27862](https://github.com/apache/flink/pull/27862) — change the overall UC restore process, JM and task initialization (introduces `bufferFilteringCompleteFuture` and the checkpointing-during-recovery transition).\n- [FLINK-38930] [apache/flink#27783](https://github.com/apache/flink/pull/27783) — filter records before processing without a spilling strategy during recovery.\n- [FLINK-39018] [apache/flink#27861](https://github.com/apache/flink/pull/27861) — support checkpoint for `LocalInputChannel`.\n\nCeleborn needs a dedicated Flink 2.3 client so users running on Flink 2.3 can use remote shuffle.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes\n\nA new Flink 2.3 client is available via the `-Pflink-2.3` profile, the release binary now ships a `celeborn-client-flink-2.3-shaded` jar, and the docs list Flink 2.3 as a supported version.\n\n### How was this patch tested?\n\nThe existing Flink client UT suites are ported to the new `flink-2.3` module and run under the `-Pflink-2.3` profile. The `deps`, `license`, and `style` CI workflows are extended to cover `flink-2.3`.\n\nCloses #3751 from SteNicholas/CELEBORN-2373.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "38f823141cbb794d754185d9d24623535fd63302", "tree": "c63b7bac1dacabd4d24d74e8d9488ac183a347af", "parents": [ "a1ecf1926e21590a0c1df816a9e6093b2ebbf69c" ], "author": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Jun 30 10:15:03 2026 +0800" }, "committer": { "name": "Shuang", "email": "lvshuang.xjs@alibaba-inc.com", "time": "Tue Jun 30 10:15:03 2026 +0800" }, "message": "[CELEBORN-2355] Return an empty stream from C++ ShuffleClientImpl::readPartition for an empty partition\n\n### What changes were proposed in this pull request?\n\nThe C++ `ShuffleClientImpl::readPartition` now returns an empty stream when a shuffle has no locations for the requested partition, instead of constructing a `CelebornInputStream` over zero locations. Specifically:\n\n- Add a `CelebornInputStream::empty()` factory that builds a no-op stream whose `read()` immediately yields `-1`.\n- In `readPartition`, when `locations` is empty, log a warning and return `CelebornInputStream::empty()`; the shuffle key is now built only on the non-empty path.\n\nTests:\n- A unit test for `CelebornInputStream::empty()` (immediate EOF, repeated reads, zero-length read).\n- Two `readPartition` tests (empty file groups, missing partition) backed by a `VisibleForTesting` seam that seeds the cached file-group response so no RPC to the `LifecycleManager` is needed.\n\n### Why are the changes needed?\n\nFor a partition with no shuffle data, the previous code constructed a full reader over an empty location list (also building the decompressor when compression is enabled). Returning an empty stream short-circuits this and makes the empty-shuffle case explicit, consistent with the read path\u0027s intent.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nAdded unit tests under `cpp/celeborn/client/tests/`:\n- `CelebornInputStreamTest.emptyStreamReadsReturnEof`\n- `ShuffleClientImplTest.readPartitionEmptyWhenFileGroupsEmpty`\n- `ShuffleClientImplTest.readPartitionEmptyWhenPartitionMissing`\n\nRun with:\n\n```\ncd cpp \u0026\u0026 mkdir -p build \u0026\u0026 cd build \u0026\u0026 cmake .. \u0026\u0026 make celeborn_client_test\n./celeborn/client/tests/celeborn_client_test \\\n --gtest_filter\u003d\u0027CelebornInputStreamTest.*:ShuffleClientImplTest.readPartition*\u0027\n```\n\nCloses #3748 from SteNicholas/CELEBORN-2355.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Shuang \u003clvshuang.xjs@alibaba-inc.com\u003e\n" }, { "commit": "a1ecf1926e21590a0c1df816a9e6093b2ebbf69c", "tree": "402f375826a314df632e39eeebc9a5f0d3452b77", "parents": [ "38202449af6652f2dcb86061c509daa3bedaceac" ], "author": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Sun Jun 28 14:34:52 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Sun Jun 28 14:34:52 2026 +0800" }, "message": "[CELEBORN-2372] Remove Hadoop 2 client support\n\n### What changes were proposed in this pull request?\n\nThis PR removes Hadoop 2 client support from Celeborn. Specifically:\n\n- Remove the `hadoop-2` Maven profile from the root `pom.xml`, which was activated when `hadoop.version` matched `2.*` and pulled in `hadoop-client` with a long list of exclusions.\n- Remove the corresponding `hadoop-2` profiles (activated via the `hadoop-2-deps` property) from `common/pom.xml`, `master/pom.xml`, and `client-mr/mr/pom.xml`.\n- Update `README.md` to reflect that Celeborn supports Hadoop MapReduce 3 (instead of 2/3).\n\n### Why are the changes needed?\n\nHadoop 2 is an old release line that is no longer actively maintained. Dropping it reduces the build/dependency matrix and maintenance burden, allowing the project to focus on Hadoop 3.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes\n\nCeleborn no longer provides the `hadoop-2` build profile. Users building against Hadoop 2 must either upgrade to Hadoop 3 or stay on a previous Celeborn release.\n\n### How was this patch tested?\n\nExisting CI. No new functionality is introduced; the change only removes build profiles and references.\n\nCloses #3747 from SteNicholas/CELEBORN-2372.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "38202449af6652f2dcb86061c509daa3bedaceac", "tree": "45dc1228e687589437bbe06aea0c712ee2d9694e", "parents": [ "a956feea9fee33009cd300f3d24024f4f79688b6" ], "author": { "name": "sychen", "email": "sychen@ctrip.com", "time": "Thu Jun 25 22:25:45 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Thu Jun 25 22:25:45 2026 +0800" }, "message": "[CELEBORN-2368][FOLLOWUP] Remove leftover dependency list files for Spark 2.4\n\n### What changes were proposed in this pull request?\n\nRemove leftover dependency list files for Spark 2.4.\n\n### Why are the changes needed?\n\nRemove leftover dependency list files for Spark 2.4.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nGHA\n\nCloses #3750 from cxzl25/CELEBORN-2368_f.\n\nAuthored-by: sychen \u003csychen@ctrip.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "a956feea9fee33009cd300f3d24024f4f79688b6", "tree": "96f352235f7104f1740a542d5232e613814afc3d", "parents": [ "581a0c0f00a3f050d819ce67344eb24f4a3525f0" ], "author": { "name": "sychen", "email": "sychen@ctrip.com", "time": "Thu Jun 25 22:24:11 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Thu Jun 25 22:24:11 2026 +0800" }, "message": "[CELEBORN-2367][FOLLOWUP] Remove leftover dependency list files for Flink 1.16 and 1.17\n\n### What changes were proposed in this pull request?\n\nRemove leftover dependency list files for Flink 1.16 and 1.17.\n\n### Why are the changes needed?\n\nRemove leftover dependency list files for Flink 1.16 and 1.17.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nGHA\n\nCloses #3749 from cxzl25/CELEBORN-2367_f.\n\nAuthored-by: sychen \u003csychen@ctrip.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "581a0c0f00a3f050d819ce67344eb24f4a3525f0", "tree": "16ec15b6df64cd00058d262ea29ded26119f4964", "parents": [ "5f49ed90f7fefcde81876ea056241dd9340b7788" ], "author": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Thu Jun 25 12:46:50 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Thu Jun 25 12:46:50 2026 +0800" }, "message": "[CELEBORN-2368] Remove Spark 2.4 client support\n\n### What changes were proposed in this pull request?\n\nThis PR removes Spark 2.4 client support from Celeborn. Specifically:\n\n- Delete the `client-spark/spark-2` and `client-spark/spark-2-shaded` modules, including the shuffle manager/reader/writer implementations (`SparkShuffleManager`, `CelebornShuffleReader`, `HashBasedShuffleWriter`, `SortBasedShuffleWriter`), `SparkUtils`, the fallback policy runner, shuffle handle, license/notice resources, and test suites.\n- Remove the `spark-2.4` Maven profile from `pom.xml` (and the related `tests/spark-it` profile) and the `Spark24` SBT project from `project/CelebornBuild.scala`.\n- Drop `spark-2.4` from the build and release scripts (`build/make-distribution.sh`, `build/release/release.sh`, `dev/dependencies.sh`, `dev/reformat`) and from CI workflows (`deps.yml`, `license.yml`, `maven.yml`, `sbt.yml`, `style.yml`).\n- Update documentation (`README.md`, `docs/developers/sbt.md`) to reflect the supported Spark versions (3.0/3.1/3.2/3.3/3.4/3.5/4.0).\n\n### Why are the changes needed?\n\nSpark 2.4 is an old release that is no longer maintained upstream. Dropping it reduces the build/test matrix, maintenance burden, and release artifacts, allowing the project to focus on currently supported Spark versions (3.0 and above).\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes\n\nCeleborn no longer ships or builds the Spark 2.4 client jar. Users running Spark 2.4 must either upgrade to a supported Spark version (3.0+) or stay on a previous Celeborn release.\n\n### How was this patch tested?\n\nExisting CI for the remaining Spark versions (3.0/3.1/3.2/3.3/3.4/3.5/4.0) via the updated Maven/SBT/style/license/deps workflows. No new functionality is introduced; the change only removes code, build profiles, and references.\n\nCloses #3743 from SteNicholas/CELEBORN-2368.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "5f49ed90f7fefcde81876ea056241dd9340b7788", "tree": "d5cc48744c59fc29fa40381ef3cd5cabce720037", "parents": [ "441d6b45a5b5e71310c90ef6099626f97c99ef47" ], "author": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Jun 23 20:44:31 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Jun 23 20:48:51 2026 +0800" }, "message": "[CELEBORN-2367] Remove Flink 1.16 and 1.17 client support\n\n### What changes were proposed in this pull request?\n\nThis PR removes Flink 1.16 and Flink 1.17 client support from Celeborn. Specifically:\n\n- Delete the `client-flink/flink-1.16`, `client-flink/flink-1.16-shaded`, `client-flink/flink-1.17`, and `client-flink/flink-1.17-shaded` modules, including their shims (`CelebornFlinkShimV116`/`CelebornFlinkShimV117`), service descriptors, license/notice resources, and test suites.\n- Remove the `flink-1.16` and `flink-1.17` Maven profiles from `pom.xml` and the `Flink116`/`Flink117` SBT projects from `project/CelebornBuild.scala`.\n- Drop `flink-1.16`/`flink-1.17` from the build and release scripts (`build/make-distribution.sh`, `build/release/release.sh`, `dev/dependencies.sh`, `dev/reformat`) and from CI workflows (`deps.yml`, `license.yml`, `maven.yml`, `sbt.yml`, `style.yml`).\n- Remove the `v1_16`/`v1_17` entries from the `FlinkVersion` test enum.\n- Update documentation (`README.md`, `docs/README.md`, `docs/deploy.md`, `docs/developers/overview.md`, `docs/developers/sbt.md`) to reflect the supported Flink versions (1.18/1.19/1.20/2.0/2.1/2.2).\n\n### Why are the changes needed?\n\nFlink 1.16 and 1.17 are older releases that are no longer maintained upstream. Dropping them reduces the build/test matrix, maintenance burden, and release artifacts, allowing the project to focus on currently supported Flink versions (1.18 and above).\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [x] Yes\n\nCeleborn no longer ships or builds Flink 1.16/1.17 client jars. Users running Flink 1.16 or 1.17 must either upgrade to a supported Flink version (1.18+) or stay on a previous Celeborn release.\n\n### How was this patch tested?\n\nExisting CI for the remaining Flink versions (1.18/1.19/1.20/2.0/2.1/2.2) via the updated Maven/SBT/style/license/deps workflows. No new functionality is introduced; the change only removes code, build profiles, and references.\n\nCloses #3742 from SteNicholas/CELEBORN-2367.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "441d6b45a5b5e71310c90ef6099626f97c99ef47", "tree": "ab404d940df3939b0af297c1704992ba62f038f0", "parents": [ "8903a31d81dedf3b9bd52e5773cbb98220d6b14d" ], "author": { "name": "Zemin Piao", "email": "pzm6391@gmail.com", "time": "Tue Jun 23 17:38:30 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Jun 23 17:38:30 2026 +0800" }, "message": "[CELEBORN-2354] Fix Helm chart indentation, PodMonitor flags and cluster-scoped PriorityClass namespace\n\n### What changes were proposed in this pull request?\n\nFix helm chart indentation, podmonitor flags and remove the namespace option for priorityClass.\n\n### Why are the changes needed?\n\nCleanup the helm chart .\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nNo.\n\nCloses #3723 from zemin-piao/fix_helm_chart_indentation_podmonitor_flags_priorityclass.\n\nAuthored-by: Zemin Piao \u003cpzm6391@gmail.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "8903a31d81dedf3b9bd52e5773cbb98220d6b14d", "tree": "c0001b55ce53d3da1941b96e55a7e016dd8f39e5", "parents": [ "0198bcf53a602f585df3a72cc1e5f147dff419a1" ], "author": { "name": "afterincomparableyum", "email": "224495379+afterincomparableyum@users.noreply.github.com", "time": "Tue Jun 23 10:56:50 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Jun 23 10:56:50 2026 +0800" }, "message": "[CELEBORN-2360] Fix ReviseLostShuffles RPC deserialization\n\n### What changes were proposed in this pull request?\n\nCELEBORN-1601 added ReviseLostShuffles support but only wired up the encode side (toTransportMessage), the master handler, and the client sender. The corresponding decode cases in ControlMessages.fromTransportMessage were never added. Since that match has no default case, receiving a REVISE_LOST_SHUFFLES (89) request on the master or a REVISE_LOST_SHUFFLES_RESPONSE (90) on the client throws a MatchError, so the feature is broken over RPC. The existing test exercises handleReviseLostShuffles directly and bypasses the RPC round-trip, which is why this was probably not caught.\n\nTo fix, I added the missing REVISE_LOST_SHUFFLES_VALUE and REVISE_LOST_SHUFFLES_RESPONSE_VALUE cases so the messages deserialize into PbReviseLostShuffles / PbReviseLostShufflesResponse.\n\nI also added a round trip test in UtilsSuite covering both messages.\n\n### Why are the changes needed?\n\nTo fix an existing bug.\n\n### Does this PR resolve a correctness bug?\n\n- [x] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nCI/CD\n\nCloses #3735 from afterincomparableyum/CELEBORN-2360.\n\nAuthored-by: afterincomparableyum \u003c224495379+afterincomparableyum@users.noreply.github.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "0198bcf53a602f585df3a72cc1e5f147dff419a1", "tree": "a680389c17823de8300db252a387dcf3ea8d7fc3", "parents": [ "d17489bc6310616a1cbd614cdba49ef10c84307d" ], "author": { "name": "AmandeepSingh285", "email": "mailto.amandeep.singh.28@gmail.com", "time": "Mon Jun 22 10:48:46 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Mon Jun 22 10:48:46 2026 +0800" }, "message": "[CELEBORN-2334] Automatically restore RocksDB in case of failures\n\n### What changes were proposed in this pull request?\n\nThe patch re-instantiates RocksDB in case of failures. In the current implementation, when RocksDB enters a read-only mode due to failures, Celeborn metadata operations fail and remain blocked until manual intervention or restart. This pull request adds logic to detect such RocksDB failures and re-instantiate the RocksDB instance so that metadata operations can recover automatically and continue functioning without prolonged disruption. RocksDB can enter a read-only or unusable state under scenarios such as: corruption in files, errors from underlying file system. In such cases, RocksDB prevents further writes to protect data consistency, which causes Celeborn metadata operations to fail.\n\n### Why are the changes needed?\n\nOnce RocksDB enters a read-only or error state, Celeborn metadata operations become unavailable because the existing RocksDB instance remains unusable, which could lead to failures in metadata updates.\n\n### Does this PR resolve a correctness bug?\n\nNo.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nUnit tests.\n\nCloses #3695 from AmandeepSingh285/auto-recover-rocks-db.\n\nLead-authored-by: AmandeepSingh285 \u003cmailto.amandeep.singh.28@gmail.com\u003e\nCo-authored-by: amandeeps.28 \u003camandeeps.28@uber.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "d17489bc6310616a1cbd614cdba49ef10c84307d", "tree": "32f41a04fb5e3ca7d9b1214b16145c0d88564483", "parents": [ "3a11d1e11950314c3692b84996af2e76b261b971" ], "author": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Jun 16 21:46:20 2026 +0800" }, "committer": { "name": "Shuang", "email": "lvshuang.xjs@alibaba-inc.com", "time": "Tue Jun 16 21:46:20 2026 +0800" }, "message": "[CELEBORN-2353] Align C++ shuffle client push/merge data path with Java ShuffleClient\n\n### What changes were proposed in this pull request?\n\nThis PR aligns the C++ shuffle client\u0027s push/merge data path with the Java `ShuffleClient`:\n\n- Refactor `pushData`/`mergeData`/`pushMergedData` to share one push prologue (`prepareBatch`) and a uniform synchronous-failure route (`pushWithFailureRouting`), and move throttling (`limitMaxInFlight`) to the caller before the initial push so the retry path never re-throttles.\n- Exclude push workers on connection/timeout failures (`celeborn.client.push.excludeWorkerOnFailure.enabled`), routing pushes away from them until a successful revive or re-assignment lifts the exclusion, and clear it wholesale on shutdown.\n- Track push-failed batches and report them at `MapperEnd`; the reader dedups duplicate batches for the adaptive skewed-partition read optimization.\n- Derive specific failure causes from transport error messages (`getPushDataFailCause`) and propagate them through revive/retry; add `StatusCode::toString`.\n\n### Why are the changes needed?\n\nThe C++ shuffle client lacked the push-side resiliency the Java `ShuffleClient` already provides: excluding failed push workers and tracking data-push failures so the reader can dedup duplicate batches for the adaptive skewed-partition read optimization. Bringing the C++ push/merge data path to parity keeps the native and JVM clients behaviorally consistent and honors the existing `celeborn.client.push.excludeWorkerOnFailure.enabled` / adaptive skewed-partition read configs.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nCovered by new/extended C++ unit tests:\n\n- `PushDataCallbackTest`, `PushMergedDataCallbackTest` — push / merged-push callback failure routing and cause propagation.\n- `ShuffleClientImplTest` — push worker exclusion and data-push-failure tracking.\n- `CelebornInputStreamRetryTest` — reader-side dedup of duplicate push-failed batches.\n- `PushStateTest`, `ControlMessagesTest` — failed-batch state and `MapperEnd` serialization round-trip.\n\nCloses #3724 from SteNicholas/CELEBORN-2353.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Shuang \u003clvshuang.xjs@alibaba-inc.com\u003e\n" }, { "commit": "3a11d1e11950314c3692b84996af2e76b261b971", "tree": "1b59fba93f31b6a4b00b9b9e8d56eb3d4ceb634a", "parents": [ "3eee443fe6e4f66a878150b19bc1a4f71d61e238" ], "author": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Jun 16 10:41:53 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Tue Jun 16 10:41:53 2026 +0800" }, "message": "[CELEBORN-2365] Bump Spark version from 4.0.1 to 4.0.3\n\n### What changes were proposed in this pull request?\n\nBump the Spark 4.0 dependency version from `4.0.1` to `4.0.3` in both the Maven build (`pom.xml`) and the SBT build (`project/CelebornBuild.scala`).\n\n### Why are the changes needed?\n\n[Apache Spark 4.0.3](https://spark.apache.org/news/spark-4-0-3-released.html) is a maintenance release containing security and correctness fixes, and the Spark community strongly recommends all 4.0 users upgrade to this stable release. See the [release notes](https://spark.apache.org/releases/spark-release-4-0-3.html) for details.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nPass GitHub Actions.\n\nCloses #3741 from SteNicholas/CELEBORN-2365.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" }, { "commit": "3eee443fe6e4f66a878150b19bc1a4f71d61e238", "tree": "ce74f51bc739d072ecf889c6187597adde319aed", "parents": [ "8c3108ddbfe0e260af6115fd087fc0a5b981704a" ], "author": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Mon Jun 15 16:24:41 2026 +0800" }, "committer": { "name": "Nicholas Jiang", "email": "programgeek@163.com", "time": "Mon Jun 15 16:24:41 2026 +0800" }, "message": "[CELEBORN-2363] Bump Flink version to 1.20.5, 2.0.2 and 2.1.3\n\n### What changes were proposed in this pull request?\n\nBump the supported Flink versions to their latest patch releases:\n\n- Flink 1.20: `1.20.3` → `1.20.5` (and the default `flink.version` property `1.20.1` → `1.20.5`)\n- Flink 2.0: `2.0.1` → `2.0.2`\n- Flink 2.1: `2.1.1` → `2.1.3`\n\nUpdated in both `pom.xml` and `project/CelebornBuild.scala`.\n\n### Why are the changes needed?\n\nKeep the Flink clients aligned with the latest patch releases, which include bug fixes and security fixes from upstream Flink.\n\n### Does this PR resolve a correctness bug?\n\n- [ ] Yes\n\n### Does this PR introduce _any_ user-facing change?\n\n- [ ] Yes\n\n### How was this patch tested?\n\nExisting CI (Maven and SBT builds plus the Flink integration tests `tests/flink-it`).\n\nCloses #3738 from SteNicholas/CELEBORN-2363.\n\nAuthored-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\nSigned-off-by: Nicholas Jiang \u003cprogramgeek@163.com\u003e\n" } ], "next": "8c3108ddbfe0e260af6115fd087fc0a5b981704a" }