)]}' { "log": [ { "commit": "39e9dc5b209e0f77e3ffe2a509087a2bc8dd59ae", "tree": "0c02c3a518c36678f342dbab882d53dabba3457e", "parents": [ "d56e6747f43c7388d0cdd6e27ef5afab82de360e" ], "author": { "name": "Danny Chan", "email": "yuzhao.cyz@gmail.com", "time": "Tue Jul 28 13:31:57 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 28 13:31:57 2026 +0800" }, "message": "docs(flink): update default profile in README (#19386)" }, { "commit": "d56e6747f43c7388d0cdd6e27ef5afab82de360e", "tree": "e53e9a8a4c6acca6d53715535bb0509fd166ce53", "parents": [ "0774d91f893fc3a123d944fa9e7c6cd828adcca1" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Tue Jul 28 09:55:08 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 28 09:55:08 2026 +0800" }, "message": "feat(flink): enable LSM reader for Flink metadata table (#19373)\n\n* feat(flink): enable LSM reader for Flink metadata table" }, { "commit": "0774d91f893fc3a123d944fa9e7c6cd828adcca1", "tree": "a4d11dff6d0f639471bd7333b80a9527d332a931", "parents": [ "4a5d5b0e0410b1e2313fb7ad42b049bc6e36e417" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Mon Jul 27 07:24:10 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 27 14:24:10 2026 +0000" }, "message": "chore(ci): modernize Codecov config for the coverage initiative (#19138)\n\n* chore(ci): modernize Codecov config for the coverage initiative\n\nRefresh .codecov.yml so per-PR coverage is accurate and actionable:\n\n- ignore: drop stale entries pointing at pre-2021 paths and files that no\n longer exist; ignore non-production code that Codecov still counts today\n (hudi-examples, packaging, the hudi-integ-test harness, the vendored\n HoodieHadoopFSUtils, and the incubating hudi-platform-service). Generated\n code (Avro/Thrift/Protobuf/ANTLR) has no source in git and is already\n absent from the report, so it needs no exclusion.\n- flag_management: carry a flag\u0027s coverage forward when its CI job is skipped\n by the path filter, so a partial run does not report a false drop. Replaces\n the old flags block whose names no longer matched the uploads.\n- component_management: report coverage per ownership area so each coverage\n subtask can read its own number on every PR.\n- comment: post a per-PR summary (project, components, patch); previously off.\n- status: project and patch statuses are informational for now (report, do\n not block); can be flipped to enforcing per component as coverage climbs.\n\nAlso document the Codecov per-PR flow in scripts/jacoco/README.md.\n\n* Restore per-class exclusions, fix docs, trigger CI\n\n- .codecov.yml: restore the individual Java-class ignores from the prior config\n for the classes that still exist (standalone main()-style tools and legacy JSON\n helpers), so their exclusion is unchanged; entries for since-deleted classes and\n stale paths are left dropped.\n- scripts/jacoco/README.md: clarify that the committed Avro model path is still\n ignored defensively, so the doc and config agree.\n- Base64CodecUtil: temporary no-op comment to trigger CI/Codecov upload (DNM, revert before merge).\n\n* [MINOR] Address review: revert DNM comment, drop stale ignore entry\n\n- Revert the temporary DNM comment in Base64CodecUtil (was only to trigger CI).\n- Drop the ignore entry for MercifulJsonConverter: the reorg moved it to\n org.apache.hudi.common.avro and it now has TestMercifulJsonConverter, so let\n it count toward coverage.\n- Document that modules whose CI jobs do not upload a JaCoCo report\n (e.g. hudi-trino-plugin) are intentionally absent from ignore/components.\n- README nits: drop redundant -Djacoco.skip\u003dfalse from the local example and\n add the missing trailing newline." }, { "commit": "4a5d5b0e0410b1e2313fb7ad42b049bc6e36e417", "tree": "54387177b219a4a5860a8c349e0ec09700b6d90d", "parents": [ "483dd878b4ef1b3df1c6135b5daae3fd0274c892" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Mon Jul 27 20:31:32 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 27 13:31:32 2026 +0000" }, "message": "fix(hive-sync): drop the unreachable HMS lock timeout-recovery path (#19371)\n\n* fix(hive-sync): drop the unreachable HMS lock timeout-recovery path\n\n* addressed review comments: stub checkLock and pin the abandoned late grant\n\n---------\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e" }, { "commit": "483dd878b4ef1b3df1c6135b5daae3fd0274c892", "tree": "f35b48514d1790386188fa232a6abbf117421ad7", "parents": [ "c3c9367907273aeae19ec072871bb9a99dc7e3d8" ], "author": { "name": "Sivabalan Narayanan", "email": "n.siva.b@gmail.com", "time": "Mon Jul 27 01:35:41 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 27 16:35:41 2026 +0800" }, "message": "feat(hive-sync): batch and parallelize HiveQL partition operations (#18984)\n\n* feat(hive-sync): batch and parallelize HiveQL partition operations\n\nAdds an opt-in pool that splits HiveQL partition DDL (add/update/touch/drop)\ninto batches of `hoodie.datasource.hive_sync.batch_num` and dispatches them\nin parallel across a pool of single-thread workers. Each worker owns its\nown Hive `Driver` + `SessionState` (both thread-bound in Hive 2.x), so the\nfan-out is implemented as a fixed pool of dedicated single-thread executors\nrather than a shared thread pool.\n\nTable-level operations (create/alter table, last commit time, writer version)\ncontinue to use the single session `Driver`. Partition-phase SQL lists run\nthrough the pool only when `hoodie.datasource.hive_sync.batching.enabled`\nis set to true; default off, existing behavior unchanged.\n\nHive 2.x\u0027s `ALTER PARTITION SET LOCATION` ignores db.table qualifiers and\nuses the connection\u0027s current database, so each worker is primed with the\ncorrect USE statement before any partition ALTER is dispatched.\n\n* fix(hive-sync): address review feedback on HiveQL partition batching\n\n- Gate TOUCH batch-splitting on hoodie.datasource.hive_sync.batching.enabled\n so the default (off) HiveQL path emits a single ALTER TABLE ... TOUCH\n statement as before, instead of always splitting into batch_num chunks.\n- Give each HiveDriverPool worker its own exclusively-owned SessionState\n instead of sharing one SessionState object across all workers. Concurrent\n Driver.run() calls against a shared session risked corrupting session-\n scoped state (current db, scratch dirs, txn/lock manager), and closing\n the shared session once per worker on teardown closed it multiple times\n while racing worker Driver.close() calls. Workers now bootstrap one at a\n time (each SessionState construction no longer races another).\n- Close the driver pool in HiveQueryDDLExecutor\u0027s constructor failure path,\n so a failed SessionState/Driver bootstrap doesn\u0027t leak the pool\u0027s worker\n threads, Drivers, and sessions (the pool is constructed by the caller\n before this constructor runs, and no one else can close it if we throw).\n- Fix testHiveQLTouchPartitionsWithBatching to drive touchPartitionsToTable\n directly; the previous resync-based version never reached the batched\n TOUCH path because incremental sync short-circuits with no new commit.\n- Make TestHiveDriverPool\u0027s cancel-on-first-error assertion race-free by\n parking pending tasks on a latch and asserting isCancelled() || !isDone().\n\n* fix(hive-sync): isolate per-worker HiveConf; document ADD parallel dispatch\n\n- HiveDriverPool.DefaultDriverFactory now builds a per-worker HiveConf copy\n (new HiveConf(hiveConf)) instead of sharing one HiveConf instance across\n all workers, and passes that copy to both SessionState and Driver. Hive\u0027s\n QueryState/Driver mutate per-query keys (e.g. HIVEQUERYID) on the conf\n during run(), so a shared HiveConf let concurrent Driver.run() calls\n overwrite each other\u0027s query-scoped configuration even though each\n worker already had its own SessionState object.\n- Reworded HIVE_SYNC_BATCHING_ENABLED\u0027s doc to explicitly include ADD in\n the parallel dispatch scope. addPartitionsToTable routes through\n runSQLs, so ADD batches are dispatched across the pool same as TOUCH/\n SET_LOCATION when the flag is on; only the batch size (not the fan-out)\n was already unchanged before this flag existed.\n\n* fix(hive-sync): abort dispatch on first error; scope TOUCH batching to parallel paths\n\nTwo review fixes on the HiveQL partition batching path.\n\n1. awaitAll did not actually cancel pending work on first error.\n\nThe futures returned by dispatchAll belong to N independent single-thread\nexecutors, each draining its own queue. awaitAll blocked on Future.get() in\nsubmission order, so a failure on a fast worker went unobserved while the\nawaiting thread was parked on a slow worker\u0027s earlier future -- and the failed\nworker kept pulling and applying more partition DDL from its own queue. The\nadvertised \"cancel pending futures on first error\" behavior did not hold.\n\ndispatchAll now returns a Dispatch handle carrying a shared abort flag. Each\ntask checks the flag on entry and bails with CancellationException without\ntouching its Driver; the first task to fail sets it. awaitAll blocks on a latch\nthat trips on either all-settled or first-abort, then sweeps cancel(false)\nbefore walking the futures. Cancelling from the awaiting thread is inherently\nlate here, so the in-task check is what bounds how much extra DDL a failed sync\ncan apply. mayInterruptIfRunning\u003dfalse is preserved, so in-flight statements\nstill run to completion rather than leaving a Driver mid-statement.\n\n2. hoodie.datasource.hive_sync.batching.enabled leaked into JDBC mode.\n\nQueryBasedDDLExecutor is also the base class for JDBCExecutor, which does not\noverride runSQLs. With the flag on in JDBC mode, TOUCH was split into batch_num\nstatements that then executed serially -- changing statement count and\npartial-application semantics for no benefit, and contradicting the documented\n\"JDBC is unaffected\" contract.\n\nThe config read in constructPartitionAlterStatements is replaced by a\ngetTouchBatchSize(int) hook. The base implementation returns the full partition\ncount (one statement, the long-standing behavior); only HiveQueryDDLExecutor\noverrides it, and only when a driver pool is actually present. Keying on pool\npresence rather than on the config means the split can never take effect on a\npath that would just execute the batches serially.\n\nTests:\n- awaitAllStopsLaterWorkerWhenEarlierFutureIsSlow: pins the interleaving the bug\n needs (slow statement on worker 0, fast failure on worker 1) and asserts the\n statement queued behind the failure never reaches a Driver.\n- awaitAllCancelsPendingFuturesOnFirstError: rewritten. The prior version\n asserted isCancelled() || !isDone() to hedge around the race; the abort flag\n makes the outcome deterministic, so it now asserts on what actually executed.\n- New TestQueryBasedDDLExecutorTouchBatching: asserts a serial executor emits a\n single TOUCH statement with the flag on, that its SQL is byte-identical with\n the flag on and off, and that a parallel-dispatch executor still splits.\n\nBoth new pool tests were verified to fail on all surefire attempts against the\nprior logic and pass against the fix. Full hudi-hive-sync suite: 303/303.\n\n* fix(hive-sync): close driver pool if sync client construction fails\n\nHiveQueryDDLExecutor\u0027s own catch closes the pool, but it only covers throws from\ninside its try block. QueryBasedDDLExecutor\u0027s super(config) runs the\nPartitionValueExtractor reflection first, so a bad\nhoodie.datasource.hive_sync.partition_extractor_class throws before that try is\nentered -- the executor\u0027s catch never runs, HoodieHiveSyncClient\u0027s catch just\nrethrows, and the already-constructed pool leaks its worker threads and Drivers.\n\nClose the pool in HoodieHiveSyncClient\u0027s constructor catch, which covers every\nwindow between building the pool and handing ownership to the executor. close()\nis idempotent, so overlapping with the executor\u0027s own cleanup is harmless." }, { "commit": "c3c9367907273aeae19ec072871bb9a99dc7e3d8", "tree": "b80ea0e2c851ad6b929142bb632f28855e5a8b64", "parents": [ "42e938dbd2874086c07234faffe278b3be3354c7" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Mon Jul 27 00:02:04 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 27 00:02:04 2026 +0800" }, "message": "feat(trino): Migrate the Trino-Hudi connector into the Hudi repo (RFC-105) (#18837)\n\n* feat(trino): Migrate the Trino-Hudi connector into the Hudi repo (RFC-105)\n\nSquash of the 55-commit review history. The connector implementation moves\ninto the Hudi repo as the hudi-trino module (published as\norg.apache.hudi:hudi-trino), targeting Trino SPI 481 on JDK 25, behind the\nopt-in hudi-trino Maven profile. The Trino repo keeps only a thin trino-hudi\nshim depending on the published artifact. Removes hudi-trino-bundle and the\ntrino{base,coordinator,worker} docker modules.\n\nCo-authored-by: vamsikarnika \u003cvamsikarnika@gmail.com\u003e\nCo-authored-by: Aditya Goenka \u003cad1happy2go@gmail.com\u003e\n\n* fix(trino): read RFC-103 native parquet log-block footers via Trino\u0027s parquet reader\n\nMOR tables written at table version 10 store delta-log blocks as native parquet\nfiles (RFC-103). Reading one calls\nHoodieIOFactory.getFileFormatUtils(PARQUET).readFooter(...) to pull the log-block\nheader out of the parquet footer, but HudiTrinoIOFactory stubbed getFileFormatUtils\nout, so any realtime read of a v10 MOR table failed with \"FileFormatUtils not\nsupported in HudiTrinoIOFactory\" (surfaced by TestHudiCustomMerger and\nTestHudiCustomMergerEndToEnd, which write their tables at the default version).\n\nAdd HudiTrinoParquetFileFormatUtils, implementing readFooter with Trino\u0027s own\nparquet reader (MetadataReader.readFooter -\u003e FileMetadata.getKeyValueMetaData),\nmirroring ParquetUtils.readFooter semantics, and return it from\nHudiTrinoIOFactory.getFileFormatUtils for PARQUET; non-parquet native log formats\nnow throw a clear message. Record decoding already flows through Trino\u0027s\npage-source reader, so the footer read was the only missing piece. No new\ndependency is added (hudi-hadoop-common stays excluded).\n\n* fix(trino): read native parquet delta-log records for MOR merge\n\nThe reader context ignored its path/schema arguments and re-served the\npre-built base page source, so RFC-103 native (parquet) log records were\nnever read and v10 MOR reads returned base-only values. Build a Trino\nparquet page source on demand for native log files, projected on the\nfile-group reader\u0027s requiredSchema with predicate pushdown off, via a\nfactory injected from HudiPageSourceProvider.\n\nSynthesize VARCHAR handles for Hudi meta columns the file-group reader\nadds to requiredSchema beyond the connector projection (e.g.\n_hoodie_commit_time); all meta columns are UTF8 strings on disk and\nparquet columns are resolved by name, so the placeholder ordinal is\nnever read (refactor tracked by apache/hudi#19249).\n\nCustom mergers must be projection compatible: a non-projection-compatible\nmerger makes the file-group reader demand the full table schema, whose\nunprojected data columns cannot be typed at this layer; fail such reads\nwith a clear NOT_SUPPORTED error and declare isProjectionCompatible() on\nthe test mergers, which merge on their declared mandatory fields only.\n\nAlso adapt the TestHudiSmokeTest footer-read call site to the\nparameterized createPageSource helper.\n\n* fix(trino): keep the inherited Lombok annotation processor off the JDK 25 compile\n\nMaster commit b0a2563e25e5 declares Lombok in the root pom\u0027s\nannotationProcessorPaths (a parallel-build deflake). hudi-trino inherits\nthat path but compiles on JDK 25, where Lombok 1.18.36 crashes javac with\nExceptionInInitializerError: com.sun.tools.javac.code.TypeTag :: UNKNOWN --\nthis broke the hudi-trino CI on every PR merge ref once that commit landed.\nhudi-trino uses no annotation processors, so set proc\u003dnone and override the\ninherited processor path away.\n\n* fix(ci): run trino workflow on every PR so the required check reports\n\ntest-hudi-trino-plugin is a required status check in .asf.yaml. Gating the\nworkflow with on.pull_request.paths meant it was never instantiated on PRs\nthat do not touch hudi-trino, so the required context stayed pending and\nblocked merges.\n\nDrop the paths filter on pull_request and add a detect-trino-changes job\nthat guards the expensive build/test steps with an if:, mirroring what\nbot.yml does via needs.changes. The push trigger keeps its paths filter.\n\n* refactor(trino): rename orderingColumns to mergeRequiredColumns\n\nThe field is populated by getMergeRequiredColumnHandles, which for a CUSTOM\nmerge mode also contributes the merger\u0027s mandatory fields on top of the\nordering fields. The old name understated what it holds.\n\nRenames lazyOrderingColumns, getOrderingColumns, the orderingColumns JSON\nproperty, and HudiUtil.prependHudiMetaAndOrderingColumns, and documents the\ngetter.\n\n* test(trino): cover the non-projection-compatible merger guard\n\nThe NOT_SUPPORTED branch in HudiTrinoReaderContext.buildRequiredColumnHandles\nhad no coverage because both test mergers report isProjectionCompatible()\u003dtrue.\n\nAdds NonProjectionCompatibleTestRecordMerger (KeyBasedTestRecordMerger with\nthe flag flipped, so it resolves against the same test table) and a negative\ntest in TestHudiCustomMerger that overrides the merger per session and asserts\nthe guard fires. Exposes record_merger_impls via SessionBuilder so the case\nreuses the existing query runner instead of a new test class.\n\n* fix(ci): make the nightly Trino SPI drift check actually test master\n\nThree problems with the nightly compat job:\n\n- The version parse took the first \u003cversion\u003e in trinodb/trino\u0027s root pom,\n which belongs to the \u003cparent\u003e (io.airlift:airbase). It yielded 395 instead\n of Trino\u0027s own 484-SNAPSHOT. Ask Maven for project.version instead.\n- The trino checkout was never built, so trino artifacts resolved from Maven\n Central. Combined with stripping -SNAPSHOT, the job compiled against a\n stale release rather than master. Install the modules hudi-trino needs\n from the master checkout and keep the SNAPSHOT version.\n- The failure handler filed a new issue on every run, so persistent drift\n would open an issue a night. Comment on the existing report instead,\n found via a marker in the body.\n\nAlso declares the issues: write permission the handler needs.\n\n* chore(trino): add ASF license headers to hudi-trino markdown files\n\nRAT flagged 17 unapproved files: hudi-trino/README.md and the 16\nhudi-testing-data table docs. The repo has no *.md exclusion in the\napache-rat config, and every other authored markdown carries the header\nas an HTML comment, so add it rather than widen the excludes.\n\n* fix(trino): surface wrapped TrinoException as the query failure\n\nHudi\u0027s log scanning wraps connector errors in a generic\nHoodieException (\"Exception when reading log file\"), so the\nactionable message (e.g. the non-projection-compatible merger\nguard) was buried in the cause chain. In HudiPageSource\u0027s\nconstructor, rethrow a TrinoException found in the cause chain so\nits error code and message surface as the query failure. Tighten\nthe negative merger test to assert the top-level message.\n\n* fix(ci): gate the Trino check on the connector\u0027s upstream modules\n\nThe detect-trino-changes filter matched only hudi-trino/*, so a PR\ntouching hudi-common (or any other module the connector build\ninstalls) reported the required test-hudi-trino-plugin check green\nwithout compiling the connector. Cover the full -am closure of the\nupstream install step plus the root and hudi-client poms, in both\nthe PR filter and the push paths. The full job is about 12 minutes\n(the from-source Trino install is only about 4), well inside the\nAzure CI window such PRs already wait on.\n\n* fix(ci): only file the SPI drift issue on a connector compile failure\n\nThe failure() guard on the issue-filing step is job-scoped, so a\nfailed checkout or a broken trinodb/trino master build also filed\n\"SPI drift detected\", with an empty version in the title when the\nfailure was before the Read Trino version step. Guard on the\ncompile step\u0027s outcome instead.\n\n* fix(build): revert stray maven-shade-plugin bump to 3.6.2\n\nLeftover from when this branch still carried the legacy\npackaging/hudi-trino-bundle, which the migration commit deleted.\nhudi-trino is non-shaded and no remaining module shades JDK 25\nbytecode, so restore 3.5.3 and keep the MSHADE-461 comment\naccurate; a bump can go in its own PR.\n\n* docs(trino): correct the README claim about CI and hudi-trino-tests\n\nCI does enable the hudi-trino-tests profile; it installs the Trino\ntest-jars from a source checkout of the pinned tag first.\n\n---------\n\nCo-authored-by: vamsikarnika \u003cvamsikarnika@gmail.com\u003e\nCo-authored-by: Aditya Goenka \u003cad1happy2go@gmail.com\u003e" }, { "commit": "42e938dbd2874086c07234faffe278b3be3354c7", "tree": "77b997809ae83273f5b58166d104871735c9b7f5", "parents": [ "26b1ebc4e9384d15c5599a4d97d522f781063b35" ], "author": { "name": "Lin Liu", "email": "141371752+linliu-code@users.noreply.github.com", "time": "Sun Jul 26 03:50:50 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Jul 26 10:50:50 2026 +0000" }, "message": "fix: Fix lock expiration metric (#18492)\n\n* Fix metric\n\n* Address review: read the clock once when reporting the renewed lock deadline\n\nThe SUCCESS branch of renewLock called getCurrentEpochMs() three times, so the\nvalue fed to updateLockExpirationDeadlineMetric and the two values in the log\nline could each be computed against a different clock read. Capture the clock\nonce and derive both the metric and the log arguments from it.\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "26b1ebc4e9384d15c5599a4d97d522f781063b35", "tree": "3f4a89ed9bd3fb137e917130f37216fd33306135", "parents": [ "48fe2c6bd0a44fa1ef5693aa7f7460c32f0cfd61" ], "author": { "name": "Nada", "email": "nada.attia@uber.com", "time": "Fri Jul 24 10:10:19 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 24 14:10:19 2026 +0000" }, "message": "fix(record-index-bootstrap): sort record index keys by UTF-8 bytes to match HFile sorting (#18941)\n\n* fix(HUDI-8898): sort record index keys by UTF-8 bytes to match HFile ordering\n\n* test(HUDI-8898): add record index bootstrap test for binary record keys\n\n* Address review: fix UTF-8 key ordering in RI multi-slice lookup and MDT compaction paths\n\n* test(HUDI-8898): add TestStringUtils coverage for UTF-8 comparator\n\n* Fix HoodieSortedMergeHandle to sort keys by UTF-8 bytes, not UTF-16\n\n* Fix checkstyle: avoid escaped unicode chars in test keys\n\n* Address review: note per-call UTF-8 encoding cost in compareUtf8Bytes Javadoc\n\n* Address review: restore record lambda param name in JavaHoodieMetadataBulkInsertPartitioner\n\n* Address review: sort Flink MDT bulk-insert keys by UTF-8 bytes, re-sort filterRowKeys candidates, document unpaired-surrogate caveat\n\n* Address review: merge duplicate binary-key bootstrap tests into a parameterized test and correct the failure-mode comments to the read-side seek miss\n\n* Address review: add focused unit tests for Java and Spark MDT partitioners, sorted buffer merge order, and comparator serialization\n\n* Address review: add secondary index test with non-ASCII secondary key values\n\n* Address review: sort native-log MDT delta-commit keys by UTF-8 bytes across engines, cover log-file lookups and assert write statuses in binary-key tests\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "48fe2c6bd0a44fa1ef5693aa7f7460c32f0cfd61", "tree": "7edc2357132992d200069014da744b4ca98e8960", "parents": [ "503963910d84224b094d7ecfb8ca59d6881f8dc8" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Fri Jul 24 19:50:07 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 24 12:50:07 2026 +0000" }, "message": "fix(hive-sync): stop the HMS lock heartbeat once the metastore drops the lock (#19370)\n\n* fix(hive-sync): stop the HMS lock heartbeat once the metastore drops the lock\n\n* addressed review comments: scope the lock-loss callback to its lock id\n\n* addressed review comments: document the unlock throw, snapshot lock reads and extend lock-loss coverage\n\n* addressed review comments: skip the heartbeat for a lock that was never granted\n\n---------\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e" }, { "commit": "503963910d84224b094d7ecfb8ca59d6881f8dc8", "tree": "2161958163d8258c62e8797c1193086f12e518ca", "parents": [ "d4d8bfa08a270fa9377fdf6c0c6c84ca46b59c09" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Fri Jul 24 12:23:47 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 24 12:23:47 2026 +0000" }, "message": "chore(deps): bump org.apache.commons:commons-configuration2 (#18801)\n\nBumps org.apache.commons:commons-configuration2 from 2.11.0 to 2.15.0.\n\n---\nupdated-dependencies:\n- dependency-name: org.apache.commons:commons-configuration2\n dependency-version: 2.15.0\n dependency-type: direct:production\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "d4d8bfa08a270fa9377fdf6c0c6c84ca46b59c09", "tree": "4b12b2e68548f50a1fe10d6286334a343358ce1a", "parents": [ "dfb763ec2ef5c09ee04dda5e9aa5d332b52ff4c9" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Fri Jul 24 11:15:08 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 24 11:15:08 2026 +0000" }, "message": "chore(deps): bump org.apache.thrift:libthrift (#18910)\n\nBumps [org.apache.thrift:libthrift](https://github.com/apache/thrift) from 0.14.0 to 0.23.0.\n- [Release notes](https://github.com/apache/thrift/releases)\n- [Changelog](https://github.com/apache/thrift/blob/master/CHANGES.md)\n- [Commits](https://github.com/apache/thrift/compare/v0.14.0...v0.23.0)\n\n---\nupdated-dependencies:\n- dependency-name: org.apache.thrift:libthrift\n dependency-version: 0.23.0\n dependency-type: direct:production\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "dfb763ec2ef5c09ee04dda5e9aa5d332b52ff4c9", "tree": "1bf6ca082c7f81765fbf9718ffd9c5b603e624c7", "parents": [ "e5452c6ff38d71da6cf43a9f664e61e6e4aadc9b" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Fri Jul 24 10:44:26 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 24 10:44:26 2026 +0000" }, "message": "chore(deps): bump io.airlift:aircompressor from 0.27 to 2.0.3 (#18247)\n\nBumps [io.airlift:aircompressor](https://github.com/airlift/aircompressor) from 0.27 to 2.0.3.\n- [Release notes](https://github.com/airlift/aircompressor/releases)\n- [Commits](https://github.com/airlift/aircompressor/compare/0.27...2.0.3)\n\n---\nupdated-dependencies:\n- dependency-name: io.airlift:aircompressor\n dependency-version: 2.0.3\n dependency-type: direct:production\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "e5452c6ff38d71da6cf43a9f664e61e6e4aadc9b", "tree": "4be52222b6f064f7344bae8118df326a5ef9e5fa", "parents": [ "506740a2c2659b4a12c3f812d0ab5687495d96ae" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Fri Jul 24 01:57:35 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 24 08:57:35 2026 +0000" }, "message": "feat(spark): upgrade Spark 4.2 dependency to GA 4.2.0 (#19365)\n\n* chore(spark): upgrade Spark 4.2 dependency to GA 4.2.0\n\nBump the spark4.2 profile from 4.2.0-preview4 to the official 4.2.0 GA\nrelease. Align the transitive versions that changed in GA: hadoop\n3.4.3 -\u003e 3.5.0 and lz4-java 1.10.4 -\u003e 1.11.0. Rename the CI bundle\nvalidation base image tag flink1200hive313spark420previewscala213 -\u003e\nflink1200hive313spark420scala213, rebuild it with SPARK_VERSION\u003d4.2.0\nand HADOOP_VERSION\u003d3.5.0, and update the Spark 4.2 runtime string to\nspark4.2.0 across the CI scripts, bot.yml, and .asf.yaml required checks.\n\ncloses #19364\n\n* Adapt hudi-spark4 modules to Spark 4.2.0 GA Catalyst API changes\n\nSpark 4.2.0 GA made three source-breaking Catalyst changes relative to\n4.2.0-preview4:\n\n- TableOutputResolver.resolveOutputColumns replaced the boolean\n supportColDefaultValue flag with a defaultValueFillMode enum. Moved the\n call into a per-version resolveOutputColumns seam on the shared\n hudi-spark4-common base: 4.0/4.1 pass supportColDefaultValue \u003d true, 4.2\n passes defaultValueFillMode \u003d FILL (the exact equivalent of the old true).\n- Removed the geospatial GeographyVal/GeometryVal types and\n SpecializedGetters.getGeography/getGeometry. Dropped the corresponding\n overrides from the Spark 4.2 InternalRow and partition-values wrappers.\n- Added SpecializedGetters.getBinaryView(int): BinaryView. Implemented it on\n the Spark 4.2 wrappers; the partition-mapping subclasses inherit it.\n\n* Move .asf.yaml required-check updates to a separate PR\n\nThe .asf.yaml required_status_checks rename (spark4.2.0-preview4 -\u003e\nspark4.2.0) must land before this PR so the renamed Spark 4.2 jobs can\nsatisfy the required checks. It is handled in its own PR; revert the\n.asf.yaml change here to keep this PR free of it.\n\n* chore(spark): address review comments on Spark 4.2 GA upgrade\n\n- validate.sh: collapse the three per-version Spark 4 checks to a\n `[[ \"$SPARK_VERSION\" \u003d\u003d 4.* ]]` glob so future Spark 4 bumps don\u0027t\n touch this file.\n- release_candidate_validation.yml: add the spark4.2 / spark4.2.0 entry\n to validate-release-candidate-bundles-spark4 so the\n hudi-spark4.2-bundle_2.13 gets bundle-validated at RC time." }, { "commit": "506740a2c2659b4a12c3f812d0ab5687495d96ae", "tree": "eed82f04b4c28134fddc9a3fe849c6135150d0b0", "parents": [ "a4373f75bad4e8175ab00b121706417b1cfaed7c" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Fri Jul 24 00:38:42 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 24 15:38:42 2026 +0800" }, "message": "chore(ci): update .asf.yaml required checks for Spark 4.2.0 GA (#19367)\n\nRename the two Spark 4.2 required status checks from spark4.2.0-preview4 to\nspark4.2.0 ahead of the Spark 4.2 GA dependency upgrade (#19365). .asf.yaml\nrequired_status_checks only take effect once merged to master, so this must\nland first: otherwise the job rename in #19365 produces spark4.2.0 checks\nwhile master still requires the spark4.2.0-preview4 names, and #19365 can\nnever satisfy them.\n\nissue: #19364" }, { "commit": "a4373f75bad4e8175ab00b121706417b1cfaed7c", "tree": "5cce842d69cb1cd0ffe900a7c585c98425f44eea", "parents": [ "ebbfcc500dc138fb32d33e2778d6ba3ed0b88c37" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Fri Jul 24 13:55:54 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 24 13:55:54 2026 +0800" }, "message": "test(hudi-client): improve metadata table writer coverage (#19363)\n\n* test(hudi-client): improve metadata table writer coverage" }, { "commit": "ebbfcc500dc138fb32d33e2778d6ba3ed0b88c37", "tree": "f013bcf73dfe20eb90c0ef6520ce04322c38fbf5", "parents": [ "4d11a60552e43aae88e5dfcfbbdb31d287505273" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Thu Jul 23 21:27:15 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 23 21:27:15 2026 -0700" }, "message": "fix(timeline-service): fail marker creation requests when marker flush fails (#19368)\n\nMarkerDirState.flushMarkersToFile closed the marker file stream with\n closeQuietly, so a close()-time flush failure (e.g. S3A, which performs\n the object PUT in close()) was silently swallowed. The pending marker\n creation request was then acknowledged as successful, letting the\n writer create a data file with no durable marker. Marker-based\n rollback then missed such files, and a retry under the same instant\n could leave duplicate file groups behind.\n\n - Close the writer within try-with-resources so a close() failure\n propagates as HoodieIOException instead of being swallowed.\n - On a flush failure, roll back the markers buffered by the failed\n batch from in-memory state and fail the pending futures exceptionally,\n so a retried request can recreate the markers.\n - Release the marker file index in a finally block so a failure\n doesn\u0027t leak it.\n\n Adds TestMarkerDirState covering the success, flush-failure, and\n retry-after-failure paths." }, { "commit": "4d11a60552e43aae88e5dfcfbbdb31d287505273", "tree": "3426495fa21f5bf0ce13a1bd6efad31c7e92214f", "parents": [ "427efb2654296cd78c93e2602ca06f6f82bf0733" ], "author": { "name": "Danny Chan", "email": "yuzhao.cyz@gmail.com", "time": "Thu Jul 23 18:12:29 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 23 10:12:29 2026 +0000" }, "message": "fix(flink): deduplicate delete keys and cover write paths (#19354)\n\n* fix(flink): deduplicate delete keys and cover write paths" }, { "commit": "427efb2654296cd78c93e2602ca06f6f82bf0733", "tree": "7a5ef3acbdf0aeecea38b280053b8f5f98057e53", "parents": [ "5f19cf32b039eb64828af8752346cdde31444a24" ], "author": { "name": "Danny Chan", "email": "yuzhao.cyz@gmail.com", "time": "Thu Jul 23 17:42:02 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 23 09:42:02 2026 +0000" }, "message": "fix(flink): normalize row logical conversions and improve coverage (#19351)\n\n* test(flink): improve row conversion coverage" }, { "commit": "5f19cf32b039eb64828af8752346cdde31444a24", "tree": "b28e788900d846dae9da431823af5e33b789ffa4", "parents": [ "30a4ba33e38595eb6c1ef8327cba7ae0296aa92a" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Thu Jul 23 16:22:21 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 23 08:22:21 2026 +0000" }, "message": "fix(metadata-table): follow-ups for the zero-size file skip (#18611) (#19355)\n\n* test(metadata): add Scala test for upsert after skipping zero-size file on MDT initialize\n\nAdds testUpsertAfterSkippingZeroSizeFileOnInitialize to TestRecordLevelIndex\nto cover the scenario where SKIP_ZERO_SIZE_FILES_ON_INITIALIZE is enabled:\n- corrupts the sole base file to zero-size and forces MDT rebootstrap\n- verifies upsert succeeds and writes to a new file group (not the zero-size one)\n- validates RLI completeness and that the zero-size fileId is absent from MDT\n\nUses GLOBAL_RECORD_LEVEL_INDEX so bootstrap always creates the minimum 10 file\ngroups for record_index even when all data files were skipped as zero-size.\n\n* fix(metadata-table): address review follow-ups for the zero-size file skip\n\n- honor hoodie.metadata.skip.zero.size.files.on.initialize in the\n restore-sync relisting so restore does not re-add files that were\n skipped at initialization\n- document that skipped files remain on storage untracked by MDT and\n the cleaner, and must be removed manually\n- correct the config sinceVersion to 1.3.0\n- move the metric name into HoodieMetadataMetrics and only emit it\n when files were actually skipped\n- aggregate the per-file WARN into one WARN per partition, with the\n individual paths at DEBUG\n\n---------\n\nCo-authored-by: Lokesh Jain \u003cljain@apache.org\u003e" }, { "commit": "30a4ba33e38595eb6c1ef8327cba7ae0296aa92a", "tree": "d7b221a711b16a82c6b05e62270504b2ea819a86", "parents": [ "5a42f8957f0971593572663ba9dedb94398545e3" ], "author": { "name": "201573", "email": "78600363+201573@users.noreply.github.com", "time": "Thu Jul 23 16:17:11 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 23 08:17:11 2026 +0000" }, "message": "docs(docker): document build_docker_images.sh flags (#18687)\n\n* docs(docker): document build_docker_images.sh flags\n\n* docs(docker): clarify multi-arch prerequisites\n\n* docs(docker): align demo example with setup_demo.sh\n\n* docs(docker): mention setup_demo.sh compose override\n\n* docs(docker): address review findings on demo build notes\n\n- note the default versions built when no flags are passed\n- clarify build_docker_images.sh builds with docker directly, unlike\n build_local_docker_images.sh which goes through Maven\n- warn that a plain setup_demo.sh run pulls Docker Hub images over\n locally built ones; use ./setup_demo.sh dev to keep local images\n- use shell language tag on the buildx code fence for consistency\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "5a42f8957f0971593572663ba9dedb94398545e3", "tree": "36874ed4052d0f71053455f5af2c6844e0ca2bbb", "parents": [ "9f53c46d0de8476f880129dfdf0daaf279c06045" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Thu Jul 23 14:35:41 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 23 07:35:41 2026 +0000" }, "message": "test(spark): Fix flaky TestSparkFilterHelper by giving it its own SparkSession (#19356)\n\ntestConvertInExpression calls org.apache.spark.sql.functions.expr(...), which lazily builds a SessionState on the active SparkSession. TestSparkFilterHelper extended HoodieSparkClientTestHarness directly without wiring @BeforeEach initSparkContexts(), so it relied on a leaked active session from a prior test in the same Surefire fork and intermittently failed with \"LiveListenerBus is stopped\" when that session\u0027s context was already stopped. Add @BeforeEach/@AfterEach that create and stop the session, matching HoodieClientTestBase and the sibling TestHoodieDataSourceHelper.\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e" }, { "commit": "9f53c46d0de8476f880129dfdf0daaf279c06045", "tree": "2a96ef888588540e95078c1626d69821490e6597", "parents": [ "f25e0799c5b73e8b77d777910dab78859406b1c5" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Thu Jul 23 14:21:48 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 23 14:21:48 2026 +0800" }, "message": "feat(flink): enable LSM end-to-end for Flink by default (#19307)" }, { "commit": "f25e0799c5b73e8b77d777910dab78859406b1c5", "tree": "294cb8c86d81f7ca8946e82c5a7a7ab320217e9f", "parents": [ "a44829aa6ed99f29847432536d6b7ec160e99720" ], "author": { "name": "Vinish Reddy", "email": "vinishreddygunner17@gmail.com", "time": "Thu Jul 23 10:30:27 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 23 05:00:27 2026 +0000" }, "message": "fix: Handle map/array-nested leaf columns in column stats collection during MOR log-append (#19126)\n\n* fix: Handle map/array nested leaves in Avro column-stats value navigation\n\nCollecting column stats for a primitive nested inside a MAP or ARRAY\n(e.g. `my_map.key_value.value`, `my_array.list.element`) crashed the MOR\ninline log-append path at table version 9:\n\n IllegalStateException: Cannot get field from schema type: MAP\n at HoodieSchema.getField\n at AvroRecordContext.getFieldValueFromIndexedRecord\n at HoodieAvroIndexedRecord.getColumnValueAsJava\n at HoodieTableMetadataUtil.collectColumnRangeFieldValueV2\n at HoodieInlineLogAppendHandle.collectColumnStats\n\n#17694 taught the schema-side navigator (HoodieSchema.getNestedField) and\nthe base-file (Parquet) path to resolve the Parquet-style `.key_value.key`,\n`.key_value.value` and `.list.element` synthetic accessors, so such leaves\npass isColumnTypeSupported. The record value-side navigator\n(AvroRecordContext.getFieldValueFromIndexedRecord), used by the column-stats\nV2 collection path (collectColumnRangeFieldValueV2 -\u003e getColumnValueAsJava),\nwas never updated: it assumes every path segment is a RECORD field and calls\nHoodieSchema.getField on the MAP/ARRAY schema, which throws.\n\nMake the value navigator return null when a path segment cannot be resolved\nas a plain RECORD field (a MAP/ARRAY intermediate or a null intermediate),\ninstead of throwing. This mirrors HoodieAvroUtils.getNestedFieldVal, which is\nwhy the V1 path at table version 6 already tolerated these paths. A map/array\nleaf is multi-valued per record and has no single value to fold into a\nmin/max, so returning null (no stats from the record path) is correct;\nstatistics for such leaves are still collected from the base-file (Parquet)\nfooter path added in #17694.\n\nAlso guard the intermediate downcast so a non-record value (java Map/List)\ndegrades to null rather than throwing ClassCastException.\n\n* Rename test constant COMPLEX_SCHEMA to MAP_AND_ARRAY_SCHEMA for clarity\n\n* Move the schema constants to the top of TestAvroRecordContext\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "a44829aa6ed99f29847432536d6b7ec160e99720", "tree": "2d1bc81537f0199f4b138240e836f364ba9c8398", "parents": [ "76f81ec558211bce6074529578b86189b4b00e27" ], "author": { "name": "vinoth chandar", "email": "vinothchandar@users.noreply.github.com", "time": "Wed Jul 22 20:26:42 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 23 11:26:42 2026 +0800" }, "message": "feat: agentic lakehouse v1 - hudi-lakehouse charts and hudi-agent-gateway (#19265)\n\n* feat: add hudi-k8s-lakehouse - hudi-trino Helm chart and local-dev Kubernetes environment\n\nKubernetes packaging for the Hudi lakehouse, split by concern:\n\n- charts/hudi-trino: a deployable product chart - Trino (server 472, pinned by\n the hudi-trino-plugin\u0027s strict SPI check) with the Hudi connector built from\n this repository. Values point at existing infrastructure: Hive Metastore\n (thrift) or AWS Glue, S3/GCS via Trino\u0027s native filesystem; credentials via\n Secret-backed env substitution, never in ConfigMaps.\n- local-dev/: a minikube developer environment that stands up what the chart\n points at - MinIO, a Derby-backed Hive Metastore (stock apache/hive image\n configured entirely via manifest: single-authority schema init, Recreate\n strategy for the Derby singleton, HADOOP_CLASSPATH for S3A as non-root) and\n the Apache Spark Kubernetes Operator - then installs the same hudi-trino\n chart with local values.\n- local-dev/example/: a copy-me template for submitting any Spark job as a\n SparkApplication; the sample writes a partitioned Hudi COW table to MinIO\n and registers it in HMS via hive-sync, immediately queryable from Trino.\n- images/ + scripts/: thin Dockerfiles over official images; jars staged from\n local maven builds into gitignored target/ dirs (no binaries in git).\n\nTables Trino must read are written with hoodie.write.table.version\u003d8 and\nauto-upgrade disabled: the plugin currently reads with a Hudi 1.0.x runtime\n(table versions 6/8). This pin is documented and goes away when the plugin\u0027s\nHudi dependency is upgraded.\n\nVerified end to end on minikube from a clean cluster via\nlocal-dev/scripts/smoke-test.sh: up -\u003e Spark write+sync -\u003e Trino serves 100\nrows with correct partition aggregates.\n\n* feat: add one-command idempotent quickstart for hudi-k8s-lakehouse\n\nscripts/quickstart.sh converges the entire local-dev lakehouse in one run:\npreflight -\u003e minikube (started if absent) -\u003e maven artifacts (spark bundle +\ntrino plugin; a Temurin JDK 23 is auto-provisioned into a gitignored .tools/\ndir when none is installed) -\u003e container images into minikube\u0027s docker\ndaemon -\u003e MinIO + HMS + Spark Operator + hudi-trino chart. Every step checks\nwhether its work is already done, so re-running is always safe;\n--run-example additionally runs the example Spark job so a Hudi table is\nregistered and ready to query from Trino. --rebuild-jars/--rebuild-images\nforce refreshes.\n\nup.sh made fully idempotent (immutable bucket Job deleted before re-apply);\nREADME restructured around the one-command path with a scripts reference\ntable.\n\nVerified: quickstart.sh --run-example green on an already-provisioned\nmachine (all builds skipped, releases converged, example re-ran, Trino\nserved correct results).\n\n* feat: add hudi-ai-gateway - the lakehouse AI serving layer (agent loop, MCP server, chat UI)\n\nA new top-level Python service module implementing \u0027Surface 5\u0027 of the\nAI-native lakehouse: ONE deployable process serving the lakehouse to AI.\n\nThree surfaces over one guarded tool set:\n- POST /v1/chat: the agent loop (LangGraph ReAct over lakehouse tools) with\n multi-turn sessions (checkpointer-backed, TTL/LRU-bounded in-memory store)\n and SSE streaming (token / tool_call / tool_result / done / error events).\n- /mcp: an MCP server (streamable HTTP) exposing the same tools to external\n agents such as Claude; tools are defined once and derived for both the\n agent loop and MCP from a single registry, with every invocation logged as\n structured JSON (the seed of the gateway\u0027s trace collection).\n- /ui: a first-party ChatGPT-style chat UI (vanilla JS/CSS, zero third-party\n code - no CDN, no npm, license-clean and air-gap friendly) with streamed\n tokens, collapsible tool-activity chips, and markdown table rendering.\n\nPluggable LLM providers selected by config: anthropic, openai, ollama (local\nlaptop default), and openai-compatible endpoints (vLLM etc.). Construction\nnever touches the network; /ready reports per-dependency status (trino, llm)\nand gates the Kubernetes readiness probe.\n\nv1 tools query Trino: query_lakehouse (AST-level guardrails via sqlglot -\nsingle-statement, SELECT-only, fail-closed parse, row cap injected as LIMIT,\nresult truncation with explicit notice), list_tables and describe_table via\ninformation_schema (works for thrift HMS and Glue alike).\n\nDeployment: charts/hudi-ai-gateway (sibling of hudi-trino, same conventions;\nAPI keys via Secret/existingSecret only; replicas pinned to 1 for the\nin-memory session store) wired into local-dev up.sh (default ON, no --wait\nsince readiness reflects LLM reachability) and build-images.sh/quickstart.sh.\n\nVerified: 74 offline tests (fake Trino + scripted model through the real\ncreate_react_agent), ruff + mypy clean; live end-to-end on the local-dev\nstack with both anthropic (claude-haiku-4-5) and fully-local ollama\n(qwen3:4b) - grounded answers with correct counts from hudi.default.trips,\nguardrail LIMIT visible in executed SQL, browser-verified chat UI, MCP\ninitialize over HTTP; in-cluster e2e green (pod Ready with trino + host\nollama checks, chat through the k8s service).\n\n* feat: provider-aware model selection, standard secret convention, and optional vllm chart\n\nOut-of-box experience stays Ollama everywhere; switching providers is a\nvalues change; and the UI now adapts to whatever the deployment is\nconfigured with:\n\n- GET /v1/models discovers the configured provider\u0027s models LIVE (Anthropic\n and OpenAI model APIs, Ollama\u0027s local tags, vLLM\u0027s served models; 60s\n cache; falls back to the configured default when listing fails).\n- POST /v1/chat accepts an optional \u0027model\u0027 (within the provider); agents\n are built lazily per model and share the session checkpointer, so\n conversations survive model switches.\n- The chat UI gains a model picker fed by /v1/models (persisted per\n browser; single-model deployments like vLLM render it disabled).\n- Chart secrets standardized: the deployment mounts one Secret via envFrom\n and keys use the providers\u0027 standard env-var names (ANTHROPIC_API_KEY,\n OPENAI_API_KEY; future TOGETHER_API_KEY/FIREWORKS_API_KEY/BASETEN_API_KEY/\n HF_TOKEN need no chart changes). Compatible with External Secrets/Vault\n CSI via existingSecret.\n- New optional charts/vllm: vLLM (v0.25.0) serving one open-weight model\n behind an OpenAI-compatible API; ungated Qwen/Qwen3-8B default (no HF\n token needed), tool-calling flags preconfigured, model-cache PVC, GPU\n resources, generous startup probe. INSTALL_VLLM\u003d1 in local-dev up.sh\n installs it and points the gateway at it (VLLM_MODEL\u003d to override).\n GPU-only: verified via helm lint/template, not a live pod.\n\nVerified: 78 offline tests + ruff + mypy; live /v1/models against Anthropic\n(real model list) and per-request model switch (claude-sonnet-5) answering\nfrom the trips table; UI picker rendered in browser; both charts lint.\n\n* chore: cache the gateway image\u0027s dependency layer across source changes\n\nInstall deps against a stub package first, then overlay real sources with a\nno-deps reinstall - source edits no longer re-download the dependency tree.\n\n* feat: connect panel in the gateway UI - one place for all lakehouse endpoints\n\n/v1/info now reports the configured Trino URL and MCP state; the UI gains a\n\u0027Connect\u0027 panel listing the three ways into the lakehouse with\ncopy-to-clipboard: the Trino SQL endpoint (for any SQL client/BI tool), the\ninference API (POST /v1/chat), and the MCP server URL (with the\nclaude mcp add snippet). Makes the chat UI the single pane of glass for the\nwhole stack.\n\n* refactor: rename modules for the agentic-lakehouse positioning\n\n- hudi-k8s-lakehouse/ -\u003e hudi-lakehouse/ (the distribution is the product;\n Kubernetes is the deployment target, not the identity)\n- hudi-ai-gateway/ -\u003e hudi-agent-gateway/ (module, python package, chart,\n CLI, image name). \u0027AI gateway\u0027 now names a specific infra category -- LLM\n proxy/routers -- which this is not: the server HOSTS the agent and gateways\n external agents into the lakehouse via MCP. \u0027agent-gateway\u0027 reads correctly\n in both directions.\n\nMechanical rename: no behavior changes. All references swept (python package\nhudi_agent_gateway, pyproject/console script, chart names + helpers, image\nhudi-lakehouse-agent-gateway, scripts incl. INSTALL_AGENT_GATEWAY toggle,\nREADMEs, UI display strings). Verified: 78 tests + ruff + mypy green in a\nfresh venv; all three charts lint; scripts syntax-checked; zero stale\nreferences.\n\n* fix: repair streamed adaptive-thinking blocks in the agent loop\n\nAnthropic\u0027s adaptive-thinking models (claude-fable-5, claude-opus-4-8) can\nreturn a thinking block whose text is empty (signature only). When such a\nturn is streamed, langchain-anthropic (\u003c\u003d1.4.8) merges the chunks into a\nblock with no \u0027thinking\u0027 key at all, and the API rejects the message on the\nnext agent-loop call with \u0027thinking.thinking: Field required\u0027 -- so every\ntool-using conversation on a thinking model died on turn two, while\nnon-thinking models (haiku) worked.\n\nWork around it in the pre-model hook: restore the empty \u0027thinking\u0027 field on\naffected blocks before each model call (lossless; copies the message rather\nthan mutating checkpointed history). Verified live: opus-4-8 and fable-5 now\ncomplete multi-round tool loops over SSE.\n\n* chore: add GitHub Actions CI for hudi-agent-gateway and lakehouse charts\n\nThe Python gateway module and the Helm charts were previously untested in\nCI. Adds a path-filtered workflow that runs the offline pytest suite (80\ntests, no live LLM/Trino needed), ruff, and mypy on Python 3.11 and 3.12,\nplus helm lint/template over every chart (hudi-trino renders with local-dev\nvalues since its defaults fail fast on the required metastore uri).\n\n* fix: rename \u0027New chat\u0027 to \u0027Clear chat\u0027 in the gateway UI\n\nThe button clears the message list and rotates the session id; nothing is\npersisted, so \u0027New chat\u0027 over-promised. Session/chat persistence is tracked\nas follow-on work.\n\n* fix: make UI example prompts catalog-agnostic\n\nThe welcome suggestions referenced the local-dev demo \u0027trips\u0027 table; phrase\nthem so the agent\u0027s own catalog introspection adapts them to whatever tables\nexist.\n\n* feat: add Trino web UI to connect panel, raise default SQL timeout to 120s\n\nLoad-testing against ~60M rows surfaced both: the connect panel had no link\nto Trino\u0027s own UI for watching queries run, and analytical aggregations can\nlegitimately exceed 60s on small clusters. Also give timed-out queries an\nactionable hint (narrow the query or raise GATEWAY_SQL_TIMEOUT_SECONDS)\ninstead of the misleading \u0027Fix the SQL\u0027 -- the SQL was fine.\n\n* fix: satisfy release source-license validation for new module files\n\nvalidate_source_copyright.sh flagged four files from the new modules:\n.helmignore, .dockerignore, and py.typed now carry the ASF header (PEP 561\nignores py.typed content, so the marker stays valid -- mypy verified), and\n.png joins .jpg in the script\u0027s binary exemptions for the gateway UI logo.\nWith the header inside py.typed the rat exclude added earlier in this branch\nis unnecessary; the root pom.xml is back to fully untouched.\n\n* fix: use preinstalled helm in agent gateway CI\n\nazure/setup-helm is not on the apache org\u0027s Actions allowlist and\nstartup-failed the whole workflow; GitHub-hosted ubuntu runners ship helm\npreinstalled, so the third-party action is unnecessary. Remaining actions\n(checkout, setup-python) are GitHub-created and allowlisted.\n\n* fix: address review feedback on tool input validation and result sizing\n\n- list_tables/describe_table interpolated catalog/schema/table names into\n SQL unvalidated; these arrive over HTTP and MCP, so quoted values could\n break out of the identifier/literal. Names are now validated against a\n strict identifier pattern and rejected with a hint before any SQL is built.\n- shape_result could exceed the configured size cap even after dropping all\n rows (huge sql text / many columns / tiny cap); it now degrades to a\n bounded metadata payload, and past that to a bare error payload.\n- README documented the old 60s default for GATEWAY_SQL_TIMEOUT_SECONDS;\n now matches the 120s code default.\n\n* fix: address second-round review feedback on gateway robustness and quickstart\n\n- llm.py: wire GATEWAY_LLM_TIMEOUT_SECONDS into the default (ollama)\n provider via httpx client_kwargs, so a stalled Ollama no longer hangs\n /v1/chat unboundedly\n- chat.py/registry.py: replace the substring-based tool-error detection\n with a shared JSON-parsing is_tool_error(); a successful result with a\n column named \"error\" is no longer reported as a failed tool call\n- trino_tools.py: results that fill the row cap now report\n truncated: true with a notice, so the agent warns instead of\n presenting a capped answer as complete\n- trino_client.py: queries run on a dedicated bounded executor and the\n timeout path cancels the Trino query server-side, so repeated\n timeouts cannot pile up stuck threads; /ready\u0027s SELECT 1 ping is now\n cached (10s) like the LLM check\n- gateway chart: explicit readinessProbe timeoutSeconds (k8s\u0027 1s\n default flapped against the check\u0027s own multi-second budget)\n- build-jars.sh: the JDK-23 guard probes the JDK Maven will actually\n use (honours JAVA_HOME) instead of PATH\u0027s java, unbreaking the\n one-command quickstart on machines whose default JDK is older\n- quickstart/build scripts: drop the --spark-version flag; the\n quickstart supports exactly the Spark 3.5 line the example manifest\n pins, so a 4.1 image that nothing templates can no longer be selected\n- UI: hotlink the logo from hudi.apache.org (no vendored binary) and revert the repo-wide *.png\n exemption added to validate_source_copyright.sh\n- README: document that v1 ships with no authn/authz\n- registry.py: narrow input_schema derivation to pydantic v2 models\n (newer langchain-core widened tool_call_schema\u0027s type; mypy)\n- guardrails: add a SELECT-rooted rejection test (CTE wrapping INSERT)\n so the defense-in-depth tree walk is actually exercised" }, { "commit": "76f81ec558211bce6074529578b86189b4b00e27", "tree": "4a849390bdb769786cab6c210283d88678d8534f", "parents": [ "e7b0ba818df93476be4f339b74a2aae970046494" ], "author": { "name": "Shihuan Liu", "email": "skywalker0618@gmail.com", "time": "Wed Jul 22 18:55:54 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 23 09:55:54 2026 +0800" }, "message": "fix(flink): make hudi-flink-bundle built with flink-bundle-shade-hive usable for Hive sync (#19330)" }, { "commit": "e7b0ba818df93476be4f339b74a2aae970046494", "tree": "9ef4892bc7de10eed0e6db6f8af6aa3b6b9f7adf", "parents": [ "fb07a2dfa5c8b03b714d67d85644407e1519bb57" ], "author": { "name": "Danny Chan", "email": "yuzhao.cyz@gmail.com", "time": "Thu Jul 23 09:55:23 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 23 09:55:23 2026 +0800" }, "message": "test(common): add LSM file group read path coverage (#19347)" }, { "commit": "fb07a2dfa5c8b03b714d67d85644407e1519bb57", "tree": "4bcb3816eca43e2c33b0c770e11dc4f8710bb3b5", "parents": [ "e4071ce5407656e6acd6214b77fcc66813bce69b" ], "author": { "name": "Lokesh Jain", "email": "ljain@apache.org", "time": "Thu Jul 23 05:26:00 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 22 23:56:00 2026 +0000" }, "message": "fix(reader): derive pre-v9 CDC delete markers from the effective payload class (#19348)\n\nCo-authored-by: Y Ethan Guo \u003cethan.guoyihua@gmail.com\u003e" }, { "commit": "e4071ce5407656e6acd6214b77fcc66813bce69b", "tree": "c64a1c372436553801753a2da30f789804d356b7", "parents": [ "76a95f5d629bceff89eab081c768438f2fef1e87" ], "author": { "name": "Davis-Zhang-Onehouse", "email": "169106455+Davis-Zhang-Onehouse@users.noreply.github.com", "time": "Wed Jul 22 03:35:14 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 22 10:35:14 2026 +0000" }, "message": "chore(utilities): add SQS backlog / in-flight visibility to S3 events source (#19333)\n\n* logging: add SQS backlog / in-flight visibility to S3 events source\n\nAdds a lifecycle log line to S3EventsSource.onCommit recording how many SQS\nmessages are being deleted at commit and the checkpoint, improving operational\nvisibility into the SQS-backed S3 events ingestion path.\n\nPlaceholder / first increment of a broader observability change tracked for an\nOnehouse internal hotfix; the full receive-loop / in-flight logging will be\nupstreamed here subsequently.\n\n* chore(utilities): drop redundant class/method prefix from onCommit log\n\nSLF4J emits the logger name already (every log4j2 pattern in the repo uses\n%c), so the \"S3EventsSource.onCommit:\" prefix duplicated it. Also names the\nqueue as SQS in the message.\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "76a95f5d629bceff89eab081c768438f2fef1e87", "tree": "f47f9dfa20ef63278735beb590be8b54885b344b", "parents": [ "9f8376e3723d6e51f5e1a27f52f396b6181a6f6f" ], "author": { "name": "Nikhil Kumar", "email": "nikhilkumar@onehouse.ai", "time": "Wed Jul 22 15:47:46 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 22 10:17:46 2026 +0000" }, "message": "fix(utilities): guard Source.releaseResources() against transient RDD unpersist failures (#19328)\n\nreleaseResources() runs in StreamSync.syncOnce()\u0027s finally block, after the\nwrite/commit has already completed. A transient Spark NullPointerException in\nBlockManagerMaster.removeRdd (thrown when the SparkContext is mid-teardown or\nstopping and the driver endpoint is null) during the post-write RDD unpersist\nwas propagating out and failing otherwise-successful ingestion rounds. A\ncleanup-only failure must not fail the round: wrap the unpersist in try/catch\nand log at WARN. Extract the unpersist into a protected unpersistCachedSourceRdd()\nseam (VisibleForTesting) and add TestSource covering both the swallow and\nhappy paths." }, { "commit": "9f8376e3723d6e51f5e1a27f52f396b6181a6f6f", "tree": "8af236a8e30ce4fcc89b95d1980a7eac19f59cc4", "parents": [ "a18f22126dfa029b59d68cf53c4d1a3b75be0e33" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Wed Jul 22 17:03:41 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 22 18:03:41 2026 +0800" }, "message": "fix(hive-sync): keep HMS lock heartbeat alive and release its thread pool on close (#19334)\n\n* fix(hive-sync): keep HMS lock heartbeat alive and release its thread pool on close\n\nFixes three defects in HiveMetastoreBasedLockProvider and its Heartbeat (package transaction/lock):\n\n- Heartbeat.run() no longer rethrows on failure. Under ScheduledExecutorService.scheduleAtFixedRate a thrown exception permanently cancels all subsequent executions (observable only through the unread ScheduledFuture), so one transient HMS/network hiccup silently stopped lock renewal while the writer still believed it held the lock. It now logs a warning with the cause and lets the next tick retry; the previous throw also discarded the cause.\n\n- close() moves executor.shutdown() into a finally block so the 2-thread scheduled pool is always released even when unlock()/Hive.closeCurrent() throws, and passes the caught exception to log.error instead of dropping it.\n\n- acquireLock recovery path: when the client times out on Future.get but the lock is granted server-side and recovered via checkLock, a heartbeat is now scheduled (via the extracted scheduleHeartbeat()) so the recovered lock is renewed instead of being left to expire mid-write.\n\nAdds unit tests TestHeartbeat and TestHiveMetastoreBasedLockProviderClose.\n\n* addressed review comments: drop the hardcoded thread count from the executor shutdown comment\n\n---------\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e" }, { "commit": "a18f22126dfa029b59d68cf53c4d1a3b75be0e33", "tree": "fa05154a94cae552c4f8bf9565c19511f5c2a88d", "parents": [ "0acf20fed0cc376ed68680b67df71ddbb0db423b" ], "author": { "name": "Prashant Wason", "email": "pwason@uber.com", "time": "Wed Jul 22 02:38:55 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 22 17:38:55 2026 +0800" }, "message": "[HUDI-18060] Improve error message when ordering field value is null (#18061)\n\nWhen records have a null value in the ordering (precombine) field, Hudi\njobs previously failed with a cryptic \"Ordering value is null for record\"\nerror that gave no actionable context. This change fails fast in\nHoodieCreateRecordUtils with a clear message identifying the offending\nordering field and record key, and suggesting remediation.\n\nMerge modes that do not depend on the ordering value are exempted:\nCOMMIT_TIME_ORDERING and OverwriteWithLatestAvroPayload fall back to\nOrderingValues.getDefault() instead of failing. The payload-class check\nis retained so table version 6 (which may not have a merge mode set)\nstill bypasses the failure for OverwriteWithLatestAvroPayload.\n\nThe gating flag is computed once at driver scope rather than per record\nto avoid repeated work on the per-record hot path.\n\nCloses #18060" }, { "commit": "0acf20fed0cc376ed68680b67df71ddbb0db423b", "tree": "e5b2e913a14cffcb45b7d46f0d1cf1061db16acc", "parents": [ "55becc7b03f4cf8ec3e1e5303d14fd46bfc8f9b9" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Wed Jul 22 02:10:30 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 22 17:10:30 2026 +0800" }, "message": "refactor(spark): share the Spark 3.x legacy parquet file format via a common base (#19341)" }, { "commit": "55becc7b03f4cf8ec3e1e5303d14fd46bfc8f9b9", "tree": "0dc98cb203fa413a04cb564cc46380d6823bac3d", "parents": [ "7f786d434fa8e1d60880fac9bc5a9323b99c0f0a" ], "author": { "name": "vamshipasunuru1", "email": "127545665+vamshipasunuru1@users.noreply.github.com", "time": "Wed Jul 22 01:53:52 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 22 08:53:52 2026 +0000" }, "message": "fix: do not fall back to timeline server markers on transient HDFS failures (#18887)\n\n* fix: do not fall back to timeline server markers on transient HDFS failures\n\nWhen MARKERS.type is absent, MarkerBasedRollbackUtils tries DIRECT markers\nand catches IOException to fall back to TIMELINE_SERVER_BASED. This was too\nbroad: a transient \"Server too busy\" RetriableException is also an IOException,\ncausing rollback to use the timeline server marker path which finds 0 markers\nand deletes nothing, leaving orphan data files behind.\n\nOnly catch IllegalArgumentException (marker path format mismatch) for the\nfallback. Let IOException propagate so the rollback fails and retries rather\nthan silently producing an incorrect result.\n\n* update log\n\n* Address review: use parameterized SLF4J logging and fix storage reference\n\n- Switch String.format to SLF4J parameterized form so the exception\n stack trace is preserved in the log output.\n- Fix compile error: getTimelineServerBasedMarkers takes HoodieStorage\n (storage), not the undefined fileSystem variable.\n\n* Address review: rewrite test to exercise the actual code path\n\nThe prior test stubbed metaClient.getFs() and HoodieWrapperFileSystem.listStatus(),\nbut the production code goes through table.getStorage() and HoodieStorage.listDirectEntries().\nThe injected IOException never reached the catch block under test.\n\nRewrite to mock the correct seams:\n- table.getStorage() returns a mock HoodieStorage\n- storage.exists(MARKERS.type) → false to trigger the fallback branch\n- storage.exists(markerDir) → true so allMarkerFilePaths() lists entries\n- storage.listDirectEntries(markerDir) throws the failure of interest\n\nVerified by inspecting the surefire stack trace: the exception now originates\nin FSUtils.processFiles → DirectWriteMarkers.allMarkerFilePaths →\nMarkerBasedRollbackUtils.getAllMarkerPaths, confirming the fix is exercised.\n\nAlso add a companion test asserting that IllegalArgumentException still falls\nback to TIMELINE_SERVER_BASED (preserving the original intended behavior).\n\n---------\n\nCo-authored-by: vamshi_UBER \u003cvamshi@uber.com\u003e\nCo-authored-by: vamshipasunuru1 \u003cvamshipasunuru1@users.noreply.github.com\u003e" }, { "commit": "7f786d434fa8e1d60880fac9bc5a9323b99c0f0a", "tree": "5900d8b1665142aa37a7cde76fc5e1390c6ee305", "parents": [ "58dd532e64a7631971c0f735b82edb87285ad326" ], "author": { "name": "Danny Chan", "email": "yuzhao.cyz@gmail.com", "time": "Wed Jul 22 15:02:39 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 22 15:02:39 2026 +0800" }, "message": "chore: upload Flink integration-test coverage (#19343)\n\n* ci: upload Flink integration-test coverage\n\n* chore: clarify Flink integration-test coverage" }, { "commit": "58dd532e64a7631971c0f735b82edb87285ad326", "tree": "adab48779b45e1119854a2cc3be5d79a02cfe348", "parents": [ "d8ebba4974e74d990af9f0c71ebd6073205e603e" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Wed Jul 22 10:41:10 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 22 10:41:10 2026 +0700" }, "message": "fix(hive-sync): set HMS table createTime in seconds instead of milliseconds (#19335)" }, { "commit": "d8ebba4974e74d990af9f0c71ebd6073205e603e", "tree": "c3262f28cc37b6009a3cb75bf17df3a092a2676a", "parents": [ "cc75051fc4efa5449f1e0d9e6f6bffe29e2b14f6" ], "author": { "name": "vamsikarnika", "email": "vamsikarnika@gmail.com", "time": "Wed Jul 22 07:59:16 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 21 19:29:16 2026 -0700" }, "message": "fix: relax existing column to nullable in reconcileSchema when source made it nullable (#19337)\n\nIn reconcileSchema, detect existing columns whose incoming schema is nullable but the table is required (nullabilityRelaxColumns), exclude them from the early-return short-circuit, and relax them to nullable in the result via updateColumnNullability(col, true). This only ever widens, never tightens (a nullable→required incoming leaves the table nullable), consistent with reconcileSchemaRequirements, and still null-fills genuinely missing columns." }, { "commit": "cc75051fc4efa5449f1e0d9e6f6bffe29e2b14f6", "tree": "ef5ea9a4624af043d2461c49e7ea5c7add43d1d8", "parents": [ "b90627ae393dd3d9b4b846ee85beb96c3aa8d5a8" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Tue Jul 21 23:45:52 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 22 00:45:52 2026 +0800" }, "message": "fix(hive-sync): fix partition-value parsing on \u0027\u003d\u0027 and pushdown comparator overflow (#19336)\n\nTwo edge-case defects in the partition path of hudi-hive-sync:\n\n- PartitionFilterGenerator.ValueComparator sorted int/bigint partition values with subtraction (i1 - i2, Long.signum(l1 - l2)) to derive the pushdown min/max bounds. The subtraction overflows when the difference exceeds the type range (e.g. Integer.MAX_VALUE and a negative value), giving a wrong ordering and wrong bounds that can exclude valid partitions or trip the Comparator contract check. Use Integer.compare/Long.compare.\n\n- MultiPartKeysValueExtractor and HiveStylePartitionValueExtractor split a hive-style key\u003dvalue segment with split(\"\u003d\") (no limit), so a value that itself contains \u0027\u003d\u0027 (base64 padding or an embedded \u0027\u003d\u0027) was rejected (aborting the sync) or silently truncated (trailing \u0027\u003d\u0027 dropped), causing drift from the metastore. Split on the first \u0027\u003d\u0027 only with split(\"\u003d\", 2).\n\nExtends TestPartitionFilterGenerator (extreme int/bigint min/max bounds), TestMultiPartKeysValueExtractor and TestPartitionValueExtractor (values containing \u0027\u003d\u0027).\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e" }, { "commit": "b90627ae393dd3d9b4b846ee85beb96c3aa8d5a8", "tree": "c2b93cc4d84a7319113dd8098f24b7f333b4ee87", "parents": [ "41a8fc881ac8465f6716908a8816208d6f20e17a" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Tue Jul 21 01:16:28 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 21 16:16:28 2026 +0800" }, "message": "test(common): add unit coverage for metrics reporters and schema utilities (#19221)\n\nAdds unit tests for three under-covered hudi-common areas:\n- TestValueType: per-type conversion, cast, and round-trip coverage across\n numeric, string, bytes, decimal, UUID, date, time, and timestamp helpers,\n plus fromParquetPrimitiveType, fromSchema, and fromOrdinal.\n- TestHoodieSchemaTypePromotion: promotion matrix (widening, narrowing,\n string\u003c-\u003ebytes, unrelated types) and decimal widening rules.\n- TestM3ScopeReporterAdaptor: registry-to-scope mapping for counters,\n gauges, histograms, meters, and timers using a mocked Scope.\n\nCo-authored-by: sivabalan \u003cn.siva.b@gmail.com\u003e" }, { "commit": "41a8fc881ac8465f6716908a8816208d6f20e17a", "tree": "bc030fc35df9ee940148eef48cd2f84be238cfd3", "parents": [ "d78a7b5baa42eb5cc4d493ed2101e0af35a441e2" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Tue Jul 21 09:24:45 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 21 09:24:45 2026 +0800" }, "message": "fix(mdt): use native log format for file group initialization (#19312)" }, { "commit": "d78a7b5baa42eb5cc4d493ed2101e0af35a441e2", "tree": "351f7798d75e3f22fa9f4f503901f33fd290eb3c", "parents": [ "afec0c9adcce1e67dea0a9fb475834bb6a1bdec9" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Tue Jul 21 09:20:16 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 21 09:20:16 2026 +0800" }, "message": "fix(flink): avoid reusing split reader functions across fetchers (#19315)" }, { "commit": "afec0c9adcce1e67dea0a9fb475834bb6a1bdec9", "tree": "da31ac97ce35968bddede94997dfa99c0db79ff8", "parents": [ "f558c0039457de30bcf4fd6d8ecf1972dd3bc4af" ], "author": { "name": "Shihuan Liu", "email": "skywalker0618@gmail.com", "time": "Mon Jul 20 18:16:21 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 21 09:16:21 2026 +0800" }, "message": "fix(hive-sync): close proxied IMetaStoreClient in HoodieHiveSyncClient.close() to prevent HMS connection leak (#19331)" }, { "commit": "f558c0039457de30bcf4fd6d8ecf1972dd3bc4af", "tree": "ac7f60d2ddd637a222dc7cd40451feddf9861087", "parents": [ "bd2888c6370800b2386cad7daf1e8dd0788519bb" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Mon Jul 20 08:22:45 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 20 23:22:45 2026 +0800" }, "message": "test(client): add unit coverage for client utilities and services (#19222)\n\n* test(client): add unit coverage for client utilities and services\n\n* fix(client): fix TSM client default config and lock owner info; address review findings\n\n- Fix two compile errors in TestHoodieTableServiceManagerClient: import\n HoodieTableType from common.model, and declare the checked\n UnsupportedEncodingException from URLDecoder.decode via throws\n IOException.\n- TABLE_SERVICE_MANAGER_DEPLOY_EXTRA_PARAMS now defaults to the empty\n string: its getter uses getStringOrDefault, which throws for no-default\n properties, so every compact/clean/cluster call through\n HoodieTableServiceManagerClient failed with HoodieException unless\n hoodie.table.service.manager.deploy.extra.params was explicitly set.\n- FileSystemBasedLockProvider.acquireLock writes the owner lock info\n unconditionally after the atomic create: the previous\n \u0027if (!storage.exists(lockFile))\u0027 guard ran right after create() and was\n always false, so lock files were empty and lock-conflict messages from\n LockManager and TimeGeneratorBase carried no owner info.\n- Consolidate filesystem lock provider tests in hudi-client-common: move\n the default-lock-path and non-reentrancy cases from hudi-spark-client\u0027s\n TestFileBasedLockProvider (which used no Spark) into\n TestFileSystemBasedLockProvider and delete the old file.\n- Test nits: drop an inert retry-wait property, import TypedProperties\n instead of inline qualification, use assertEquals over\n assertTrue(equals).\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "bd2888c6370800b2386cad7daf1e8dd0788519bb", "tree": "da64b077f0e87b2ec87ffecd3c7ad54011379e09", "parents": [ "93f2d4c16aaeaf152a9e9f3fc2b67a9607ffb9be" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Mon Jul 20 08:22:31 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 20 23:22:31 2026 +0800" }, "message": "test(common): add unit coverage for config and utility classes (#19220)\n\n* test(common): add unit coverage for config and utility classes\n\n* fix(common): add missing READER group description and address review findings\n\n- Fix checkstyle ImportOrder violation in TestDateTimeUtils\n (java.time.temporal.ChronoUnit must follow java.time.format imports).\n- Add the missing READER case to ConfigGroups.getDescription, surfaced\n by the new placeholder test: the Reader Configs group fell through to\n the \u0027Please fill in the description\u0027 default, which leaks into the\n generated config docs.\n- Strengthen testFormatUnixTimestamp to re-parse the formatted output\n instead of asserting only its length, matching its comment.\n- Import cleanup in TestCollectionUtils and an intent comment on the\n emptyProps singleton assertion.\n- Rename copy-paste-leftover locals in generateChecksum (database/table\n to indexType/indexName) and correct the checksum format comment; the\n produced checksum is byte-identical.\n\n* test(common): make new test classes package-private and align index names with types\n\n- Drop the public modifier from TestConfigGroups and\n TestHoodieIndexingConfig and their test methods, matching the JUnit 5\n package-private idiom used by TestCollectionUtils. TestDateTimeUtils\n keeps public to match its pre-existing file style.\n- Use index names that match the configured index type (idx_bloom,\n idx_record, idx_original) instead of reusing column_stats for\n non-column-stats types, which read confusingly.\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "93f2d4c16aaeaf152a9e9f3fc2b67a9607ffb9be", "tree": "9f4f167cbbe135ea8917b2636ca981d05aeae076", "parents": [ "59ed745c08fd37cd0a43bde4a4bd5650508e128d" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Mon Jul 20 07:49:49 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 20 22:49:49 2026 +0800" }, "message": "test(spark): add unit coverage for Spark row, schema and sort utilities (#19219)\n\n* test(spark): add unit coverage for Spark row, schema and sort utilities\n\n* test(spark): fix single-column range test and address review cleanups\n\n- Replace the single-column linear-order assertion, which relied on\n repartitionByRange sorting within partitions (it does not, per the\n Spark javadoc), with assertions on the actual contract: no Index\n column, non-overlapping ranges across partitions, both requested\n partitions populated, and full row preservation.\n- Reuse BinaryUtil.compareTo as the Z-curve ordering comparator\n instead of a local reimplementation.\n- Stop reaching into cross-module non-public ValueMetadata members:\n construct metadata through the public getValueMetadata factory and\n assert decimal precision and scale via getValueTypeInfo().\n- Import cleanups (assertArrayEquals static import, Metadata import).\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "59ed745c08fd37cd0a43bde4a4bd5650508e128d", "tree": "d0d56d5b827c56b0c5e485933eb078ae6b4f2585", "parents": [ "5178a3bedae848e1322842b1db5f1b03ee439b83" ], "author": { "name": "Ranga Reddy", "email": "rangareddy.avula@gmail.com", "time": "Mon Jul 20 15:40:38 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 20 18:10:38 2026 +0800" }, "message": "fix(hive-sync): sync column and partition column comments to HMS (#19289)\n\nWhen hoodie.datasource.hive_sync.sync_comment is enabled, comments of\npartition columns were never synced to the Hive metastore:\n\n- In hms sync mode, HMSDDLExecutor.updateTableComments only updated the\n columns in the storage descriptor and ignored the table\u0027s partition\n keys, silently dropping partition column comments.\n- In jdbc/hiveql sync modes, the generated ALTER TABLE ... CHANGE COLUMN\n statement targeted the partition column, which Hive rejects with\n \"Invalid column reference\", failing the whole sync.\n- createTable never populated any comments, so even regular column\n comments only appeared from the second sync onwards.\n- The spark schema serialized into the spark.sql.sources.schema table\n property never carried the field docs, so spark DESCRIBE showed no\n comments even when they were present in HMS.\n\nChanges:\n- HMSDDLExecutor.createTable populates column and partition column\n comments from the storage schema field docs when sync_comment is on.\n- HMSDDLExecutor.updateTableComments also updates the partition keys,\n matching AWSGlueCatalogSyncClient.updateTableComments.\n- HiveSchemaUtil.generateCreateDDL (jdbc/hiveql modes) appends\n COMMENT clauses to regular and partition column definitions when\n sync_comment is on.\n- QueryBasedDDLExecutor.updateTableComments skips partition columns\n with a warning since HiveQL has no DDL to alter partition column\n comments, instead of failing the sync.\n- SparkSchemaUtils.convertToSparkSchemaJson can now include field docs\n as the comment of the field metadata; HiveSyncTool enables it for the\n spark.sql.sources.schema table property when sync_comment is on.\n\nFixes #17359 (HUDI-8843), follow-up of #11922 which fixed the same\nproblem for AWS Glue." }, { "commit": "5178a3bedae848e1322842b1db5f1b03ee439b83", "tree": "086c2c44940889d156c13617a351ec6f0ee38d63", "parents": [ "4394513a8cb7a96d03e301255d6da39310d86b47" ], "author": { "name": "Nada", "email": "nada.attia@uber.com", "time": "Sun Jul 19 14:11:08 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Jul 19 11:11:08 2026 -0700" }, "message": "fix(metadata-table): add config to skip zero-size data files on MDT initialization (#18611)\n\nAdds an opt-in config `hoodie.metadata.skip.zero.size.files.on.initialize`\n (default false, advanced) that skips zero-size data files while listing the\n data table during MDT initialization. Prevents init failures on tables \n carrying leftover zero-byte files, and emits `skipped_zero_size_files_on_initialize`\n with the skipped count. \n \n Skip is scoped to the initialize path only — the restore-sync caller of \n `listAllPartitionsFromFilesystem` passes `skipZeroSizeFiles\u003dfalse` \n unconditionally, so files already tracked in MDT are not spuriously deleted. \n Existing `DirectoryInfo` constructors delegate with `false` to preserve \n current behavior. " }, { "commit": "4394513a8cb7a96d03e301255d6da39310d86b47", "tree": "c21ae72a85e6c8997426da0ab50c66b90aeb28bd", "parents": [ "a806d3968488a05a75c756e4dd436fde2f717755" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Sun Jul 19 10:09:30 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Jul 19 17:09:30 2026 +0000" }, "message": "test(common): add log-format reader and scanner coverage (#19223)\n\n* test(common): add log-format reader and scanner coverage\n\n* test(common): fix scanner close/lifecycle and restrict BRAF tests to read path\n\n- Drop non-existent HoodieUnMergedLogRecordScanner.close() calls that\n caused Azure CI to fail compilation of hudi-hadoop-common tests.\n- Refactor TestBufferedRandomAccessFile to seed fixtures via Files.write\n and only exercise the read path. Production only opens the class in\n read mode (\"r\") (BitCaskDiskMap / LazyFileIterable). Exercising its\n write path across buffer boundaries hangs in an infinite loop (dormant\n writer-side bug in expandBufferToCapacityIfNeeded not addressed here).\n\n---------\n\nCo-authored-by: sivabalan \u003cn.siva.b@gmail.com\u003e\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "a806d3968488a05a75c756e4dd436fde2f717755", "tree": "3e0c7e2b337f28e68290cceaa8b08853edff0f5b", "parents": [ "5f7f7660a08bc8f11fd1904fbd9abccf983eb50b" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Sat Jul 18 19:56:39 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Jul 18 19:56:39 2026 +0800" }, "message": "[HUDI-8970] Change scheduleandexecute in RunCompactionProcedure to compact all pending plans too (#12794)" }, { "commit": "5f7f7660a08bc8f11fd1904fbd9abccf983eb50b", "tree": "c8f242e8f0109eea501d9c66c870b9ca712e137b", "parents": [ "d8b75259e7f4ec675dbc1fdc1efda4154def91ee" ], "author": { "name": "Lokesh Jain", "email": "ljain@apache.org", "time": "Fri Jul 17 17:16:01 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 17 11:46:01 2026 +0000" }, "message": "fix(test): stabilize flaky testReattemptOfFailedClusteringCommit (#19120)\n\nTestJavaHoodieBackedMetadata.testReattemptOfFailedClusteringCommit was\nintermittently failing in CI. The post-clustering insert used hardcoded\ncommit time \"0000003\", which sorts lexicographically before the\nreal-timestamp clusteringCommitTime. Writing an out-of-order commit after\nclustering corrupts the metadata table\u0027s file listing state, causing\nvalidateMetadata to fail on affected partitions.\n\nFix: use WriteClientTestUtils.createNewInstantTime() for monotonically\nincreasing commit times." }, { "commit": "d8b75259e7f4ec675dbc1fdc1efda4154def91ee", "tree": "78524b0926ef752cb6f0950b4a083e27753ca1a2", "parents": [ "f6590309b42ae4316e21d3a0b49d6bb1eef4e29b" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Fri Jul 17 16:48:17 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 17 17:48:17 2026 +0800" }, "message": "fix(ci): let validate-source\u0027s RAT check fail on unlicensed files outside Maven modules (#19275)\n\n* fix(ci): let validate-source\u0027s RAT check fail on unlicensed files outside Maven modules\n\nInside a Maven module RAT works: it is declared in ~55 module poms and bound\nto the compile phase, so an ordinary build job fails on a missing ASF header.\nOutside a module nothing checks. Those runs are each scoped to their own\nmodule\u0027s basedir, and the root aggregator does not declare the plugin, so the\nonly run that scans the repository root is the direct `mvn apache-rat:check`\ngoal in validate_source_rat.sh. That script always exits 0:\n\n (bash -c \"mvn apache-rat:check\") || (echo \"Rat Check Failed. [ERROR]\" \u0026\u0026 exit 1)\n echo \"RAT Check Passed [OK]\"\n\nThe `exit 1` runs in a subshell, so it exits the subshell rather than the\nscript, and with no set -e control falls through to the success message.\nOn master the job prints BUILD FAILURE, then \"Too many files with unapproved\nlicense: 19\", then \"Rat Check Failed. [ERROR]\", then \"RAT Check Passed [OK]\",\nand stays green.\n\nPropagate the failure with an if-block instead of the subshell, and point the\nerror at target/rat.txt rather than at a --verbose flag this script does not\naccept.\n\n17 of those 19 violations are phantom. The job stages a copy of the tree into\nhudi-tmp-repo before the RAT step, and the root-anchored exclude\n`hudi-trino-plugin/**` does not cover `hudi-tmp-repo/hudi-trino-plugin/**`, so\nRAT scans the copy. Match at any depth instead. The intent of the exclusion,\nthat hudi-trino-plugin\u0027s licensing is handled by airlift, is unchanged.\n\nThe remaining 2 are real: rfc/rfc-99/variant-appendix.md and\nrfc/rfc-99/vector-appendix.md are the only 2 of the 54 rfc markdown files with\nno ASF header. Add it, copied verbatim from the sibling rfc-99.md.\n\nCloses #19274\n\n* addressed review comments: keep the CI scratch copy out of the tree RAT scans\n\n`mvn apache-rat:check` scans the filesystem from the repository root, so the full copy of the tree that the validate-source job stages into `hudi-tmp-repo` is scanned as well, and every root-anchored exclude in the root pom silently stops applying to it. That is where the 17 phantom violations came from: all of them sit under `hudi-tmp-repo/hudi-trino-plugin`.\n\nBroadening the exclude to `**/hudi-trino-plugin/**` treated the symptom and loosened the pattern for good. Stage the copy in the runner temp directory instead, so RAT sees only the real tree: no exclude has to change and license coverage stays exactly as it is on master. The release path already behaves this way, since validate_source_rat.sh runs inside the extracted source tarball, which contains no nested copy. The pom exclude is reverted to `hudi-trino-plugin/**`.\n\n---------\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e" }, { "commit": "f6590309b42ae4316e21d3a0b49d6bb1eef4e29b", "tree": "c5b41b515b78f5a8515edea5131c4a764cc26356", "parents": [ "79908efc59c4b4231cf1405252ed58ea89bcca88" ], "author": { "name": "Danny Chan", "email": "yuzhao.cyz@gmail.com", "time": "Fri Jul 17 16:09:13 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 17 16:09:13 2026 +0800" }, "message": "fix: fix the file size calibration for native log format (#19293)\n\n* fix: fix the file size calibration for native log format\n\n* fix variable name\n\n* remove the check for parquet log block since we have native log now\n\n* fix log file size calibration by block format\n\n* address Spark file sizing review comments" }, { "commit": "79908efc59c4b4231cf1405252ed58ea89bcca88", "tree": "57dead58489ee6d9fe0c1a9fd7e09f7a41f7e6dd", "parents": [ "d10b868d90c97bd7297c0ac18fcb7fcf3deac8af" ], "author": { "name": "ashokkumar-allu", "email": "65997235+ashokkumar-allu@users.noreply.github.com", "time": "Fri Jul 17 03:05:55 2026 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 17 16:05:55 2026 +0800" }, "message": "docs: claim RFC-108 Multi-dataset incremental reads in Hudi Streamer (#19308)\n\nAdd the RFC-108 reservation to rfc/README.md.\n\nCo-authored-by: gallu \u003cgallu@uber.com\u003e" }, { "commit": "d10b868d90c97bd7297c0ac18fcb7fcf3deac8af", "tree": "9d72eda0da56ec32183df74614089a03ccaf8428", "parents": [ "442261014f396a24cdbddb5df2b7e80e7bcce991" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Fri Jul 17 07:51:08 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 17 08:51:08 2026 +0800" }, "message": "fix(flink): remove the Source V2 read teardown race via materialized bounded minibatches (#19202)\n\n* fix(flink): make CDC read iterator teardown thread-safe to de-flake stream-read ITs\n\nCdcFileSplitsIterator is drained on the Flink task thread but can be closed from the split-fetcher thread during job teardown, racing on its non-volatile recordIterator/imageManager fields (and the nested BaseImageIterator.cdcItr) and surfacing an NPE that flakes testStreamReadFromSpecifiedCommitWithChangelog.\n\nSynchronize hasNext()/next()/close() on the iterator and add a closed flag so a concurrent close() cannot null out state mid-read and the iterator reports a well-defined end-of-stream afterwards. Extend the test\u0027s isAcceptableTerminalFailure teardown-race tolerance to the CDC-iterator frame, since BatchRecords drives hasNext()/next() as separate calls and a close() between them still ends a force-terminated drain with a benign NoSuchElementException.\n\n* addressed review comments: split next() guard and use class refs for CDC teardown-frame check\n\n* test(flink): retry a bare await timeout in the CDC stream-read IT\n\nThe flink2.1 CI shard failed testStreamReadFromSpecifiedCommitWithChangelog with\n\"Unexpected job failure\" caused by a TimeoutException from tableResult.await(): the\nstreaming read had not collected its expected rows within the window (CI-load slowness,\nnot the teardown race this PR already tolerates and retries). A bare TimeoutException\nwas rethrown as an AssertionError, bypassing submitAndFetchWithRetry, which only retries\nshort results.\n\nTreat a bare await timeout as a retryable short read: cancel the still-running job and\nreturn the rows collected so far so the retry loop re-submits a fresh job. Also widen\nthe await window from 30s to 60s so a slow shard is less likely to time out at all.\nTest-only; no production change.\n\n* addressed review comments: use equals for CDC teardown-frame class match\n\n* addressed review comments: redesign Source V2 read path to materialize bounded minibatches\n\nThe reviewers asked for a root-cause fix rather than the CdcFileSplitsIterator synchronization mitigation. The real flaw was that BatchRecords did lazy reading: it held a live ClosableIterator drained on the Flink task thread, while a forced-cancel teardown closed that same iterator on the split-fetcher thread, so read and close genuinely ran on different threads.\n\nThis reworks the Source V2 read path so a reader function is a per-split cursor that owns the record iterator, the CdcImageManager and the file-group readers. HoodieSourceSplitReader.fetch() opens a split, reads one bounded minibatch (DEFAULT_MINI_BATCH_SIZE), and closes the split\u0027s resources on the same split-fetcher thread on EOF, failure or cancellation. BatchRecords now carries a materialized list instead of a live iterator, so read and close happen on one thread and the cross-thread teardown race is removed at its source; this also covers the MOR/COW Parquet teardown race, not just the CDC one.\n\nBecause Flink\u0027s columnar readers and the CDC/MOR row projections return the same reused RowData on every next(), readBatch deep-copies each record before buffering it and re-applies the RowKind, which the serializer does not round-trip.\n\nWith read and close single-threaded, the CdcFileSplitsIterator synchronization and closed-flag guards are reverted, and the test-side teardown-race tolerance in ITTestHoodieDataSource is removed: isAcceptableTerminalFailure now accepts only the SuccessException happy path. The independent await-timeout retry is kept.\n\n* addressed review comments: narrow the await-timeout retry to a bare top-level TimeoutException\n\nTableResult.await(long, TimeUnit) throws its own timeout bare, so only a top-level TimeoutException means the await window elapsed with the job still running. A genuine job failure arrives wrapped in an ExecutionException that may itself embed a TimeoutException (checkpoint expiry, RPC timeout); walking the whole cause chain misclassified such a failure as a slow shard, cancelled and retried it, and then reported a row-count mismatch with the real cause discarded. isAwaitTimeout now inspects only the top-level exception.\n\n* addressed review comments: regroup copySerializer field and rename test open-count accessor\n\n* addressed review comments: drop redundant RowKind re-apply after RowDataSerializer.copy\n\n* addressed review comments: unblock fetch() on wakeUp and close the file-group reader on init failure\n\nP1: fetch() now drains an I/O-backed minibatch, so wakeUp() sets a volatile flag the drain polls between records; on a wake-up fetch() returns promptly without finishing/closing, keeping the split-close on the fetcher thread.\nP2: HoodieSplitReaderFunction.createRecordIterator retains the HoodieFileGroupReader in a local and closes it when getClosableIterator() fails, suppressing the close error onto the original exception.\n\n---------\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e" }, { "commit": "442261014f396a24cdbddb5df2b7e80e7bcce991", "tree": "6857affb76c2bbee2b482f24efe73bcf7606c596", "parents": [ "89da9601b149c32b364db84f6ad092d26e8e76fd" ], "author": { "name": "vinoth chandar", "email": "vinothchandar@users.noreply.github.com", "time": "Thu Jul 16 12:09:36 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 16 12:09:36 2026 -0700" }, "message": "refactor(common): utils consolidation sweep remnants (#19284)\n\nFollow-ups from the #19212 review, all behavior-preserving:\n\n- SecondaryIndexKeyUtils: rename getUnescapedSecondaryKey[Prefix]FromSecondaryIndexKey\n to getEscapedSecondaryKey[Prefix]FromSecondaryIndexKey -- both return the\n still-encoded segment (the sibling that actually unescapes calls them), so\n the old names said the opposite of what the methods do.\n- Delete HoodieHiveUtils#getNthParent, a byte-identical copy of\n TablePathUtils\u0027 private walker; make the latter public and point the one\n remaining caller (HoodieROTablePathFilter) at it.\n- Fold CommonClientUtils#generateWriteToken into FSUtils as a\n makeWriteToken(TaskContextSupplier) overload next to its 3-arg sibling;\n migrate the single caller (RollbackHelperV1).\n\nAlso audited all HoodieAvroUtils#unwrapNullable call sites for multi-member\nunion exposure (follow-up from the nullable-union dedup): no reachable\nmis-resolution -- variant-shredding sites operate on internally-constructed\nschemas, and the one arbitrary-schema site (projectFields) either discards\nthe unwrapped result or immediately validates it is a RECORD and throws." }, { "commit": "89da9601b149c32b364db84f6ad092d26e8e76fd", "tree": "1992ac192c3c59bcdd0a397aee233f03e3efa1bd", "parents": [ "e098da78c501072e20da4cef4212ed84c3918cf6" ], "author": { "name": "chrevanthreddy", "email": "27821245+chrevanthreddy@users.noreply.github.com", "time": "Thu Jul 16 01:47:13 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 16 13:47:13 2026 +0800" }, "message": "docs: claim RFC-109 Hudi Native Vector Index (#19303)\n\nCo-authored-by: Revanth Chandupatla \u003crevanth.chandupatla@walmart.com\u003e" }, { "commit": "e098da78c501072e20da4cef4212ed84c3918cf6", "tree": "d0f8a3e480db1a1bf0de0468df66dcee714d5456", "parents": [ "b0a2563e25e533d195b24f3b88103b99d91ade28" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Wed Jul 15 20:04:41 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 15 21:04:41 2026 +0800" }, "message": "fix(ci): make the source-release binary-file check actually run (#19267)\n\n* fix(ci): make the source-release binary-file check actually run\n\nvalidate_source_binary_files.sh pipes every file through `file -I`, the\nBSD/macOS spelling of the MIME option. GNU file rejects it, printing usage\nto stderr and nothing to stdout, so the pipeline\u0027s `wc -l` always counts\nzero. errexit does not catch it because the pipeline sits in a command\nsubstitution and pipefail is not set. The check has therefore never fired on\nLinux, which is every CI runner.\n\nAsk libmagic for the charset instead of the MIME type. `file --mime` is the\nlong option for both GNU\u0027s `-i` and BSD\u0027s `-I`, and its charset\u003dbinary field\nanswers the question directly, without depending on libmagic\u0027s growing type\nvocabulary: the previous MIME allowlist already yields five false positives\non master, where libmagic labels the JSON test data application/x-ndjson and\nmisdetects one .java source as application/javascript. Empty files, which\nlibmagic also reports as charset\u003dbinary, are exempted alongside the existing\nrelease/ and src/test/ exemptions.\n\nAdd pipefail so a broken `file` invocation fails loudly rather than reading\nas \"no binary files found\", run `file` once over a batched find/xargs instead\nof one process per path, and build the report from that single pass.\n\nRun the check where it is meant to run. Its contract is the root of a source\nrelease tree, which is how validate_staged_release.sh invokes it, but the\nvalidate-source job ran it against the raw checkout, where rfc/, docker/images/,\nhudi-notebooks/ and .idea/ still exist. bot.yml now creates the source release\ndirectory once and runs both the binary and the copyright check inside it.\n\nCloses #19266\n\n* addressed review comments: make binary-file grep chain byte-safe and pin the release/ exemption to top level\n\n---------\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e" }, { "commit": "b0a2563e25e533d195b24f3b88103b99d91ade28", "tree": "f0bf93fd5d809af42096f520eefb2624150c2136", "parents": [ "6543aec2553ecbc0dec84d5a88aa7b09292ffc9b" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Wed Jul 15 19:04:18 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 15 12:04:18 2026 +0000" }, "message": "fix(build): declare lombok in annotationProcessorPaths to deflake parallel-build compilation (#19020)\n\n* fix(build): declare lombok in annotationProcessorPaths to deflake parallel-build compilation\n\nParallel reactor builds (mvn -T) intermittently fail testCompile with \"Provider lombok... could not be instantiated\" because the processor is discovered from the shared compile classpath via ServiceLoader. Declaring lombok explicitly in the compiler plugin\u0027s annotationProcessorPaths gives each compilation an isolated processor path, removing the race. Lombok is the only annotation processor with work in the build, so no other processing is affected.\n\n* addressed review comments: note JDK 23+ processor-path benefit in the compiler-plugin comment\n\n---------\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e" }, { "commit": "6543aec2553ecbc0dec84d5a88aa7b09292ffc9b", "tree": "d822b1ddb6722cbfe1d478fabb9cfd094275209e", "parents": [ "59e62dd488cd964c3125e9cb96d0089f9148c0be" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Wed Jul 15 18:42:43 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 15 18:42:43 2026 +0800" }, "message": "feat(lance): enable lance as native log format (#19283)\n\n* feat(lance): enable lance as native log format" }, { "commit": "59e62dd488cd964c3125e9cb96d0089f9148c0be", "tree": "e3b812eaf3a8a7109c91966cfff12eb481fa5438", "parents": [ "502f0cea086663c41ac0268c4299fda6ad579478" ], "author": { "name": "Vinish Reddy", "email": "vinishreddygunner17@gmail.com", "time": "Wed Jul 15 12:12:54 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 15 06:42:54 2026 +0000" }, "message": "fix: Improve error message for conflict resolution (#18119)\n\n* Improve error message for conflict resolution\n\n* Fix build\n\n* Address comments" }, { "commit": "502f0cea086663c41ac0268c4299fda6ad579478", "tree": "315da530b4311f893191b6727854e41cbc368386", "parents": [ "ac179e48a0c111532fe37682016f46132e96cc6d" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Wed Jul 15 14:14:30 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 15 14:14:30 2026 +0800" }, "message": "docs(rfc): [HUDI-9315] Mark RFC-94 (Timeline UI) as IN PROGRESS (#19294)\n\nImplementation of the Hudi Timeline UI has landed, so move RFC-94 from\nUNDER REVIEW to IN PROGRESS in the RFC index." }, { "commit": "ac179e48a0c111532fe37682016f46132e96cc6d", "tree": "0be70a9170cfc6c178ffbad5cf886598a8a02dfb", "parents": [ "1dfbdcb265ea470837a30f432079245f3a76fb45" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Wed Jul 15 12:58:26 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 15 12:58:26 2026 +0800" }, "message": "feat(spark): Support data skipping based on partitioned RLI (#19013)" }, { "commit": "1dfbdcb265ea470837a30f432079245f3a76fb45", "tree": "27dbb6df2ab1f12a431c286e7abb25f807a6ecf9", "parents": [ "29a4925a0f01f121f17623e19a9afa562bcef567" ], "author": { "name": "vamsikarnika", "email": "vamsikarnika@gmail.com", "time": "Tue Jul 14 13:52:56 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 14 08:22:56 2026 +0000" }, "message": "fix: reuse Inflater/Deflater in BitCaskDiskMap to avoid JDK8 finalizer contention (#18818)\n\n* fix(common): reuse Inflater/Deflater in BitCaskDiskMap to avoid JDK8 finalizer contention\n\nCompressionHandler currently allocates a new Deflater on every compressBytes()\nand a new Inflater on every decompressBytes(). On JDK 8 both classes register\na Finalizer on construction. Under sustained, multi-threaded disk-map traffic\n(observed during MDT/RLI compaction merging millions of records across several\nSpark task threads on the same executor), the rate of zlib allocations exceeds\nthe rate at which the single Finalizer thread can drain its queue. Native\nZStreamRef handles pile up in old gen, the heap saturates, and G1 enters a\nmixed-GC death spiral while application threads make no progress.\n\nCompressionHandler is already held in a ThreadLocal, so a single Deflater/\nInflater pair per worker thread is sufficient. This change:\n\n - Adds transient Deflater/Inflater fields and lazy accessors (transient so\n the class remains Serializable; lazy so deserialized instances rebuild\n the codecs on first use).\n - Calls reset() on the cached codecs at the start of each call.\n - Passes the user-supplied codecs to DeflaterOutputStream(out, def) and\n InflaterInputStream(in, inf), which sets usesDefaultDeflater/Inflater\n to false so close() does not call end() on the codec — the codec\n survives the try-with-resources for reuse on the next call.\n\nOn-disk format, compression level, and error semantics are unchanged.\nAllocation rate drops from O(records) to O(threads). On JDK 9+ this also\nremoves per-call Cleaner registration overhead.\n\n* remove extra comments\n\n* address comments\n\n* address nits\n\n* test(common): add micro-benchmark for Inflater/Deflater reuse\n\nAdds InflaterDeflaterReuseRLIBenchmark to quantify the gain from reusing\nDeflater/Inflater per worker thread vs allocating per call, using a realistic\nRLI HoodieMetadataPayload (kryo-serialized HoodieRecord wrapper, ~156 bytes --\nmatches what BitCaskDiskMap actually compresses). InspectRLISize captures the\nper-record serialized + deflated sizes.\n\n* chore: re-trigger CI" }, { "commit": "29a4925a0f01f121f17623e19a9afa562bcef567", "tree": "f46ea66e7289be308e4f8c49393e17eacd3b549c", "parents": [ "43b2926ce3970fe9230afaf8d669c9f65bf7d2f9" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Tue Jul 14 12:44:17 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 14 12:44:17 2026 +0800" }, "message": "feat(flink): support nested data type for flink lance reader/writer (#19242)\n\n* feat(flink): support nested data type for flink lance reader/writer" }, { "commit": "43b2926ce3970fe9230afaf8d669c9f65bf7d2f9", "tree": "5fd0218d8bba3e22d9cf0b0a3c4be4c296a2c007", "parents": [ "2bed01a98acd11371a0cc2cf16ca32d0d08a1350" ], "author": { "name": "Sivabalan Narayanan", "email": "n.siva.b@gmail.com", "time": "Mon Jul 13 18:20:50 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 14 06:50:50 2026 +0530" }, "message": "feat(tools): add TimelineInspector standalone Java tool for timeline forensics (#19238)\n\n* feat(tools): add TimelineInspector standalone Java tool for timeline forensics\n\nAdds `org.apache.hudi.tools.TimelineInspector`, a pure-Java CLI for inspecting a\nHudi table\u0027s active + archived timeline without needing Spark. Lives in\n`hudi-hadoop-common` since it uses Hadoop `FileSystem` APIs for raw archive log\nscanning and per-instant mtime lookups.\n\nFive modes:\n- `--show-instant` — dump one instant\u0027s content (all states + timeline location)\n- `--find-file-id` — find which instants touched a file id / partition / file group\n- `--commit-stats` — per-instant stats (numWrites/numInserts/numUpdateWrites/numDeletes,\n bytes, partitions touched) with optional filters\n- `--parse-filename` — decode a base or log file name into {fileId, commitTime,\n writeToken, ...}\n- `--raw-archive` — dump raw archive log entries for a specific instant\n- `--phase-timings` — per-instant phase durations (requested→inflight→completed for\n data table + MDT sub-table) for latency forensics\n\nModifiers: `--lifecycle`, `--output table|json`, `--state`, `--actions`,\n`--start-instant`, `--end-instant`, `--limit`, `--sort asc|desc`, `--quiet`,\n`--no-archived`.\n\nRobust against archived-timeline storage quirks: Avro-container vs JSON-string sniff,\nunion-tag fixup walker for malformed JSON, `instantsRollback` checked alongside legacy\n`commitsRollback`, and a JSON-tree aggregator fallback for `--commit-stats` when the\nPOJO path yields an empty shell.\n\nHandles V2 timeline layout: reads instant files under `\u003cbase\u003e/.hoodie/timeline/`,\nresolves completed-instant filenames with the `_\u003ccompletionTime\u003e` suffix via the\nmetaclient\u0027s `InstantFileNameGenerator`. For the phase-timings mode, the MDT\ncompleted-file lookup globs the timeline dir by `\u003crequestedTime\u003e` prefix +\n`.deltacommit` suffix (we don\u0027t know the completion time a priori).\n\nTest coverage: `TestTimelineInspector` — 32 tests covering every mode + argument\nvalidation + phase-timings happy path / limit / drop-on-missing-state / MDT\nfallback / replacecommit opt-in / JSON aggregates. Uses `@TempDir` and synthetic\ninstants — no Spark dependency.\n\nRunbook in `hudi-hadoop-common/src/main/java/org/apache/hudi/tools/TimelineInspector.md`\n(classpath setup, per-mode usage + examples, troubleshooting).\n\nSigned-off-by: sivabalan \u003cn.siva.b@gmail.com\u003e\n\n* fix(tools): improve TimelineInspector correctness, testability, and storage abstraction\n\n - Fix MDT timeline path for V1 layout tables by constructing MDT metaClient\n - Fix commitStats totals computed over all rows instead of only capped subset\n - Remove dead skipReasons variable and side-effect on Args\n - Replace Hadoop FileSystem casts with HoodieStorage abstraction\n - Extract run() method to eliminate deprecated SecurityManager in tests\n\nSigned-off-by: codope \u003csagarsumit09@gmail.com\u003e\n\n* fix(tools): fix ExitException swallowed by catch(Exception), remove dead code\n\n- Narrow catch(Exception) to catch(IOException | RuntimeException) so\n ExitException propagates with its original exit code instead of being\n re-wrapped as exit code 1\n- Add V1/V2 fallback path probing when MDT hoodie.properties is absent\n- Remove unreachable helpRequested guard after the arg-parse loop\n\nSigned-off-by: codope \u003csagarsumit09@gmail.com\u003e\n\n---------\n\nSigned-off-by: sivabalan \u003cn.siva.b@gmail.com\u003e\nSigned-off-by: codope \u003csagarsumit09@gmail.com\u003e\nCo-authored-by: codope \u003csagarsumit09@gmail.com\u003e" }, { "commit": "2bed01a98acd11371a0cc2cf16ca32d0d08a1350", "tree": "1a4d1c4d8f8475c7387696f6841a9ea89070f4df", "parents": [ "72c9558ca9ca859a36888e8cedc4a28c0f744f18" ], "author": { "name": "vinoth chandar", "email": "vinothchandar@users.noreply.github.com", "time": "Mon Jul 13 18:07:53 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 13 21:07:53 2026 -0400" }, "message": "refactor(common): consolidate *Utils classes (dead code, fold-ins, renames, dedup) (#19212)\n\n* refactor(common): remove dead code and duplicate helpers from *Utils classes\n\nBehavior-preserving cleanup from the *Utils consolidation sweep across\nhudi-io, hudi-common, and hudi-hadoop-common:\n\n- Delete ActiveTimelineUtils (empty class) and fix stale comment reference\n- Delete HadoopFSUtils#getFileSize(FileSystem,Path) and\n #getFileIdFromFilePath(Path) (no callers)\n- Delete HadoopConfigUtils#getBooleanWithAltKeys (test-only) and its test\n- Delete TableServiceUtils#isStale(long,long) overload (test-only) and its test\n- Dedupe IOUtils#copy(InputStream,OutputStream) / #readAsByteArray against\n the byte-identical FileIOUtils twins in the same package (keep FileIOUtils)\n- HoodieSchemaUtils#convertBytesToBigDecimal(byte[],int,int) now delegates to\n the byte-identical HoodieAvroUtils implementation\n- Collapse StringUtils private stringIsNullOrEmpty into public isNullOrEmpty\n\nVerified no reflection-based use: repo-wide scan for deleted names as string\nliterals is clean. CommitUtils#buildMetadataFromInstant and\nHadoopFSUtils#getAllDataFilesInPartition were candidates but are kept: each\nhas a single Scala caller (TimelineRelation, ShowInvalidParquetProcedure);\nmigrating those callers belongs to the follow-up consolidation PRs.\n\n* refactor(common): fold six tiny single-use *Utils classes into natural homes\n\nBehavior-preserving continuation of the *Utils consolidation sweep; each\nfold deletes a class with 1-3 caller edits:\n\n- ArrayUtils (one method) -\u003e CollectionUtils#toPrimitive\n- HadoopInLineFSUtils -\u003e callers use InLineFSUtils directly; buildInlineConf\n inlined into HadoopStorageConfiguration#getInline; the Hadoop-Path adapter\n becomes a private helper in InLineFileSystem, its only user\n- HoodieColumnStatsIndexUtils (two thin delegates) -\u003e HoodieMetadataWriteUtils\n- InputSplitUtils -\u003e writeString/readString become static interface methods\n on RealtimeSplit (its only caller); trivial boolean wrappers replaced with\n direct DataOutput/DataInput calls. Kept Java-8 compatible since\n hudi-hadoop-mr pins compiler release 8.\n- HoodieIncrSourceCheckpointValUtils -\u003e folded into package sibling\n checkpoint.CheckpointUtils; tests merged into TestCheckpointUtils\n- ThreadUtils -\u003e collectActiveThreads inlined into PulsarSource, its only\n caller\n\nVerified no reflection-based use (repo-wide string-literal scan) and no\nremaining references across all file types including Scala.\n\n* refactor(common): rename colliding *Utils class names\n\nFive pairs of same-name/different-purpose utility classes made call sites\nambiguous. Pure renames, no logic changes:\n\n- common.util.CheckpointUtils -\u003e KafkaCheckpointUtils (parses Kafka-offset\n checkpoint strings; distinct from table.checkpoint.CheckpointUtils which\n translates Streamer checkpoint V1/V2 and keeps the name). Its test becomes\n TestKafkaCheckpointUtils, resolving a TestCheckpointUtils collision too.\n- org.apache.hudi.io.hfile.HFileUtils (hudi-io) -\u003e HFileByteUtils (low-level\n byte/key primitives; the FileFormatUtils subclass in hudi-common keeps\n HFileUtils, symmetric with ParquetUtils/OrcUtils/LanceUtils)\n- org.apache.hudi.io.IOUtils (hudi-client-common) -\u003e MergeUtils\n (merge/compaction memory sizing + runMerge; distinct from hudi-io\u0027s\n byte-primitives IOUtils). TestSparkIOUtils -\u003e TestSparkMergeUtils.\n- client.heartbeat.HeartbeatUtils -\u003e WriterHeartbeatUtils (writer-side\n delete/abort; distinct from hudi-common\u0027s read/expiry HoodieHeartbeatUtils)\n- index.secondary.SecondaryIndexUtils -\u003e SecondaryIndexDefinitionUtils\n (JSON serde of index definitions, unrelated to the SI key codec)\n\n* refactor(config): collapse HadoopConfigUtils alt-key reader onto ConfigUtils\n\nHadoopConfigUtils re-implemented the key-then-alternatives walk (with its own\ncopy of the deprecation warning) against Hadoop Configuration. Extract the\nwalk as ConfigUtils.getRawValueWithAltKeys(Function\u003cString,Object\u003e, ...) --\nmirroring the Properties overload\u0027s no-default-fallback semantics -- and make\nthe Hadoop variant a one-line adapter over conf::get. HadoopConfigUtils\nshrinks to createHadoopConf + the adapter. Behavior-preserving: same lookup\norder, same Option semantics, same deprecation warning text.\n\n* refactor(common): delegate hadoop-mr table-path resolution to TablePathUtils\n\n- HoodieInputFormatUtils#getTableMetaClientForBasePathUnchecked re-implemented\n TablePathUtils#getTablePath\u0027s two-branch walk (partition metadata -\u003e nth\n parent; else walk up to .hoodie). Now delegates, throwing\n TableNotFoundException on empty like before.\n- Fix latent bug in TablePathUtils#getTablePathFromPartitionPath exposed by\n the delegation: the parent walk assumed StoragePath#getParent returns null\n at the root, but it throws IllegalStateException, so non-Hudi paths crashed\n instead of returning Option.empty(). Guard on depth()\u003d\u003d0; add unit test.\n- Migrate ShowInvalidParquetProcedure off the FileSystem-based\n HadoopFSUtils#getAllDataFilesInPartition to the HoodieStorage-based\n FSUtils variant (it already had a HoodieStorage in scope) and delete the\n Hadoop twin, which was kept alive solely for this caller.\n\nVerified: TestInputPathHandler (exercises non-Hudi path classification),\nTestTablePathUtils (incl. new root-walk test), TestShowInvalidParquetProcedure\n(ScalaTest, end-to-end incl. delete path), TestHoodieRealtimeInputFormatUtils.\n\n* refactor(common): move record reconstruction from SpillableMapUtils to HoodieAvroUtils\n\nThe three convertToHoodieRecordPayload overloads (+ private getPreCombineVal)\nbuild a HoodieAvroRecord from an Avro record: extract key/partition from meta\nfields, compute the ordering value, optionally strip meta fields, and\ninstantiate the payload. Nothing about them is specific to spillable maps --\nthey were only historically located there, and their sole production caller\nis HoodieAvroUtils#createHoodieRecordFromAvro (the dispatcher).\n\nRelocate them next to the dispatcher in HoodieAvroUtils; SpillableMapUtils\nnow contains only spill I/O (readBytesFromDisk/spillToDisk/computePayloadSize).\nBehavior-preserving; one test call site updated.\n\n* refactor(core): dedupe commit-metadata reads and reverse-instant scans in timeline utils\n\n- RepairUtils#getBaseAndLogFilePathsFromTimeline hand-rolled the same\n commit-vs-replace-commit metadata dispatch that TimelineUtils#getCommitMetadata\n encapsulates; collapse the two switch arms into one that delegates. The\n load-bearing default -\u003e Option.empty() arm for unsupported actions stays.\n- Extract TimelineUtils#findLatestInCommitMetadata: scan reverse-ordered\n instants, read each commit metadata, return the first non-empty extraction.\n TimelineUtils#getLatestInstantAndCommitMetadataWithValidCheckpointInfo and\n CommitUtils#getValidCheckpointForCurrentWriter were near-identical copies of\n this loop; both become 5-line specializations.\n- RestoreUtils#getRestorePlan used getRollbackRequestedInstant instead of\n getRestoreRequestedInstant. Both InstantGenerator implementations (V1/V2)\n are currently identical for the two, so this is behavior-preserving, but\n the restore-specific hook is the correct one to call.\n\nConsidered and rejected: a generic requested-plan reader template across\nCompaction/Cleaner/Clustering/Rollback/RestoreUtils -- after recent upstream\nrefactoring those methods share only ~3 lines each with differing checked\nexception contracts; a template would add indirection without removing\nmeaningful duplication.\n\n* refactor(common): eliminate private duplicates of nullable-union unwrapping\n\nTwo private copies of the lenient nullable-union resolver existed alongside\nthe canonical helpers:\n\n- Spark4VariantShreddingProvider#unwrapNullable(Schema) was byte-identical to\n HoodieAvroUtils#unwrapNullable; use the canonical one via static import.\n- HoodieVariantReconstruction#unwrapNullable(HoodieSchema) computed\n isNullable() ? getNonNullType() : schema, which is provably equivalent to\n HoodieSchema#getNonNullType() alone in every case (non-union and non-null\n unions return `this`); call it directly.\n\nThe remaining resolvers each keep distinct, load-bearing semantics and are\nintentionally NOT merged: AvroSchemaUtils#getNonNullTypeFromUnion (strict,\nthrows on multi-member unions), HoodieAvroUtils#unwrapNullable (lenient,\nfirst non-null member), HoodieAvroUtils#getActualSchemaFromUnion (data-driven\nresolution for complex unions), HoodieSchemaUtils#resolveUnionSchema\n(by-fullname match), and AvroOrcUtils#getActualSchemaType (ORC-specific:\nall-null union maps to a NULL schema).\n\n* refactor(core): pull duplicated iterator boilerplate up into FileFormatUtils\n\nThree methods on the FileFormatUtils contract had byte-identical bodies in\nParquetUtils, OrcUtils, and LanceUtils; convert them from abstract to\nconcrete on the base class:\n\n- getHoodieKeyIterator(storage, filePath) delegates to the 4-arg overload\n- fetchRecordKeysWithPositions(storage, filePath) delegates to the 4-arg one\n- fetchRecordKeysWithPositions(storage, filePath, keyGen, partitionPath)\n wraps getHoodieKeyIterator in a position-tracking CloseableMappingIterator\n\nOrcUtils keeps its 4-arg fetch override (adds an existence guard) and\nHFileUtils keeps its UnsupportedOperationException overrides so behavior is\nunchanged everywhere; only the redundant Parquet/Orc/Lance copies are gone.\n\n* fix(core): checkState(boolean, Supplier) throws IllegalStateException\n\nThe Supplier overload of ValidationUtils#checkState threw\nIllegalArgumentException -- a copy-paste slip from the checkArgument family;\nboth sibling checkState overloads and the javadoc\u0027d contract throw\nIllegalStateException. Repo-wide audit found no callers of this overload and\nno code catching IllegalArgumentException around checkState call sites, so\nthe thrown-type change has no observable impact today. Add a test pinning\nthe exception contract of all checkArgument/checkState overloads.\n\n* style(common): fix import ordering nits from utils consolidation sweep\n\nSeven checkstyle violations (ImportOrder / blank-line) introduced by the\nscripted import swaps in the preceding commits; found by running the\ncheckstyle+scalastyle lifecycle gates locally across all touched module\nchains, which now pass with zero violations.\n\n* refactor(core): reconcile utils sweep with latest master (Vortex format)\n\n- VortexUtils (new FileFormatUtils subclass on master) carried the same three\n iterator boilerplate methods this branch pulled up into the base class;\n drop the now-redundant overrides (byte-identical to the inherited impls).\n- Restore HoodieCommitMetadata/HoodieException imports in\n HoodieMetadataWriteUtils: master no longer uses them in this file but the\n relocated updateColsToIndex does (rebase fallout).\n\n* refactor(common): address utils-sweep review feedback\n\n- Rename HoodieAvroUtils#convertToHoodieRecordPayload -\u003e convertToRecord\n (review: name described the payload era, not what it returns).\n- Add direct unit test for convertToRecord covering meta-field key/partition\n extraction, preCombine ordering value, explicit partition-name override,\n and meta-field stripping via schemaWithoutMetaFields.\n- getTableMetaClientForBasePathUnchecked: map FileNotFoundException from the\n TablePathUtils lookup to TableNotFoundException, preserving the historical\n contract for nonexistent input paths; add test." }, { "commit": "72c9558ca9ca859a36888e8cedc4a28c0f744f18", "tree": "3a47b3855e6cd534bb68cd132d720e3e2f135a1b", "parents": [ "a4cde3a1d0bcba5c4a179edd0d0a78faa5f0f556" ], "author": { "name": "Trivedhi", "email": "85494284+Rajeev-01@users.noreply.github.com", "time": "Tue Jul 14 06:37:41 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 14 09:07:41 2026 +0800" }, "message": "[MINOR] Add restore action in all commits timeline (#19277)" }, { "commit": "a4cde3a1d0bcba5c4a179edd0d0a78faa5f0f556", "tree": "8c5554f14433bcbbb5b8ca1766effac2448221c9", "parents": [ "770e8aabf367b51b56912e4f9120fe0500334198" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Mon Jul 13 09:19:25 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 13 16:19:25 2026 +0000" }, "message": "test(spark): cover the legacy parquet read path with file-group reader disabled (#19133)\n\n* test(spark): cover the legacy parquet read path with file-group reader disabled\n\n* test(spark): exercise legacy parquet read path on nested struct/array columns\n\nAddress review feedback on #19133:\n- Add a nested struct and an array column so the legacy parquet reader is\n driven through its complex-type branch (the vectorized nested-column path,\n historically fragile per HUDI-7190/#10265), not just flat scalar columns.\n- Sort compared rows by the named \u0027id\u0027 column instead of a positional index.\n\n* test(spark): keep vectorized nested-column coverage on spark3.3\n\nThe nested struct/array columns flip the legacy parquet reader\u0027s\nsupportBatch off on spark3.3, where\nspark.sql.parquet.enableNestedColumnVectorizedReader defaults to false\n(true only from spark3.4). That made testCowSnapshotReadEqualsFileGroupReader\nand testCowSnapshotReadWithoutVectorizedReader both fall back to the\nparquet-mr row path on 3.3, dropping the vectorized nested-column branch\nthe suite targets.\n\nEnable nested-column vectorization in setUp so the vectorized branch runs\non every Spark profile, and assert supportBatch is true in the vectorized\ncase and false in the non-vectorized case so a Spark default change cannot\nsilently collapse the two paths again.\n\n* test(spark): cover legacy reader implicit int-\u003elong type-change path\n\nThe prior cases all wrote and read the same schema, so typeChangeInfos\nstayed empty and shouldUseInternalSchema never became true in the\nSparkXXLegacyHoodieParquetFileFormat shims. The suite never reached the\nbranch that swaps Spark\u0027s stock VectorizedParquetRecordReader for Hudi\u0027s\nHoodieVectorizedParquetRecordReader -- the Hudi-specific reason these\nper-version copies exist.\n\nAdd testCowSnapshotReadWithImplicitTypeChange: commit 1 writes `value` as\nint32; commit 2 upserts only the p0 rows with a widened long value too\nlarge for an int, promoting the table schema to long. In COW the p1/p2\nbase files keep the narrower physical int, so reading them against the\nlong table schema goes through HoodieParquetFileFormatHelper\n.buildImplicitSchemaChangeInfo and HoodieVectorizedParquetRecordReader.\nAsserts the vectorized branch is engaged, the values widen correctly, and\nthe legacy path still matches the file-group reader.\n\n* test(spark): cover schema-on-read and remaining type-change branches of the legacy read path\n\nAddress review on the legacy-read-path suite:\n- testCowSnapshotReadWithSchemaOnRead: the one production shape in which\n DefaultSource#resolveBaseFileOnlyRelation returns BaseFileOnlyRelation\n itself (buildScan ships), flips shouldExtractPartitionValuesFromPartitionPath\n off (shims built with shouldAppendPartitionValues \u003d false), and drives the\n explicit internal-schema branch (InternalSchemaCache + InternalSchemaMerger).\n The InternalSchema is seeded without DDL via hoodie.schema.on.read.enable\n plus hoodie.datasource.write.reconcile.schema on the writes.\n- testCowSnapshotReadWithImplicitTypeChangeWithoutVectorizedReader: the\n row-based Cast/GenerateUnsafeProjection widening the shims run when the\n vectorized reader is disabled.\n- testCowSnapshotReadWithNestedTypeChange: int-\u003elong inside the nested struct;\n vectorized reads must fail fast with the documented IllegalArgumentException\n and the advertised workaround (vectorized reader off) must read the promoted\n struct correctly through the row-based Cast branch.\n\n---------\n\nCo-authored-by: voonhous \u003cvoonhousu@gmail.com\u003e" }, { "commit": "770e8aabf367b51b56912e4f9120fe0500334198", "tree": "d38e5d797c8fc9dddcbbc51cabf977eba610f4f6", "parents": [ "014358a3e6c8a954e3d72eeaa7033e2aa18adc43" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Mon Jul 13 21:18:35 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 13 21:18:35 2026 +0700" }, "message": "fix(spark): match the staged table, not LogicalWriteInfo, in BasicStagedTable.newWriteBuilder (#19251)" }, { "commit": "014358a3e6c8a954e3d72eeaa7033e2aa18adc43", "tree": "4834e311ae36987af573550269b48b01e24d3698", "parents": [ "2a5c83b7307509045dc75c68cd8589cc0d92611a" ], "author": { "name": "Lokesh Jain", "email": "ljain@apache.org", "time": "Mon Jul 13 18:09:20 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 13 12:39:20 2026 +0000" }, "message": "[HUDI-18075] Cap files per batch in cloud incremental source (#18076)\n\nThe cloud incremental source only limits bytes per batch, so with\nsmall files the driver can hold unbounded file metadata and OOM.\n\nAdds hoodie.streamer.source.cloud.data.num.files.per.batch to cap\nfiles per sync. Applied deterministically via a window predicate so\nthe checkpoint advances consistently with the emitted batch." }, { "commit": "2a5c83b7307509045dc75c68cd8589cc0d92611a", "tree": "ab183250638993f44d8a42206a2842a59947ded0", "parents": [ "bbe73d3e2ee4771b0fdaae39c7f5250e626c6c74" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Mon Jul 13 20:02:23 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 13 20:02:23 2026 +0800" }, "message": "feat(ui): [HUDI-9315] Add RFC-94: Hudi Timeline UI (#13152)\n\n* feat: [HUDI-9315][RFC-94] Add RFC-94: Hudi Timeline UI\n\n* Update RFC-94 on how users are expected to use it\n\n* Add more details to ui\n\n* Update RFC\n\n* Added additional future work and also rephrase\n\n- Also added rahil-c and yihua as approvers\n- Bumped release target version\n\n* Add embedded mode startup and Spark UI tab integration to RFC-94\n\n* Address RFC-94 review feedback and reflow to 120 cols\n\n- Scope: add threat-model note (UI does not widen exposure surface; opt-in, off by default)\n- Frontend: serve bundled vis-timeline by default, drop CDN-first delivery\n- API spec: note on response size and additive from/to time-range pagination\n- Handler design: cache HoodieTableMetaClient per basepath, not per request\n- Spark UI tab: add \"Multiple Tables in One Application\" subsection\n- Add \"Alternatives Considered\" section\n- Reflow whole file to 120-column wrap\n\n* docs(rfc-94): align RFC with implementation; defer embedded mode + Spark UI tab\n\n- Update API spec to the as-built /v2/hoodie/view/* routes and TimelineDTOV2\n response; add the table-config and schema-history endpoints.\n- Reflect TimelineHandler (not UiHandler) holding the v2 methods, UiHandler\n serving the /ui entry page, and registerTimelineV2API() gated by --enable-ui.\n- Document static serving at the server root plus the /ui entry page.\n- Add Table Config and Schema History as read-only views in Scope and Feature.\n- Mark embedded-mode UI enablement and the Spark UI tab as deferred to a\n follow-up (standalone-first keeps the initial PR small; the linking lands\n next), retaining their design for that follow-up.\n- Align residual references (architecture diagram, standalone config, rollout,\n test plan, manual checklist).\n\n* docs(rfc-94): account for all three vendored frontend libraries\n\nThe RFC only counted vis-timeline and quoted ~300KB, but the UI also\nvendors Bootstrap 5 and renderjson under lib/. Update the File Structure\ntree and Dependency Impact to list all three with sizes and licenses,\ncorrect the footprint to ~890KB total, and add a LICENSE/NOTICE bullet\nnoting each needs a bundling stanza in LICENSE (Category A; no NOTICE\nchanges required).\n\n* docs(rfc-94): correct v2 DTO rationale (v1 already has completion time)\n\nv1 InstantDTO.fromInstant already populates requestedTime and\ncompletionTime (HUDI-9332), so the old rationale that v1 lacks completion\ntime was wrong. Restate the DTO Design section: the UI could consume the\nv1 timeline DTO directly, and the v2 DTOs exist only to give the /v2/ API\na cleaner JSON contract (requestTs/completionTs keys, dropping the\nredundant legacy ts field).\n\n* docs(rfc-94): fix Spark UI tab dependency inversion (won\u0027t-compile as written)\n\nEmbeddedTimelineService is in hudi-client-common; hudi-spark-client\ndepends on it, not the reverse, so it cannot reference a Spark\nWebUITab/WebUIPage or HoodieSparkEngineContext directly. Specify a\nServiceLoader SPI (TimelineServerUITabProvider defined in\nhudi-client-common, implemented in hudi-spark-client), mirroring how\nHoodieTableConfig discovers HoodieTableFormat. Keeps hudi-client-common\nfree of any Spark compile-time dependency.\n\n* docs(rfc-94): namespace UI static assets under /ui/static/\n\nMounting static files at the server root reserved /js, /css, /lib\nprefixes on the shared Javalin instance. Serve them under a /ui/static/\nhostedPath instead so the whole UI surface sits under /ui and cannot\ncollide with /v1/, /v2/, or future module-registered routes. Update the\narchitecture diagram, request-flow description, JavaScript-delivery\npaths, and the API-spec static-file note.\n\nImplementation follow-up (hudi-ui branch): set Javalin\nstaticFiles.hostedPath\u003d/ui/static and repoint index.html asset links.\n\n* docs(rfc-94): specify schema-history reconstruction, cost, no-evolution case\n\nThe schema/history endpoint was described only as \"schema-change history\nfrom recent commits\", leaving the source ambiguous. Document that\ngetSchemaHistory reads both sources: per-commit HoodieCommitMetadata\nSCHEMA_KEY (deduped across consecutive commits, most recent \u0027limit\u0027\ninstants) and the .hoodie/.schema InternalSchema history when present.\nAdd the cost model (one resolve + \u003c\u003dlimit metadata reads + one .schema\nread, bounded by the active timeline) and the behavior for tables that\nnever evolved schema (single/no history entry, no internalSchemaHistory,\nno error). Matches the first-cut impl in PR #13147.\n\n* docs(rfc-94): correct threat model for config/schema exposure\n\nThe threat model claimed the UI does not widen the exposure surface and\nthat /v2/ serves the same data as /v1/. That is false for two views: no\n/v1/hoodie/view/* route serves hoodie.properties (they return\nfile-slice/timeline DTOs only), so getTableConfig (full\nHoodieTableConfig.getProps()) and the schema-history view are net-new\nread exposure. Rewrite the note to acknowledge this, call out that table\nprops can reference sensitive material (KMS endpoints, lock-provider\nconnection strings, key/vault paths), and state the control (--enable-ui\ngate + trusted network); flag a redacting/allowlisted config view as a\nfuture refinement rather than adding a denylist now (table props rarely\ncarry secrets). Addresses review feedback from hudi-agent and\nwombatu-kun.\n\n* docs(rfc-94): reload-on-request freshness for config/schema views + Refresh control\n\nThe per-basepath metaClient was built once and never reloaded, so\ngetTableConfig/getSchemaHistory could serve stale hoodie.properties /\nschema for the life of a long-lived embedded driver. Since a\nHoodieTableMetaClient snapshots tableConfig/activeTimeline at\nconstruction, each config/schema request now reloads the relevant\nsnapshot (reloadTableConfig / reloadActiveTimeline) under a per-basepath\nguard before reading. These are click-frequency views, so the reload\ncost is negligible and the response is always current; no TTL (it would\nonly re-add a staleness window). Timeline/instant-detail are unaffected\n(server-synced FileSystemView). Also document a UI Refresh control on\nboth tabs. Addresses hudi-agent + wombatu-kun feedback.\n\n* docs(rfc): link RFC-94 row in rfc/README.md index\n\nThe doc-landing commit left the RFC-94 index row as the un-linked\nreservation stub. Match the convention peer RFCs follow (RFC-93, RFC-101)\nand point the row at ./rfc-94/rfc-94.md; status stays UNDER REVIEW.\n\n* docs(rfc): make RFC-94 v2 DTO naming consistent (InstantDTOV2)\n\nThe DTO design mixed a suffix-free InstantDTO with TimelineDTOV2 in the\nsame dto.v2 package. Use the V2 suffix on both to match the repo\u0027s\nversioned-class convention (InstantGeneratorV1/V2, BaseTimelineV1/V2) and\nto avoid a simple-name clash with the existing dto.InstantDTO that\nTimelineHandler keeps importing for the v1 routes.\n\n* docs(rfc): read the active timeline via metaClient in RFC-94 handler\n\ngetFileSystemView(basePath).getTimeline() returns the write timeline\nfiltered to completed + (log)compaction instants, so it cannot render\nclean/rollback/savepoint/restore/indexing or any requested/inflight\nstate -- contradicting the vis-timeline spec (one group per\nVALID_ACTIONS_IN_TIMELINE action, requested/inflight point items) and the\ntest plan.\n\nSource the timeline from the per-basepath HoodieTableMetaClient\u0027s\ngetActiveTimeline() (already introduced for config/schema), which carries\nall 11 actions in every state. Reconcile the architecture diagram, the\nrequest-category description, the freshness/reload paragraph and the\nmulti-table section accordingly.\n\n* docs(rfc): spec RFC-94 instant read against non-deprecated APIs\n\nItem 2 of the handler design specified the deprecated\nHoodieInstantReader.getInstantDetails() and an \"own layout-aware\nInstantGenerator\" accessor that does not exist (the field is protected on\nBaseHoodieTimeline). Point it at getContentStream(instant) and the public\nmetaClient.getInstantGenerator(), which the handler already holds.\n\n* docs(rfc): correct embedded-port discovery in RFC-94\n\nhoodie.embed.timeline.server.port defaults to 0 (pick a free port) and\nthe bound port is never written back to it: EmbeddedTimelineService keeps\nit in serverPort, logs it, and propagates it only via\nFileSystemViewStorageConfig.withRemoteServerPort(). Reading the property\nback returns 0, not the live port. Point users at the startup log line\nand the view-storage config, and drop the \"retrieved from the Spark\nconfiguration\" claim.\n\n* docs(rfc): add flag-gating negative test to RFC-94 test plan\n\nThe threat model names --enable-ui (off by default) as the primary\ncontrol for the two views that widen the read surface, but every test\nstarted the service with the flag set. Add the negative integration case:\nwith the flag unset, /ui, /ui/static/* and /v2/hoodie/view/* must 404\nwhile the /v1/ routes still respond -- the assertion the Scope section\u0027s\nsecurity argument depends on.\n\n* docs(rfc): build a per-request metaClient in RFC-94 handler\n\nThe cached-per-basepath metaClient plus per-request reloadActiveTimeline()\n/reloadTableConfig() shared one mutable HoodieTableMetaClient across\nJavalin request threads. Its accessors are synchronized, but a compound\nreload-then-read is not one critical section, so a concurrent same-basepath\nreload could swap the snapshot mid-request. Since the data is re-read every\nrequest anyway, the cache saved little. Switch to a short-lived metaClient\nbuilt per request (no shared mutable state, no lock, equally cheap at\nclick frequency) and reconcile the architecture, handler, feature and\nmulti-table sections.\n\n* docs(rfc): correct the RFC-94 threat model read-surface claim\n\nThe threat model claimed the timeline and instant-detail views were\n/v1-parity. Neither is:\n\n- /v1\u0027s timeline/instants/all is served from the FileSystemView\u0027s write\n timeline (completed instants plus pending (log)compaction, write\n actions only), while /v2 reads the full active timeline and so also\n exposes clean/rollback/savepoint/restore/indexing and every\n requested/inflight state.\n- No /v1 route returns instant content at all; /v2\u0027s instant-detail view\n returns HoodieCommitMetadata, including per-partition write stats and\n the table schema under extraMetadata.schema -- the same schema the RFC\n already counted as widening when served by the schema-history view.\n\nList all four /v2 views as widening the read surface instead of claiming\nparity for two of them.\n\n* docs(rfc): group the RFC-94 timeline by comparable action\n\nThe active timeline applies TimelineLayout.filterHoodieInstantsByLatestState\nbefore the handler sees it, collapsing each (requestedTime, comparableAction)\ngroup to its highest state. InstantComparatorV2.COMPARABLE_ACTIONS maps\ncompaction -\u003e commit, logcompaction -\u003e deltacommit and clustering -\u003e\nreplacecommit, so a completed compaction arrives as a single instant whose\naction is commit, and a raw compaction action only reaches the UI while the\ninstant is pending.\n\nThe RFC\u0027s group model gave compaction, logcompaction and clustering their own\nrows, which can therefore never hold a completed instant, and the test plan\nasked for one instant per VALID_ACTIONS_IN_TIMELINE entry, which is\nunsatisfiable.\n\nFollow what Hudi already does instead of undoing it: group rows by comparable\naction, carry both action and comparableAction on InstantDTOV2 so items keep\ntheir raw label inside the folded row, and restate the test plan around the\ncollapse. A stuck compaction stays legible as a pending compaction item with no\nfollowing range bar in the commit row.\n\n* docs(rfc): include getInstantDetails in the RFC-94 metaClient cost model\n\nThe per-request metaClient paragraph enumerated getTimelineV2,\ngetTableConfig and getSchemaHistory, but getInstantDetails builds its\ninstant from the request\u0027s metaClient and reads through the active\ntimeline\u0027s getContentStream, so it needs one too -- and it pays the\nactive-timeline listing the cost sentence attributed only to the timeline\nand schema routes.\n\nList all four methods and restate the cost per route: every route pays one\nhoodie.properties read, all but getTableConfig pay one active-timeline\nlisting, and getInstantDetails additionally reads one instant file.\n\n* docs(rfc): keep RFC-94 vendored assets out of the engine bundles\n\nThe \u0027No impact on Spark/Flink bundles\u0027 claim was wrong.\nhudi-timeline-service is a compile-scope dependency of hudi-client-common,\nand six bundles list it in their shade artifactSet: hudi-timeline-server-bundle\nplus hudi-spark-bundle, hudi-flink-bundle, hudi-utilities-bundle,\nhudi-kafka-connect-bundle and hudi-integ-test-bundle. Their \u003cfilters\u003e exclude\nonly signature files, META-INF/services/javax.* and **/*.proto, so anything\nunder src/main/resources/ is unpacked into each shaded JAR. The module has no\nsrc/main/resources today, so the ~890KB of vendored assets would be net-new\npayload in all six.\n\nThe first cut is standalone-only, so the UI is unreachable from an embedded\nserver anyway. Specify an explicit public/** shade filter on the five\nnon-server bundles, scope the LICENSE obligations to the bundles that actually\nship the assets, and add a bundle-validation check so dropping the filter fails\na build instead of silently re-bloating every engine bundle.\n\n* docs(rfc): note stream ownership on the RFC-94 instant read\n\ngetContentStream returns a raw open InputStream from\nmetaClient.getStorage().open(...), which the caller owns. The deprecated\ngetInstantDetails() read into a byte[] under try-with-resources and closed\nit for you, so switching to getContentStream moves that responsibility onto\nthe handler. Spell out that the read is under try-with-resources, so the\nhandle is released on the deserialization-failure path and not just on\nsuccess.\n\n* docs(rfc): resolve the RFC-94 instant param against the timeline\n\nThe instant-detail route only validated action/state; the instant param\nflowed unchecked into the read path. getContentStream resolves to\nnew StoragePath(metaClient.getTimelinePath(), \u003cinstant\u003e.\u003caction\u003e.\u003cstate\u003e),\nand StoragePath runs URI.normalize(), which collapses \u0027..\u0027 segments, so an\ninstant of ../../../../tmp/x resolves outside .hoodie and the handler reads\nwhatever is there. The suffix and the Avro deserialization narrow what can be\nretrieved, but the path is attacker-controlled and unvalidated.\n\nThe handler already loads the active timeline, so resolve the\n(instant, action, state) triple against it and 404 when nothing matches.\nThat is stronger than regex-validating the timestamp: it also refuses\nwell-formed timestamps absent from the table, so the route can only read\ninstants the timeline lists. Also note the 404 in the API table, cross-link it\nfrom the threat model, and add a negative test that plants a file at the\ntraversal target and asserts its content never surfaces.\n\n* docs(rfc): name the RFC-94 comparable-action source concretely\n\nThe DTO spec said comparableAction was derived from \u0027the table\u0027s\nInstantComparator (getComparableAction)\u0027, which is not callable: the\nInstantComparator interface declares only the three comparator getters, and\ngetComparableAction is a public static on InstantComparatorV1/V2, unreachable\nthrough an interface-typed reference. No version-dispatched accessor exists --\nTimelineLayout.filterHoodieInstantsByLatestState is private and metaClient\u0027s\ntimelineLayout field has no getter.\n\nPin InstantComparatorV2.getComparableAction(action) and record why that is\nsafe for layout-v1 tables: InstantComparatorV1\u0027s map omits clustering, but\nActiveTimelineV1.VALID_EXTENSIONS_IN_ACTIVE_TIMELINE omits the clustering\nextensions too, so a v1 timeline can never surface an instant the two maps\ndisagree on. Lifting the method onto InstantComparator is noted as the\nfollow-up if a second caller appears; changing a shared hudi-common interface\nfor a UI display field is not warranted here.\n\n* docs(rfc): sharpen why pinning the v2 comparable-action map is safe\n\nThe previous wording read as a coincidence: v1\u0027s map omits clustering and\nv1\u0027s extensions omit clustering, so they never disagree. The real reason is\nstronger. In 0.x, clustering and replace-commit shared a filename --\nInstantFileNameGeneratorV1.makeRequestedClusteringFileName() delegates to\nmakeRequestedReplaceFileName() -- so a v1 clustering instant is written as\n.replacecommit.requested and already carries action\u003dreplacecommit. v1 has no\nclustering action at all, so v2\u0027s clustering -\u003e replacecommit entry is inert\nthere and the two maps are equivalent over everything a v1 timeline can emit.\nInstantComparatorV1\u0027s map has no clustering entry because there is nothing to\nmap, not because something is missing.\n\n* docs(rfc): resolve RFC-94 currentSchema via getTableSchemaIfPresent\n\ngetTableSchema() cannot return null: it delegates to getTableSchema(boolean),\nwhich ends in getTableSchemaInternal(...).orElseThrow(schemaNotFoundError())\nand raises HoodieSchemaNotFoundException. On a table with no commits it throws,\nwhich contradicted both the cost model\u0027s \u0027no error\u0027 claim and the empty-timeline\nbehaviour in Error Handling -- a freshly-created table would have come back as a\n500.\n\nSwitch to getTableSchemaIfPresent(boolean), which returns Option\u003cHoodieSchema\u003e\nand does not throw. There is no no-arg overload, so pass\npopulateMetaFields() explicitly, which is what the no-arg getTableSchema() was\nsupplying implicitly. Also state the empty-table contract (200 with\ncurrentSchema: null, history: []) in Error Handling, record the 404 for an\ninstant absent from the timeline, and add a unit test pinning the empty-table\npath so a regression surfaces as a failing test rather than a 500.\n\n* docs(rfc): correct the RFC-94 schema-history cost model\n\ngetHistorySchemaStr() is not \u0027one .schema history-file read\u0027. It delegates to\ngetHistorySchemaStrByGivenValidCommits(EMPTY_LIST), which calls getValidInstants()\n-- a walk of getCommitsTimeline().filterCompletedInstants() -- at line 155,\nbefore the storage.exists(baseSchemaPath) check at line 157. So a table with no\n.schema directory still pays the walk, which is exactly the \u0027no extra scanning\u0027\ncase the cost model claimed.\n\nSpecify the FileBasedInternalSchemaStorageManager(HoodieTableMetaClient)\noverload. The (HoodieStorage, StoragePath) ctor leaves metaClient null and\ngetMetaClient() lazily builds a second HoodieTableMetaClient, costing a second\nhoodie.properties read and a second timeline listing per request. Sharing the\nrequest\u0027s metaClient makes the walk an in-memory filter over the already-cached\nactive timeline instead.\n\nAlso record why the walk exists at all: the .schema files are an append-only\nlog and the newest valid .schemacommit already carries every prior version, so\nonly one file is read. The completed-commits filter is there to discard\nresidual .schemacommit files from aborted schema changes -- not to reconstruct\nhistory -- so it must not be optimized down to a lexicographic max.\n\n* docs(rfc): gate RFC-94 UI on asset presence; size the packaging guard honestly\n\n- Stripping public/** from the engine bundles removes the assets but not the\n code, so --enable-ui stays reachable from a bundle with no assets. Javalin\n resolves a classpath static-files dir eagerly at create(), and\n startServiceOnPort treats any non-bind exception as a port problem (16\n retries, then an IOException about ports) -- so a missing resource would\n surface as a misleading port failure. Make static-file registration\n conditional on the assets being present and fail fast with a clear message.\n\n- Spark UI tab rests on Spark internals, not public spark-core API:\n WebUITab/WebUIPage/attachTab/SparkContext.ui are private[spark]. Record that\n the tab must be Scala in an org.apache.spark package (precedent:\n HoodieSparkKryoRegistrar), that Java cannot implement it across 2.12/2.13\n because WebUIPage.render\u0027s Seq[Node] erases differently, and that these\n internals carry no cross-version stability guarantee.\n\n- Pin the path-traversal negative test to REQUESTED/INFLIGHT: a COMPLETED\n probe is resolved against the timeline by getInstantFileName and throws\n before reaching StoragePath, so it would pass against the unfixed handler.\n\n- bundle-validation cannot host the packaging guard as written -- it does not\n copy the timeline-server or integ-test bundles and has no jar-content\n assertion primitive. Say so, rather than implying the check is free.\n\n- Generalise stream ownership to every timeline content read via one shared\n closing helper; the up-to-1000 schema-history reads matter more than the\n single instant-detail read.\n\n* docs(rfc): move RFC-94 implementation detail into an appendix\n\nReview drove a lot of code-level detail into the body -- which static to call,\nwhich constructor overload, how URI.normalize collapses \u0027..\u0027, how many exists()\ncalls a request makes. All of it is correct and worth recording, but it buried\nthe design under API forensics and would go stale the moment the code moved.\n\nRestructure so the body states decisions and contracts, and an appendix holds\nthe evidence behind them:\n\n A1 why a completed compaction arrives as a commit\n A2 where the comparable-action mapping comes from\n A3 why the instant route must resolve, not reconstruct\n A4 why a fresh metaClient per request\n A5 why currentSchema resolves through the non-throwing accessor\n A6 what the internal-schema-history read actually does\n A7 why the UI needs an asset-presence gate\n A8 why the vendored assets reach six bundles\n A9 why the Spark UI tab must be Scala\n A10 why the packaging guard is net-new tooling\n\nThe appendix opens by saying it is reference material, not design, and that the\nbody wins on any disagreement. No decision changed; nothing was dropped.\n\n* docs(rfc): type the oldest RFC-94 schema-history entry as a baseline\n\nhistory scans only the most recent limit completed commits, and the oldest\nscanned entry has no predecessor inside that window. Recording it under the\n\u0027differs from the previous instant\u0027 rule emits it as a schema CHANGE, so a\ntable with more than limit commits would show a change that never happened --\non the one tab whose job is saying when the schema changed.\n\nType the oldest entry \u0027baseline\u0027 (the schema as of the oldest commit scanned)\nand every later entry \u0027change\u0027, and return the scanned window as\n{ oldestInstantScanned, truncated } so the UI can tell a window edge from the\ntable\u0027s first schema. Both are response-contract changes, not implementation\nchoices: without them the truncation is silent and unrecoverable client-side.\n\n* docs(rfc): move the RFC-94 UI API under /ui/api, not /v2/hoodie/view\n\n/v1/hoodie/view is not a public REST API -- it is the private RPC namespace of\nRemoteHoodieTableFileSystemView, the wire protocol the FSV client speaks to\nthis server. Mounting the UI at /v2/hoodie/view/ would read as version 2 of the\nFSV protocol, which it is not, and would consume the URL space a real FSV v2\nwill one day want.\n\nThe UI is a different API for a different consumer (a browser, with UI-shaped\nDTOs), so give it its own namespace instead of version-bumping someone else\u0027s:\n\n- routes move to /ui/api/*, alongside the page (/ui) and assets (/ui/static/*)\n- DTOs become UiInstantDTO / UiTimelineDTO in dto.ui, since they are not\n version 2 of the FSV protocol\u0027s InstantDTO and naming them V2 implied they\n were; the Ui prefix still avoids the simple-name clash that motivated the\n suffix\n- getTimelineV2 -\u003e getUiTimeline, registerTimelineV2API -\u003e registerUiApi\n\nAlso collapses the feature onto one prefix: everything under /ui is the UI, and\n--enable-ui gates exactly that subtree. The flag previously had to gate three\nunrelated prefixes for one feature, and the gating test had to enumerate them." }, { "commit": "bbe73d3e2ee4771b0fdaae39c7f5250e626c6c74", "tree": "c1fc59f11c00c4cc5d41b85be92091b2a276267b", "parents": [ "7854419d70b5f0d8eaaaed9d77e1ab5e4d3e027c" ], "author": { "name": "Surya Prasanna", "email": "syalla@uber.com", "time": "Mon Jul 13 03:08:27 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 13 18:08:27 2026 +0800" }, "message": "fix: align log4j2 and slf4j versions to resolve IntelliJ test failures (#18177)\n\n* Fix no classdef errors on intellij\n\n* fix: bind slf4j 2.x by using the log4j-slf4j2-impl binding\n\nRaising slf4j.version to 2.0.7 in the root properties leaves every build that\ndoes not activate a spark profile (a plain `mvn` build, and all -Dflink*\nbuilds) pairing slf4j-api 2.0.7 with log4j-slf4j-impl. That artifact only ships\norg/slf4j/impl/StaticLoggerBinder, which is the slf4j 1.7 binding; slf4j 2.x\ndiscovers providers through META-INF/services/org.slf4j.spi.SLF4JServiceProvider\nand therefore finds none:\n\n SLF4J: No SLF4J providers were found.\n SLF4J: Defaulting to no-operation (NOP) logger implementation\n SLF4J: Class path contains SLF4J bindings targeting slf4j-api versions 1.7.x\n or earlier.\n\nAll logging would silently go to the NOP logger, which no test asserts on.\n\nSwap the binding to log4j-slf4j2-impl (the log4j2 binding for slf4j 2.x,\navailable since log4j 2.19) in the root dependencyManagement, in every module\nthat declares it, and in the hudi-metaserver-server-bundle shade includes.\nSpark builds already pull log4j-slf4j2-impl transitively, so this only makes\nthe declared binding match what slf4j 2.x can actually load. No source change\nis needed: Hudi uses no slf4j 2.x-only API.\n\n* fix: keep hive\u0027s slf4j 1.x binding off the classpath\n\nhive-shims-common pulls in log4j-slf4j-impl 2.17.2, and that artifact ships the\nsame classes as log4j-slf4j2-impl (org.apache.logging.slf4j.Log4jLoggerFactory,\nLog4jLogger, Log4jMarkerFactory), so both jars on one classpath means the shared\nclasses are resolved from whichever jar comes first, with no Maven mediation.\n\nThis was masked until now: the root dependencyManagement pinned log4j-slf4j-impl\nto ${log4j2.version}, so hive\u0027s copy was force-upgraded and its duplicate\nLog4jLoggerFactory happened to carry the same (Log4jMarkerFactory) constructor.\nOnce that entry became log4j-slf4j2-impl the pin was gone, hive\u0027s 2.17.2 copy\ncame in with only a no-arg constructor, and SLF4JServiceProvider.initialize()\nfailed with\n\n NoSuchMethodError: Log4jLoggerFactory.\u003cinit\u003e(Log4jMarkerFactory)\n\nsurfacing as ExceptionInInitializerError / NoClassDefFoundError on every class\nwith a static logger, e.g. HoodieConfig.\n\nExclude log4j-slf4j-impl from the four hive deps that leak it (hive-shims,\nhive-jdbc, hive-serde, hive-metastore); hive-exec, hive-common and hive-service\nalready exclude org.apache.logging.log4j:* wholesale. This mirrors the existing\nexclusions for slf4j-log4j12, hive\u0027s other slf4j 1.x binding.\n\n* fix: keep spark 3.3\u0027s slf4j 1.x binding off the classpath\n\nSpark only moved to log4j-slf4j2-impl in 3.4, so spark-core 3.3.x still brings\nlog4j-slf4j-impl. That is the same duplicate-class collision already fixed for\nhive: both binding artifacts ship org.apache.logging.slf4j.Log4jLoggerFactory,\nso the older copy shadows the one log4j-slf4j2-impl expects and\nSLF4JServiceProvider.initialize() dies with\n\n NoSuchMethodError: Log4jLoggerFactory.\u003cinit\u003e(Log4jMarkerFactory)\n\nExclude log4j-slf4j-impl from both spark-core entries in dependencyManagement\n(the main one and the tests classifier), alongside the org.slf4j:* and\nlog4j:log4j exclusions already there. Spark 3.4+ is unaffected: it ships the\nslf4j2 binding already.\n\nVerified there is no remaining log4j-slf4j-impl on the tree for spark3.3,\nspark3.4, spark3.5, spark4.0, spark4.1, spark4.2, flink1.18, flink1.19,\nflink1.20, flink2.0, flink2.1 and the default profile.\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "7854419d70b5f0d8eaaaed9d77e1ab5e4d3e027c", "tree": "466eeeca6e39a240d72dd96ec7dd897756fc5ddf", "parents": [ "541ab01b9f5a2ef039cfb90738541329fbb91bc6" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Mon Jul 13 13:02:57 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 13 13:02:57 2026 +0800" }, "message": "fix(writer): skip unsupported native log column stats metadata (#19245)" }, { "commit": "541ab01b9f5a2ef039cfb90738541329fbb91bc6", "tree": "c1009bb408a7e8c626142f2cc3c691ca0170b05d", "parents": [ "1e2736f5a51b95796947a61ab6924ab900af6896" ], "author": { "name": "Lin Liu", "email": "141371752+linliu-code@users.noreply.github.com", "time": "Sat Jul 11 01:42:02 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Jul 11 08:42:02 2026 +0000" }, "message": "Remove spaces around partition columns (#18423)" }, { "commit": "1e2736f5a51b95796947a61ab6924ab900af6896", "tree": "1a4e743fc0e0f4e83b6dcf9194dca984d337d4a3", "parents": [ "120d73fcc4dba7ddfae0b9693f487190a8410d34" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Sat Jul 11 13:59:11 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Jul 11 14:59:11 2026 +0800" }, "message": "fix(ci): retry transient Maven Central 5xx and stop seeding a 10 MB Maven cache (#19244)\n\nCI jobs intermittently die in \"Build Project\" when Maven Central rejects a plugin or plugin-dependency fetch. The retry knobs in MVN_ARGS are maven.wagon.* properties, which Maven 3.9\u0027s native resolver transport never reads, so the build effectively ran with resolver defaults and retried only 429 and 503. A single 502 from Central killed a six-minute build.\n\nReplace them with the equivalent aether.connector.http.* properties and extend the retried codes to 429,500,502,503,504. 403 is deliberately excluded: Central answers 403 when its CDN has blocked the runner\u0027s egress IP, and retrying that is what escalates the block. Azure keeps the wagon flags too, because its Maven version is not pinned and Maven 3.8 reads only those.\n\nDrop `cache: maven` from validate-source. All jobs share one cache key and the first job to finish wins the write, so that 40-second RAT check was sealing a ~10 MB local repository which every build job then discarded and re-downloaded from Central.\n\nRemove the 2019-era maven-jar-plugin 3.1.2 overrides from hudi-utilities, hudi-kafka-connect, hudi-examples-flink and hudi-flink so they inherit the 3.5.0 pin, as hudi-aws already does. 3.1.2 is the only reason maven-archiver:3.4.0, the artifact that failed, is fetched at all.\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e" }, { "commit": "120d73fcc4dba7ddfae0b9693f487190a8410d34", "tree": "f7630cff33ee70c96fe565ecd61fc46e80239a58", "parents": [ "be8a54c4ebdaf287993d24fefba60a58952a9c9e" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Sat Jul 11 13:21:09 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Jul 11 13:21:09 2026 +0800" }, "message": "fix(spark): read INLINE blobs as CONTENT on internal write-side Lance… (#19236)\n\n* fix(spark): read INLINE blobs as CONTENT on internal write-side Lance reads\n\nMOR compaction, clustering, and upsert merge read Lance base files through\nSparkReaderContextFactory -\u003e SparkFileFormatInternalRowReaderContext -\u003e\nSparkLanceReaderBase, which resolves hoodie.read.blob.inline.mode from the\nbroadcast Hadoop conf. The factory never set it, so the DESCRIPTOR default\napplied and rewrites persisted INLINE blobs with null data, silently losing\nthe bytes of every carried-over row (all rows, in the clustering case).\n\n- Pin hoodie.read.blob.inline.mode\u003dCONTENT in the conf that\n SparkReaderContextFactory broadcasts; the factory only serves internal\n write-side and index reads, never user-facing queries, which build their\n own conf from session options.\n- Add a per-row guard in HoodieSparkLanceWriter that rejects the\n descriptor-leak shape {type\u003dINLINE, data\u003dnull, reference!\u003dnull} so any\n future leak fails loudly instead of dropping bytes; {INLINE, null, null}\n stays writable.\n- Fix the stale comment in SparkLanceReaderBase claiming CONTENT is the\n config default (it is DESCRIPTOR).\n- Un-mask testBlobInlineCompactionRoundTrip by forcing all rows into one\n file group so untouched rows actually go through the compaction rewrite,\n and correct its doc about which reader compaction uses.\n- Add testBlobInlineClusteringRoundTrip and writer-guard tests.\n\nFixes #19232\n\n* test(spark): assert INLINE blob bytes via plain projection before read_blob\n\nAssert payload.data under CONTENT mode before calling read_blob() in the\ncompaction and clustering round-trip tests. On a regression the failure now\nreads as explicit data loss (data is null after the rewrite) instead of\nread_blob()\u0027s misleading DESCRIPTOR-mode IllegalStateException, which made\n#19232 look like an error by design. Also drop a stale comment crediting\nHoodieSparkLanceReader\u0027s CONTENT pin for compaction correctness.\n\n* test(spark): assert single file group in blob compaction round-trip test\n\nThe un-masking of #19232 relies on coalesce(1) plus shuffle parallelism 1\nforcing all rows into one file group, but nothing asserted it. Pin the\ninvariant before the anyHadLogs scan so drifting back to multiple file\ngroups (log-free groups compaction never rewrites) fails loudly instead\nof silently re-masking the regression.\n\n* docs(spark): fix DESCRIPTOR-mode comment; transform keeps type\u003dINLINE\n\nBlobDescriptorTransform never emits OUT_OF_LINE; it synthesizes the\nreference sub-struct while preserving type\u003dINLINE. The old wording\ndescribed the exact wrong mental model that hid #19232.\n\n* test(spark): guard CONTENT pin in TestSparkReaderContextFactory\n\nAssert hoodie.read.blob.inline.mode\u003dCONTENT on the captured broadcast\nConfiguration so dropping the pin in SparkReaderContextFactory fails a\nfast unit test instead of only the Lance functional suite.\n\n* refactor(spark): hoist blob struct layout into shared BlobStructLayout\n\nBlobDescriptorTransform and the HoodieSparkLanceWriter guard each\nre-derived the BLOB struct ordinals, arities and INLINE token. Move them\ninto a package-private BlobStructLayout holder (arities sourced from\nHoodieSchema.Blob) so the two decoders cannot drift apart.\n\n* docs(spark): correct stale CONTENT-pin attributions for Lance readers\n\nThe deltaCommits assertion message still credited the pin to\nHoodieSparkLanceReader; it lives in SparkReaderContextFactory. The test\ndocstring understated HoodieSparkLanceReader\u0027s callers (bloom-index\nlookups and legacy HoodieWriteMergeHandle merges also use it), and the\npin comment inside that reader still claimed compaction/merge/log-replay\nwhich now go through SparkLanceReaderBase.\n\n* fix(spark): widen blob guard error to cover user query round-trips\n\nAfter the SparkReaderContextFactory pin, internal rewrites cannot\nproduce the descriptor shape; the realistic trigger is a user reading a\nblob table at the DESCRIPTOR default and writing the rows back (INSERT\nINTO ... SELECT). Stop attributing the leak solely to internal rewrites\nso that user knows to set hoodie.read.blob.inline.mode\u003dCONTENT on their\nread.\n\n* test(spark): assert clustering completed and rewrote base files in blob inline clustering test\n\nAddress review on testBlobInlineClusteringRoundTrip. getLastClusteringInstant.isPresent\nis satisfied by a REQUESTED/INFLIGHT replacecommit (getTimelineOfActions filters by action\nonly), so assert the instant isCompleted. Also snapshot the first commit\u0027s base files and\nassert the post-clustering set is disjoint, proving the rewrite ran rather than the byte\nassertions reading the untouched originals (the masking mode of #19232).\n\n* fix(test): define missing engineContext in latestBaseFileNames helper\n\nThe latestBaseFileNames helper in testBlobInlineClusteringRoundTrip\nreferenced engineContext without binding it, causing a Scala compile\nerror. Define it locally via HoodieLocalEngineContext(mc.getStorageConf),\nmatching the three other FileSystemViewManager call sites in this file.\n\n* docs(test): reword CONTENT-projection comments to the shape they actually backstop\n\nThe old comments described catching a persisted DESCRIPTOR leak\n({INLINE, null data, populated reference}), but validateBlobRow in\nHoodieSparkLanceWriter now rejects that shape inside the rewrite, so the\ntest would fail at the compaction/clustering write before the projection\nruns. Reword both sites (compaction + clustering) to the shape the guard\ndeliberately allows, {INLINE, null, null}, and point at\nTestSparkReaderContextFactory as what pins the CONTENT config.\n\n* test(spark): cover CoW upsert merge path for INLINE blob preservation\n\nCompaction and clustering resolve their reader via getReaderContextFactory,\nbut a CoW upsert rewrites the base file through\nFileGroupReaderBasedMergeHandle, which resolves it via\ngetReaderContextFactoryForWrite -- a separate path that branches on the\nrecord merger type. No blob test exercised it. Add a CoW round-trip that\nbulk-inserts INLINE blobs into a single file group, upserts a subset,\nproves the merge rewrote the base file (disjoint base file names, no\ndeltacommits), and verifies touched rows carry new bytes while untouched\nrows retain the originals. Hoist latestBaseFileNames out of the\nclustering test for reuse." }, { "commit": "be8a54c4ebdaf287993d24fefba60a58952a9c9e", "tree": "d837f8d7d3f0bb3e858f6508c6074c29d827a504", "parents": [ "7b843ae2c35fa00993a2a1b102c9ad7e1e116267" ], "author": { "name": "Joy", "email": "33287603+Joy-2000@users.noreply.github.com", "time": "Sat Jul 11 07:27:59 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Jul 11 07:27:59 2026 +0800" }, "message": "fix(flink): prevent data loss on global failover for streaming writes (#19237)\n\n* fix(flink): prevent data loss on global failover for streaming writes\n\n---------\n\nCo-authored-by: jiangyu84 \u003cjiangyu84@jd.com\u003e\nCo-authored-by: danny0405 \u003cyuzhao.cyz@gmail.com\u003e" }, { "commit": "7b843ae2c35fa00993a2a1b102c9ad7e1e116267", "tree": "bdc0fd660c6404478993f7924056d290aed77927", "parents": [ "bc874774af1759c16d7ff42781c842485352cd03" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Fri Jul 10 06:32:00 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 10 21:32:00 2026 +0800" }, "message": "refactor(spark): consolidate the vendored 3.x Avro serde forks into hudi-spark3-common (#19168)\n\n* refactor(spark): consolidate the vendored 3.x Avro serde forks into hudi-spark3-common\n\nThe vendored spark-avro AvroSerializer and AvroDeserializer are duplicated\nacross hudi-spark3.3.x, hudi-spark3.4.x and hudi-spark3.5.x. The 3.3 and 3.4\ncopies are byte-identical; the 3.5 copies differ only in how they import\nLegacyBehaviorPolicy. Move a single copy of each into hudi-spark3-common,\nwhich every 3.x version module already depends on, and delete the six\nduplicates.\n\nThe one cross-version snag is the LegacyBehaviorPolicy enum: it is nested in\nSQLConf (org.apache.spark.sql.internal.SQLConf.LegacyBehaviorPolicy) on Spark\n3.3/3.4 but a top-level object (org.apache.spark.sql.internal.LegacyBehaviorPolicy)\non Spark 3.5, so no single explicit import resolves on all three (verified by\ncompiling against the 3.3.4/3.4.3/3.5.5 catalyst jars). The shared source\nimports both containers via wildcards; exactly one of them contributes the\nenum on any given version, so there is no ambiguity and the bodies stay\nbyte-identical to the originals. The per-version HoodieSpark3_xAvro{Serializer,\nDeserializer} wrappers are unchanged, since the shared classes keep the same\nconstructors they called.\n\nConsolidating also gives each file a unique repo path, so it re-enters the\ncoverage denominator: the identically-pathed copies were dropped by\nreport-path resolution and were invisible to Codecov.\n\n* refactor(spark): consolidate the vendored 4.x Avro serde forks into hudi-spark4-common\n\nThe vendored spark-avro AvroSerializer and AvroDeserializer are duplicated\nacross hudi-spark4.0.x, hudi-spark4.1.x and hudi-spark4.2.x.\n\nAvroSerializer: the 4.0/4.1/4.2 bodies are identical except for the private\nconvenience constructor that reads AVRO_REBASE_MODE_IN_WRITE from SQLConf.\nOn Spark 4.0 that ConfigEntry is typed as String (so the read wraps it in\nLegacyBehaviorPolicy.withName), while on 4.1+ it is already a\nLegacyBehaviorPolicy.Value; a single shared source cannot express both. Since\nthe only callers are the per-version HoodieSpark4_xAvroSerializer wrappers,\nmove that read into each wrapper (verbatim per version) and drop the\nconvenience constructor, leaving one shared AvroSerializer in\nhudi-spark4-common that all three versions use.\n\nAvroDeserializer: 4.1 and 4.2 are identical apart from one comment word, so\nthey collapse to a single shared copy in hudi-spark4-common. Spark 4.0 is kept\nseparate: it pulls in Avro 1.12.0 (fast reader off) and lacks the ~71-line\nread-side java.time normalization that 4.1+ need for Avro 1.12.1, and that\nblock does not parameterize cleanly. Because hudi-spark4.0.x depends on\nhudi-spark4-common, a same-named copy would collide on the classpath, so the\n4.0-only copy is renamed to Spark40AvroDeserializer (body unchanged).\n\nConsolidating also gives each file a unique repo path, so it re-enters the\ncoverage denominator: the identically-pathed copies were dropped by\nreport-path resolution and were invisible to Codecov.\n\n* Revert the 4.x Avro serde consolidation (unsound: vendored fork must shadow spark-sql per-module)\n\nReverts commit 5a8169f05bbe. On Spark 4.x the avro connector is merged into\nthe spark-sql artifact, so org.apache.spark.sql.avro.{AvroDeserializer,\nAvroSerializer,AvroUtils} ship inside spark-sql_2.13 itself (verified in the\n4.0.2/4.1.1/4.2.0-preview4 jars). spark-sql is a provided dependency of every\n4.x version module, so Spark\u0027s own AvroDeserializer/AvroSerializer are always\non the 4.x compile classpath.\n\nThe vendored fork shares Spark\u0027s FQN. While it lived as source in each version\nmodule, same-module source shadowed the spark-sql class during that module\u0027s\ncompile. Moving it into the hudi-spark4-common jar turned it into a peer\ndependency-jar class that competes with spark-sql\u0027s copy, and spark-sql wins:\nthe HoodieSpark4_xAvro{Deserializer,Serializer} wrappers then resolve the wrong\nclass (compile error in 4.2.x: the 3-arg (Schema, DataType,\nLegacyBehaviorPolicy.Value) call cannot bind to Spark\u0027s constructors). Even\nwhere it compiled, the wrapper would silently link Spark\u0027s serde on a data\npath.\n\nThe 3.x consolidation is kept: spark-sql/spark-catalyst 3.x do not contain the\navro serde classes (they live only in the separate spark-avro artifact, which\nis not a dependency of the 3.x modules), so the vendored fork in\nhudi-spark3-common is the only such class on the 3.x classpath and resolves\ncorrectly. This revert restores the six 4.x serde files to be byte-identical\nto apache/master." }, { "commit": "bc874774af1759c16d7ff42781c842485352cd03", "tree": "ae09f2a5258c39184fb4c90a23242b6803320a16", "parents": [ "14fe7393a42fc1f6ac79533fa5cac18d0812330a" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Fri Jul 10 18:05:25 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 10 11:05:25 2026 +0000" }, "message": "fix(common): name the offending expression in BindVisitor\u0027s unsupported-predicate error (#19241)\n\n* fix(common): name the offending expression in BindVisitor\u0027s unsupported-predicate error\n\nBindVisitor#visitPredicate interpolated `this`, the visitor, rather than the predicate into its IllegalArgumentException, and omitted the space before \"cannot\". Since BindVisitor has no toString() override the message read \"The expression org.apache.hudi.common.expression.BindVisitor@41a90fa8cannot be visited as predicate\". The sibling PartialBindVisitor#visitPredicate already interpolates the predicate correctly.\n\nStringStartsWithAny is the only Predicate that can reach that guard, and the only class in Predicates.java without a toString() override, so the corrected message would still have printed an identity hash. Added a null-safe toString() to it, since HoodieBackedTableMetadata constructs it with a null left operand.\n\nAdded TestBindVisitor covering the error message, and two TestPredicates cases covering the new toString(), including the null left operand shape.\n\nCloses #19240\n\n* addressed review comments: fix toString rendering in Predicates\n\nDrop the prefix and suffix from Collectors.joining in StringStartsWithAny.toString() so the rendering matches the method-call shape of the sibling StringStartsWith.toString(), and scope its null-safety comment to the left operand, which is the only one HoodieBackedTableMetadata passes as null. Correct the startWith typo in StringStartsWith.toString() and pin the rendering with a test.\n\n---------\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e" }, { "commit": "14fe7393a42fc1f6ac79533fa5cac18d0812330a", "tree": "4e7f3c987e65293fa6bd19b5b28e768a504ffe81", "parents": [ "b22a3d7af37a5c87665477b5e5775307d0f3137d" ], "author": { "name": "vinoth chandar", "email": "vinothchandar@users.noreply.github.com", "time": "Fri Jul 10 01:53:42 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 10 08:53:42 2026 +0000" }, "message": "fix(spark): read Lance BLOB columns in \u003c\u003d512-row chunks to avoid lance-core FFI abort (#19181)\n\n* fix(spark): read Lance BLOB columns in \u003c\u003d512-row chunks to avoid lance-core FFI abort\n\nlance-core 4.0.0 aborts the JVM in its Arrow C-stream export\n(arrow_array::ffi_stream::get_next: \"range end index N out of range for slice\nof length 0\") whenever a single readAll stream crosses Lance\u0027s internal BLOB\npage boundary (512 rows); the requested batchSize does not help because Lance\nre-chunks BLOB columns at 512 internally. As a result CoW Lance tables with\nBLOB columns could not be read past 512 rows: OUT_OF_LINE reads threw an Arrow\n\"should have as many children as in the schema\" error and INLINE CONTENT reads\ncrashed the JVM (SIGABRT).\n\nRead BLOB-containing Lance files in \u003c\u003d512-row row-range chunks, issuing a fresh\nreadAll per chunk so each FFI stream stays within a single BLOB page and the\nbuggy second-page export is never reached. LanceRecordIterator gains a\nchunkedBlobReader(...) factory driven by an ArrowReaderSupplier;\nSparkLanceReaderBase (SQL/DataFrame read) and HoodieSparkLanceReader (internal\nCONTENT path) route to it when the projection contains a BLOB field. Non-BLOB\nreads keep the single streamed reader and are unchanged; the per-row hot path is\nuntouched, with only small fixed per-chunk allocations.\n\nAlso document hoodie.read.blob.inline.mode option placement (read option for\nboth DataFrame and SQL) and add batch-scale regression tests: INLINE CONTENT,\nVECTOR + OUT_OF_LINE BLOB at n\u003d100/1000 (crossing the 512-row boundary), with\nvector top-k, projection/filter, and read_blob byte + SHA-256 checks.\n\n* refactor(spark): rename ArrowReaderSupplier to ArrowReaderSequence, use nonEmpty\n\n* fix(spark): recurse Lance BLOB chunking detection, pin chunk size to lance-core 4.0.0\n\nRoute reads through the chunked path when a BLOB appears at any nesting\ndepth (HoodieSchema.containsBlobType() made public for the internal reader;\nrecursive StructType walk in SparkLanceReaderBase), so a future nested-BLOB\nwriter change cannot silently skip chunking and re-introduce the lance-core\nFFI abort. Document BLOB_READ_CHUNK_ROWS as pinned to lance-core 4.0.0\u0027s\ninternal 512-row BLOB page size, revalidated on lance upgrades.\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "b22a3d7af37a5c87665477b5e5775307d0f3137d", "tree": "d1d8a79601bae11546ca6ac1935771977be728a9", "parents": [ "9df73c747453790e1fea6a8be6d8bac3c37f48ab" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Thu Jul 09 10:51:13 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 10 01:51:13 2026 +0800" }, "message": "test(spark): add extended SQL parser coverage for index DDL and Hudi column types (#19218)\n\n* test(spark): add extended SQL parser coverage for index DDL and Hudi column types\n\n* test(spark): strengthen parser coverage assertions per review\n\n- BLOB/VECTOR tests now assert the hudi_type field metadata (BLOB, canonical\n VECTOR(3)) and the mapped dataType (BlobType(), ArrayType(FloatType, false))\n instead of only checking the column name exists, so they fail if the custom\n type maps incorrectly.\n- Drop the unused create-table setup from the four index-DDL tests; parsePlan\n is purely syntactic, so the parse is identical whether or not the table\n exists (noted on the parse helper).\n\n* test(spark): fold parser coverage into TestIndexSyntax per review\n\nTestExtendedSqlParserCoverage duplicated existing coverage: BLOB/VECTOR\ncreate-table + hudi_type assertions live in TestCreateTable and\nTestBlobDataType, and CREATE/DROP/SHOW INDEX parse+resolution is covered\nby TestIndexSyntax. The only genuine gap was REFRESH INDEX, which\nTestIndexSyntax\u0027s \u0027Test Create/Drop/Show/Refresh Index\u0027 promised in its\ntitle but never exercised. Add the refresh-index parse+analyze\nassertions there, mirroring the sibling statements, and drop the\nredundant suite.\n\n* test(spark): assert catalog-stored schema retains hudi_type for BLOB/VECTOR\n\nThe dropped parser-coverage suite read the schema back via\ncatalog.getTableMetadata rather than spark.table; that was the one\nassertion without an existing home: the CatalogTable persisted by\nCreateHoodieTableCommand retains the hudi_type field metadata. Pin it in\nTestCreateTable\u0027s BLOB and VECTOR create-table tests.\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "9df73c747453790e1fea6a8be6d8bac3c37f48ab", "tree": "98aa7eb88c01eceb197bcc919d54514634e5cd3d", "parents": [ "6fa846164892d07c257376471e2e68ada83df159" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Fri Jul 10 01:49:49 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 10 01:49:49 2026 +0800" }, "message": "fix(bundle): shade parquet-variant into common bundle includes for Spark 4.1+ (#19235)\n\nParquet 1.16.0 (Spark 4.1) split VARIANT support into a separate `parquet-variant`\nmodule carrying `org.apache.parquet.variant.VariantConverters`. `parquet-avro:1.16.0`,\nshaded into the spark and utilities bundles and used on the read path, references it, but\nthe shade `\u003cincludes\u003e` allowlists omitted parquet-variant, so the class was dropped from\nthe bundles and absent at runtime.\n\nOn Spark 4.1 every read-modify write (UPDATE / MERGE / DELETE) then aborted with\nNoClassDefFoundError; only INSERT succeeded. This affects any Spark 4.1 bundle that shades\nparquet-avro: both hoodie-spark-bundle (surfaced by the hive-sync E2E, #19216) and\nhoodie-utilities-bundle (the HoodieStreamer standalone runtime).\n\nAdd parquet-variant to the shared \"common to all bundles\" artifactSet in the root pom that\nevery bundle appends to via combine.children\u003d\"append\". It is a transitive compile dep of\nparquet-avro for Parquet 1.16.0+, so it shades for Spark 4.1/4.2 and is a no-op for older\nSpark and bundles that pin pre-1.16 parquet. Verified the rebuilt spark4.1 spark-bundle and\nutilities-bundle both contain VariantConverters.\n\nFixes apache/hudi#19234" }, { "commit": "6fa846164892d07c257376471e2e68ada83df159", "tree": "e5bb2edbff1ec921b5b774cac5f162157190c552", "parents": [ "1046ec084732e76075ed8b3b677a23a48503b4ae" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Thu Jul 09 07:00:16 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 08 20:00:16 2026 -0400" }, "message": "fix(ci): cap UT_FT_10 reactor install to -T 1 to avoid GCLocker-induced compile OOM (#19119)\n\nThe Azure UT_FT_10 job (\"UT FT common \u0026 other modules\") intermittently OOMs during its initial `mvn clean install` reactor build while compiling a large packaging bundle, surfaced as a generic \"An unknown compilation problem occurred\". javac runs in-process in the job\u0027s single 8g JVM; under parallel builds, concurrent native zip work (maven-shade of large bundles plus javac reading huge classpaths via zipfs) holds JNI-critical sections that block the GC (\"Retried waiting for GCLocker too often\"), so garbage accumulates uncollected and the heap is exhausted even though the live set is small.\n\nA prior `-T 2` cap (#19008) reduced but did not eliminate this. Lower the prepended cap to `-T 1` so Maven (first -T wins) serializes this single job\u0027s install, removing the concurrent JNI-critical contention. Other jobs keep the shared -T 3.\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e" }, { "commit": "1046ec084732e76075ed8b3b677a23a48503b4ae", "tree": "3ec665d3753cb2a49264119861a0fb16c873373a", "parents": [ "0fc240bf5f95d2fa979e3c8e01f66a65724abfe1" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Wed Jul 08 23:49:56 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 08 23:49:56 2026 +0800" }, "message": "test(integ-test): add Spark 4.1.1 stack to hive-sync E2E matrix (#19216)\n\nAdds a spark4.1 row to the integration-tests-hive-sync job, backed by new\ndocker-compose_hadoop340_hive2310_spark411_{amd64,arm64}.yml files derived\nfrom the 4.0.2 pair (image tags and HDFS cluster name only; arm64 comment\nnormalized to ASCII). VARIANT tests run on this row via the existing\nspark4 compose-prefix token; no Java harness changes.\n\nRequires apachehudi/hudi-hadoop_3.4.0-hive_2.3.10-spark{base,master,worker,adhoc}_4.1.1\nimages to be published to Docker Hub before the new CI row can pass." }, { "commit": "0fc240bf5f95d2fa979e3c8e01f66a65724abfe1", "tree": "3fba4fd03d6e1409a900d30217f8283c4452cafb", "parents": [ "21a931572dfd07d8ba18c4cd64cd721aaf5e3df1" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Wed Jul 08 06:13:58 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 08 13:13:58 2026 +0000" }, "message": "test: add Vortex reader/writer round-trip test (#19182)\n\n* feat: Add VORTEX file format enum, storage configs, and Maven dependency management\n\nCloses #18624\n\n* feat: Wire VORTEX into reader/writer factory dispatch and IOFactory\n\nCloses #18625\n\n* feat: Add Vortex base writer and Spark InternalRow writer\n\nCloses #18626\n\n* feat: Add Vortex Spark reader, record iterator, and resource closer\n\nCloses #18627\n\n* feat: Add Vortex Hive InputFormat stubs and wire Hive dispatch\n\nMirror the Lance Hive integration for the VORTEX base file format:\n\n- Add HoodieVortexInputFormat (COW) and HoodieVortexRealtimeInputFormat\n (MOR) as catalog/metastore registration stubs; reading through the Hive\n InputFormat is not yet supported and throws, directing users to the Spark\n datasource path.\n- Wire VORTEX into HoodieInputFormatUtils dispatch: getInputFormat (by\n format and by extension), getInputFormatClassName, getOutputFormatClassName\n and getSerDeClassName (reusing the Parquet OutputFormat/SerDe like Lance).\n- Guard the Hive reader context so .vortex files fail fast with\n VORTEX_SPARK_ONLY_ERROR_MSG instead of being mis-read.\n\nVortexUtils (key iteration, schema reads, row counts, filterRowKeys) is\nalready implemented at parity with LanceUtils.\n\n* feat: Add Vortex Spark datasource reader and wire into SparkAdapter\n\nAdd the query-time Vortex reader for the Spark datasource path, mirroring the\nLance integration for the initial Vortex scope (standard data types, no filter\npushdown/data skipping, no BLOB/VECTOR handling).\n\n- Create SparkVortexReaderBase (hudi-spark-common) implementing\n SparkColumnarFileReader: opens the file via the dev.vortex.api Session, projects\n the requested columns, converts Arrow batches to rows through the existing\n VortexRecordIterator, and applies null-padding/cast projections plus partition\n column appending for schema evolution. Handles the partition-only / all-new-column\n case by emitting empty rows sized from DataSource.numRows().\n- Add SparkAdapter.createVortexFileReader; override in the Spark 3.5/4.0/4.1/4.2\n adapters (Some) and 3.3/3.4 (None, connector artifact unavailable), matching the\n vortex-spark test matrix in the pom.\n- Extend MultipleColumnarFileFormatReader with a fourth vortexReader and a VORTEX\n dispatch case; thread it through HoodieFileGroupReaderBasedFileFormat,\n HoodieMergeOnReadRDDV2, and SparkReaderContextFactory (plus the single-format\n VORTEX branches) and update the file-group reader test constructor.\n\n* test: add Vortex writer and reader round-trip test\n\nAdds a file-level round trip through HoodieSparkVortexWriter and\nHoodieSparkVortexReader against the real dev.vortex.api: a multi-batch\nwrite+read of long/string/double columns and a projected (column-subset)\nread. Gated by -Dvortex.skip.tests.\n\ncloses #18630\n\n* test: strengthen Vortex round-trip test per review\n\n- Wrap HoodieStorage in try-with-resources so it closes even when an assertion\n throws (previously closed only on the happy path).\n- Project the non-leading \"name\" column in testProjectedRead so the assertion\n actually pins projection; a leading-column (\"id\") projection read back at\n index 0 whether or not the reader narrowed to the requested schema.\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "21a931572dfd07d8ba18c4cd64cd721aaf5e3df1", "tree": "8c532c1252694903f9ed0f17ec05dff37bfec5b5", "parents": [ "1165eb536b019e2f2ad0df2778fd02eae6eeff8f" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Wed Jul 08 05:30:15 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 08 12:30:15 2026 +0000" }, "message": "test(client): cover low-coverage small classes across hudi-client (#19224)\n\n* test(client): cover low-coverage small classes across hudi-client\n\n* test(client): address review nits on tail-sweep coverage tests\n\n- Assert content equality instead of reference identity for the candidate-key\n and data-write-stat accessors, so the tests no longer pin the no-defensive-copy\n implementation detail.\n- Document that isKeyInRange fails fast with an NPE when bounds are unset.\n- Use assertNotEquals for the BloomIndexFileInfo inequality check, matching the\n sibling equality tests.\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "1165eb536b019e2f2ad0df2778fd02eae6eeff8f", "tree": "74b49cefcf3da5e5f54d92e1af2ab8e51931851c", "parents": [ "c18b366e3794a76aee2c61fd5b1be91126f55784" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Wed Jul 08 05:05:02 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 08 20:05:02 2026 +0800" }, "message": "fix(spark): demote spurious per-write INFO logs to debug (#19170)" }, { "commit": "c18b366e3794a76aee2c61fd5b1be91126f55784", "tree": "c5b092413c39252271f78367725c5bca72b8d766", "parents": [ "2baa29b14d372354c28056e4737000a40e56ceb8" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Wed Jul 08 04:53:20 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 08 19:53:20 2026 +0800" }, "message": "docs: update javadoc spark-submit examples to Spark 3.5 / Scala 2.12 (#19171)" }, { "commit": "2baa29b14d372354c28056e4737000a40e56ceb8", "tree": "d73d0f647b0ae4398fb87b8234ef2ed816a9c69b", "parents": [ "3015eb944083272459040dc5da5755e683674cb5" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Wed Jul 08 19:36:07 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 08 19:36:07 2026 +0800" }, "message": "feat(index): Refactor metadata table update logic based on index abstraction (#18372)\n\n* feat(index): Refactor metadata table update logic based on index abstraction" }, { "commit": "3015eb944083272459040dc5da5755e683674cb5", "tree": "b1ad361dba55633bcc54a5a8925ede540854ab09", "parents": [ "c56cca6187b1ad399bc42e987dae90baacf4c55d" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Wed Jul 08 18:57:59 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 08 18:57:59 2026 +0800" }, "message": "test(integ-test): add Testcontainers E2E for VECTOR/BLOB/VARIANT Hive… (#19203)\n\n* test(integ-test): add Testcontainers E2E for VECTOR/BLOB/VARIANT Hive sync\n\nTestcontainers-based E2E coverage that a Spark write of Hudi\u0027s custom\nlogical types produces a Hive-syncable schema queryable through a real\nHive metastore. ITTestCustomTypeHiveSync exercises VECTOR and BLOB (SQL +\nDataFrame) plus Spark 4.0 VARIANT (Spark4-gated), asserting write -\u003e\nhive-sync -\u003e Hive DESCRIBE/SHOW PARTITIONS/SELECT round-trips.\n\nRuns on both a Spark 3.5.3 (JDK 11) and Spark 4.0.2 (JDK 17) stack against\nHive 2.3.10, via a new integration-tests-hive-sync CI job. The integ2\npackage manages its own docker stack and is excluded from the legacy\nin-job compose demo to avoid host-port collisions.\n\nScoped decomposition of #18535 (Hive-sync half only): the Trino read E2E\nand two bundled production features (blob partial-struct write, Lance\nVARIANT support) are deferred to follow-up PRs.\n\n* test(integ-test): address review comments on testcontainers harness\n\n- delete unused-and-broken HiveService.runQuery (execInContainer has no\n shell, so the added literal quotes would reach hive -e and break parsing)\n- fix assertStdOutContains javadoc: asserts exactly once, not at least once\n- gate integration-tests-hive-sync CI job on the changes job\u0027s relevance\n output, matching the legacy integration-tests job\n- trim unused harness API: sparkAdhoc2, unused Containers/Paths/Network\n constants, CommandResult.expectToFail/assertExitCodeIs/assertStdErrContains,\n HiveService.executeFile/copyFile, SparkService.copyFile,\n CommandExecutor.copyFileToContainer\n- register compose temp files for deleteOnExit so the temp dir actually\n deletes; add EOF newline; fix ContainerProvider javadoc; rely on root\n dependencyManagement for the managed testcontainers artifact\n\n* test(integ-test): use jdbc hive_sync mode for VARIANT SQL fixture\n\nThe VARIANT SQL fixture was the only one syncing via hms mode while\nretaining a jdbcurl (used only by jdbc mode). Align it with every other\nfixture, including the DataFrame VARIANT fixture that syncs the identical\nVariantType, which all use jdbc.\n\n* test(integ-test): move VERBOSE_HIVECONFS into SystemProps\n\nKeep all verbose-mode config in one place. VERBOSE_HIVECONFS was the only\nconstant at the outer-class level; group it with SystemProps.HIVE_VERBOSE,\nwhich its javadoc already references." }, { "commit": "c56cca6187b1ad399bc42e987dae90baacf4c55d", "tree": "1e18dcea7aa8e5a61e22eda27052eff319bd030c", "parents": [ "a001ea87da7bc374a856a7224f3840141e203ef6" ], "author": { "name": "Danny Chan", "email": "yuzhao.cyz@gmail.com", "time": "Wed Jul 08 13:23:44 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 08 13:23:44 2026 +0800" }, "message": "feat: add FileNameParser for file name decoding (#19198)" }, { "commit": "a001ea87da7bc374a856a7224f3840141e203ef6", "tree": "21cb8d4fe030f3fd460b76d70f0fce854a4ad325", "parents": [ "b84f61823610647fbf56dba53d260741a3e38d23" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Wed Jul 08 09:55:47 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 08 09:55:47 2026 +0800" }, "message": "feat(index): Add Indexer abstraction and refactor metadata table init… (#18348)" }, { "commit": "b84f61823610647fbf56dba53d260741a3e38d23", "tree": "922eddaad344b4164fd7647554bf0f38f037fd42", "parents": [ "ee3692cf3142c293b2ec36e520c252936a8c7af3" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Tue Jul 07 19:24:17 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 07 19:24:17 2026 +0800" }, "message": "fix(reader): Remove redundant partition value conversion in RecordContext (#19201)" }, { "commit": "ee3692cf3142c293b2ec36e520c252936a8c7af3", "tree": "3d910737038be18a498d5731c8cfb83b775c8ded", "parents": [ "a1dbb5620bd126f46fb48a51287b59bbdc3899d8" ], "author": { "name": "ericyuan915", "email": "77124531+ericyuan915@users.noreply.github.com", "time": "Tue Jul 07 04:18:50 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 07 19:18:50 2026 +0800" }, "message": "fix(flink): avoid AIOOBE in NestedColumnReader across the read batch boundary (#19210)\n\nNestedColumnReader#readRow collapses a present row whose children are all null into a NULL row. That loop iterated up to rowPosition.getPositionsCount(), which on a full, non-final vectorized batch is one larger than the materialized child vectors: the Dremel level stream carries a one-record lookahead (NestedPrimitiveColumnReader#readAndNewVector reads one value past the batch, and #getLevelDelegation keeps that trailing level for the next batch). Reading a nested ROW column from a COW base file with more rows than the 2048-row batch (RecordIterators.DEFAULT_BATCH_SIZE) therefore indexed one element past a shorter child vector and threw ArrayIndexOutOfBoundsException.\n\nClamp the collapse loop to the shortest vector it indexes (the row vector and every child), unwrapping ParquetDecimalVector (a non-AbstractHeapVector DECIMAL child) via a new vectorLength helper; the phantom trailing position is never surfaced downstream (ParquetColumnarRowSplitReader caps the batch at num). Applied identically to all five flink modules (1.18.x/1.19.x/1.20.x/2.0.x/2.1.x). Adds integration test ITTestHoodieDataSource#testParquetNestedRowExceedingReadBatch covering both a heap-vector nested row and an isolated decimal-only nested row across the batch boundary.\n\ncloses #19208" }, { "commit": "a1dbb5620bd126f46fb48a51287b59bbdc3899d8", "tree": "24445903785f811c5399dccdfd4aa145c159e398", "parents": [ "aefabf79f707fe5930f4dfc02b7952c19296af00" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Tue Jul 07 19:17:56 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 07 19:17:56 2026 +0800" }, "message": "feat: Support table version 10 to 9 downgrade (#19199)\n\n* feat: Support table version 10 to 9 downgrade" }, { "commit": "aefabf79f707fe5930f4dfc02b7952c19296af00", "tree": "02de1b00d4f5a0691ab25ec850bd779290b3b600", "parents": [ "cf2566cd06c447ca86f5fbe45f9f3a949cd77ca2" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Tue Jul 07 15:40:05 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 07 16:40:05 2026 +0800" }, "message": "test(trino): de-flake testRecordLevelFileSkipping by setting the record-index wait timeout (#19213)\n\ntestRecordLevelFileSkipping enables the record-level index but sets\nwithColumnStatsTimeout(\"10s\") - a no-op here, since column stats is disabled in this\ntest - and never sets the record-index wait timeout, so the async RLI load falls back to\nthe 2s default (HudiConfig#recordIndexWaitTimeout). Under CI load the RLI is not ready in\n2s, so file skipping is incomplete and the query scans 2 splits instead of 1\n(expected: 1 but was: 2).\n\nReplace the no-op withColumnStatsTimeout(\"10s\") with withRecordIndexTimeout(\"10s\"),\nmatching the sibling RLI tests testRLIWithColumnNameUsingUppercaseLetters and\ntestMultiKeyRLIWithColumnNameUsingUppercaseLetters and the earlier de-flake #13869.\nTest-only.\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e" }, { "commit": "cf2566cd06c447ca86f5fbe45f9f3a949cd77ca2", "tree": "72a62ffd14f2d81461b79abac11f958e4d003794", "parents": [ "d291efccaad2ef3428639efe4a38e61765fb8f1d" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Tue Jul 07 11:14:39 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 07 11:14:39 2026 +0800" }, "message": "feat: Support insert-order record positions for native log files (#19200)" }, { "commit": "d291efccaad2ef3428639efe4a38e61765fb8f1d", "tree": "a796d7f97141bb46eae715a2dd03760bde01a3f6", "parents": [ "8a653836db55c1490cd3e3964d22f3e7abc780fe" ], "author": { "name": "vinoth chandar", "email": "vinothchandar@users.noreply.github.com", "time": "Mon Jul 06 09:28:58 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 06 12:28:58 2026 -0400" }, "message": "refactor(common)!: continue hudi-common core/common package reorganization (#19195)\n\n* refactor(common)!: move column stats types under org.apache.hudi.metadata.stats\n\nRelocates the column statistics vocabulary (HoodieColumnRangeMetadata,\nValueType, ValueMetadata, ParquetAdapter, and spark-client\u0027s\nSparkValueMetadataUtils) from org.apache.hudi.stats to\norg.apache.hudi.metadata.stats. These types are metadata-subsystem\nvocabulary: they depend on metadata.HoodieIndexVersion, and\nValueType.ordinal() is persisted in HoodieMetadataColumnStats records\nof the col-stats/partition-stats index. Nesting them under\norg.apache.hudi.metadata - a namespace that is permanently anchored by\npersisted FQNs (HoodieMetadataPayload in every metadata table\u0027s\nhoodie.properties) - gives them a home that never moves again.\n\nBREAKING CHANGE: source/binary break for externally compiled code\nusing these types programmatically (e.g. reading footer ranges via\nFileFormatUtils APIs); fails loudly, fixed by an import change. No\nconfig values, reflection targets, or Kryo registrations reference\nthese FQNs, and nothing persisted carries Java class names - metadata\nrecords store typeOrdinal ints and commit metadata serializes by field\nname. Note ValueType\u0027s enum ORDER is storage-format-locked via the\npersisted ordinal (independent of this move).\n\nVerified across common, hadoop-common, client-common, spark client and\ndatasource, utilities (spark4.1) and flink-client (flink2.1); value\ntype/metadata and metadata payload suites pass.\n\n* refactor(common)!: move timeline service client to org.apache.hudi.common.table.timeline\n\nRelocates TimelineServiceClient and TimelineServiceClientBase (plus\ntest) from the two-class org.apache.hudi.timeline package into\norg.apache.hudi.common.table.timeline, next to the timeline model and\nthe dto package whose payloads the client fetches. Removes the\norg.apache.hudi.timeline package from hudi-common, disambiguating it\nfrom the timeline-service server module namespace.\n\nBREAKING CHANGE: source/binary break for external code importing the\nclient directly; unlikely surface - the client is constructed\ninternally (RemoteHoodieTableFileSystemView, timeline-server-based\nmarkers), never named in configs or loaded reflectively. Fails loudly.\n\n* refactor(common)!: dissolve org.apache.hudi.util grab-bag in hudi-common\n\nContinues the client.utils dissolution from the hudi-client-common\nreorg, dispersing hudi-common\u0027s org.apache.hudi.util by function:\n\n- HoodieFileMetadataMerger -\u003e org.apache.hudi.core.io.storage: it\n merges file footer metadata (bloom filters, column ranges) for the\n parquet binary-copy write path, whose contracts live there; its test\n moves alongside, finally matching its package.\n- Lazy -\u003e org.apache.hudi.common.util (33 consumers, pure substrate).\n- LazyConcatenatingIterator -\u003e org.apache.hudi.common.util.collection,\n and the three engine-agnostic iterators recently consolidated into\n client-common\u0027s org.apache.hudi.util (LazyIterableIterator,\n ConcatenatingIterator, CloseableConcatenatingIterator) are pulled up\n into hudi-common\u0027s org.apache.hudi.common.util.collection with them,\n joining the ClosableIterator family; TestConcatenatingIterator moves\n to hudi-common accordingly. OperationConverter (JCommander-bound) and\n CommonClientUtils (client-domain) remain in client-common.\n- PartitionPathFilterUtil deleted: a single static predicate with two\n callers, now a public helper on HoodieTableMetadataUtil (it encodes\n metadata-table partition-path semantics); callers in\n HoodieBackedTableMetadata and CatalogBackedTableMetadata updated and\n its tests ported.\n- Transient deleted: zero references repo-wide.\n\nBREAKING CHANGE: source/binary break for externally compiled code\nimporting these utilities (all shipped through 1.2.0 at the old FQNs);\nfails loudly, fixed by import changes. None are config values,\nreflection targets, or Kryo-registered. hudi-trino-plugin untouched\n(builds against released Hudi).\n\nVerified across common, hadoop-common, client-common, spark client and\ndatasource, utilities (spark4.1) and flink client/datasource\n(flink2.1); merger, iterator, timeline client and metadata predicate\nsuites pass.\n\n* refactor(common)!: vacate org.apache.parquet.schema namespace from hudi-common\n\nMoves the three hudi classes squatting the parquet namespace to their\nproper homes:\n\n- LogicalTypeParquetAdapter / OriginalTypeParquetAdapter -\u003e\n org.apache.hudi.metadata.stats, next to the ParquetAdapter interface\n they implement. The namespace residency was unnecessary: both use\n only public parquet APIs, and ParquetAdapter.getAdapter() references\n them via class literals (the reflective load exists to defer linkage\n for the parquet \u003c1.11 fallback and is unaffected by the package).\n- HoodieSchemaRepair -\u003e org.apache.hudi.common.schema: it imports zero\n parquet types - it is pure HoodieSchema manipulation and belongs with\n the schema utilities it already uses; its test moves alongside.\n\nhudi-common no longer contributes any classes to org.apache.parquet.*.\nThe hadoop-common parquet-namespace classes (SchemaRepair,\nHoodieAvroParquetSchemaConverter) are intentionally untouched - that\nmodule is the parquet-facing layer; its equivalence test now imports\nthe relocated HoodieSchemaRepair explicitly.\n\nBREAKING CHANGE: source/binary break for external code importing\nHoodieSchemaRepair (shipped in 1.2.0) or the adapters; loud failure,\nimport fix. None are config values or reflection targets by name.\n\n* refactor(common)!: move avro utilities to org.apache.hudi.common.avro\n\nRelocates hudi-common\u0027s org.apache.hudi.avro root classes (16 classes:\nHoodieAvroUtils, AvroSchemaUtils, MercifulJsonConverter, reader\ncontext, serializers, variant/shredding utilities and friends) plus\nthe processors subpackage and tests to org.apache.hudi.common.avro -\navro (de)serialization substrate belongs in the common tier.\n\nTwo residents deliberately do NOT move, and the old namespace persists\nonly for them:\n- org.apache.hudi.avro.model: generated from .avsc schemas whose\n namespace is persisted inside timeline metadata files - storage\n format, immovable.\n- HoodieAvroWriteSupport (hudi-hadoop-common) and its tests: its FQN\n is the default value of hoodie.avro.write.support.class and a\n documented extension point; leaving it in place keeps user configs\n working with no compatibility stub at all. Cross-package references\n between it and the moved classes are now explicit imports.\n\nBREAKING CHANGE: source/binary break for externally compiled code\nusing these utilities programmatically (HoodieAvroUtils is widely\nimported); loud failure, import fix. No other config values,\nreflection targets, or Kryo registrations reference the moved FQNs;\navro record namespaces on storage are untouched.\n\nVerified across nine spark-side modules (spark4.1) and the flink chain\n(flink2.1); avro utils/converter/reader-context suites pass. Known\npre-existing failure TestHoodieAvroParquetWriter#testProperWriting\n(avro LocalDate logical-type cast) reproduces identically on the\nmerge base without this change - environmental, not introduced here.\n\n* refactor(common)!: move HoodieStorageUtils to common.util and index contracts to core.index\n\nTwo namespace-stray resolutions in hudi-common:\n\n- HoodieStorageUtils -\u003e org.apache.hudi.common.util: it was the sole\n hudi-common resident of org.apache.hudi.storage, a package otherwise\n owned by hudi-io - removing it eliminates that split package. It\n cannot live in hudi-io today (reads HoodieStorageConfig from\n common.config); relocating the class to the hudi-io jar remains a\n follow-up for the module split, at which point the package stays\n put (FQN-stable jar move).\n- org.apache.hudi.index.{record,expression,secondary} (6 classes:\n HoodieRecordIndex, HoodieExpressionIndex, secondary index\n manager/type/utils) -\u003e org.apache.hudi.core.index.*: index metadata\n and definition machinery is domain core. Engine implementations\n keep their packages (spark\u0027s HoodieSparkExpressionIndex and\n ExpressionIndexSparkFunctions remain in\n org.apache.hudi.index.expression, mirroring the core-contract vs\n engine-impl split used for core.io), importing the moved contracts\n explicitly.\n\nBREAKING CHANGE: source/binary break for externally compiled code\nimporting these classes; loud failure, import fix. Neither class set\nis referenced by config values, reflection, Kryo, or persisted state -\nindex definitions serialize function and column names, never Java\nclass names, and hoodie.index.class targets the client-common\nHoodieIndex hierarchy which is untouched.\n\nVerified across common, hadoop-common, client-common, spark client and\ndatasource, utilities, hadoop-mr, cli (spark4.1) and the flink chain\n(flink2.1); secondary index and spark expression index suites pass.\n\n* refactor(spark): update lance writer test to relocated avro imports\n\nReconciles new static imports introduced on master by #19118 (after\nthis branch\u0027s avro move) with org.apache.hudi.common.avro." }, { "commit": "8a653836db55c1490cd3e3964d22f3e7abc780fe", "tree": "0aae55b5a4da75a4460792492b9fc7bf9eb6e497", "parents": [ "3cc8fd128b40f3e5a6740873494e2aa83e9c90b8" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Mon Jul 06 16:39:48 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 06 16:39:48 2026 +0800" }, "message": "feat(writer): Switch default to native log format for table version \u003e\u003d 10 (#19118)" }, { "commit": "3cc8fd128b40f3e5a6740873494e2aa83e9c90b8", "tree": "4da6bfae07a77e041dc3a8fd0f4896da29359976", "parents": [ "0d760762907ead5a8f3c5a44b0f2bf1787bdec73" ], "author": { "name": "vinoth chandar", "email": "vinothchandar@users.noreply.github.com", "time": "Sun Jul 05 21:08:56 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Jul 05 21:08:56 2026 -0700" }, "message": "refactor(common)!: reorganize hudi-common packages into core and common namespaces (#19193)\n\n* refactor(common): move lock providers to org.apache.hudi.core.transaction.lock\n\nRelocates InProcessLockProvider and NoopLockProvider out of\norg.apache.hudi.client.transaction.lock in hudi-common, pre-staging the\nfuture hudi-common/hudi-core module split so the eventual module move\nstays FQN-stable.\n\n- InProcessLockProvider: the old FQN is a documented, user-configured\n value of hoodie.write.lock.provider resolved via reflection, so a\n deprecated stub is kept at the old FQN, marked with the new\n @CompatAlias annotation (hudi-io) recording the replacement and the\n 1.3.0 deprecation release. The stub preserves the reflective\n (LockConfiguration, StorageConfiguration) constructor contract.\n- NoopLockProvider: internal-only (upgrade/downgrade nested-lock path),\n moved without a stub.\n- Sites that string-compare the configured lock provider class now use\n InProcessLockProvider.isInProcessLockProvider(), which accepts both\n the current and legacy FQN: the 1ms clock-skew inference in\n HoodieTimeGeneratorConfig, the metadata-table multi-writer guard in\n HoodieMetadataWriteUtils, and the single-writer auto-adjust in\n HoodieWriteConfig. Without this, configs carrying the legacy FQN\n would silently lose those behaviors.\n\nBehavior-preserving otherwise. Legacy-FQN test configs are deliberately\nleft in place as regression coverage for the stub path; a reflective\nload test and a legacy-FQN clock-skew row are added.\n\n* refactor(common)!: move pre-commit validation framework base to org.apache.hudi.core.validator\n\nRelocates BasePreCommitValidator and ValidationContext from\norg.apache.hudi.client.validator to org.apache.hudi.core.validator in\nhudi-common, continuing the core-package staging for the future\nhudi-common/hudi-core module split.\n\nBREAKING CHANGE: both classes shipped in 1.2.0 as\n@PublicAPIClass(EVOLVING) and are the extension point for custom\npre-commit validators on the Flink path (hoodie.precommit.validators).\nA custom validator compiled against 1.2.0 fails at load with\nNoClassDefFoundError and must be recompiled with the new import. This\nis an accepted EVOLVING-API change: the config only ever names the\nuser\u0027s concrete class (FQN unchanged), all shipped validators move\ntogether, and no compatibility stub is provided deliberately - a stub\nat the old FQN would turn externally compiled overrides of\nvalidateWithMetadata(ValidationContext) into dead overloads once the\nparameter type moves, silently skipping the user\u0027s validation, which\nis worse than the loud load failure.\n\nEngine subclasses and utils in hudi-client-common, hudi-utilities and\nhudi-flink update their imports; StreamingOffsetValidator and its test\npreviously resolved the base types via same-package access and now\nimport them explicitly. The test also accessed the protected config\nfield cross-package, which the move surfaced; it now goes through an\naccessor on the test\u0027s mock subclass.\n\nVerified with the client-common, utilities and flink validator suites.\n\n* refactor(common)!: move metrics config classes to org.apache.hudi.common.config.metrics\n\nRelocates the seven metrics config classes (HoodieMetricsConfig and the\nCloudWatch/Datadog/Graphite/Jmx/M3/Prometheus variants) from\norg.apache.hudi.config.metrics to org.apache.hudi.common.config.metrics,\njoining HoodieCommonConfig and HoodieMetadataConfig in the hudi-common\nconfig namespace. The old package was a leftover from the classes\u0027\noriginal hudi-client-common home; the new location also avoids adding\na split package under hudi-io\u0027s org.apache.hudi.common.metrics.\n\nBREAKING CHANGE: source/binary break for code that imports these\nclasses to build configs programmatically (e.g.\nHoodieWriteConfig.Builder#withMetricsConfig) - fails loudly with\nNoClassDefFoundError, fixed by an import change. Regular usage is\nunaffected: all hoodie.metrics.* config keys are unchanged, reporter\nselection is enum-based (no class names in config values), and custom\nreporters via hoodie.metrics.reporter.class are instantiated with a\n(Properties, MetricRegistry) constructor that never exposes these\ntypes. No compatibility stub: inherited static builder methods would\nfail exact-descriptor resolution on the nested Builder types anyway.\nNote: mixing a new engine bundle with an older hudi-aws-bundle fails\nat metrics init for CloudWatch (reflective constructor lookup); bundle\nversions must match, as already required.\n\nVerified across hudi-common, client-common, spark/java clients,\nhudi-spark, utilities, sync-common, hive-sync and hudi-aws with the\nmetrics config/reporter test suites.\n\n* refactor(common)!: move file I/O contract layer to org.apache.hudi.core.io\n\nRelocates the 19 domain-aware file I/O contract classes in hudi-common\nfrom org.apache.hudi.io[.storage] to org.apache.hudi.core.io[.storage]:\nHoodieFileReader/Writer and their factories, HoodieIOFactory,\nHFile/bootstrap readers, HoodieParquetConfigInjector, and the ORC/HFile\nconfig holders. This establishes org.apache.hudi.core as the namespace\nfor upper-layer code that depends on both hudi-io primitives and\nhudi-common domain types (HoodieRecord, HoodieSchema), disambiguating\nit from the hudi-io module\u0027s low-level org.apache.hudi.io namespace.\nEngine implementations (org.apache.hudi.io.storage.hadoop and the\nSpark/Flink packages) do not move; hoodie.io.factory.class defaults\nand values are unchanged.\n\nBREAKING CHANGE: source/binary break for externally compiled\nHoodieIOFactory subclasses (@PublicAPIClass EVOLVING, plugged via\nhoodie.io.factory.class) and HoodieParquetConfigInjector\nimplementations (hoodie.parquet.write.config.injector.class, since\n1.2.0) - both fail loudly at load with NoClassDefFoundError and are\nfixed by an import change plus recompile. Config values themselves are\nuntouched in both cases (they name the user\u0027s or the shipped\nimplementation class). Regular property-based usage is unaffected; no\nconfig key, reflection, serialization or bundle surface changes.\n\nSame-package consumers of the moved contracts (Spark reader/writer\nclasses remaining in org.apache.hudi.io.storage, and tests) now import\nthem explicitly. The hudi-trino-plugin import updates are verified by\nits own CI (module is not in the default reactor), as is the\nhudi-integ-test import and the non-active spark version modules.\n\nVerified with -Dspark4.1 across common, hadoop-common, client-common,\nspark/java clients, hudi-spark, hadoop-mr and utilities, and with\n-Dflink2.1 across the flink modules; HFile reader/factory, file writer\nfactory and parquet config injector suites pass.\n\n* refactor(common)!: remove unused ByteBufferBackedInputFile\n\nDeletes org.apache.hudi.parquet.io.ByteBufferBackedInputFile and with\nit the org.apache.hudi.parquet.io package, which contained nothing\nelse. The class has zero references in the repository - orphaned by an\nearlier refactor of the parquet log-block read path.\n\nBREAKING CHANGE: the class shipped in releases up to 1.2.0, so\nexternally compiled code importing this helper would fail loudly at\nclass load. It is a 30-line self-contained utility; any such user can\ninline it. Never referenced by any config value or reflection path.\n\n* refactor(common)!: move ParquetAdapter next to its stats consumers\n\nRelocates org.apache.hudi.ParquetAdapter to org.apache.hudi.stats,\nthe package of its only two consumers (ValueType, ValueMetadata). The\ninterface is a parquet-version compatibility shim serving column-stats\ntyping exclusively; its two implementations remain in\norg.apache.parquet.schema (deliberate namespace residency for parquet\nAPI access) and are loaded reflectively via class literals, so they\nfollow the move automatically.\n\nBREAKING CHANGE: source/binary break for external code importing this\ninterface - unlikely in practice, since the static getAdapter() factory\nhard-wires the only two shipped implementations. Fails loudly at class\nload if it exists anywhere; never referenced by config or reflection\nby name.\n\n* refactor(common)!: move BaseHoodieTableFileIndex to org.apache.hudi.core.read\n\nRelocates the engine-agnostic file index base (and its unit test) from\nthe org.apache.hudi root package to org.apache.hudi.core.read,\nestablishing the read-side home in the core namespace for domain-aware\nquery/listing abstractions (Phase 4 read-path work lands alongside).\nIt is a consumer of the metadata subsystem, not part of it, so\norg.apache.hudi.metadata was deliberately not chosen.\n\nBREAKING CHANGE: source/binary break for external code importing\nBaseHoodieTableFileIndex or its nested types (PartitionPath,\nFileStatusCache) directly. Engine subclasses shipped by Hudi keep\ntheir FQNs (HiveHoodieTableFileIndex, SparkHoodieTableFileIndex), and\nengine bundles are self-consistent, so existing Presto/Trino\ndeployments are unaffected at runtime; connector source trees that\nimport the base class directly see a compile-time import fix at their\nnext Hudi version bump. Never a config value or reflection target.\n\nThe move surfaced several same-package couplings, now explicit: the\nspark-common scala classes in package org.apache.hudi (file index and\nindex-support family) referenced the base bare and now import it,\nHoodieBaseRelation read the protected metadataConfig field directly\nand now uses the getter, and the read-only introspection methods\ngetAllQueryPartitionPaths/areAllPartitionPathsCached used by tests\nacross the old package boundary are now public.\n\nVerified across common, hadoop-common, hadoop-mr, spark modules and\nutilities with -Dspark4.1; file index suites pass.\n\n* refactor(common)!: move expression package to org.apache.hudi.common.expression\n\nRelocates the engine-agnostic predicate/expression AST (Expression,\nPredicates, Literal, NameReference, bind visitors, StructLike/ArrayData)\nfrom org.apache.hudi.expression to org.apache.hudi.common.expression.\nThe package has no Hudi domain awareness - its only dependencies are\nthe internal type system and Option - and its consumers span the read\npath, engine bridges and metastore sync tools alike, making it shared\nsubstrate (common) rather than domain machinery (core). Keeps the\ndependency layering acyclic for the future module split:\ntypes -\u003e expressions -\u003e domain -\u003e engines.\n\nBREAKING CHANGE: source/binary break for external code constructing\nHudi expressions programmatically, e.g. callers of\nHoodieTableMetadata.getPartitionPathWithPathPrefixUsingFilterExpression.\nFails loudly; fixed by an import change. Expression classes are never\nnamed in configs and never loaded reflectively.\n\n* refactor(spark): fix scala brace import of relocated BaseHoodieTableFileIndex\n\nFollow-up to the core.read move: scala brace-group imports name classes\nwithout their package prefix, so the FQN rewrite missed this one test.\nSplit the moved class out of the brace group.\n\n* refactor(common)!: move internal.schema to org.apache.hudi.common.schema.internal\n\nRelocates the schema-evolution type system (InternalSchema, Type/Types,\ntable-change actions, avro converters, visitors, and the file-based\nschema storage manager) from org.apache.hudi.internal.schema to\norg.apache.hudi.common.schema.internal, nesting it under the existing\norg.apache.hudi.common.schema namespace. Like the expression AST that\nsits on top of it, this is shared substrate consumed across every\nengine and the sync tools - common, not domain core - and \"internal\"\nas a top-level package name was an artifact of its origin.\n\nBREAKING CHANGE: source/binary break for external code importing these\ntypes, e.g. schema-evolution integrations using InternalSchema or\nSerDeHelper directly. Fails loudly at compile/class-load; fixed by an\nimport change. No persisted format impact: serialized schemas (commit\nmetadata and .schema files) carry no Java class names, and none of\nthese classes are config values or reflection targets. All engine\nintegrations (spark, flink, hive, utilities) move in lockstep in-repo.\n\nVerified with the full schema-evolution unit suites (SerDe, avro\nevolution, table changes, converter, schema storage manager) across\nthe spark4.1 chain; version-shim modules for other spark/flink\nversions are import-only changes covered by CI.\n\n* fix(trino): revert import rewrite in trino plugin, it builds against released hudi\n\nhudi-trino-plugin is a vendored copy of the Trino connector that\ncompiles against released Hudi artifacts (dep.hudi.version\u003d1.0.2,\nparent io.trino:trino-root), not against the in-repo modules. The\norg.apache.hudi.core.io.storage package introduced by the file I/O\ncontract move does not exist in 1.0.2, so the earlier blanket import\nrewrite would have broken the plugin build. Restores the two files to\nthe released-version imports; the plugin picks up the new package\nnames whenever dep.hudi.version is bumped to a release containing\nthem.\n\n* fix(common): add compatibility stub for NoopLockProvider old class name\n\nCI (TestSevenToEightUpgrade cases with\nhoodie.write.lock.provider\u003dorg.apache.hudi.client.transaction.lock.NoopLockProvider)\nshowed the old NoopLockProvider FQN is exercised as a user config value\nin the table-upgrade path, invalidating the earlier internal-only\nassessment that let it move without an alias. Restores a deprecated\n@CompatAlias stub at the old FQN, mirroring the InProcessLockProvider\ntreatment, and adds a reflective-load regression test covering the\nexact LockManager code path that failed." }, { "commit": "0d760762907ead5a8f3c5a44b0f2bf1787bdec73", "tree": "60e62e0e9e01ee517ad4599dc2ce934792ea214d", "parents": [ "feea94ba18480cdf37e5af90d1859546e94ecd2c" ], "author": { "name": "vinoth chandar", "email": "vinothchandar@users.noreply.github.com", "time": "Sun Jul 05 18:26:20 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Jul 05 18:26:20 2026 -0700" }, "message": "refactor(client)!: reorganize hudi-client-common internal package layout (#19192)\n\n* refactor(client): remove dead HttpRequestClient and relocate one-off utility classes\n\n- Delete org.apache.hudi.util.HttpRequestClient (zero usages repo-wide)\n- Move HilbertCurveUtils to hudi-spark-client org.apache.hudi.sort, next to its\n sole consumer SpaceCurveSortingHelper; hilbert-curve dependency moves with it\n (dropped from flink/kafka-connect bundle shade lists, no longer in their tree)\n- Move DistributedRegistryUtil to hudi-spark-client org.apache.hudi.metrics,\n next to DistributedRegistry (sole consumer is SparkRDDWriteClient)\n- Move ExecutorFactory to org.apache.hudi.execution where all its consumers live\n- Removes the single-class org.apache.hudi.optimize package\n\nBehavior-preserving; no moved class is @PublicAPIClass, config-FQN-referenced,\nor Kryo-registered.\n\n* refactor(client): dissolve client.utils grab-bag and flatten timeline versioning packages\n\nDisperse org.apache.hudi.client.utils by feature (FQN-stable class names, packages change):\n- ArchivalUtils, ArchivalMetrics, LegacyArchivedMetaEntryReader -\u003e client.timeline\n- TransactionUtils -\u003e client.transaction\n- PreWriteValidatorUtils -\u003e client.validator\n- DeletePartitionUtils -\u003e table.action.commit (its consumers)\n- FileSliceMetricUtils -\u003e table.action (consumed by compaction/clustering strategies)\n- LazyIterableIterator, ConcatenatingIterator, CloseableConcatenatingIterator,\n OperationConverter -\u003e org.apache.hudi.util (joins hudi-common\u0027s iterator siblings)\n- Delete MergingIterator, ClosableMergingIterator (zero consumers repo-wide)\n\nFlatten client/timeline/versioning/{v1,v2} (1-2 files each) into client.timeline:\nTimelineArchiverV1, TimelineArchiverV2, LSMTimelineWriter; class names already\ncarry the version. Tests move to mirror; TestCommitMetadataResolver relocated to\norg.apache.hudi.client where its subject lives.\n\nBehavior-preserving; no moved class is @PublicAPIClass, config-FQN-referenced,\nor Kryo-registered. o.a.h.client.utils continues to exist in engine modules for\ntheir own residents.\n\n* refactor(client): flatten transaction lock and commit callback package trees\n\n- Fold client/transaction/lock/models (6 storage-lock classes) and\n lock/metrics (HoodieLockMetrics) into client.transaction.lock, beside\n StorageBasedLockProvider and StorageLockClient they serve; \u0027models\u0027 was a\n filler name. The audit/ subpackage stays (cohesive feature, public API).\n- Fold callback/util (HoodieCommitCallbackFactory, HoodieWriteCommitCallbackUtil)\n into org.apache.hudi.callback next to the interfaces they build.\n- Move callback/client/http/HoodieWriteCommitHttpCallbackClient to callback.impl\n next to HoodieWriteCommitHttpCallback, its only caller.\n\nFQN-frozen classes (lock providers, callback interfaces/impl/message) untouched.\nBehavior-preserving; no moved class is @PublicAPIClass, config-FQN-referenced,\nor Kryo-registered.\n\n* refactor(client): fold single-class index subpackages and move column stats index utils\n\n- HoodieInMemoryHashIndex (index/inmemory), HoodieSimpleIndex and\n HoodieGlobalSimpleIndex (index/simple) move to org.apache.hudi.index root;\n the impls are instantiated via IndexType enum switch, not by FQN, and 1-2\n class subpackages added no grouping value. bloom/ and bucket/ stay.\n- HoodieColumnStatsIndexUtils moves from org.apache.hudi.client to\n org.apache.hudi.metadata: it maintains the col-stats metadata index\n partition, beside HoodieTableMetadataUtil it delegates to.\n\nBehavior-preserving; no moved class is @PublicAPIClass, config-FQN-referenced,\nor Kryo-registered.\n\n* refactor(client): carve org.apache.hudi.io.cdc out of the io handle package\n\nMove the 7 internal CDC log-writing classes (HoodieCDCLogger, HoodieNativeCDCLogger,\nHoodieAvroNativeCDCLogger, HoodieNativeCDCFileWriter, HoodieNativeLogFormatWriter,\nHoodieCDCLogWriter, HoodieCDCLogWriterFactory) into org.apache.hudi.io.cdc; they are\nthe one distinct feature cluster in the 42-class io package. Write handles stay in\nio root (user-pluggable by FQN).\n\nHoodieCDCLogWriterFactory and its factory methods widen to public (previously\npackage-private, accessed same-package); HoodieNativeCDCFileWriter stays\npackage-private inside the cdc package.\n\nBehavior-preserving; none of these classes is @PublicAPIClass,\nconfig-FQN-referenced, or Kryo-registered.\n\n* refactor(client)!: move record size estimator classes under org.apache.hudi.execution.estimator\n\nRelocate RecordSizeEstimator (@PublicAPIClass), AverageRecordSizeEstimator and\nRecordSizeEstimatorFactory from the 3-class top-level org.apache.hudi.estimator\npackage to org.apache.hudi.execution.estimator, beside the write-execution\nmachinery that consumes them. Config docs for\nhoodie.record.size.estimator.class updated to the new FQNs.\n\nBREAKING CHANGE: users who explicitly set hoodie.record.size.estimator.class\nto org.apache.hudi.estimator.AverageRecordSizeEstimator get a loud\nClassNotFoundException at write start (fix: update the config value); custom\nestimators compiled against the old package fail with NoClassDefFoundError\n(fix: import change + recompile). Config default tracks the move\nautomatically; no compat stub by design - every failure is loud, no silent\nmisbehavior is possible. Accepted by VC 2026-07-05, tracked as BC-5.\n\n* refactor(client): rehome async delegates, commit metadata resolvers, and streaming metadata handler\n\nTidy the org.apache.hudi.client root down to genuinely client-scoped classes:\n- BaseCompactor, BaseClusterer -\u003e org.apache.hudi.async: their only consumers\n are the async table services; they are the async delegation contract.\n- CommitMetadataResolver, CommitMetadataResolverFactory,\n MarkerBasedCommitMetadataResolver, NoOpCommitMetadataResolver -\u003e\n org.apache.hudi.table.action.commit, beside BaseCommitActionExecutor which\n drives them. The factory is a pure internal version/engine switch - not a\n user plug point.\n- StreamingMetadataWriteHandler -\u003e org.apache.hudi.metadata, joining the\n HoodieTableMetadataWriter it orchestrates; getMetadataWriter widens to\n protected for the cross-package engine subclasses.\n\nClients, client mixins, the frozen WriteStatus family and its stat value\nobjects, and CommitMetadataProperties stay in org.apache.hudi.client.\n\nBehavior-preserving; no moved class is @PublicAPIClass, config-FQN-referenced,\nor Kryo-registered.\n\n* style(client): restore import ordering after package relocations\n\nRe-sort the org.apache.hudi import groups disturbed by the P0.9.11 package\nmoves; no code changes. Checkstyle and scalastyle now pass on all touched\nmodules." } ], "next": "feea94ba18480cdf37e5af90d1859546e94ecd2c" }