)]}' { "log": [ { "commit": "1d3a0affad873868e74df7207013ae8eeb4dbb8c", "tree": "e4236377cdabf33b34590d5ae4f5d76a0130cf52", "parents": [ "a46909b09bbb044acda01a41dfc90ea844fe05b3" ], "author": { "name": "zhaoyudi-creator", "email": "72803958+zhaoyudi-creator@users.noreply.github.com", "time": "Fri Sep 11 14:48:18 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 11 14:48:18 2026 +0800" }, "message": "fix(common): anchor a file group\u0027s first slice on a committed log (#19785)\n\nA MOR file group can end up with its latest file slice keyed on a base\ninstant that never committed. For example, under NBCC with the bucket\nindex the earliest delta commit on a file group fails (or is rolled back)\nwhile later delta commits on the same group succeed. Log files are\nattributed to a slice by completion time, so when the group is first\nbuilt the failed instant\u0027s log arrives first and opens a slice keyed on\nthat uncommitted instant; every later committed log is then attributed to\nthe same slice.\n\nisFileSliceCommitted only checks whether the slice\u0027s base instant itself\ncommitted, so the whole slice -- including its committed log files -- is\ntreated as uncommitted and dropped from the reader view, silently losing\ncommitted data.\n\nFix this at file-group construction instead of weakening the visibility\ngate. When the group has no slice yet, the earliest completed log now\nestablishes the initial slice, so the slice is anchored on a real\ncommitted instant. An earlier pending log then follows the normal\npending-log rule and attaches to that committed slice, instead of\ncreating its own uncommitted slice that would also hide every later\ncommitted log in the group. isFileSliceCommitted keeps its original\nsingle invariant, so all existing and future view APIs benefit without\nper-call trimming.\n\naddLogFiles now owns the sort and the re-anchoring for a batch of log\nfiles; addLogFile becomes private and the file-system view calls the\nbatch API. Sorting is done once and reused by both the re-anchor scan and\nthe attach loop.\n\nTests:\n- TestHoodieFileGroup#testUncommittedFirstLogDoesNotAnchorCommittedLogs\n verifies an uncommitted earliest log does not anchor the slice; the\n first committed log becomes the base instant and the pending log\n attaches to it.\n- TestHoodieFileGroup#testUncommittedBaseFileSliceStaysHiddenDespiteCommittedLogs\n verifies a slice with an uncommitted base file stays hidden even when\n later committed logs land in the same file group.\n- TestHoodieTableFileSystemView#testUncommittedFirstLogUsesFirstCommittedLogAsBaseInstant\n verifies the file-system view anchors the raw slice on the first\n committed log and surfaces only the committed logs.\n\nCloses #19774" }, { "commit": "a46909b09bbb044acda01a41dfc90ea844fe05b3", "tree": "0e86b9f3bf7e7af857e58dd71b2c8762d49cec79", "parents": [ "11527aecb083251d098be627cb02955d388f10d2" ], "author": { "name": "Venkateswarlu Boggavarapu", "email": "mailtoboggavarapu@gmail.com", "time": "Thu Sep 10 22:44:44 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 11 10:44:44 2026 +0800" }, "message": "fix: close InflaterInputStream in HoodieAvroDataBlock.decompress() to prevent native memory leak (#19861)" }, { "commit": "11527aecb083251d098be627cb02955d388f10d2", "tree": "e03a65823f4a96dc37b4b2bfa4bdc447ae9af230", "parents": [ "44b7ed322575ba9678ac77d6da7d122ac3635175" ], "author": { "name": "Peter Huang", "email": "huangzhenqiu0825@gmail.com", "time": "Thu Sep 10 19:39:58 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 11 10:39:58 2026 +0800" }, "message": "fix(flink): bootstrap operator for bounded source (#19844)\n\n* fix(flink): bootstrap operator for bounded source\n* refactor(flink): group bounded source bootstrap routing\n\n---------\n\nCo-authored-by: danny0405 \u003cyuzhao.cyz@gmail.com\u003e" }, { "commit": "44b7ed322575ba9678ac77d6da7d122ac3635175", "tree": "fa88fdfd8a07bd233cf96e5f390fded82d6a7d1f", "parents": [ "648996f002c3c9e24a1e2bf53f97fe03e8bc4c81" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Fri Sep 11 01:23:16 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 11 01:23:16 2026 +0800" }, "message": "chore(ci): align Codecov range with 80% target (#19896)\n\nThe dashboard bar is a three-state indicator, not a gradient: red\nbelow the range lower bound, yellow in between, and green only at\nor above the upper bound. With range \"50...100\" the bar could\nnever be green short of 100% coverage, so it sat yellow at 80.22%.\n\nSet the range to \"70...80\" so it matches the coverage target the\nproject committed to. This is cosmetic only: coverage.range does\nnot feed the status checks, which stay informational." }, { "commit": "648996f002c3c9e24a1e2bf53f97fe03e8bc4c81", "tree": "3f264125cad93d5c9824d24af7604d02d654158b", "parents": [ "8af07e1e1534c92e1ba97dbc1e18a50f73f6c65c" ], "author": { "name": "Sivabalan Narayanan", "email": "n.siva.b@gmail.com", "time": "Thu Sep 10 09:29:10 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 16:29:10 2026 +0000" }, "message": "feat(agent): add Hudi Architect design-time skill (#19380)\n\n* feat(agent): add Hudi Architect design-time skill\n\nAdds a specialized, design-time agent skill that turns workload\nrequirements into a Hudi table architecture: an Architecture Decision\nRecord, a grouped hoodie.* config bundle, and a runnable submit command.\nIt asks workload questions (\"how often does data land?\", \"what do\nconsumers filter on?\") rather than Hudi questions (\"which index type?\"),\nand derives the Hudi decisions from the answers.\n\nLands under hudi-agent-gateway/skills/ as a first specialist agent for\napache/hudi#19263. The gateway\u0027s own loop is grounded and read-only --\nit answers from tool results over tables that exist. Table design is the\nopposite shape: generative, interrogative, and about a table that does\nnot exist yet, so there is nothing to query. That is the narrower-\ntoolset argument in #19263 applied to design time.\n\nContents:\n- SKILL.md -- flow control, tier gating, question-delivery contract\n- references/question-flow.md -- round-by-round questions with gating\n- references/decision-tables.md -- derivation per decision domain\n- references/warnings.md -- 14 named traps with trigger + firing point\n- references/config-templates.md -- hoodie.* templates and sample bundles\n- references/adr-template.md -- ADR output structure\n- RUNBOOK.md -- operator guide for data engineering / ETL teams\n- validate_config_keys.py -- validates every hoodie.* key referenced by\n the skill against ConfigProperty definitions in the source tree\n (83 keys, exit 1 on unknown), plus an allowlist with a documented\n reason per entry\n\nScope: this is deliberately knowledge-and-flow only, with no rule engine.\nConfig correctness rests on the reference tables plus the key validator\nrather than on model inference. A hallucinated hoodie.* property becomes\na durable table property, not a re-runnable wrong answer, so the\nderivation logic is owed as a follow-up before this drives anything\nunattended.\n\nFour workload archetypes were driven end-to-end while writing this;\nthe fixes from those runs are folded in, including three config defects\nthat would otherwise have shipped: hoodie.compaction.target.io is\ndenominated in MB rather than bytes, hoodie.metadata.record.index.* are\ndeprecated aliases for the global properties rather than the partitioned\nones, and an initialized record index file-group count is durable rather\nthan resizable.\n\nCo-Authored-By: codope \u003csagarsumit09@gmail.com\u003e\n\n* feat(agent): derive multi-writer concurrency in the Hudi Architect skill\n\nMulti-writer was the largest gap declared in this PR\u0027s \"where M1 stops\ntoday\" table: the flow asked whether a second writer existed, then emitted\nan OCC skeleton with a `\u003clock provider class\u003e` placeholder and a blocking\nopen question. It now derives the concurrency mode, selects a lock\nprovider, and emits a complete runnable block.\n\nEvery config key and class name here is verified against 1.2.0 source\nrather than the published docs, which disagree in three places:\n`hoodie.cleaner.policy.failed.writes` is a deprecated alias for\n`hoodie.clean.failed.writes.policy`; DynamoDB validates `table` + `region`\nrather than the documented `endpoint_url`; and `InProcessLockProvider`\nhas moved to `org.apache.hudi.core.transaction.lock`.\n\nDecision rubric (decision-tables.md -\u003e Concurrency, placed after all\ntable services since a standalone service job is itself a writer):\n\n- Mode: OCC by default. NBCC only when the design has already landed on\n MOR + simple bucket index for its own reasons, gated on the constraint\n HoodieWriteConfig enforces. Table type and index are never reshaped to\n reach NBCC -- bucket count is durable and concurrency mode is not, so\n that trade runs the wrong way.\n- Provider: ordered by maturity, not convenience. DynamoDB (0.10.0) on\n AWS, an existing ZooKeeper quorum or Hive Metastore otherwise. The\n storage-based provider (1.0.2) is offered only where no lock\n infrastructure exists, with its maturity stated -- a lock provider is\n the wrong place to be an early adopter, since the cost of an\n undiscovered edge case is silent corruption.\n- Identity: emit the implicit provider variants, which derive the lock\n from `hoodie.base.path`. The explicit variants require operators to\n keep a lock key identical across every writing job by hand; when two\n disagree, each takes its own lock, both succeed, and nothing logs.\n\nThe writer question moves from Round 3 to Round 1 and fires at every tier\nbut EXPLORATION. A second writer is a correctness fact rather than a scale\none, and it costs a single question about something the user already\nknows. Round 3 keeps a refinement step, because a standalone compactor or\nasync clustering job derived later is itself an uncounted writer.\n\nSix warnings added: NBCC_INELIGIBLE, NBCC_CLUSTERING_CONFLICT,\nSTORAGE_LOCK_MATURITY, LOCK_PROVIDER_MISMATCH (fires only on explicit\nproviders, the sole configuration where the failure is reachable),\nFILESYSTEM_LOCK_UNSAFE, OCC_INSERT_DUPLICATES.\n\nThe ADR\u0027s blocking open question becomes a pre-launch checklist: the\nprovider and config are decided, so what remains is applying them to\nevery writing job.\n\nvalidate_config_keys.py gains the hudi-aws and hudi-gcp source globs.\nWithout them the DynamoDB lock keys resolve as unknown and correct\ncontent fails the check. 100 keys verified, up from 83.\n\n* feat(agent): ask what lock infrastructure exists before recommending one\n\nThe provider question presented four locking backends as a menu, as though\nthe user were choosing fresh. Most arrive with the answer already settled\nby their platform -- their org runs ZooKeeper, or standardizes on DynamoDB\nlocking, or a sibling Hudi table made the call. Handing that user a menu\nasks them to re-decide something settled, and invites changing a working\nanswer. It also breaks the skill\u0027s own rule against asking questions whose\nanswer would not change the recommendation.\n\nRestructured into three steps:\n\n- Step A asks what they already have, always, before naming any backend.\n An existing platform standard is taken as the answer and not\n re-litigated; the only override is a genuine incompatibility, such as\n filesystem-based locking on cloud storage, which is a correctness\n objection rather than a preference.\n- Step B educates only when Step A shows they need it, and leads with what\n a lock provider does in operational terms before naming products. The\n comparison table carries a maturity column and is filtered to what\n applies -- no DynamoDB row for a GCP user.\n- Step C discloses the lock defaults and takes an acknowledgement, reusing\n the skill\u0027s existing disclosed-defaults mechanism. Values are stated in\n operational terms rather than as key names (\"wait up to 60s to acquire a\n lock, 15 retries a second apart\"), with a multi-select router for anyone\n who wants to own specific values. A value the user explicitly chooses is\n emitted even when it equals the default, since it now records a decision\n rather than restating a default. Step C is skipped when Step A found a\n platform standard, which already owns these values.\n\nConfig coverage, checked against every lock ConfigProperty in source: all\neleven no-default keys are accounted for -- emitted, deliberately omitted\nbecause an implicit provider derives them, or documented as optional.\n`dynamodb.table` and `.region` carry defaults but are hard-validated in\nDynamoDbBasedLockConfig, so they are emitted as required.\n\nAlso documents that `hoodie.write.lock.storage.heartbeat.poll.secs` is a\ndeprecated alias for `renew.interval.secs`, notes why DynamoDB\u0027s default\nPAY_PER_REQUEST billing suits a lock table, and extends the not-emitted\ntable to cover the ZooKeeper timeouts, DynamoDB capacity keys, `app_id`,\nand `endpoint_url` with reasons.\n\n* feat(agent): derive catalog and metastore sync in the Hudi Architect skill\n\nCatalog sync was declared out of scope in RUNBOOK.md: a user whose table\nis read by Trino or Athena got a design that would produce a table those\nengines cannot see, with a pointer to the docs. The flow now asks which\nengines query the table and derives the sync config from the answer.\n\nCovers Hive Metastore, AWS Glue, BigQuery, and DataHub, with Polaris\nrouted. HMS is the default and Glue is the default on AWS, matching where\nusers actually are; the rest are reached only when named.\n\nThe question is \"which engines query this table\", never \"do you want Hive\nsync\". Spark and Flink read by path, so a Spark-only pipeline correctly\ngets no catalog at all -- stated explicitly, since silence there reads as\nan omission rather than a decision. As with the lock provider, the flow\nasks what metastore the user already runs before recommending one.\n\nConstraints that are easy to get wrong, each verified against source:\n\n- Partition extractor must match the partitioning. The common cases are\n inferred, but a TimestampBasedKeyGenerator producing yyyy/MM/dd needs\n SinglePartPartitionValueExtractor, or the catalog gets three partition\n columns instead of one date and date filters stop pruning. This is a\n sync that succeeds and then produces a table nobody can query correctly.\n- MOR registers two tables, _ro and _rt. Both are named in the ADR along\n with which consumers should use which; the wrong suffix gives stale data\n or unexpected latency with no error either way.\n- BigQuery accepts MOR -- BigQuerySyncTool handles both table types -- but\n its manifest lists base files only, so BigQuery reads read-optimized\n rather than snapshot. The published docs describe this as MOR being\n unsupported, which understates what happens: the sync succeeds and the\n data merely looks stale.\n- Glue writes a new catalog version per commit, since\n hoodie.datasource.meta_sync.condition.sync defaults to false. At a\n 15-minute cadence that is ~96 versions a day for an unchanged schema.\n- A sync failure does not fail the write, so the catalog falls behind\n silently. This is an operational-playbook item, not a writer concern.\n- Each non-HMS sync tool lives in its own module, so a missing bundle\n fails at the first commit rather than at submit time.\n\nDataHub is treated as additive rather than an alternative: it publishes\nmetadata over REST and registers the table nowhere a query engine looks,\nso a user who calls it their catalog and needs Trino needs both. Multiple\ntools comma-separate in hoodie.meta.sync.client.tool.class.\n\nFive warnings added: PARTITION_EXTRACTOR_MISMATCH, GLUE_SYNC_VERSION_CHURN,\nCATALOG_SYNC_SILENT_STALENESS, BIGQUERY_MOR_READ_OPTIMIZED, and\nCATALOG_BUNDLE_REQUIRED.\n\nOne more published-docs correction: there is no\nhoodie.gcp.bigquery.sync.base_path. The BigQuery docs list it as a config;\nBigQuerySyncConfig defines table_name and takes the location from the\nstandard base-path config. Allowlisted as negation prose, since the\nreference now warns users away from it.\n\n130 config keys verified against source, up from 100.\n\n* feat(agent): put cloud bundles in the emitted submit command\n\nTwo fixes from a dry run of the catalog-sync flow.\n\n**Cloud bundles were named in prose but never in the submit command.** A\ndesign with DynamoDB locking or Glue sync needs hudi-aws-bundle on the\nclasspath, and the properties bundle is valid without it -- so the job\nsubmits, starts, and dies at the first commit with a ClassNotFoundException\non a class the properties file names. Correct config that cannot possibly\nwork is the worst shape of output this skill can produce, and burying the\nrequirement in reference prose means the user reads it after the failure\nrather than before.\n\nThe bundle now belongs in the emitted `--packages` line. Added a mapping\nfrom derived decision to required artifact, covering the DynamoDB and\nstorage-based lock providers alongside the Glue, BigQuery, and DataHub sync\ntools; hudi-aws-bundle serves both AWS cases, which is stated explicitly so\na reader seeing one jar justified twice does not think they need two. The\nHoodieStreamer template carries a `\u003cCLOUD_BUNDLE\u003e` slot that is substituted\nat emission rather than left for the user -- unlike `\u003cMEM\u003e` or `\u003cSCALA\u003e`,\nthis one is derived, and a literal placeholder would leave them guessing.\n\nThe Spark DataSource path gets explicit treatment because it is easier to\nmiss there: the skill emits `.option(...)` calls and never sees the user\u0027s\nlaunch command, so the requirement is shown against spark-shell /\nspark-sql / spark-submit directly.\n\nArtifact coordinates verified against packaging/: all three bundles are\nunsuffixed, unlike hudi-spark\u003cversion\u003e-bundle_\u003cscala\u003e.\n\n`CATALOG_BUNDLE_REQUIRED` renamed to `CLOUD_BUNDLE_REQUIRED`, since locking\nneeds the same jar and the old name implied catalog sync was the only\ntrigger. It fires once per bundle rather than once per decision.\n\n**Storage scheme now precedes the query-engine question.** The dry run\nproduced a contradiction -- GCS storage with Athena, which is AWS-only and\ncannot read GCS -- because the two were asked side by side. A widget\u0027s\noptions are fixed when it renders, so a question whose option set depends\non a gate cannot share a screen with that gate. Storage is now Q2.1a and\nfilters Q2.1b\u0027s engine list; Athena is separated from Trino/Presto, which\nneed the same kind of registration but are not interchangeable across\nclouds. If a contradiction still arrives the flow names it and asks rather\nthan guessing, since the two readings imply different catalogs.\n\nThe general rule is in SKILL.md, alongside the existing never-split-a-gate\nrule: never merge a gate with a question whose options it narrows either.\n\n* feat(agent): gate catalog sync behind one question before asking for detail\n\nThe catalog section derived whether sync was needed from the engine list,\nwhich inverted the order: a user who had already decided they wanted no\ncatalog answered storage and engine questions before anything established\nthey wanted sync at all. Plenty of tables need none -- Spark-only\npipelines, intermediate tables in a chain, anything whose consumers read\nby path -- and deriving that from three questions the user did not need to\nanswer is the same waste as one unnecessary question, multiplied.\n\nQ2.1a now gates the section. Storage becomes Q2.1b and engines Q2.1c,\nreached only when the gate opens.\n\nTwo things make the gate honest.\n\nIt is phrased as an outcome rather than a Hudi feature: \"does this table\nneed to show up in a catalog so SQL engines and BI tools can find it by\nname?\", not \"do you want meta sync?\". A user may have no name for the\nthing while knowing perfectly well whether their tables appear in Athena.\n\nAnd it states that the decision is reversible. Catalog sync is not durable\n-- it can be enabled later against a live table by running the standalone\nsync tool once and then adding the properties, with nothing rewritten. A\nuser who does not know that may say yes defensively to avoid being locked\nout of an option they were never at risk of losing.\n\nThere is one genuine exception, now surfaced at the gate rather than\ndiscovered later: BigQuery sync requires hive-style partitioning, and that\nIS fixed at table creation. A table that might later be read from BigQuery\nshould set hoodie.datasource.write.hive_style_partitioning\u003dtrue today even\nwith no sync configured, since it costs nothing and preserves the option.\nRaised only on GCS or when the user hints at a future BigQuery consumer.\n\n\"Not sure\" is the one answer that still derives from the engine list, which\nis the right behavior -- a user who names Athena has answered yes without\nknowing the term. Storage is captured even when the gate closes the\nsection, because the lock-provider derivation needs it.\n\nThe general rule is in SKILL.md beside the existing never-ask-a-question-\nwhose-answer-changes-nothing principle: gate optional sections with one\nplain question, phrase it as an outcome, and say whether declining is\nreversible.\n\n* docs(agent): explain Apache XTable and hand off, rather than configure it\n\nCross-format interop was the remaining gap in the catalog section. It gets\na mention rather than a rubric, because XTable is not a Hudi sync target in\nthe way Hive Metastore or Glue are:\n\n- The sync class is org.apache.xtable.hudi.sync.OneTableSyncTool, in\n XTable\u0027s repo, not Hudi\u0027s.\n- The hoodie.onetable.* keys are defined in XTable\u0027s codebase, so\n validate_config_keys.py cannot check them. Every other hoodie.* key this\n skill emits resolves to a ConfigProperty in this source tree; emitting\n unverifiable keys beside verified ones would misrepresent how much of\n the output is actually checked.\n- The bundle is built and versioned by XTable (still a SNAPSHOT artifact),\n so its coordinate moves independently of the Hudi version targeted here.\n- XTable is incubating at the ASF, so its config surface is less settled\n than the sync tools that ship in this repo.\n\nFour XTable mentions exist in the whole Hudi tree, one of them RFC-41\nmarked ABANDONED, and no XTable code. Writing a config rubric would mean\ntranscribing another project\u0027s docs with no way to keep them honest --\nprecisely the failure mode this skill exists to avoid.\n\nSo: no question in the flow, no config templates. A short section that\nfires only when the user\u0027s own answers make it relevant -- they name an\nIceberg or Delta consumer, or ask about interoperability -- explaining that\nXTable translates metadata rather than copying data, naming the two\nintegration shapes, and pointing at xtable.apache.org, the incubator repo,\nand Hudi\u0027s own syncing_xtable page for follow-up.\n\nOne design-time coupling is worth surfacing, because it is durable: the\nXTable FAQ lists MOR tables as unsupported for both Hudi and Iceberg,\nCopy-on-Write only. Hudi\u0027s page does not mention this. If it holds, a\nworkload needing Iceberg or Delta interop should probably be CoW, and\ntable type is fixed at creation. New XTABLE_MOR_UNSUPPORTED surfaces that\ntension with both options rather than silently reshaping the table type.\n\nThe claim is attributed rather than asserted: it appears on their FAQ but\nnot in the repo README, so it cannot be corroborated against code, and the\nwarning tells the user to confirm it against current docs since an\nincubating project\u0027s support matrix moves.\n\nAlso notes that XTable is not a catalog -- it translates format metadata\nand registers the table nowhere a SQL engine looks, so cross-format reads\nplus engine discovery needs XTable and a catalog sync both.\n\n* fix(agent): add ASF license header to hudi-architect allowlist\n\nSigned-off-by: codope \u003csagarsumit09@gmail.com\u003e\n\n---------\n\nSigned-off-by: codope \u003csagarsumit09@gmail.com\u003e\nCo-authored-by: codope \u003csagarsumit09@gmail.com\u003e" }, { "commit": "8af07e1e1534c92e1ba97dbc1e18a50f73f6c65c", "tree": "ce5f9a0d3014068b3653c5679ea73daf15770bfa", "parents": [ "634110752b2f0bd1fbcba265bb3e4f3252368269" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Thu Sep 10 16:59:31 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 16:59:31 2026 +0800" }, "message": "fix(utilities): validator Config.equals and drop-partitions hive sync, with tests for the standalone tools (#19875)\n\nThree bugs in the standalone hudi-utilities tools, found while adding\ntests for tools that had no coverage at all: HoodieDropPartitionsTool,\nHoodieDataTableValidator, TableSizeStats and HoodieTTLJob.\n\n- HoodieDataTableValidator.Config.equals cast its argument to the\n sibling HoodieMetadataTableValidator.Config, so comparing any two\n instances threw ClassCastException. It now casts to its own Config.\n All four tools compare basePath through Objects.equals (a default\n instance NPEd) and drop help from hashCode, which equals never\n compared, so every Config hashes over exactly the fields it\n compares.\n- HoodieDropPartitionsTool --sync-hive-meta could never work:\n buildHiveSyncProps stored two ConfigProperty objects as Properties\n keys, so HiveSyncConfig threw ClassCastException before any\n metastore contact. It now uses the keys and string values for the\n boolean flags.\n- HoodieDropPartitionsTool validated the hive arguments only after\n masking the partitions, so a typo in --hive-database cost the\n partitions first. The DELETE arm now verifies them before anything\n is dropped (dry runs unchanged), and --hive-partition-field is\n required with --sync-hive-meta: its empty default was written into\n the sync props verbatim, which blocked the inference from the table\n config and made HiveSyncTool skip every partition.\n\nTests, one class per tool, each writing a small three-partition COW\ntable and running the tool in-process:\n\n- TestHoodieDropPartitionsTool (10): dry-run reports the file ids of\n exactly the named partitions and changes nothing; delete writes one\n replacecommit masking exactly those partitions; props sourcing;\n unsupported mode; hive arguments verified before the drop; a failing\n metastore connection after the drop is committed; Config contracts.\n- TestHoodieDataTableValidator (7): healthy table; dangling base file\n with and without --ignore-failed, asserted through the validator\u0027s\n log; extra file for a completed commit; continuous mode stopping on\n the first failure, bounded by a timeout; Config contracts.\n- TestTableSizeStats (15): table and partition stats; the date\n interval filters including the half-open bounds and a partition\n dated yesterday for --num-days; base paths from --props-path; the\n error branches; Config contracts.\n- TestHoodieTTLJob (2): both constructors; expired partitions are\n replaced and the fresh one survives.\n- Shared test utilities: CapturingLogAppender for tools whose only\n output is their log, and ToolTestUtils for the helpers the classes\n had in common.\n\nEach fix has a test that fails without it. main() of each tool and\nHudiHiveSyncJob (needs a metastore) stay uncovered on purpose." }, { "commit": "634110752b2f0bd1fbcba265bb3e4f3252368269", "tree": "2d79b00a96383bf9c5a5becb81a08380813d09cf", "parents": [ "4be4864082d7d787d70dec6cf2a91f7f04866eb8" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Thu Sep 10 16:56:41 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 16:56:41 2026 +0800" }, "message": "test(cli): cover timeline, export, metadata commands and the SparkMain helpers (#19877)\n\nSecond round of hudi-cli coverage after #18816 wired the module into\nCI. TimelineCommand, ExportCommand and most of MetadataCommand had no\ntests, and the SparkMain helpers were only reached through the\nspark-submit child process the shell commands launch, which JaCoCo\nnever sees.\n\n- TestTimelineCommand (new, 8): timeline show active / incomplete and\n metadata timeline show active / incomplete with every option; the\n fixture holds completed, requested and rolled-back commits plus a\n pending rollback so both rollback annotations render.\n- TestExportCommand (new, 2): export instants, decoding the exported\n files through the table serde and asserting the written partitions,\n and the invalid-folder guard.\n- TestMetadataCommand (+4): stats, list-partitions, list-files with\n and without a partition, validate-files in both verbose arms with a\n planted stray base file, create and init, set.\n- TestCompactionCommand (+6): compaction validate on a healthy plan\n and on one with a deleted log file, repair, unschedule plan and\n unschedule file with and without dryRun, and schedule-and-execute\n compaction on a MOR table with log files.\n- TestClusteringCommand (new, 1), TestRollbacksCommand (+2) and\n TestSavepointsCommand (+1): the cluster, rollback and savepoint\n entry points, including the failure return codes.\n- CLIFunctionalTestHarness gains renderedRows, a parser for\n HoodiePrintHelper tables so tests assert on cell values, and\n createTableAndConnect, which skips the five-second existing-table\n probe in TableCommand.createTable.\n\nThe only production change is visibility: ten SparkMain helpers go\nfrom private static to package-private @VisibleForTesting static,\nmatching the existing archive helper, so they can be called\nin-process. No behaviour change." }, { "commit": "4be4864082d7d787d70dec6cf2a91f7f04866eb8", "tree": "d5d01eb46e2260cf32681229a60a29e601b8f713", "parents": [ "96858170d02cf72a83243be3db199ee888efd672" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Thu Sep 10 16:50:11 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 16:50:11 2026 +0800" }, "message": "chore(docker): move demo/IT images to Spark 3.5.9 (#19888)\n\nThe docker demo stack and the docker-cluster ITs were still on\nSpark 3.5.3 while the root pom moves to 3.5.9 (#19884), so the\ncontainers ran a bundle compiled against a newer Spark than their\nruntime. Move every pin in one go: the four Spark Dockerfiles,\ndocker.spark.version, the build_docker_images.sh default, both\ncompose sets (renamed spark353 -\u003e spark359), setup/stop_demo.sh,\nhudi-integ-test\u0027s dockerCompose.file, the README, and the parked\nintegration-tests-hive-sync matrix.\n\nThe 3.5.9 images do not exist on Docker Hub yet. integration-tests\nruns docker compose up on the hadoop284 file every run, so this\nmust not merge until they are pushed; #19887 has the build and\npush steps." }, { "commit": "96858170d02cf72a83243be3db199ee888efd672", "tree": "b28e21d2ecb8499036c06792887610f98c45523f", "parents": [ "91f32af2403e1f315e665e19ea4f1e319f1ed969" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Thu Sep 10 16:11:22 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 16:11:22 2026 +0800" }, "message": "docs(flink): document Lance runtime dependencies (#19866)\n\n* docs(flink): document Lance runtime dependencies" }, { "commit": "91f32af2403e1f315e665e19ea4f1e319f1ed969", "tree": "3a4abcc91afffda35b3cd3cdb02bd0b8a3483517", "parents": [ "e6b8ef6c92f84b4412429199c3392847c4abfe58" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Thu Sep 10 13:55:15 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 13:55:15 2026 +0800" }, "message": "test(aws): cover the Glue sync client, DynamoTableUtils and HoodieAWSConfig (#19876)\n\ntest(aws): cover the Glue sync client, DynamoTableUtils and HoodieAWSConfig (#19876)\n\nTest-only change, all with Mockito and no AWS access.\n\n- TestAWSGlueSyncClient: 42 new tests on the mocked GlueAsyncClient\n fixture covering partition-index management (enable flag, stale\n index drop, missing index create, the three-index cap, and the\n steady state that re-reads nothing), updateLastCommitTimeSynced\n including the swallowed partition-index failures and the empty\n timeline, updateSerdeProperties on all four exits and both values\n of useRealtimeFormat, getMetastoreSchema, getStorageFieldSchemas,\n getPartitionsFromList, the endpoint, region and default arms of\n buildAsyncClient, and the failure-wrapping branches of create,\n drop, exists, update and addPartitions. Each asserts the request\n sent to Glue, not only the return value.\n- TestDynamoTableUtils (new, 13): waitUntilExists and waitUntilActive\n with short explicit timeouts, the default-overload path bounded by\n @Timeout, createTableIfNotExists and deleteTableIfExists on both\n exits.\n- TestHoodieAWSConfig (new, 4): every builder method, build()\n defaults, fromProperties, fromFile.\n- TestS3StorageLockClient: two more branches of the upsert error\n handling.\n- GlueTestUtil wrote its commit as raw JSON to .hoodie/101.commit,\n which the 1.x timeline layout never reads, so the fixture table had\n an empty timeline and no schema. It now writes the commit through\n DefaultCommitMetadataSerDe under .hoodie/timeline/ and records the\n table schema." }, { "commit": "e6b8ef6c92f84b4412429199c3392847c4abfe58", "tree": "6c44cc93ffaaf8d9bf97566901a5d212fd8ff125", "parents": [ "58c9e64e58a040d5c608b0893c2709e8c231534b" ], "author": { "name": "Danny Chan", "email": "yuzhao.cyz@gmail.com", "time": "Thu Sep 10 13:30:24 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 13:30:24 2026 +0800" }, "message": "fix(core): use incoming record keys in native log writers (#19874)\n\n* fix(core): use incoming record keys in native log writers" }, { "commit": "58c9e64e58a040d5c608b0893c2709e8c231534b", "tree": "71a6a614b4b1d4e0716fc23f982ade10565da693", "parents": [ "4c6e75b2d2bedc2e8ccee61d768a72a21bad292b" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Thu Sep 10 12:30:30 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 12:30:30 2026 +0800" }, "message": "chore(spark): bump Spark 3.5 to 3.5.9 and fast-path the CI download (#19884)\n\nspark35.version moves 3.5.5 -\u003e 3.5.9 and the integration-tests Spark\narchive pin moves 3.5.3 -\u003e 3.5.9 to match, along with the five\nspark-avro spark-submit examples. The old split compiled against 3.5.5\nand ran the ITs on 3.5.3, the direction that can hit a missing API.\n\nThe stale pin is also why the download was slow: 3.5.3 has rolled\noff the ASF CDN, so the job pulled the 382MB tarball from\narchive.apache.org, a single un-CDN\u0027d origin measured at 42s to 1h08m\nfor the same file (#19883). 3.5.9 is the current 3.5 release and is\nserved by dlcdn.apache.org, so the job now tries the CDN first and\nfalls back to the archive for pins that have rolled off it. Plain\n--retry rather than --retry-all-errors, so a CDN 404 falls through\nat once instead of sleeping through five retries.\n\ncurl gains -f so an error page is a failure rather than a corrupt\ntarball, and --speed-limit 1000 --speed-time 120 to abort a dead\nconnection. The floor sits far below the archive\u0027s real throughput\non purpose: --retry truncates the output back to byte 0, so a floor\nnear it would re-download 382MB per abort. -C - is not used; the\nfile never exists when curl starts, so it could never resume. tar\ndrops -v, which printed 4000 filenames into the job log.\n\nThe three Spark sources Hudi copies from are byte-identical between\nv3.5.5 and v3.5.9, and every in-tree Spark version gate is\nminor-level, so the patch bump crosses none of them." }, { "commit": "4c6e75b2d2bedc2e8ccee61d768a72a21bad292b", "tree": "78b87ac89c8164e25f9ddfcaa2bc972940fadadb", "parents": [ "eccfee6dfe7daf3666290e8112052f86753f979e" ], "author": { "name": "Toyeshh Medikonda", "email": "toyeshhm@gmail.com", "time": "Wed Sep 09 21:29:14 2026 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 10:29:14 2026 +0800" }, "message": "Fix truncated return type in RFC-101 sample and typos across docs (#19885)\n\nThe HoodieRecordMerger snippet in RFC-101 declared fullOuterMerge as\nreturning `ist\u003cPair\u003cHoodieRecord, Schema\u003e\u003e`; restore `List\u003c...\u003e` so the\nsample compiles as written.\n\nAlso correct spelling in RFC design docs, the HFile format spec, the\nKafka Connect and integration-test READMEs, and the release guide\n(e.g. PostgresSQL -\u003e PostgreSQL, intial -\u003e initial, requried -\u003e required,\nture -\u003e true, taskmanger -\u003e taskmanager)." }, { "commit": "eccfee6dfe7daf3666290e8112052f86753f979e", "tree": "050f06373871fe86dd558647268779efd79ce045", "parents": [ "7b4ad9d6c96284399bf9ed6d1ebdf4dccb9cc7e6" ], "author": { "name": "chrevanthreddy", "email": "27821245+chrevanthreddy@users.noreply.github.com", "time": "Wed Sep 09 19:18:53 2026 -0600" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 18:18:53 2026 -0700" }, "message": "docs(rfc): RFC-109 native vector search support in Apache Hudi (#19309)\n\n* docs(rfc): RFC-109 native vector search support in Apache Hudi\n\n* docs(rfc): address vector index review feedback\n\n* docs(rfc): keep vector consistency self-contained\n\n* docs(rfc): define vector option defaults\n\n* docs(rfc): define incremental vector maintenance semantics\n\n* docs(rfc): separate vector correctness from maintenance\n\n* docs(rfc): address vector index review feedback\n\n* docs(rfc-109): include partition pruning in initial scope\n\n* docs: address and clarify last round of review\n\n* docs(rfc-109): tighten maintenance and API sections\n\nClarify §7 maintenance semantics in Hudi terms: table services as the\nautomated maintenance runtime, MDT read expectations consistent with it\nbeing a MoR table, and vector-index correctness stated for the index\nrather than as a v1 caveat.\n\nScope §7.3 to the built-in MDT compaction strategy for v1, with the\ndemand-aware vector-aware strategy called out as longer term and\ndependent on query telemetry collected elsewhere. Mark §7.6 LIRE as a\nfuture extension to revisit after v1 performance is evaluated.\n\nNote in §8 that vector search should eventually be expressed in query\nfilter predicates rather than requiring a join against a TVF result.\n\nLeaves an open VC: question in §7.2 on how to state rewrite/relocation\ncost in Hudi terms.\n\n* docs(rfc-109): postings locate rows by record key\n\nMake the record locator concrete instead of leaving its form open. A\nposting stores the record key and nothing else about where the row\nlives; exact fetch resolves that key through RLI and then reads the data\nfile. The index stores no file group, base instant, or row position.\n\nMerge the former S4 (row locators) into S6 (record keys), leaving five\nposting-block families: S4 is now the partition dictionary needed for\npartition pruning, S5 the record keys. Bootstrap still orders packing by\n(cluster, fileGroup, rowPosition), but that ordering is applied at build\ntime rather than stored.\n\nRecord keys are stable across clustering and compaction, so a data-file\nrewrite requires no vector-index update at all -- RLI alone absorbs the\nrelocation, as it already does for the secondary index. Drop the locator\nrefresh branch from the incremental write path, the locator maintenance\nfrom MDT compaction and affinity clustering, and the large-rewrite\ntrigger.\n\nExact fetch is by key within the RLI-resolved file slice. Once RLI\npopulates row position, that becomes a single-page positional read;\nuntil then the positional fast path is not available.\n\n* Revert \"docs(rfc-109): postings locate rows by record key\"\n\nThis reverts commit ae4d6f5cd4393ff3816f0ec0be0f28b8cddf2570.\n\n* docs(rfc-109): align storage model with the implementation\n\nCorrect the posting-block and key-scheme description against the vector\nindex implementation in #19802.\n\nS1-S6 are sections laid end to end in one posting block record\u0027s\npayload, not key-suffixed entries and not a storage-format column-family\nfeature. Drop the familyId suffix from the posting key, which is\n0x10|generation|cluster|segment|block. Note that \"family\" already means\nthe leading key byte, so S1-S6 are not key families. The two-pass scan\ntherefore bounds decode work rather than bytes read, since the block is\nfetched as one record.\n\nDefine the row locator where S4 is introduced: a file-group index, an\ninstant index, and a row position, with the indices pointing into the S5\ndictionaries and the file-group dictionary carrying the partition path\nused for pruning. S5 holds three dictionaries, not two.\n\nGive the row-family table its one-byte tags, and split quantizer state\n(0x02) from centroid chunks (0x03) rather than folding quantizer\nmetadata into the generation manifest.\n\nDocument the delta key: the posting prefix with the reserved DELTA block\nid 0xFFFFFFFF followed by the record key, so one delta addresses exactly\none record and repeated writes to a record collapse on that key.\n\n* docs(rfc-109): plainer wording for two-pass scan cost\n\n* docs(rfc-109): rewrite in plain technical prose\n\nThe document read as a product pitch rather than an ASF design\ndocument. Rewrite the prose without changing any design claim.\n\nDrop the promotional framing: \"one storage innovation\", \"the core\nstorage contribution of this RFC\", \"make results trustworthy\",\n\"artifacted measurements\", \"a naive index would\", and the pull-quote\ncallout in the abstract. State the measured recall with its caveats\nrather than as a headline. Replace the named design-principle bullets\nwith the four constraints stated plainly.\n\nUse Hudi\u0027s own vocabulary. A query resolves a snapshot rather than\npinning one; writers commit rather than publish; readers read rather\nthan consume. \"RLI arbitration\" becomes the RLI lookup, \"the vector\nhook\" the vector indexer, \"layer-2 skip semantics\" the standard MDT\nbehavior for a partition a writer does not handle. Drop \"contract\"\nwhere it meant behavior, and \"seam\" where it meant call site.\n\nRemove restatements that the MDT commit protocol already implies,\nincluding that generation activation is atomic and that no\nintermediate step is visible.\n\nNo technical content is added or removed.\n\n* docs(rfc-109): state relocation cost of a physical rewrite\n\nReplace the open question in 7.2 with the answer for the storage model\nas implemented. A posting stores a row locator, so clustering and\ncompaction both invalidate the locators of rows they move and the index\nmust refresh them, as the record-level index already does; the secondary\nindex is unaffected since it maps to record keys. Neither rewrite\nrequires re-routing or re-encoding.\n\nDistinguish clustering, which is a replacecommit into new file groups,\nfrom compaction, which rewrites a file group\u0027s base file in place at a\nnew instant. Rename the pure-motion optimization to relocation-only.\n\n* docs(rfc-109): reflow section 8 preamble, fix typo\n\n---------\n\nCo-authored-by: Revanth Chandupatla \u003crevanth.chandupatla@walmart.com\u003e\nCo-authored-by: vinoth chandar \u003cvinoth@apache.org\u003e" }, { "commit": "7b4ad9d6c96284399bf9ed6d1ebdf4dccb9cc7e6", "tree": "61e96e157c6bff3c1d5fbf4f5b98d36f722389f1", "parents": [ "bb6f52c185d3b690328df340da396623f7f63243" ], "author": { "name": "deepakpanda93", "email": "deepakpanda93@gmail.com", "time": "Wed Sep 09 21:25:11 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 15:55:11 2026 +0000" }, "message": "fix(common): scan jar entries in ReflectionUtils#getTopLevelClassesInClasspath (#19624)\n\nFollow-up to #19784 (HUDI-736), which simplified the method but did\nnot add jar-entry scanning.\n\nA jar: classpath entry is non-hierarchical, so toDirectory logs the\nIllegalArgumentException and drops it, and a scan run from inside a\nshaded jar returns nothing. Every caller is a packaging bundle Main\nclass, which runs exactly that way; before #19784 the same case threw.\nClass#getPackage is also null for arrays and primitives, which was\ndereferenced without a check.\n\nclassNamesIn now dispatches on protocol. A jar: entry is read through\nJarURLConnection with setUseCaches(false), so the scan opens its own\nJarFile instead of closing one shared through the JVM-wide cache, and\nthe names are collected before the jar is closed because the returned\nstream outlives the method. The entry prefix stays derived from the\npackage name: anchoring on getEntryName() would make the result JDK\ndependent on a multi-release jar (the loader resolves the package to\nMETA-INF/versions/N/\u003cpkg\u003e/ on JDK 9-23 but to \u003cpkg\u003e/ on 8 and 24+) and\nwould drop classes that exist only in the base directory. A connection\nthat is not a JarURLConnection is skipped, a class with no package\nyields an empty stream, and findClasses skips a directory whose\nlistFiles() is null instead of throwing.\n\nOnly hudi-cli-bundle ships slf4j, so the other bundle Main classes\nstill fail in ReflectionUtils.\u003cclinit\u003e when run from the bare jar; the\nscan works wherever slf4j is on the classpath.\n\nTestReflectionUtils is 13 tests. Each guard is pinned by a test that\nfails without it: the jar dispatch by an empty result, the null-package\nguard and the listFiles guard by NullPointerException, the instanceof\ncheck by ClassCastException, and setUseCaches(false) by\n\"IllegalStateException: zip file closed\" on a JarFile another reader\nholds open through the cache." }, { "commit": "bb6f52c185d3b690328df340da396623f7f63243", "tree": "0c7887f0edf11aa86e3fceb834ca151e153e8dc0", "parents": [ "6ac27904f71b1393a985e7bd417d32fe55e168dc" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Wed Sep 09 19:18:40 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 19:18:40 2026 +0800" }, "message": "feat(timeline-ui): [HUDI-9315] Add Hudi Timeline UI (#13147)\n\nReading a Hudi timeline from the console or a file browser is hard:\ninstants sort by file name, and the completion time embedded in\ncompleted instant names makes concurrent or long-running writes\nespecially difficult to group by eye. This adds a timeline visualizer\nto hudi-timeline-service, implementing RFC-94.\n\nThe UI is served at /ui behind the new --enable-ui flag, which also\nregisters four read-only JSON endpoints under /ui/api:\n\n timeline/instants/all all instants\n timeline/instant instant details (action and state validated)\n table/config table configuration\n table/schema/history schema evolution history, newest first\n\nInstant details are decoded per state: a completed replacecommit reads\nthe replace-commit metadata, an inflight one reads the workload profile\nsaved to the inflight file, and anything else falls back to the\nrequested twin. The page drops responses from superseded loads, so a\nfast second submit cannot render the previous table\u0027s data.\n\nStatic assets are served from the classpath under /ui/static. The\nvendored third-party libraries (Bootstrap, renderjson, vis-timeline)\nlive in resources/public/lib, are recorded in LICENSE and excluded\nfrom apache-rat, and hudi-timeline-server-bundle now stamps\nMETA-INF/LICENSE into its shaded jar.\n\nThe UI runs only from the standalone timeline server, so the engine\nbundles (spark, flink, utilities, kafka-connect, integ-test) exclude\npublic/** from the shaded hudi-timeline-service artifact.\nscripts/release/validate_bundle_ui_assets.sh asserts both halves of\nthat split and runs in the bundle-validation CI jobs.\n\nCovered by TestUiApi: 22 tests against a live service instance." }, { "commit": "6ac27904f71b1393a985e7bd417d32fe55e168dc", "tree": "f3c5ec944af0b32aa653e7a4b770fe0754cfd2de", "parents": [ "64904e8941913f7701909cedde8483e21c410eb7" ], "author": { "name": "Ranga Reddy", "email": "rangareddy.avula@gmail.com", "time": "Wed Sep 09 11:17:03 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 13:47:03 2026 +0800" }, "message": "test(clustering): re-enable testClusteringPlanInflight and fix its timeline transition (#19865)\n\nCloses #17333 (HUDI-8686).\n\nThe test was disabled in #9717 citing AVRO-3789, which avro 1.11.3\nfixed; Hudi is on 1.11.4. Dropping the annotation alone still fails:\ncreateRequestedClusterInstant builds a CLUSTERING_ACTION instant, but\nthe test transitioned it with transitionReplaceRequestedToInflight,\nwhose precondition is REPLACE_COMMIT. The mismatch entered in #11553\n(HUDI-7905), which migrated every other transition in this file but\nskipped this one because it was disabled.\n\nSwitch to transitionClusterRequestedToInflight and parameterize the\ntest over table version. Table version 6 has no clustering action and\nschedules clustering as a replacecommit (see\nClusteringPlanActionExecutor), so that lane drives a replacecommit\ninstant through the REPLACE_COMMIT arm of isClusteringInstant, which\nno test pinned before: deleting the arm left the class green. The\nhelper now picks the action from the timeline layout version the way\nproduction does.\n\nAlso assert that the inflight file is empty, which the comment claimed\nbut nothing checked, and that the plan is still readable from the\ncompleted instant, covering the InstantGeneratorV2 fork HUDI-8610\n(#12375) added without a test.\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "64904e8941913f7701909cedde8483e21c410eb7", "tree": "0f97efd8f3086d07fe9e2a13b55950059a2cc457", "parents": [ "eccbef97f33941f7c2826a3f59f59bc235a82ad5" ], "author": { "name": "Danny Chan", "email": "yuzhao.cyz@gmail.com", "time": "Wed Sep 09 13:25:20 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 13:25:20 2026 +0800" }, "message": "fix(flink): repair test compilation after metadata field mode migration (#19873)" }, { "commit": "eccbef97f33941f7c2826a3f59f59bc235a82ad5", "tree": "dce71cadd0e7965ba112a13c72662f9988f210f5", "parents": [ "56eae7b48089478616e5109a15efa11aae23e30e" ], "author": { "name": "Danny Chan", "email": "yuzhao.cyz@gmail.com", "time": "Wed Sep 09 12:06:49 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 12:06:49 2026 +0800" }, "message": "feat(flink): support metadata field modes for COW tables (#19858)\n\n* feat(flink)!: support metadata field modes for COW tables" }, { "commit": "56eae7b48089478616e5109a15efa11aae23e30e", "tree": "bee6a79c936c1a79342d30d052390e5ffff1ce51", "parents": [ "08af5869addbdf3249464cf8771f60150e0bd0d2" ], "author": { "name": "Ranga Reddy", "email": "rangareddy.avula@gmail.com", "time": "Tue Sep 08 22:36:20 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 01:06:20 2026 +0800" }, "message": "test(utilities): report why the continuous-mode wait timed out (#19485)\n\ntest(utilities): diagnose wait timeouts (#19485)\n\nHoodieDeltaStreamerTestBase.TestHelpers.waitTillCondition polled its\ncondition every two seconds and swallowed whatever it threw, so when a\ncontinuous-mode test timed out the failure named only the helper. Every\nreport of the flaky multi-writer test in #16228 (HUDI-6843) is that\nbare TimeoutException, and none says which of the four assertions in\nits condition was still failing.\n\nThe wait now keeps the last Throwable the condition threw and attaches\nit as the cause, counts completed evaluations, and reports one of three\nshapes on timeout: the condition\u0027s own error, no evaluation completed,\nor evaluations that returned false. The condition helpers name\nthemselves in their messages. The \"Condition completed successfully\"\nlog on a false result is gone, and the exit where the streamer future\nfinishes first is logged at warn with the same detail.\n\nA failure no longer leaves the streamer running into the next test\n(surefire runs this module with forkCount\u003d1, reuseForks\u003dtrue). The\nrunner stops it on any failure, each wait bounded at 30s with at most\ntwo in sequence, force-stops the ingestion service if the stop outlives\nits bound, and surfaces a streamer that died instead of waiting two\nminutes for \"Deltastreamer should have shutdown by now\". The polling\nexecutor is shut down, InterruptedException is handled ahead of the\ncatch-all so that shutdown takes effect, and waitFor is bounded at 120s\nand restores the interrupt.\n\nJavaTestUtils.checkNestedExceptionContains tolerates a null message in\nthe chain. TestHoodieDeltaStreamerWithMultiWriter records whether its\nbackfill prerequisite held, so an expected-conflict run whose writers\nnever overlapped says so instead of blaming conflict handling, and its\ntwo sibling prep jobs get the MOR compaction settings HUDI-6445 gave\nthe ForConflicts one.\n\nTestDeltaStreamerTestHelpers (14 tests, no Spark) and TestJavaTestUtils\ncover the helpers directly. This does not fix the flake in #16228; it\nmakes the next occurrence diagnosable.\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "08af5869addbdf3249464cf8771f60150e0bd0d2", "tree": "35528a1f1b38a92ac8773903244d468d2e657da7", "parents": [ "cc9de4d1174e75ae7db2dde5f2ea9e15e22f428a" ], "author": { "name": "Kavya Popat", "email": "kavyapopat95@gmail.com", "time": "Tue Sep 08 18:20:05 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 20:50:05 2026 +0800" }, "message": "fix(cli): repair the hudi-cli tests and re-enable the module in CI (#18816)\n\nThe hudi-cli tests have not run in CI since #10805 excluded the module.\n#11866 then repurposed the fifth parameter of TableCommand.createTable\nfrom --layoutVersion to --tableVersion. Both are Integer, so the 14 call\nsites passing TimelineLayoutVersion.VERSION_1 kept compiling and quietly\ncreated table-version-1 tables, which Hudi 1.x rejects.\n\nTests: pass the current table version at those call sites, and fix the\ntimeline path, file-name generator and sort order that the v9 layout\nchanged. New coverage for archived clean and replacecommit rows, a v6\ntable read over the legacy archive, an empty timeline under show fsview\nlatest, and the clean-plan corruption cases below.\n\nRepairsCommand: repair corrupted clean files left empty .clean.requested\nfiles in place, because reading one raises \"unable to read commit\nmetadata\" rather than the \"Not an Avro data file\" it matched. That\nmessage cannot be matched either, since the serde wraps a transient read\nfailure in it too. The command now reads the plan bytes in full, so a\nread failure propagates and the instant stays, and only the in-memory\ndecode judges corruption -- through any runtime failure, since Avro\nraises UnsupportedOperationException past its array ceiling and a\nversionless plan leaves the migrator unboxing a null. A corrupt plan now\ntakes the whole pending action with it, inflight file first, instead of\nleaving the requested file behind for a second pass.\n\nArchivedCommitsCommand: read through the archived timeline on the v8+\nLSM layout instead of globbing .commits_.archive*, uncached so that a\ntrigger archival earlier in the session is visible. Tables below v8 keep\nthe legacy reader, which now renders the completed entry of each instant\nand skips entries with no metadata, where a writer\u0027s empty requested\nfile used to NPE it. Actions with no typed reader render through the\nschema embedded in their payload instead of failing the listing, and\nwithout --sortBy the printer keeps timeline order and cuts at --limit,\nso only the instants that can be rendered are loaded. Output change: the\nlegacy format archived each state separately, so show archived commits\nprinted three rows per commit and now prints one on both layouts.\n\nCleansCommand renders clean partitions in partition order rather than\nmap iteration order, and FileSystemViewCommand guards the active\ntimeline lookup so view commands do not throw on an empty timeline.\n\nCI: drop !hudi-cli from azure-pipelines-20230430.yml, and run the module\ninside test-spark-client-and-hadoop-common in bot.yml, since Codecov is\nfed only by bot.yml. That job\u0027s -am reactor grows from 10 to 23 modules\nand it runs about 43 min, roughly 30 min under the critical path.\n\nThe hudi-cli functional suite goes from 113 tests with 12 failures and\n24 errors to 118 with none, all 20 classes green, and every HUDI-7614\ndisable lifted.\n\nCloses #16448" }, { "commit": "cc9de4d1174e75ae7db2dde5f2ea9e15e22f428a", "tree": "319d01ff4a8fb0d11888181d867ab0f570da3c31", "parents": [ "75678093b3eb6af46b8dcfa1c1e41ccc384a9afd" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Tue Sep 08 18:28:25 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 18:28:25 2026 +0800" }, "message": "fix(schema): stop union walks looping forever (#19834)\n\nHoodieSchema.getNonNullType() returns this for a union with no null\nbranch, and a union of the non-null branches when there are two or\nmore of them. Six recursive schema walkers called themselves on that\nresult inside their UNION arm, so any schema carrying a field typed\n[\"null\",\"string\",\"int\"] or [\"string\",\"int\"] recursed until\nStackOverflowError:\n\n- HoodieSchemaUtils.hasDecimalField (Streamer JSON sources)\n- HoodieSchemaRepair.hasTimestampMillisField\n- InternalSchemaConverter.collectColNamesFromSchema\n- ValueType.fromSchema (column stats)\n- HoodieTableMetadataUtil.coerceToComparable (column stats)\n- HoodieSchemaUtils.findNestedField\n\nThe predicates now iterate every branch. The two walkers that need a\nsingle answer throw an explicit unsupported-union error, and\nfindNestedField returns empty, since such a union has no one type to\ndescend into. The name collector walks the branch\nvisitSchemaToBuildType keeps, the only one the internal schema\ncarries ids for. HoodieSchema#isComplexUnion() names the shape once,\nand getNonNullType()\u0027s javadoc spells out that its result can itself\nbe a union.\n\nEnding the loop only moved the failure, so the read and stats paths\nfollow.\n\nColumn stats reached ValueType#fromSchema and #coerceToComparable\nwith such a union because isColumnTypeSupported\u0027s deny lists never\nnamed UNION. Both throw, and readColumnRangeMetadataFrom catches\nException, so one union column silently dropped the stats for every\ncolumn in the file. Skip the column up front the way RECORD, MAP and\nARRAY are.\n\nOn the Spark read path, pruneDataSchema rejected a union on either\nside. A union is a leaf for pruning -- Avro picks a branch by its\ntype, so dropping one changes the column\u0027s type instead of narrowing\nit -- so the data schema passes through whole. Spark reads a union as\na struct of nullable member0..memberN fields and prunes it like any\nother struct, so the scan\u0027s output projection is now bound to the\nshape the reader emits and drops the fields Spark did not ask for by\nname at every depth. That also covers the BLOB and VARIANT columns\npruneDataSchema keeps whole.\n\nThe member-struct heuristic ran on the root of the requested schema\ntoo, where an all-memberN projection is the row and not a union. It\nhanded the reader the entire table schema, so SELECT member0 came\nback with _hoodie_commit_time, and two such columns of one type\nfailed the conversion outright; it is guarded on depth now.\n\nCloses #19825" }, { "commit": "75678093b3eb6af46b8dcfa1c1e41ccc384a9afd", "tree": "5e808386135fc44234025284daba3fc34f173740", "parents": [ "8a41388d76a89f8316dfbf787b8f3188f717a8f4" ], "author": { "name": "Joy", "email": "33287603+Joy-2000@users.noreply.github.com", "time": "Tue Sep 08 16:20:32 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 16:20:32 2026 +0800" }, "message": "fix(lock): fix lock config derivation for Spark procedure (#19794)\n\n* fix(lock): unify default FS lock path and make CLI lock config take effect\n\nTwo related defects in the metadata-table auto-lock path:\n\n1. Lock-path inconsistency: FileSystemBasedLockProvider.getLockConfig used\n the .aux folder as the default lock path, while the constructor fell back\n to the .hoodie meta folder. The two defaults could diverge, silently\n placing the lock file at different paths and breaking mutual exclusion\n across engines/tasks. Unify both on the table metadata path (.hoodie) by\n reusing a single defaultLockPath() helper.\n\n2. Lock config never applied: RunClustering/RunCompaction appended lock\n options to `confs` AFTER the write client was already built, so they had\n no effect. Move the auto-lock injection down into\n HoodieCLIUtils.createHoodieWriteClient, applied after the final parameters\n are merged and before the client is built, guarded on the fully-merged\n params so any explicitly-configured lock provider is respected. This also\n extends the coverage from just clustering/compaction to all write\n procedures that go through createHoodieWriteClient.\n\nRemove the now-dead lock-injection blocks and unused imports from the two\nprocedures. Add/extend tests covering the unified path and getLockOptions.\n\n* fix(lock): keep metadata lock derivation in clustering and compaction\n\n---------\n\nCo-authored-by: jiangyu84 \u003cjiangyu84@jd.com\u003e\nCo-authored-by: danny0405 \u003cyuzhao.cyz@gmail.com\u003e" }, { "commit": "8a41388d76a89f8316dfbf787b8f3188f717a8f4", "tree": "0de315b73158d142099fabae2e109347e1d23e0b", "parents": [ "d9e6f42045cdf69f0d3f6f9627125338cbfeee39" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Tue Sep 08 13:38:05 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 13:38:05 2026 +0800" }, "message": "fix(trino): keep none() when splitting predicates (#19863)\n\nPort of trinodb/trino#30958 (03074fb642d9). HudiPredicates.from only\nread TupleDomain.getDomains(), which is empty for both none() and all(),\nso a none() constraint came back as all() on both the partition and the\nregular side. Return none() for both when the input is none(), and add\nupstream\u0027s TestHudiPredicates.\n\nOne addition over upstream: HudiSplitManager.getSplits now returns an\nempty split source when either predicate on the table handle is none().\nWithout it, the preserved none() would reach MetastoreUtil\u0027s\ncomputePartitionKeyFilter, whose checkArgument rejects a none() domain,\nturning a query that should produce no splits into a failure. The\nplanner already turns a none() domain into an empty ValuesNode before\napplyFilter, so neither path is reachable through the engine today; the\nguard is the same defense HiveSplitManager carries." }, { "commit": "d9e6f42045cdf69f0d3f6f9627125338cbfeee39", "tree": "e9f5cd3928c87adc0c3d1cf46da6a908459716af", "parents": [ "6c666ee57985b4823ea1bfa8a0c2fd2fc4025cf7" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Tue Sep 08 12:18:47 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 12:18:47 2026 +0800" }, "message": "feat(flink): support vector columns in Lance reader (#19842)\n\n* feat(flink): support vector columns in Lance reader" }, { "commit": "6c666ee57985b4823ea1bfa8a0c2fd2fc4025cf7", "tree": "857028a048459a52e626ed05a2cfdc92bdf1f0e8", "parents": [ "cebbd6a0d56423383ffe5c8bd947db14c646e3d5" ], "author": { "name": "LeonxLJX", "email": "51880185+LeonxLJX@users.noreply.github.com", "time": "Tue Sep 08 12:17:20 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 12:17:20 2026 +0800" }, "message": "[HUDI-736] Simplify ReflectionUtils#getTopLevelClassesInClasspath (#19784)\n\n* [HUDI-736] Simplify ReflectionUtils#getTopLevelClassesInClasspath\n\nReplace the imperative loop with a stream pipeline over the package\nresources and fix a latent NPE: if ClassLoader#getResources throws an\nIOException, the previous code left \u0027resources\u0027 null and then called\nObjects.requireNonNull on it. The simplified version returns an empty\nstream on error instead.\n\n* refactor(common): minimize classpath discovery changes against master\n\n---------\n\nCo-authored-by: LeonxLJX \u003cliuzhaoxing373@gmail.com\u003e\nCo-authored-by: danny0405 \u003cyuzhao.cyz@gmail.com\u003e" }, { "commit": "cebbd6a0d56423383ffe5c8bd947db14c646e3d5", "tree": "741c1c780166e4fb5c803ed909f85ee9d5f84349", "parents": [ "73e6872fedd600a8c60876a4cb8270fa9103612c" ], "author": { "name": "Nikulin Nikita", "email": "42353747+w3lld1@users.noreply.github.com", "time": "Mon Sep 07 19:54:36 2026 +0300" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 16:54:36 2026 +0000" }, "message": "fix(spark): widen procedure filter numeric comparisons (#19836)\n\nProcedure filters cast a Long column down to Int when compared with\nan Int literal and left every other mixed numeric pair unresolved, so\na value past the Int range matched the wrong rows, a literal on the\nleft failed validation, and mixed-type operands were rejected.\n\nWiden operands to their common type instead, using TypeCoercion or\nAnsiTypeCoercion as SQLConf.get.ansiEnabled selects, and apply\nSpark\u0027s DecimalPrecision rules before the generic widening so decimal\nliterals keep their precision and decimal arithmetic keeps its result\nscale. The same widening covers IN, \u003c\u003d\u003e, coalesce and arithmetic;\nnon-decimal \"/\" promotes to Double and \"div\" promotes narrow integrals\nto Long, mirroring the analyzer\u0027s Division and IntegralDivision rules;\na null operand takes the type of its peers.\n\nBeyond the widening: an ANSI arithmetic or cast error now fails the\nprocedure instead of silently dropping the row, and a filter whose\nresult is not boolean is rejected at validation the way Spark raises\nFILTER_NOT_BOOLEAN. Binding and resolution run once per batch rather\nthan per row, and unresolved operands are left alone so validation\nkeeps its own messages and Or still short-circuits.\n\nTests compare the evaluator with df.filter across both ANSI modes and\nthe decimal precision settings, pin the Spark 3 overflow and Spark 4\ncast-down outcomes of a DECIMAL(38,0) vs DECIMAL(38,18) comparison,\nand add procedure-level filters on show_fsview_all and\nshow_metadata_column_stats_overlap, the two reproducers in the issue.\n\nFilters still run after limit in the show_* procedures (#19862), and\ndecimal parity under the legacy retain-fraction and precision-loss\nsettings is tracked in #19860.\n\nFixes #19632\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "73e6872fedd600a8c60876a4cb8270fa9103612c", "tree": "7e4f0f49ab31cf4221c7a49c1dfdd097f0d97249", "parents": [ "9903b6d83fbb59eecf15a5a76206c11797b6239b" ], "author": { "name": "Ryu", "email": "177669517+ryux1@users.noreply.github.com", "time": "Mon Sep 07 18:45:37 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 16:45:37 2026 +0000" }, "message": "refactor(spark): collapse HoodieTableState (#19849)\n\nHoodieTableState retained eight fields after the FileGroupReader\nmigration, but HoodieMergeOnReadRDDV2 read only the query\u0027s target\ninstant. Since the case class was an RDD constructor field, every MOR\ntask serialized the table path, merge configuration and metadata\nconfiguration without consuming them.\n\nRemove HoodieTableState and pass only that Option[String] to\nHoodieMergeOnReadRDDV2 from the snapshot and incremental relations.\nThe instant lives on BaseMergeOnReadSnapshotRelation, the common\nparent of the three MOR relations that read it, so the base relation\ngains no member; HoodieBaseRelation.queryTimestamp widens to\nprotected to allow that. It stays a lazy val, so the capture-once\ntiming of the removed lazy val tableState is unchanged.\n\nDropping the initializer left BaseHoodieTableFileIndex#getMetadataConfig\nwith no caller anywhere in the repo, so remove it. The metadataConfig\nfield it wrapped is protected final and subclasses already read it\ndirectly. Its only test planted that field by reflection and asserted\nthe accessor returned it, so replace it with a truth table over the\nthree-way conjunction that actually builds the config, which nothing\ncovered before.\n\nImpact: removing the public HoodieTableState class and\ngetMetadataConfig are source- and binary-compatibility changes for\nout-of-tree code. Query results are unchanged. Initialization timing\nmay change: constructing the removed table state forced the\nHoodieFileIndex constructor, so on the MOR incremental non-full-scan\npath the metadata-table reader open and the active-timeline reload no\nlonger precede composeRDD. Planning paths that consult sizeInBytes\nstill initialize the file index.\n\nCloses #19465.\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "9903b6d83fbb59eecf15a5a76206c11797b6239b", "tree": "a12fb9519b2ee9f4e203b963d5d20ce8f7b89d45", "parents": [ "d36c1ab32bba30a85c5a8584a976dbacb0e34610" ], "author": { "name": "Peter Huang", "email": "huangzhenqiu0825@gmail.com", "time": "Sun Sep 06 23:50:39 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 14:50:39 2026 +0800" }, "message": "feat(flink): support record index handling in dynamic bucket assign function (#19837)" }, { "commit": "d36c1ab32bba30a85c5a8584a976dbacb0e34610", "tree": "e2bae7f9c89a348f51f5616d537104071da77912", "parents": [ "b71b10bf477f45ac161fc4ef4fd8da71e6c3ec8a" ], "author": { "name": "hutiefang76", "email": "137664623+hutiefang76@users.noreply.github.com", "time": "Mon Sep 07 10:28:57 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 10:28:57 2026 +0800" }, "message": "fix(build): remove redundant hadoop mr bundle dependencies (#19856)" }, { "commit": "b71b10bf477f45ac161fc4ef4fd8da71e6c3ec8a", "tree": "7cef3c25b66689aec3a50da1ffcd07f2636ae75b", "parents": [ "7e518bc88eaec2378e1a76f7dfdc735b94e291fc" ], "author": { "name": "Ryu", "email": "177669517+ryux1@users.noreply.github.com", "time": "Sun Sep 06 18:32:58 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 00:32:58 2026 +0800" }, "message": "fix(spark): reject unsupported procedure filter functions (#19850)\n\n* fix(spark): reject unsupported procedure filter functions\n\n* fix(spark): validate resolved filter expressions\n\n* fix(spark): align filter validation coercion" }, { "commit": "7e518bc88eaec2378e1a76f7dfdc735b94e291fc", "tree": "28181db92455e7e0a66ac272e885e39eef5049ed", "parents": [ "a7deb61f74263d5fa4d1b71fb32b17d70263ada9" ], "author": { "name": "Sagar Sumit", "email": "sagarsumit09@gmail.com", "time": "Sun Sep 06 19:20:45 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Sep 06 21:50:45 2026 +0800" }, "message": "docs: claim RFC-111 non-blocking updates during clustering (#19851)" }, { "commit": "a7deb61f74263d5fa4d1b71fb32b17d70263ada9", "tree": "f41373344d07c98aee7e727f43c4c827969c4d2d", "parents": [ "aeaa7751e15b7e784749f4082896df085495c4e7" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Sat Sep 05 17:09:27 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Sep 05 17:09:27 2026 +0800" }, "message": "fix(schema): lowercase projected names with Locale.ROOT (#19835)\n\n* fix(schema): lowercase projected names with Locale.ROOT\n\nHoodieSchemaUtils.generateProjectionSchema lowercased both the schema\nfield names and the requested names with the JVM default locale, while\nHiveHoodieReaderContext pre-lowercases the requested names with\nLocale.ROOT. Under a tr_TR or az_AZ default locale an upper-case I in a\ncolumn name becomes dotless-i on the schema side only, the lookup\nmisses and the Hive read fails with \"Field id not found in log schema\".\n\nThe legacy realtime reader path lowercased the same way and is aligned\ntoo: the projection lookup, getNameToFieldMap and addPartitionFields in\nHoodieRealtimeRecordReaderUtils, and constructHiveOrderedSchema in\nAbstractRealtimeRecordReader. The projection test now uses a name with\nan I and a second test pins the lookup under a Turkish default locale.\n\nCloses #19826\n\n* fix(test): point the locale-toggle note at the restore\n\nThe comment credited surefire\u0027s single-forked, sequential run for\nkeeping the tr-TR default-locale toggle from leaking. That is the\ncondition that would let an unrestored toggle reach later tests in\nthe same JVM, not what prevents it, and forkCount/reuseForks are\npinned only in the functional-tests profiles anyway. Point the note\nat the finally restore, which is what actually makes this safe." }, { "commit": "aeaa7751e15b7e784749f4082896df085495c4e7", "tree": "4fc24d38f50e412978ab9f727cea1791ea190553", "parents": [ "e29d4d6a8289101af1251dcd5df058b36c5abd11" ], "author": { "name": "Shihuan Liu", "email": "skywalker0618@gmail.com", "time": "Fri Sep 04 18:16:38 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Sep 05 09:16:38 2026 +0800" }, "message": "fix(hive-sync): close the SessionState HiveQL sync starts (#19812)\n\n* fix(hive-sync): close the SessionState HiveQL sync starts\n\nHiveQueryDDLExecutor starts a SessionState in its constructor and never\nclosed it. Hive derives a session\u0027s four scratch directory roots from\nhive.session.id and reclaims them only in SessionState.close(), so every\nsync left a directory set behind, along with the session\u0027s registry and\nclass loaders. A HiveSyncTool is built per sync, so on a long-running\nstreaming job this accumulates for the life of the JVM.\n\nclose() now closes the session, after the Driver teardown because\nDriver.destroy() can reach SessionState.get() while releasing locks, and\nin a finally so a Driver close that throws cannot skip it. The\nconstructor\u0027s error path uses the same helper, which also stops a\nRuntimeException from a failing teardown masking the construction error.\n\nClosing the session detaches it from the calling thread, which Hive does\nunconditionally for whichever session is attached. The single-session SQL\npath therefore re-asserts its own session before running statements, as\nHive documents a thread running several sessions must, instead of relying\non a thread local another executor may have replaced or cleared.\n\n* Rename closeQuietly parameter to state to distinguish it from the field\n\n* Bind the owned session while tearing down the Driver\n\nDriver.close() and Driver.destroy() both act on whichever SessionState the\nthread currently holds, so an executor closed while a later-constructed\nexecutor\u0027s session is attached would clear that session\u0027s lineage state,\nrelease locks through its transaction manager, and then detach it.\n\n* Scope the session binding to the statements that need it\n\nLeaving this executor\u0027s session in the thread local made the caller\u0027s later\nHive work run under our database, configuration and transaction state, which\nis wrong for an interleaved executor and for an application that embeds the\nsync and owns its own session.\n\n* Give the thread back to the session construction displaced\n\nSessionState.start() attaches the session it starts, so the constructor was\ntaking a thread that belongs to the caller and never returning it. The session\nthe caller held was then never recovered: the statement path and the teardown\nboth saw this executor\u0027s own session as the one to put back, and the teardown\nclosed and detached it, leaving the caller with nothing.\n\n* Restore the context class loader with the session\n\nBinding a session also swaps the thread\u0027s context class loader for the one on\nthe session conf, a UDFClassLoader belonging to that single session, and\ndetachSession() does not undo it. The thread was therefore left holding a\nloader it does not own -- and after SessionState.close(), which closes that\nloader, one it can no longer load classes through.\n\n* Keep the teardown reachable when the session cannot be bound\n\nThe bind sat outside the try, so a failure in it skipped the Driver destroy\nand the session close -- the shutdown hook and the scratch directories this\nteardown exists to reclaim -- and left the thread bound to us. The teardown\nnow reports a failed bind and proceeds; the statement path keeps failing, as\nrunning SQL under another session is worse than not running it.\n\n* Give the session a conf of its own\n\nA session does not just read its conf: the constructor writes hive.session.id\ninto it and swaps in a UDFClassLoader, and close() deletes the scratch\ndirectories that id names and closes that loader. Passing config\u0027s HiveConf,\nwhich outlives the executor, meant close() left the caller holding a closed\nloader, and made correctness depend on the driver and metastore client pools\nbeing built before this executor -- built after, their copies of that conf\nwould carry our session id, so their sessions would share, and delete, our\nscratch directories. HiveDriverPool\u0027s workers own their conf for this reason.\n\n* Say why the session is closed after the Driver\n\n* Let a failed bind fail the teardown\n\nSwallowing it bought nothing the finally did not already give -- the session\nis closed and the thread handed back there either way -- and it cost the\nDriver calls their guarantee, running close() and destroy() against whatever\nsession the thread held instead. Those act on the thread\u0027s session, so with\nours unbound they would clear another session\u0027s lineage and release our locks\nthrough its transaction manager. Skip them and report, as close() already\ndoes when the Driver itself fails.\n\n* Say what a failed bind really leaves behind\n\nHive assigns the session to the thread before the conf and class loader swap\nthat can throw, so a failed bind leaves the thread holding ours, which the\nclose and restore then take off. The Driver is skipped because a half-applied\nbind promises nothing about what the thread holds, not because another\nsession would be attached -- ours is.\n\n* Close the session before clearing this thread\u0027s Hive\n\nSessionState.close() reaches for the thread-local Hive to uncache DataNucleus\nclass loaders, and ends by clearing it itself. Clearing it first left nothing\nto find, so the session opened a metastore connection of its own for that one\nlookup -- a connect per sync, and a retry loop when the metastore is down, in\nthe middle of a teardown. Closing the session first lets it reuse the client\nthe Driver has been using. Hive.closeCurrent() moves to a finally, where it\nstill clears the thread when there was no session to do it.\n\n* Tighten the comments to the invariants\n\nEach inline block now states what must hold and why in a line or three, with\nthe reasoning that outgrew that in the method javadocs -- one of which\nupdateHiveSQLs was missing. Also drops the claim that a leftover session\nloader breaks class loading: a UDFClassLoader is created with no URLs of its\nown, so a closed one still delegates. What is true is that the thread keeps a\nloader belonging to a session it does not own.\n\n* fix(hive-sync): avoid inheriting the caller session ID\n\n---------\n\nCo-authored-by: danny0405 \u003cyuzhao.cyz@gmail.com\u003e" }, { "commit": "e29d4d6a8289101af1251dcd5df058b36c5abd11", "tree": "f46a70920226e609adb5e28d3c2f788238fc7495", "parents": [ "9b5742cc961e8756261e82254cbb0dd5fc0605b4" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Fri Sep 04 21:52:15 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 04 21:52:15 2026 +0800" }, "message": "chore(deps): bump org.apache.thrift:libthrift (#19824)\n\nBumps [org.apache.thrift:libthrift](https://github.com/apache/thrift) from 0.23.0 to 0.24.0.\n- [Release notes](https://github.com/apache/thrift/releases)\n- [Changelog](https://github.com/apache/thrift/blob/master/CHANGES.md)\n- [Commits](https://github.com/apache/thrift/compare/v0.23.0...v0.24.0)\n\n---\nupdated-dependencies:\n- dependency-name: org.apache.thrift:libthrift\n dependency-version: 0.24.0\n dependency-type: direct:production\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "9b5742cc961e8756261e82254cbb0dd5fc0605b4", "tree": "0cd41c2bda031cdd9e3d2e5fa566b86bb5d0f810", "parents": [ "b290301a35d177c065156486fcbd275fd33e39ba" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Fri Sep 04 19:37:49 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 04 19:37:49 2026 +0800" }, "message": "refactor(schema): dissolve AvroSchemaUtils, dedupe compat (#19810)\n\nPart 2 of #16639. Part 1 (#19809, 93f1f711e065) wrote down where\nschema helpers belong; this removes the class that has no place in\nthat rule and the logic the compatibility cluster had implemented\ntwice. Behavior-preserving, internal API only.\n\nAvroSchemaUtils is gone. Its three remaining uses were all inside\nHoodieAvroUtils, so the helpers move there verbatim:\ngetNonNullTypeFromUnion stays public, isNullable becomes private and\ncreateNewSchemaFromFieldsWithReference package-private.\nHoodieSchema.Blob builds its fields with a local nullable() helper and\nderives REFERENCE_FIELD_COUNT from a REFERENCE_SCHEMA constant instead\nof unwrapping the union around it. The caller-less\nHoodieSchemaUtils.createNullableSchema shim goes with it.\n\nHoodieSchemaUtils.asNullable runs the InternalSchema nullability\nupdate on the HoodieSchema directly. The old path was HoodieSchema -\u003e\nAvro -\u003e HoodieSchema -\u003e InternalSchema -\u003e HoodieSchema -\u003e Avro -\u003e\nHoodieSchema, and the outputs are identical. Two details are kept:\nAvro\u0027s Schema.isNullable is true for a bare NULL type where\nHoodieSchema.isNullable is not, so NULL-typed fields are excluded\nexplicitly, and the all-nullable case returns the input instance. A\nnon-RECORD argument is now rejected with IllegalArgumentException\nrather than an Avro \"Not a record\" error.\n\nCompatibility cluster:\n- lookupWriterField was implemented twice; the facade keeps its\n stricter RECORD precondition and delegates to the checker.\n- The checker\u0027s LONG/FLOAT/DOUBLE/BYTES/STRING cases collapse into one\n HoodieSchemaTypePromotion.canPromote call, against the same table\n the projection checker uses.\n- TIMESTAMP over LONG and UUID over STRING stay checker-only.\n isCompatibleProjectionOf(source, target) tests canPromote(target,\n source), so adding them would make a timestamp a compatible\n projection of a bare long, and writer-schema deduction would then\n keep the table\u0027s long as the writer schema and silently drop the\n logical type.\n- Decimal widening is documented, not unified: the projection checker\n requires fixed-size parity, the compatibility checker does not.\n- The 3-arg isSchemaCompatible named its parameters reader/writer\n while routing them prev/new, so they are renamed, not reordered.\n\nTests pin what the refactor could have moved: the persisted Blob\nschema JSON as a literal, the promotion table from both sides\nincluding the five reverse numeric narrowings and canPromote(LONG,\nDATE/TIME), lookupWriterField (direct, alias, ambiguous, absent,\nnon-record), the argument order of areSchemasCompatible, and the three\npre-existing losses of the asNullable round trip (non-null default,\nENUM, null-last union) with a VECTOR column surviving it.\nTestAvroSchemaUtils folds into TestHoodieAvroUtils.\nTestHoodieTableSchemaEvolution.testFieldWithAlias gains a type change\nso the alias match, not the writerField !\u003d null guard, decides its\noutcome.\n\nTwo javadocs said the surviving toAvroSchema/fromAvroSchema\ndelegations were being retired under #16639. Where a helper belongs is\nthat issue; whether its body round-trips is #14263, which allows\nconversions only at the memory-to-disk, disk-to-memory and engine\nboundaries. They now point at #14263.\n\nSix review items raised on part 1 but not applied before it merged\ncome along: a bloom-filter fixture spelled out three times in\nTestHoodieMetadataPayload, a note on the assertSame there that is\nimplied by the instance check above it, the missing explanation for why\nan equal-but-distinct class schema throws\nArrayIndexOutOfBoundsException, a note on why\ncollectColumnRangeFieldValueV1\u0027s AVRO branch calls the static helper\nrather than dispatching (HoodieAvroIndexedRecord reports type AVRO and\noverrides getColumnValues), two \"@return Column value.\" on methods\nreturning an Object[], and javadoc for createSchemaErrorString.\n\nFound on the way and filed rather than fixed here: asNullable on a\ntable with a BLOB column returns the wrong field, because\nInternalSchemaConverter gives the blob\u0027s nested reference fields ids\n0..3 into a flat id map (#19833).\n\nCloses #16639" }, { "commit": "b290301a35d177c065156486fcbd275fd33e39ba", "tree": "95dc4664adb690d72b40bde32d5d83e4c14d31ac", "parents": [ "5e58dd363aed21aa9c048b573023fff94bd798f6" ], "author": { "name": "Ranga Reddy", "email": "rangareddy.avula@gmail.com", "time": "Fri Sep 04 16:45:25 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 04 19:15:25 2026 +0800" }, "message": "refactor(config): deprecate hoodie.parquet.outputtimestamptype; no-op since 1.1.0 (#19815)\n\nThe config has had no effect on any write path since 1.1.0, while its\ndescription still promised it set spark.sql.parquet.outputTimestampType.\n\nSince #13882 (4d95b2c2d165, first released in 1.1.0)\nHoodieRowParquetWriteSupport extends parquet\u0027s WriteSupport instead of\nSpark\u0027s ParquetWriteSupport and builds the Parquet schema itself, taking\nthe timestamp unit from the writer schema\u0027s logical type. The Avro\nParquet writer has always derived it from the writer schema too. The\nconstructor kept copying the value into the Hadoop conf, but nothing has\nread that key since, which is why the config still read as live.\n\nChanges:\n\n- HoodieStorageConfig.PARQUET_OUTPUT_TIMESTAMP_TYPE: description\n rewritten to say it has no effect since 1.1.0 and to point at the\n writer schema; adds @Deprecated and deprecatedAfter(\"1.1.0\"). The key\n and its default are untouched.\n- HoodieRowParquetWriteSupport: drops the dead\n hadoopConf.set(\"spark.sql.parquet.outputTimestampType\", ...). The only\n other reference in the tree, SparkReaderContextFactory, reads Spark\u0027s\n own SQLConf on the read path and is unaffected.\n- HoodieWriteConfig.parquetOutputTimestampType() and\n HoodieStorageConfig.Builder.parquetOutputTimestampType(): marked\n @Deprecated with javadoc rather than removed, so downstream\n integrations keep compiling. Matches how the other deprecated members\n of those classes are handled.\n- TestHoodieInternalRowParquetWriter: new test pinning both directions,\n so the description cannot drift back.\n\nNo behaviour change: the same Parquet footers are written before and\nafter.\n\nThis does not make the config work, so it is not a fix for #17302.\nOffering a supported way to set the Parquet timestamp unit without\nhand-writing a full writer schema is a separate storage-format\ndiscussion, and that issue stays open for it.\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "5e58dd363aed21aa9c048b573023fff94bd798f6", "tree": "6f9b1041070ab6e39782b285946a21581a761ee7", "parents": [ "40a41c93b5563f3d7f75dc175dac9455f4335047" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Fri Sep 04 17:08:16 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 04 17:08:16 2026 +0800" }, "message": "feat(flink): support vector columns in Lance writer (#19831)" }, { "commit": "40a41c93b5563f3d7f75dc175dac9455f4335047", "tree": "a8994c239417970e5cd0403dda3fd2dbcde79302", "parents": [ "d9e08f0763708dd937f211a6a883ba0b7e1939be" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Fri Sep 04 09:51:51 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 04 09:51:51 2026 +0800" }, "message": "feat(storage): support separate zstd level for native parquet logs (#19781)\n\n* feat(storage): support separate zstd level for native parquet logs" }, { "commit": "d9e08f0763708dd937f211a6a883ba0b7e1939be", "tree": "9fdb5ac9c0c13809ea1c58a9a4b4a2e94e71b296", "parents": [ "93f1f711e065e704d06b26f837865933a0e43208" ], "author": { "name": "Peter Huang", "email": "huangzhenqiu0825@gmail.com", "time": "Thu Sep 03 05:14:16 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 03 20:14:16 2026 +0800" }, "message": "feat(flink): add partitioned rli boostrap operator (#19805)\n\n* feat(flink): add partitioned rli boostrap operator" }, { "commit": "93f1f711e065e704d06b26f837865933a0e43208", "tree": "24d3aceef9eff3f1efb494e8991ecbc539eea27d", "parents": [ "7a3e2f3ad69b7b6727e36c9dc64acdda89bc7ffe" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Thu Sep 03 18:09:28 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 03 18:09:28 2026 +0800" }, "message": "refactor(schema): schema util contracts, dead API and folds (#19809)\n\nrefactor(schema): schema util contracts, dead API and folds\n\nPart 1 of #16639. The HoodieSchema migration moved schema helpers out\nof AvroSchemaUtils and HoodieAvroUtils but never wrote down what\nbelongs where, so helpers kept being re-implemented instead of found.\nBehavior-preserving.\n\nThe routing rule, now in the class javadocs: HoodieSchema answers\nquestions about one schema and holds the factories; HoodieSchemaUtils\nholds table-schema transforms and the Hudi record shapes;\nHoodieSchemaCompatibility is the only compatibility and projection\nentry point; HoodieAvroUtils holds Avro record and value operations;\ncommon.schema.internal is the InternalSchema domain. Look-alike\nhelpers with different semantics cross-reference each other instead\nof being merged.\n\nCode changes to match it:\n- HoodieSchemas folds into HoodieSchemaUtils (createDeleteLogSchema).\n- AvroSchemaCache deleted: its one caller interned a generated-class\n static, a no-op since #18967 removed the last consumer.\n- LocalHoodieSchemaCache moves from common.util to common.schema;\n getInstance() becomes create(), which is what it always did.\n- convertBytesToBigDecimal(byte[], HoodieSchema) moves to\n HoodieAvroUtils; the (byte[], int, int) shim and the caller-less\n HoodieSchemaUtils.convertValueForSpecificDataTypes and\n hasSmallPrecisionDecimalField are deleted.\n- HoodieAvroUtils.getRecordColumnValues takes HoodieSchema, since both\n callers unwrapped one to call it; it still interns through\n HoodieAvroSchemaCache and now reuses the static PROPERTIES and a\n fixed array instead of allocating per call.\n- 13 caller-less members narrowed; test-only ones get\n @VisibleForTesting.\n- Stale \"equivalent to X\" javadocs fixed, cross-references added.\n\nTests pin delete-log field order and logical types, case-insensitive\nprojection, nested column values, the HoodieMetadataPayload\nreference-equality fast path, the getRecordColumnValues intern, and\nLocalHoodieSchemaCache (test restored from #17740).\n\nPre-existing bugs found on the way are filed, not fixed here:\n#19823, #19825, #19826.\n\nPart 2 (#19810) implements asNullable natively, dissolves\nAvroSchemaUtils and dedupes the compatibility cluster.\n\nCloses #15908" }, { "commit": "7a3e2f3ad69b7b6727e36c9dc64acdda89bc7ffe", "tree": "4d74549ea1ce63ea395b7063d92a0b5fee325be5", "parents": [ "72989b47d36f841252d5943000ae2e7b2c6094f4" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Wed Sep 02 21:06:23 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 03 12:06:23 2026 +0800" }, "message": "fix(tests): reach TFramedTransport at the package libthrift 0.14 moved it to (#19830)\n\nHiveTestService imports TFramedTransport from the pre-0.14 package while\noverriding TServerTransport.accept(), which is final below 0.14, so no single\nlibthrift satisfies it. The build only gets away with it because the full\nhive-exec bundles a 0.14-era transport package; consumers of the tests jar that\ntake hive-exec with the core classifier resolve TServerTransport from libthrift\nand fail verification.\n\nThe libthrift test dependency moves to hive.libthrift.version, which already\nmodels the thrift the Hive client is built against, leaving thrift.version to\nthe metaserver." }, { "commit": "72989b47d36f841252d5943000ae2e7b2c6094f4", "tree": "43ce4166a91e9cfe65bd0d7e470b4cdfdff15783", "parents": [ "31904acd2a1d3a215b6bd12e73fb81c9423a71e8" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Thu Sep 03 11:58:25 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 03 11:58:25 2026 +0800" }, "message": "fix(flink): run mapGroupsByKey in dedicated fork join pool (#19819)\n\n* fix(flink): run mapGroupsByKey in dedicated fork join pool" }, { "commit": "31904acd2a1d3a215b6bd12e73fb81c9423a71e8", "tree": "3c3f1ddf9d2f5d1e552ca620d7f2bf2c764b7bcc", "parents": [ "ef834187ce47eb235110f35160ccc961e577fca9" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Wed Sep 02 11:51:44 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 02 11:51:44 2026 -0700" }, "message": "fix(build): exclude bundled Avro from the DataHub sync bundle (#19814)" }, { "commit": "ef834187ce47eb235110f35160ccc961e577fca9", "tree": "8a12beeff690f97c2b2d2b732fa5f8ba00405d5f", "parents": [ "a2788eef922e2375432ead3fe50af7fe5f55faf7" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Wed Sep 02 18:19:36 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 02 18:19:36 2026 +0800" }, "message": "feat(variant): infer shredding schemas by default on Spark 4.1+ (#19808)\n\nhoodie.parquet.variant.shredding.schema.inference.enabled now\ndefaults to true, matching Spark 4.1\u0027s own\nspark.sql.variant.inferShreddingSchema and writeShredding defaults.\nThe key is new in 1.3.0, so no released table changes behavior on\nupgrade; flipping it after a release is what would have.\n\nOnly writers with a Spark 4.1+ inferrer on the classpath change.\nSpark 3.x, Spark 4.0, Flink and Java keep writing unshredded. The\nschema is inferred per file from a sample of its first records\n(4096 rows / 64MB), for top-level variant columns only, in base\nfiles and in native parquet log files on table version 10+; Avro\nlog blocks stay unshredded and shred at compaction. Shredded-ness\nis therefore per file, never a table-level property, and mixed\nlayouts within a table are supported.\n\nShredded files can only be read back by Spark 4.1+. Spark 4.0,\nHive and Flink already failed fast; Spark 3.x now does too. Its\ndocumented way to read a variant table declares the column as\nstruct\u003cvalue: binary, metadata: binary\u003e, and parquet matched that\nrequest against a shredded group by name, leaving typed_value\nbehind and returning a null value for every shredded row.\nParquetSchemaEvolutionUtils.validateNoShreddedVariantStructs\nrejects that read from Spark33/34/35ParquetReader and the legacy\nfile format for base files, and from the new\nSpark3HoodieParquetReadSupport, returned by BaseSpark3Adapter, for\nlog blocks, which HoodieSparkParquetReader reads through the read\nsupport instead. The anchor is two-sided: on the request, binary\nmembers named metadata and value and nothing else - a subset too,\nsince nested schema pruning narrows SELECT v.value to one member -\nresolved case-insensitively because the Spark 3.x readers force\nspark.sql.caseSensitive\u003dfalse; on the file, typed_value beside a\nbinary metadata, the anchor the Hive and Spark 4.0 guards use. A\nplain user struct, an unshredded file and a query that does not\nproject the column are untouched.\n\nThe row-writer factory now orders its inference gates by cost -\nstatic inferrer lookup, StructType scan for a top-level variant,\nthen the config-schema parse - so a classpath that cannot infer,\nand any table without a variant, no longer pays an uncached parse\nof hoodie.avro.schema per file handle.\n\nConfig docs: the reader list names Spark 3.x, and the un-shred\nrecipe names hoodie.parquet.variant.write.shredding.enabled\u003dfalse,\nthe key that actually strips typed_value from a schema read back\noff shredded files. Opt out with either that key or the inference\nkey on tables other engines read; clustering with it off rewrites\nalready shredded files unshredded.\n\nTests: TestVariantSchemaUtils and the Avro writer factory pin the\ndefault; TestParquetSchemaEvolutionUtils covers the schema walk\n(nesting, pruning, case, exemptions); TestSpark3HoodieParquetReadSupport\ncovers the read-support wiring over a hand-built InitContext;\nTestVariantDataType gains a Spark 3.x fail-fast leg over the mixed\nshredded fixture, full and pruned, and its COW inference leg now\nrelies on the default." }, { "commit": "a2788eef922e2375432ead3fe50af7fe5f55faf7", "tree": "f849bdceb38b55ab61bca0b72075a5d54f6801df", "parents": [ "ca28fdb9877e34d19d29bb4bef001686bc4e5309" ], "author": { "name": "Lin Liu", "email": "141371752+linliu-code@users.noreply.github.com", "time": "Tue Sep 01 21:02:59 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 01 21:02:59 2026 -0700" }, "message": "fix(spark-sql): resolve MERGE INTO partition columns so records are not mis-partitioned (#19610)\n\nCo-authored-by: Y Ethan Guo \u003cethan.guoyihua@gmail.com\u003e" }, { "commit": "ca28fdb9877e34d19d29bb4bef001686bc4e5309", "tree": "8bebe28bebac79c701f054004f7ab0a69eb123fe", "parents": [ "c59987a024cd021cbb9aa34bdc74d341ea834b4d" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Tue Sep 01 21:08:16 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 01 21:08:16 2026 +0800" }, "message": "chore(trino): advance trino master pin to 40b70400a8a6 (#19807)\n\nCo-authored-by: github-actions[bot] \u003c41898282+github-actions[bot]@users.noreply.github.com\u003e" }, { "commit": "c59987a024cd021cbb9aa34bdc74d341ea834b4d", "tree": "403060f4137a8b192fddc906922baf1ce07787d4", "parents": [ "1e59c9eb6d9e341d5fd10e1418c0c3deb34b6579" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Tue Sep 01 18:59:39 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 01 18:59:39 2026 +0800" }, "message": "fix(variant): project PushVariantIntoScan struct paths in the ... (#19783)\n\nSpark 4.1+ rewrites a variant reached through a struct path into a\nprojection struct nested in the scan schema, e.g. s: struct\u003cinner:\nstruct\u003c\"0\": string\u003e\u003e. SparkFileFormatInternalRowReaderContext overlaid\nand projected that struct for top-level fields only, so on a MOR table\nwith log files the merged row still held a raw VariantVal at s.inner\nwhile the plan read it as the projected struct: SIGBUS, InternalError,\nOOM or silent nulls, on any log format and record type. COW was fine.\n\n- SparkFileFormatInternalRowReaderContext: the projection overlay and\n its detection recurse into struct members, mirroring Spark\u0027s\n VariantInRelation.rewriteType; arrays and maps stay native.\n- BaseSpark4Adapter: buildVariantProjector hoisted from the identical\n 4.1/4.2 copies and made recursive. A null variant in an avro log\n record projects to a NULL struct, not a struct of nulls, since\n PushVariantIntoScan rewrites IsNull/IsNotNull onto that struct.\n- SparkSchemaTransformUtils.addMissingFields: a projection struct\n requested over a file VariantType stays the reader type when a\n sibling member has an implicit type change; it used to be folded\n back to VariantType, and the type-change Cast then cast the variant\n value to the projected struct.\n- SparkAdapter.containsVariantProjection: one definition for the\n reader context and the adapter.\n- The deferral comments claiming a native VariantType request clips a\n shredded file to {metadata, value} now state the verified mechanism:\n the Spark 4.1+ parquet reader reconstructs a shredded variant at any\n depth. The full-variant rewrite stays top-level as a contract.\n\nTests, over tables whose only variant is the nested one.\nTestVariantShreddingMixedLayouts covers MOR merge, read-optimized,\ncompaction and clustering on both writer paths, CDC images, nested\nvariant_get and cast on COW and on MOR over native parquet logs and\navro blocks with pushVariantIntoScan on and off (each arm pins whether\nthe scan carries the projection struct), a null variant through the\nlog, implicit widening of a sibling, and an array\u003cvariant\u003e element\nshredded through a declared write schema.\nTestBaseSpark4AdapterVariantMethods covers the projector and the\nimplicit-change reconciliation as units. TestStreamingSource runs the\nlegacy-RDD leg with and without a top-level variant beside the nested\none.\n\nCloses #19775. Part of #18937." }, { "commit": "1e59c9eb6d9e341d5fd10e1418c0c3deb34b6579", "tree": "30a94a3bd7e46e8c89f887b81e444fa3cfc19dac", "parents": [ "886ebc7d4c1daf5f277cd4231036e1a799d69196" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Mon Aug 31 21:30:39 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 31 21:30:39 2026 -0700" }, "message": "fix(core): record the ordering field when upgrading a table from version 1 (#19803)" }, { "commit": "886ebc7d4c1daf5f277cd4231036e1a799d69196", "tree": "3b5d44319f2d1ced7ae88743708ccb1e5c8e889a", "parents": [ "0db4d3a05fd358d5b6ff93bce1e94f686eb8183e" ], "author": { "name": "Ranga Reddy", "email": "rangareddy.avula@gmail.com", "time": "Mon Aug 31 20:46:50 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 31 23:16:50 2026 +0800" }, "message": "fix(build): tidy shade config drift ahead of hoisting the POM flags (#19492)\n\nThree shade-plugin inconsistencies found while scoping #19466, none of which changes what any\njar contains today.\n\nhudi-io sets createDependencyReducedPom\u003dfalse with no explanation, which reads like an\noversight next to the packaging bundles that all set it true. It is not: the shaded jar is\nattached under the \"shaded\" classifier and the primary jar stays unshaded, so the published\nPOM must keep declaring protobuf-java. Reducing it would strip a dependency the primary jar\nreally needs. Recorded as a comment so the next person does not \"fix\" it.\n\nhudi-timeline-server-bundle was the only packaging bundle setting createDependencyReducedPom\nwithout also setting dependencyReducedPomLocation, so its reduced POM was written to the\nmodule root instead of target/. Added, matching the wrapped formatting the other 16 bundles\nuse.\n\nhudi-aws-bundle carried a duplicate \u003cartifactSet\u003e include block; removed the repeat.\n\nPartial fix for #19466 - deliberately not \"Closes\", since the hoist of the shade flags into the\nroot POM, which is the main ask, is not included here." }, { "commit": "0db4d3a05fd358d5b6ff93bce1e94f686eb8183e", "tree": "1ac2874ffb79d76212ce8168243368bf2f24f17d", "parents": [ "6103a52e5c8b272095e3f8eae241fb1fda33bf17" ], "author": { "name": "Stefan Wang", "email": "1fannnw@gmail.com", "time": "Mon Aug 31 08:39:39 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 31 20:39:39 2026 +0800" }, "message": "fix(common): prevent multi-table commits failing with metrics enabled (#19787)\n\nSigned-off-by: 1fanwang \u003c1fannnw@gmail.com\u003e" }, { "commit": "6103a52e5c8b272095e3f8eae241fb1fda33bf17", "tree": "19509264534dd640800a1aa7a784b6f786461fa5", "parents": [ "6db0dded833263c29c2a530c71638da4e38fb30c" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Mon Aug 31 16:27:22 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 31 16:27:22 2026 +0800" }, "message": "fix(variant): shred nested variants on the Avro write path for parity… (#19777)\n\n* fix(variant): shred nested variants on the Avro write path for parity with the row writer\n\nHoodieRowParquetWriteSupport shreds a variant at any depth, while\nHoodieAvroWriteSupport only touched top-level fields, so a nested\nvariant silently declined to the unshredded layout on the AVRO record\ntype. Same table, same config, two on-disk layouts.\n\n- VariantSchemaUtils.applyForcedShredding reaches variant record\n members at any depth, mirroring the row writer; a bare array element\n or map value stays unforced on both paths. Generated records get a\n Hudi-owned namespace, so a reused record type or a DDL field named\n like a user type no longer trips Avro\u0027s \"Can\u0027t redefine\". A record\n type with a variant member reached both at a forced position and\n inside a multi-branch union, which the DDL does not reach on either\n the write or the read side, is rejected at splice time with a clear\n error instead of failing at footer stamping.\n- HoodieAvroWriteSupport shreds values wherever the effective schema\n declares typed_value, through a Shredder tree with by-name field\n matching. A declared nested typed_value used to crash at the first\n write.\n- stripVariantShreddingByShape (the footer fallback) recurses, so a\n nested typed_value cannot leak into the table schema and Hive sync.\n- supportBatch applies the top-level variant policy at any depth, so\n nested-variant tables read row-based on Spark 4.1+ like top-level\n ones (#18605).\n- HoodieSchema.createShreddedFieldStruct and\n createVariantShreddedObject take the namespace explicitly; the\n overloads that minted records with a null namespace are gone.\n\nOut of scope: inference at depth; arrays keep their per-writer list\nlayout; the parquet-avro \u003c 1.14 single-field-element read limitation\nis #19782. The Spark-native internal read path is #19775.\n\nTests extend TestHoodieAvroWriteSupportShredding (with a value-level\nparquet round trip), TestVariantSchemaUtils, TestHoodieSchema and\nTestVariantShreddingMixedLayouts.\n\nCloses #19689. Part of #18937." }, { "commit": "6db0dded833263c29c2a530c71638da4e38fb30c", "tree": "e86601bc305374278da343bd1dbbec2ddfb57bab", "parents": [ "1593524e3244a7718fe5af1d75a19c703f71933a" ], "author": { "name": "hutiefang76", "email": "137664623+hutiefang76@users.noreply.github.com", "time": "Mon Aug 31 11:50:31 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 31 11:50:31 2026 +0800" }, "message": "fix(spark): handle empty projections with internal schema (#19796)\n\n* fix(spark): handle empty internal schema projections\n\n* fix(spark): pin the empty-projection fix on every profile\n\nThe empty internal-schema sentinel is load-bearing but unexplained,\nand the only end-to-end assertion for it sits behind\nassumeTrue(gteqSpark4_1), so it runs in the single spark4.2 lane and\nnever on Azure (-Dspark3.5).\n\n- HoodieBaseRelation: say why the empty sentinel is deliberate, so it\n is not \"aligned\" with the unpruned idiom that the four\n *LegacyHoodieParquetFileFormat copies use for the same input.\n- TestHoodieRelations: assert the schema-on-read arm lands on the same\n projection as the plain arm, which is the parity the fix is for.\n- TestLegacyParquetReadPath: add an ungated count() leg to\n testCowSnapshotReadWithSchemaOnRead, so the fix is pinned on every\n Spark profile rather than only 4.1+.\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "1593524e3244a7718fe5af1d75a19c703f71933a", "tree": "5af0ea603dea4e8d35ba784d8c69d5acc56c01b8", "parents": [ "8e884960820a081e34cdce8e78505e9ed54dff28" ], "author": { "name": "Ranga Reddy", "email": "rangareddy.avula@gmail.com", "time": "Sun Aug 30 19:59:47 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Aug 30 22:29:47 2026 +0800" }, "message": "fix(hadoop-mr): ignore blank Hive projection ids and report both lists on mismatch (#19463)\n\nRelates to #14673 (HUDI-1286).\n\nFor SELECT COUNT(*) on Hive before 3.0.0 the read-column ids arrive\nempty and Hive combines them with Hudi\u0027s required projection ids into\ne.g. \",2,0,3\" (HIVE-22438). Every consumer parses those with\nInteger.parseInt, so a blank entry failed with a bare\nNumberFormatException carrying none of the projection lists.\n\ncleanProjectionColumnIds is the single point all of them read from, so\nthe filter goes there. It stripped one leading comma, which leaves\n\",,2,0\" and cannot reach an interior blank at all: Hive prepends ids\nwhile appending names, so an id prepended after an empty one gives\n\"3,,2,0\". It now trims and drops every blank entry, writes the joined\nvalue back only when it changed, and no longer NPEs when the key is\nunset. The read-modify-write is synchronized on the conf, the way\naddProjectionToJobConf guards its own writes; f41539a9cb5f (#3630)\nmoved this call out of that latch without restoring a lock.\n\nHoodieParquetInputFormat#getRecordReader now cleans the conf as well.\nThat is where the bootstrap reader and the parquet schema-evolution\nreader are built, and neither goes through a realtime input format, so\non COW those consumers were still reached with the blank in place.\n\nSchemaEvolutionContext read the same key with no default and no filter\nin setColumnNameList and setColumnTypeList, so an unset key was an NPE\nand a blank id a bare NumberFormatException, in methods that already\nreport a size mismatch. Both now parse the non-blank tokens and report\nthat mismatch instead, carrying the counts actually compared rather\nthan only the raw conf value. getRequireColumn, which runs just before\nthem, gets the same missing-default fix.\n\norderFields keeps a filter of its own as defence in depth, for callers\nthat assemble the csv without going through the conf. Its mismatch\nmessage compared de-duplicated counts while printing the raw name\ncount, so duplicate names produced two equal numbers; the counts are\nnow named #distinctFieldNames and #distinctFieldPositions and both\nprojection lists are included.\n\nNot addressed here and filed as #19506: names and ids are de-duplicated\nindependently and then paired positionally, which is unsound because\nHive prepends ids while appending names. The fullColNamelist.get(id)\nIOOBE in setColumnNameList is out of scope for the same reason.\n\nThe mechanism behind #14673 is unconfirmed. An earlier revision\nasserted CombineHiveInputFormat accumulating projection entries across\nsplits, which the issue does not support: the reported frame is\nHiveInputFormat.getRecordReader, the base class, and addProjectionField\nalways sets both keys together. HIVE-22438 and HUDI-313 are what this\nrepo documents for that query.\n\nTests: new coverage for orderFields, cleanProjectionColumnIds and\nSchemaEvolutionContext, none of which had any before; a blank-id arm on\nTestHoodieRealtimeRecordReader#testIncrementalWithReplace driving\ncleanProjectionColumnIds and orderFields together; and blank ids in\nTestHiveTableSchemaEvolution, the only test that reaches\nHoodieParquetInputFormat#getRecordReader on the COW schema-evolution\npath. The bootstrap arm remains uncovered, since no test constructs a\nBootstrapBaseFileSplit.\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "8e884960820a081e34cdce8e78505e9ed54dff28", "tree": "6b450ebf60eb3622f9d1e7301a7cc384382e5660", "parents": [ "6fd80965527f4a5c8476ac188a52d8d293c22807" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Sun Aug 30 01:45:58 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Aug 30 01:45:58 2026 +0800" }, "message": "fix(spark): keep the embedded Derby metastore out of the RAT check (#19792)\n\nSince #19162 runs TestSparkCatalogMetaStoreClient in CI, its enableHiveSupport()\nsession writes Derby\u0027s metastore_db/ into the hudi-spark module directory, and the\nnext maven invocation in the same job fails the RAT check on the unlicensed database\nfiles (\"Files with unapproved licenses: .../hudi-spark/metastore_db/db.lck ...\").\n\nExclude **/metastore_db/** from RAT in the root pom so any Hive-enabled test is safe,\nand point the test\u0027s metastore at a temp directory (javax.jdo.option.ConnectionURL\nand derby.system.home, deleted in afterAll) so nothing is left in the tree at all." }, { "commit": "6fd80965527f4a5c8476ac188a52d8d293c22807", "tree": "53fc1d8fe21e0e22ba901a46beb50e856722c5fd", "parents": [ "01935d148fae80c07972d9363043182942e755f5" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Sat Aug 29 03:47:31 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Aug 29 18:47:31 2026 +0800" }, "message": "fix(spark): make SparkCatalogMetaStoreClient sync-safe and cover the Spark catalog DDL paths (#19162)\n\n\nProduction (SparkCatalogMetaStoreClient, the IMetaStoreClient used when\nhoodie.datasource.hive_sync.use_spark_catalog\u003dtrue):\n- getSchema returned every partition column twice (CatalogTable.schema already\n contains them) and rendered column comments as \"Some(x)\"/\"None\";\n HoodieHiveSyncClient.updateTableComments then threw \"Duplicate key\" for any\n partitioned table with sync_comment\u003dtrue.\n- fromCatalogTable dropped CatalogTable.comment, so a \"comment\" table property never\n round-tripped and updateTableProperties re-altered the table on every sync.\n- close() threw UnsupportedOperationException although HoodieHiveSyncClient.close\n calls it on every sync; it is now a no-op.\n- getPartition(db, table, values) threw UnsupportedOperationException although\n HMSDDLExecutor.dropPartitionsToTable reaches it through\n HivePartitionUtil.partitionExists, which only handles NoSuchObjectException; it is\n now implemented on ExternalCatalog.getPartitionOption.\n- alter_table forwarded to HiveExternalCatalog.alterTable, which keeps the stored\n schema by design, so a column added by HMSDDLExecutor.updateTableDefinition never\n reached the catalog. It now merges the synced columns conservatively and writes\n the result through alterTableDataSchema: new columns are appended and non-empty\n comments applied, while an existing column keeps its type, nullability and field\n metadata (the catalog holds the logical Spark schema, including the hudi_type\n markers for VECTOR/BLOB and the original VariantType, whereas the sync only speaks\n Hive type strings). The merged schema is also handed to alterTable itself so the\n behaviour holds under InMemoryCatalog. toCatalogTable carries column comments\n across so the getTable -\u003e alter_table round trips do not erase them.\n- alter_table pinned the identifier to the incoming name, so the rename step of\n HoodieHiveSyncClient.createOrReplaceTable (force-recreate, recreate-on-error,\n base-path change) dropped the real table and left only \u003ctable\u003e_temp; a Table\n carrying a different name now goes through ExternalCatalog.renameTable first.\n- HoodieCatalog.alterTable\u0027s unsupported-change message printed\n \"class java.lang.Class\" instead of the change class.\n\nTests:\n- TestHoodieCatalogDDL: HoodieCatalog create/load/alter/rename/drop through the V2\n TableCatalog API (external location survives drop, unsupported change rejected),\n SHOW CREATE TABLE through ShowHoodieCreateTableCommand (under\n spark.sql.legacy.useV1Command, the only route Spark gives that command),\n existing-location config conflict, HoodieInternalV2Table capabilities and the\n V2-to-V1 relation conversion for inserts under schema-on-read.\n- TestSparkCatalogMetaStoreClient: pins the merge contract (comment lands, property-\n only alter keeps comment and schema, metadata and catalog type survive a retype\n through the alterTableDataSchema write, comment-less column list keeps comments,\n shorter list drops nothing, rename round trip), getPartition hit and miss, close\n and setMetaConf no-ops, the warehouse fallback, and the UnsupportedOperationException\n contract for the rest of IMetaStoreClient. Helpers register input/output formats\n and a serde like HMSDDLExecutor, which the pre-existing tests needed to run at all.\n- TestSparkCatalogSync: forced table recreation through HiveSyncTool against the\n in-memory catalog.\n- TestHoodieSqlCommonUtils: partition-style detection incl. multi-column tables,\n config filtering, meta fields, partition-spec normalization, path qualification.\n- TestCreateTable: the illegal-CTAS cleanup test also covers an explicit location.\n\nCI: org.apache.spark.sql.hive is added to the scalatest wildcard lists in bot.yml and\nazure-pipelines-20230430.yml; TestSparkCatalogMetaStoreClient had never executed in\nCI and three of its tests had been failing on master unnoticed.\n\nNot covered: HoodieStagedTable. SQL CTAS on the session catalog is rewritten to the\nV1 CreateHoodieTableAsSelectCommand, so the staged path is only reachable when\nHoodieCatalog is registered under a non-session catalog name.\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "01935d148fae80c07972d9363043182942e755f5", "tree": "8b5670381d46db11d26c8727647166ce15992cfc", "parents": [ "1112c37baf68fbe48409046ad9b6ca003a846844" ], "author": { "name": "Rahil C", "email": "32500120+rahil-c@users.noreply.github.com", "time": "Fri Aug 28 13:00:01 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Aug 29 04:00:01 2026 +0800" }, "message": "fix(spark): restore the java.util.Map import in TestDataSourceUtils (#19789)\n\nTest compilation of hudi-spark_2.12 fails on master: the file uses\nMap\u003cString, String\u003e at line 136, but the import was dropped alongside an\nunrelated cleanup that removed the file\u0027s other use of Map." }, { "commit": "1112c37baf68fbe48409046ad9b6ca003a846844", "tree": "c296d7cf69fa18c154dc4e6c6aec5564f5f7609f", "parents": [ "9db33ba3302245b5bd077a5be0e76c16858f1461" ], "author": { "name": "Rahil C", "email": "32500120+rahil-c@users.noreply.github.com", "time": "Fri Aug 28 09:06:38 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Aug 29 00:06:38 2026 +0800" }, "message": "feat(metrics): report record index lookup counters to the metrics reporter (#19575)\n\nThere was no way to tell how much work a record index lookup did: how many\nkeys a commit looked up, what fraction were already in the table, or how many\nindex file groups were read. Each task computes those numbers and returns\nonly the matches, and a miss produces no output row, so the driver cannot\nrecover them afterwards. HoodieMetadataMetrics has declared\nlookup_record_index_key_count and lookup_record_index_key_hit_count for a\nlong time with nothing referencing them (see HUDI-9544). Part of #19063.\n\nEvery Spark write that tags records now reports, through the configured\nmetrics reporter and nothing else:\n\n \u003ctable\u003e.rli.lookup.lookup_record_index_key_count\n \u003ctable\u003e.rli.lookup.lookup_record_index_key_hit_count\n \u003ctable\u003e.rli.lookup.lookup_record_index_key_miss_count\n \u003ctable\u003e.rli.lookup.lookup_record_index_shards_read\n \u003ctable\u003e.rli.lookup.lookup_record_index_time\n\nCounting is per record, not per distinct key, so hits + misses \u003d\u003d key_count\nholds exactly. Lookup time is summed across shards (they are read in\nparallel), so it measures per-commit read effort; divide by shards_read for\na mean. Nothing is written to the timeline.\n\nHow it works:\n- Counters are collected on executors into an AccumulatorV2-backed\n DistributedRegistry. The driver resolves the registry\n (RecordIndexLookupMetrics.resolveRegistry) and the lookup closure captures\n it as a field, on both the global and partitioned record index paths;\n nothing resolves a registry by name on the executor.\n- Registries are owned by HoodieSparkEngineContext, keyed by a normalized\n base path (authority + path, so Spark SQL\u0027s file:///data/t and the\n DataSource\u0027s /data/t resolve to one entry while s3://a/t and s3://b/t stay\n distinct). They are never published into Registry.REGISTRY_MAP; an entry\n exists only between the lookup that created it and the commit that drains\n it, so its presence is the record that this write looked something up.\n- SparkRDDWriteClient.postCommit publishes and releases the registry once\n the commit has landed, covering DataSource, Spark SQL DML and StreamSync\n without a new hook on the base write client. A commit that never lands\n publishes nothing, each commit reports only its own lookups (an abandoned\n attempt\u0027s counters are discarded when the next write resolves the\n registry), a commit that collected nothing zeroes the names it previously\n published instead of re-reporting them, and publishing is wrapped in a\n catch-all so a reporting problem cannot fail a completed write.\n- INSERT drop-duplicates dedup now runs on the committing client\u0027s engine\n context and config instead of a throwaway HoodieSparkEngineContext\n (DataSourceUtils.resolveDuplicates is removed), so the lookup it performs\n is attributed to the commit rather than stranded.\n- DistributedRegistry ignores executor-side set/release instead of failing\n the job, and failed task attempts are not merged, so a retry does not\n inflate the counts. Duplicate successful attempts under speculation still\n can; exactness is tracked in #19759.\n\nGated by the new hoodie.metrics.rli.lookup.enable (default false, not\ninferred from hoodie.metrics.on, which it also requires). Off, nothing is\ncollected and no accumulator is registered. Spark only; counters aggregate\nper table per JVM. hudi-io and BaseHoodieWriteClient are untouched;\nhudi-client-common gains one config accessor.\n\nTests: unit coverage for the registry (including under task retry), the\nconfig gate and RecordIndexLookupMetrics; functional coverage on the Spark\nDataSource, Spark SQL, across a failed commit, and on the DeltaStreamer\npath, all reading the counters off a capturing reporter. The accumulator\nmerge itself was certified with spark-submit on local-cluster[2,2,2560]\n(two forked executor JVMs): 10000 / 5000 / 5000 across 10 shards on both of\ntwo upsert rounds, the second reading 10000 rather than 20000." }, { "commit": "9db33ba3302245b5bd077a5be0e76c16858f1461", "tree": "ed0500a418d468b9ec2bef5c25f619f8640565db", "parents": [ "6b5ea33228bb52efe3dc373da7524e06180222b7" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Fri Aug 28 08:50:00 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 28 15:50:00 2026 +0000" }, "message": "test(core): add file format dispatch exhaustiveness coverage (#19253)\n\nPer-HoodieFileFormat dispatch is duplicated across factory overloads and\nutility switches, so a new format can be wired into one dispatch point and\nmissed in another. That shipped with Lance (the getLogBlockType arm landed\n16 days after #17731, in #17768) and twice with Vortex (the StoragePathInfo\nreader overload and getLogBlockType, both still pending in #19252). These\ntests sweep HoodieFileFormat.values() instead of a hardcoded list so such\ngaps fail mechanically.\n\n- TestHoodieHadoopIOFactory: testGetFileFormatUtilsEntryPointsCoverEveryBaseFileFormat\n drives every format except HOODIE_LOG through both\n HoodieIOFactory#getFileFormatUtils(StoragePath) (the extension if-chain)\n and HoodieHadoopIOFactory#getFileFormatUtils(HoodieFileFormat) (the format\n switch); a case missing from either surfaces as\n UnsupportedOperationException, so a format wired into neither fails too.\n The hardcoded testGetFileFormatUtils gains its missing VORTEX rows and the\n class builds its factory from HoodieTestUtils.getDefaultStorage().\n- TestCommonClientUtils: testGetLogBlockTypeMapsEveryBaseFileFormat pins the\n expected log block type per base file format through a switch with a\n failing default (HFILE -\u003e HFILE_DATA_BLOCK, the rest -\u003e AVRO_DATA_BLOCK).\n VORTEX is excluded with a TODO(#19252) until its case lands; the expected\n VORTEX arm is already in place, so #19252 only drops the exclusion.\n\nNot covered: the HoodieFileReaderFactory overload sweep is owned by\nTestHoodieFileReaderFactory in #19255; shouldWriteNativeLogs is\nformat-independent since #19283; HoodieInputFormatUtils (hudi-hadoop-mr)\nremains the unswept dispatch family.\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "6b5ea33228bb52efe3dc373da7524e06180222b7", "tree": "1a03d9a5f43d5a4f1e0513f339c76dd39d9dc68c", "parents": [ "816bd96a583c29aafa18982ab8d30187eb48490a" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Fri Aug 28 22:47:25 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 28 22:47:25 2026 +0800" }, "message": "feat(storage): default parquet codec to zstd for flink and spark 3.5+ (#19685)\n\n* feat(storage): default parquet codec to zstd for flink and spark 3.5+" }, { "commit": "816bd96a583c29aafa18982ab8d30187eb48490a", "tree": "ff9ed93fa2edf3e1834a25cae1173212a734bece", "parents": [ "7b377a51ae1b3727795b73fafae76d7d67d32d90" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Fri Aug 28 07:01:16 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 28 22:01:16 2026 +0800" }, "message": "test(spark): cover schema-on-read evolution reads through the file-group reader (#19165)\n\n\nSchema-on-read evolution had no DataFrame-level test that reads an\nold-schema base file through the file-group reader: the existing\nadd-column and type-promotion tests either rewrite every file group\nbefore reading or drive the reader API directly, and no test asserts an\nincremental query after an evolution.\n\nTwo cases are added to TestBasicSchemaEvolution, which carried a TODO for\nupcasting; no new suite.\n\n- testSchemaOnReadAddColumnSnapshotAndIncrementalRead (COW and MOR):\n commit 2 adds a nullable `bonus` column and touches partition p1 only,\n so p2\u0027s base file keeps the old schema (on MOR the p1 base file does\n too and gains a log file). Asserts the null fill for rows served from\n old-schema files, pushed-down filters over the added column against a\n file that lacks it, and the exact record set plus `bonus` values of\n the incremental read after the evolution.\n\n- testSchemaOnReadTypePromotionOnMorBaseAndLogMerge (MOR, parameterized\n on promoteNested): commit 2 promotes `age` from int to long in p1 only,\n so the read merges an int base file with a long log file and widens\n the base-only int file in p2. The second arm also promotes `nested.a`:\n the vectorized read must fail fast with the ParquetSchemaEvolutionUtils\n non-atomic type-change rejection, which only engages when `nested` is\n projected, and with spark.sql.parquet.enableVectorizedReader\u003dfalse the\n row-based read must widen `nested.a` across both file shapes. COW\n promotion is already covered by TestLegacyParquetReadPath.\n\nTestHoodieFileIndex\u0027s multi-column pruning test with complex expressions\nadditionally asserts the parsed int partition value with getInt; the\nexisting toSeq(...).mkString rendering cannot tell an Integer from a\nUTF8String.\n\nTest-only change; no production code is modified.\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "7b377a51ae1b3727795b73fafae76d7d67d32d90", "tree": "945609c961209b23b04a980fbeae480a04f797f1", "parents": [ "efe02e1621c17aa39a61716371a1857b4c086f79" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Fri Aug 28 02:45:33 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 28 17:45:33 2026 +0800" }, "message": "test(spark): cover low-coverage small classes in spark datasource (#19164)\n\nOne suite per class, each in the package of the class it covers:\n\n- TestCachingIterator (hudi-spark-common, org.apache.hudi.util): full\n iteration and hasNext idempotency against a counting source iterator.\n The trait has had no in-repo user since #17457 moved the datasource\n read paths onto the file-group reader; removing it is left to a\n separate cleanup.\n- TestHoodieCatalog: HoodieCatalog.convertTransforms for identity\n columns, a single bucket transform, a sorted bucket transform keeping\n its sort columns, and the two rejections (multiple bucket transforms,\n unsupported transform). HoodieSparkCatalogUtils.MatchBucketTransform\n is covered through it, including the sorted_bucket arm.\n- TestBasicStagedTable (added by #19251): commit leaves the catalog\n alone, abort drops the staged table through it.\n- TestHoodieSparkValidateDuplicateKeyRecordMerger: the strategy id\n pinned as the literal, since a table that loads the merger through\n hoodie.write.record.merge.custom.implementation.classes selects it by\n that id and persists it, plus the pre-combining fallback to\n DefaultSparkRecordMerger. The merger was never wired in-repo: from\n #7345 it appeared in production sources only as a TODO in\n ProvidesHoodieConfig, beside the ValidateDuplicateKeyPayload\n selection that #12588 deleted; the class doc now says so instead of\n pointing at that deleted payload.\n- TestProcedureParameterImpl (added by #19161): the factories\u0027\n required/default semantics, toString field presence, a hashCode\n inequality case, and an equals case that isolates required from\n default.\n- TestCliSchemaProvider: the CLI bootstrap SchemaProvider in\n org.apache.hudi.cli (distinct from the Streamer one; named to avoid\n the TestSchemaProvider stubs in hudi-utilities and\n hudi-kafka-connect), shaped like flink\u0027s\n TestSchemaProviderCompatibility: legacy provider,\n modern provider whose target falls back to the overridden source\n HoodieSchema, and null handling. The getTargetHoodieSchema catch\n comment now names the two provider shapes that reach it.\n\nProduction changes are limited to those two comments; no behavior\nchange.\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "efe02e1621c17aa39a61716371a1857b4c086f79", "tree": "c9d45bfbda5f6d7ae7f67bd6e6ffaac349bd27dd", "parents": [ "3a3a32fb48f476be80735cbea5b5eb3dd71b6e01" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Fri Aug 28 14:50:49 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 28 14:50:49 2026 +0800" }, "message": "fix(variant): close the shredded-read gaps exposed by a mixed-layout test matrix (#19687)\n\nShredded-variant tests forced one typed_value layout per table; per-file\ninference (#18961) makes mixed layouts routine (different layouts per file,\nshredded bases under unshredded logs, typed and residual rows in one file).\nAuditing the paths that meet these mixes found six defects, several of them\nsilent data loss. Each fix is pinned by a test that was red before it.\n\n- schema: declare the shredded typed_value nullable, per the shredding spec.\n An Avro write of any non-object row under an object shredding schema crashed\n (\"Null-value for required field: typed_value\"); such rows now land in the\n residual. Old files stay readable, detection is shape-based.\n- hive: fail fast, naming the column, on a shredded variant column the query\n selects or one only the record merger reads (a CUSTOM merge whose merger is\n not projection compatible reads the whole table schema), at any nesting\n depth, on the file-group-reader path and the legacy MapredParquetInputFormat\n fall-through (fgr disabled, schema-on-read, bootstrap splits). Both guards\n honour Hive nested column pruning; the merge-only arm yields to skip-merge\n reads (no merger runs, the widened columns are projected away); the legacy\n guard anchors on the parsed Hive type, matches the footer by shape and stays\n best-effort on malformed projections. Before: silent nulls.\n- bootstrap: the one-arg getRecordIterator requests the caller\u0027s schema. The\n data file was read at its own footer schema, so reconstruction never engaged\n and typed_value was dropped; on a partitioned table whose external files\n omit the partition column the narrower record also made prependMetaFields\n shift every data column by one.\n- clustering: reject sort columns without an ordering (VARIANT and MAP at any\n depth, mirroring Spark\u0027s RowOrdering.isOrderable) up front on every entry\n point (run_clustering order/options, plan-strategy sort columns, bulk_insert\n sort columns, Flink clustering.sort.columns inside the task) via a shared\n SortUtils.validateSortableColumns that resolves dotted paths as the\n partitioners do; every sink trims the configured list; the ZORDER/HILBERT\n arm additionally requires exact top-level names, which the space-curve\n helpers otherwise dropped silently. Before: AnalysisException or\n ClassCastException deep in the job, or an unordered result.\n- spark: schema-on-read guard, recursing into nested types, shared with the\n four per-version Legacy parquet file formats; count(*) keeps working; the\n rewrite arm names both producers of the full-variant shape\n (PushVariantIntoScan on a query, Hudi\u0027s own base-file reads for inline\n compaction, inline clustering and CDC). Before: silent nulls with\n pushVariantIntoScan\u003dfalse, and those service reads died inside pruning.\n Real reconstruction is #18285.\n- spark4.0: rejectShreddedVariants in the read support, anchored on the\n catalyst schema and recursing into structs, arrays and maps, run from\n Spark40ParquetReader.read before either reader is built. Before:\n reorderVariantType could drop typed_value from the requested schema and\n nested variants were never reached.\n\nTests: new TestVariantShreddingMixedLayouts plus a VariantShreddingTestSupport\ntrait extracted from TestVariantDataType; matrix in #19688 (same-file\ntyped-vs-residual placement, four layouts in one table across snapshot,\ntime-travel, incremental and RO reads, three-layout MOR compaction, table\nversion 9, ordering modes, heterogeneous clustering). The Spark 4.0 module has\nno CI lane; its tests run locally under -Dspark4.0.\n\nNo config changes. Closes #19688. Part of #18937; unblocks #19689." }, { "commit": "3a3a32fb48f476be80735cbea5b5eb3dd71b6e01", "tree": "ce1364fb625004aecd5b6f979e468a95a6990320", "parents": [ "452b237501df5187f486856bc55369f83f1b6c64" ], "author": { "name": "Gonzalo Peña-Castellanos", "email": "goanpeca@gmail.com", "time": "Thu Aug 27 23:13:47 2026 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 28 04:13:47 2026 +0000" }, "message": "docs(hudi-notebooks): note S3A works with any S3-compatible object store (#19423)\n\n* docs: note S3A works with any S3-compatible object store\n\nSigned-off-by: Gonzalo Peña-Castellanos \u003cgoanpeca@gmail.com\u003e\n\n* docs(hudi-notebooks): make the S3A snippet valid XML and scope path-style access\n\nAngle-bracket placeholders inside \u003cvalue\u003e do not parse if the block is pasted into\ncore-site.xml, so use plain placeholder text. The path-style property is what MinIO and\nmost other providers need but Amazon S3 does not, so say so next to it the way the\nendpoint property already does.\n\n---------\n\nSigned-off-by: Gonzalo Peña-Castellanos \u003cgoanpeca@gmail.com\u003e\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "452b237501df5187f486856bc55369f83f1b6c64", "tree": "3baa6efa32b15ffe39c0a346e7610d87442bc222", "parents": [ "2d73fe40dd8a106af58454139484cbff64aca899" ], "author": { "name": "Praveen Gajulapalli", "email": "13733716+pkgajulapalli@users.noreply.github.com", "time": "Fri Aug 28 09:25:17 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 28 11:55:17 2026 +0800" }, "message": "fix(lock): name the cause on FAILED_TO_RELEASE in StorageBasedLockProvider (#19574)\n\n* fix(lock): name the cause on FAILED_TO_RELEASE in StorageBasedLockProvider\n\nThree distinct failures in StorageBasedLockProvider#unlock() threw a\nbyte-identical HoodieLockException message, so logs could not tell them\napart. All three also share a single updateLockReleaseFailureMetric\ncounter, leaving no way to attribute a release failure to a cause.\n\nEach throw now names its cause, and each logs the context needed to act\non it:\n\n- HEARTBEAT_STOP_FAILED: the heartbeat task would not stop, so the lock\n is deliberately left un-expired (the task could still renew it after\n we return). Logs the interrupted flag to separate the two sub-cases in\n LockProviderHeartbeatManager#stopHeartbeat.\n- INTERRUPTED_DURING_THROTTLE_BACKOFF: interrupted mid-backoff. Now also\n passes the InterruptedException so the stack trace survives.\n- THROTTLE_RETRIES_EXHAUSTED vs EXPIRE_WRITE_FAILED: distinguishes an\n exhausted retry budget against a storage rate limit (e.g. the GCS\n 1-write/sec per-object limit) from a terminal UNKNOWN_ERROR /\n ACQUIRED_BY_OTHERS outcome.\n\nThe four cause strings are declared as constants next to the other lock\ntunables, so the full set is visible in one place and both the call sites\nand the test assertions reference them rather than raw literals.\n\nOn ACQUIRED_BY_OTHERS, also log how long ago our lease should have\nended. A positive value means we overran our own lease, pointing at a\nstarved heartbeat (long GC, thread-pool starvation); a negative value\nmeans the lease had not elapsed by our clock, pointing at clock skew\nbetween nodes instead. Those two causes are indistinguishable today and\ncall for different fixes.\n\nEvery new message carries lockFilePath, so a lock left dangling in\nstorage can be joined back to the writer that failed to release it.\n\nBehaviour is unchanged: control flow is untouched, and each edit either\nadds a logger.error call or appends \", cause \u003cLABEL\u003e\" to an existing\nexception message. Metrics are unchanged.\n\nAdds a test for the interrupted-during-backoff path, which had no\ncoverage, and tightens the three existing FAILED_TO_RELEASE assertions\nto pin the specific cause label.\n\n* Include cause in the logs" }, { "commit": "2d73fe40dd8a106af58454139484cbff64aca899", "tree": "defcd35eefc63ae2dbc59e1b4eef564e0afd6138", "parents": [ "a0d4670ad829669876448db46484f5cacad99641" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Fri Aug 28 11:24:46 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 28 11:24:46 2026 +0800" }, "message": "feat(spark): support bucket index for LSM tables (#19727)\n\n* feat(spark): support bucket index for LSM tables" }, { "commit": "a0d4670ad829669876448db46484f5cacad99641", "tree": "5322f91321270fccde6ee6d4d5b303a9ef5f60b0", "parents": [ "863f84a08864ac4916c31b1412ec1eab737a1b73" ], "author": { "name": "Venkateswarlu Boggavarapu", "email": "mailtoboggavarapu@gmail.com", "time": "Thu Aug 27 22:28:20 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 28 10:28:20 2026 +0800" }, "message": "fix: close JDBC connection in UtilHelpers.getJDBCSchema to prevent connection leak (#19772)" }, { "commit": "863f84a08864ac4916c31b1412ec1eab737a1b73", "tree": "427af992a414bb83c25145f90fc9561a3688891a", "parents": [ "d4f9eb930a47ae0c0125fdd9da3e25139a361411" ], "author": { "name": "niranjan-1408", "email": "niranjan@onehouse.ai", "time": "Thu Aug 27 19:22:41 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 28 10:22:41 2026 +0800" }, "message": "fix(sync): drop the duplicate UpdateTable in updateTableSchema (#19762)\n\nupdateTableSchema issued the same UpdateTable request object twice. The second call\nsat outside the if (cascade) block, so it fired unconditionally, including on\nnon-partitioned tables where cascade is false.\n\nNothing between the two calls mutates the request, and the cascade branch writes\npartitions rather than the table, so the second call was a no-op that Glue still\ncounted as a new table version. Every schema change therefore consumed two table\nversions where one would do.\n\nThe first call is kept rather than the second because the cascade re-reads the table\nto source the columns it propagates: with the update first it sees the new columns,\nwhereas keeping only the second would have it propagate stale ones." }, { "commit": "d4f9eb930a47ae0c0125fdd9da3e25139a361411", "tree": "25d3b8f6808f63404034a2171a2575da67151f6f", "parents": [ "1895de8a06480506793577563f29b93a46317206" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Fri Aug 28 10:16:27 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 28 09:16:27 2026 +0700" }, "message": "refactor(variant): self-align log-block variant rows, drop buffer-level projection hook (#18923)\n\n* refactor(reader): push variant projection into log readers, drop buffer hook\n\nRemove the engine-neutral FileGroupRecordBuffer variant-projection composition\n(#18674\u0027s getLogBlockRecordProjection hook) so the merge buffer stays\nformat-agnostic; each log reader now emits rows already aligned to the projected\nread schema (#18739).\n\n- Parquet log blocks: thread the variant-overlaid StructType into a new\n HoodieSparkParquetReader.getUnsafeRowIterator(HoodieSchema, StructType, filters)\n overload so SPARK_ROW_REQUESTED_SCHEMA carries VariantMetadata and parquet-mr\n decodes variants into the projected struct shape natively (mirrors the base-file\n path). Wired in SparkFileFormatInternalRowReaderContext.getFileRecordIterator.\n- Avro log blocks: new no-op HoodieReaderContext.projectLogBlockRecords hook,\n invoked from HoodieAvroDataBlock.deserializeRecords; Spark overrides it to apply\n the VariantGet rewrite (relocated from the deleted buffer hook).\n- Both paths gated by a single shouldProjectVariants predicate (variant projection\n present AND merger not PAYLOAD_BASED), preserving the buffer\u0027s custom-payload skip.\n- FileGroupRecordBuffer/PositionBased now call getSchemaTransformerWithEvolvedSchema\n directly; getProjectedTransformer and getLogBlockRecordProjection deleted.\n- Sub-task 4: documented why the sparkRequiredSchema overlay must stay (HoodieSchema\n can\u0027t carry VariantMetadata); kept Spark-side, no schema-model change.\n\nbuildVariantProjector / isVariantProjectionStruct unchanged (caller moved).\n\nAddresses #18739.\n\n* refactor(variant): address review nits on log-block variant projection\n\n- HoodieSparkParquetReader: rename variant-overload parameter to\n structSchema and drop the no-op alias.\n- SparkFileFormatInternalRowReaderContext: extract isPayloadBased and\n drop the double negation in shouldProjectVariants.\n\n* refactor(variant): address review nits on parquet reader param and merger guard\n\n- Rename getUnsafeRowIterator param structSchema to projectedStructSchema\n- Add inline comment explaining the merger !\u003d null guard\n\n* review: address nit" }, { "commit": "1895de8a06480506793577563f29b93a46317206", "tree": "b71ba6b05e7e5004e17a7d9e342057d4dfa5212e", "parents": [ "fe82430e694ad00d1e405198054efb2a88aef56d" ], "author": { "name": "fhan", "email": "aaron.han.1986@gmail.com", "time": "Fri Aug 28 10:11:00 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 28 10:11:00 2026 +0800" }, "message": "perf(flink): preempt inactive write buckets on memory exhaustion (#19728)\n\n* perf(flink): preempt inactive write buckets on memory exhaustion\n* refactor(flink): simplify preemptive memory reclamation\n* refactor(flink): refine memory exhaustion recovery\n\n---------\n\nCo-authored-by: fhan \u003cyfhanfei@jd.com\u003e" }, { "commit": "fe82430e694ad00d1e405198054efb2a88aef56d", "tree": "6748588f0a73f4ddf73e93537184282830440457", "parents": [ "87778deec6f65d3d47d361e4ab57d8e3a7258a2e" ], "author": { "name": "niranjan-1408", "email": "niranjan@onehouse.ai", "time": "Thu Aug 27 17:12:52 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Aug 27 17:12:52 2026 -0700" }, "message": "fix(sync): keep each partition\u0027s recorded location when cascading Glue schema changes (#19761)" }, { "commit": "87778deec6f65d3d47d361e4ab57d8e3a7258a2e", "tree": "c88542fdc6bf46a27183dad67ac893ddb1430596", "parents": [ "85033e339f667c0d0bd8ed16de465bf27c517e3a" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Thu Aug 27 10:59:35 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Aug 27 10:59:35 2026 -0700" }, "message": "fix(streamer): Include start commit in S3/GCS IncrSource incremental query (#18949)" }, { "commit": "85033e339f667c0d0bd8ed16de465bf27c517e3a", "tree": "8cddfebacaf6107d5cda34c78d7fed1fe484ebdc", "parents": [ "3eef646d3afdc692b84c432120900f86e7ac3a58" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Thu Aug 27 06:56:02 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Aug 27 21:56:02 2026 +0800" }, "message": "test(spark): add write and commit path coverage (#19163)\n\nThe row-writer bulk_insert overwrite path in hudi-spark-common had no\nSQL-level coverage of rejectIfOverlappingPendingClustering (#18829 only\ndrives the RDD path) or of the static, dynamic and unpartitioned arms of\nDatasetBulkInsertOverwriteCommitActionExecutor.resolveTargetPartitions.\n\nTests, both in TestInsertTable2 and pinned to the row writer via\nhoodie.datasource.write.row.writer.enable\u003dtrue:\n\n- \"Test bulk insert with insert overwrite against pending clustering\":\n two partitions, clustering scheduled on one via run_clustering,\n parameterized over static and dynamic overwrite mode. Overwriting the\n non-clustered partition succeeds (INSERT_OVERWRITE, exactly one file\n id replaced, plan still pending); overwriting the clustered one is\n rejected by SparkRejectUpdateStrategy before any data file is written.\n The static arm also rejects \"partition (dt)\" without a value, which\n resolves to every partition.\n- \"Test bulk insert with insert overwrite on unpartitioned table against\n pending clustering\": a forced insert_overwrite (rejected), the deduced\n INSERT_OVERWRITE_TABLE through\n SparkInsertOverwriteTableCommitActionExecutor (rejected), and\n SparkAllowUpdateStrategy (goes through, plan left pending).\n\nFix: HoodieSparkSqlWriter.bulkInsertAsRow now closes the write client\nwhen executor.execute throws, mirroring the non-row-writer path. Every\nrejection above used to leak the client and its embedded timeline\nserver.\n\nTest base: HoodieSparkSqlTestBase clears the INMEMORY index\u0027s JVM-static\nrecord-location map after every test, before the catalog cleanup so it\nholds even if a drop throws, and gains getLastReplaceCommitMetadata next\nto getLastCommitMetadata.\n\nNot covered: the row-writer INSERT_OVERWRITE_TABLE variant deletes and\nre-initializes the table before the check runs (#19771). Found on the\nway, not fixed here: ALTER TABLE RENAME COLUMN / DROP COLUMN fail under\ndefault configs on a schema-on-read table since #13595 (#19766), and the\npending-clustering check is bypassed when meta fields are not populated\n(#19770).\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "3eef646d3afdc692b84c432120900f86e7ac3a58", "tree": "c66fa0eedf606ff15e2169a60722439cd2e13648", "parents": [ "98c61edfc674a7b08fb0dcff48b41a73f53ec855" ], "author": { "name": "Ranga Reddy", "email": "rangareddy.avula@gmail.com", "time": "Thu Aug 27 19:10:34 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Aug 27 21:40:34 2026 +0800" }, "message": "fix(hive): read the skeleton file when a bootstrap query projects no columns (#19510)\n\nA BootstrapBaseFileSplit carries two files: the split\u0027s own path is the skeleton, inside the\ntable root, and getBootstrapFileSplit() is the external source file, which is not.\ncreateBootstrappingRecordReader opens one of them when only one is needed, and it tested\n\"no Hudi meta column projected\" first, which resolves to the external file.\n\nSELECT COUNT(*) projects no columns at all, so both single-file conditions hold and that\nfirst branch won: Hive was handed a path outside the table root. Its vectorized reader\nresolves partition values by looking the split path up in pathToPartitionInfo, which only\nholds the table\u0027s own partition directories, so the query failed with\n\"cannot find dir \u003d \u003cexternal path\u003e in pathToPartitionInfo\".\n\nTest the external-column condition first, so a projection of no columns reads the skeleton.\nBootstrap keeps a one-to-one row correspondence between the two files, so the count is\nunchanged, and the skeleton is inside the table root and much smaller.\n\nScope, after review:\n\n - This fixes the no-projection shape only. Projecting data columns still opens the\n external split, and so does the stitch branch, so a partitioned bootstrap table still\n fails for those - filed as #19643 with the\n \"set hive.vectorized.execution.enabled\u003dfalse\" workaround.\n - It is not Hive-3 specific. hive-exec 2.3.10 ships the same vectorized reader; what\n differs is the default of hive.vectorized.execution.enabled, false in hive-common 2.3.10\n and true in 3.1.3 (javap on HiveConf$ConfVars: iconst_0 vs iconst_1). Hive 2 with\n vectorization enabled hits it too.\n - MOR is reached only for a bootstrap file slice with no log files. When a\n HoodieRealtimeBootstrapBaseFileSplit has delta logs, addVirtualKeysProjection injects the\n meta columns through the 3-arg addProjectionField, which does not consult LIST_COLUMNS\n and so always succeeds, leaving this branch unreachable.\n\nThe two booleans are now derived from the projected column names alone. The previous code\nzipped getReadColumnIDs with getReadColumnNames, which is the pairing\nHoodieColumnProjectionUtils warns about in getReadColumnIDs itself: the id list is\nde-duplicated and parsed with Integer.parseInt while the name list is neither, so a blank id\n(the HIVE-22438 shape) threw NumberFormatException and a duplicated name misaligned the zip.\nThe Integer half was never read.\n\nTests live in TestHoodieParquetInputFormat rather than a new class. The one that matters goes\nthrough getRecordReader: a bootstrap split whose skeleton holds 3 rows and whose external file\nholds 7, read with nothing projected, must yield 3. It yields 7 without the fix. The remaining\nthree combinations are a parameterised truth table over the two booleans.\n\nCloses #15676" }, { "commit": "98c61edfc674a7b08fb0dcff48b41a73f53ec855", "tree": "24f6ba5cfa170245ee71e3ae894b88520074d5b2", "parents": [ "0c3f00ee0958eb1c72889d7b593dd10cf8f92b35" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Thu Aug 27 18:19:17 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Aug 27 10:19:17 2026 +0000" }, "message": "test(utilities): cover the KafkaAvroSchemaDeserializer schema-refresh... (#19760)\n\n...branch in AvroKafkaSource\n\nAvroKafkaSource#readFromCheckpoint re-configures\nKafkaAvroSchemaDeserializer before every read: it stamps the schema\nprovider\u0027s current schema into\nhoodie.streamer.source.kafka.value.deserializer.schema, rotates group.id\nwith a hash of that schema so Spark does not hand back a cached\nKafkaConsumer whose deserializer still holds the old schema, and\nrebuilds KafkaOffsetGen. #10118 introduced the per-fetch re-stamp and\n#12111 the group.id rotation; neither came with a test that fetched\nthrough the branch.\n\nAdd testSchemaDeserializerRefreshesSchemaBetweenFetches to\nTestAvroKafkaSource. It produces Confluent-framed records under the\nsimple fixture schema through an in-JVM mock:// registry scoped to the\ntopic, fetches, evolves the SchemaTestProvider to the evolved fixture,\nproduces a second batch of which part is still written under the old\nschema, and fetches again on the same source instance from the first\ncheckpoint. It asserts that every record of the second batch carries the\nevolved schema, with the old-schema records resolving the new phone\nfield to its null default and the rest carrying the written value, that\nthe deserializer config holds the evolved schema, and that group.id is\nthe first fetch\u0027s group.id suffixed with exactly that schema\u0027s hash. The\ntopic has one partition so both fetches share a consumer cache key;\ndropping either the per-fetch re-configure or the group.id rotation\nmakes the second fetch decode with the stale reader schema.\n\nNo new fixtures: the simple and evolved *-test-with-default-value.avsc\nschemas, SchemaTestProvider, SchemaTestUtil and the class\u0027s Kafka\ntestcontainer harness are reused.\n\nCloses #19421" }, { "commit": "0c3f00ee0958eb1c72889d7b593dd10cf8f92b35", "tree": "2e0699e52714d0983be8553b0fbf1312609ece9a", "parents": [ "d43fa7fb63199a11bc0f88707b70694b5021b7a2" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Thu Aug 27 18:02:10 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Aug 27 18:02:10 2026 +0800" }, "message": "fix(trino): read LSM archived-timeline parquet files through a Trino-… (#19732)\n\nfix(trino): read LSM archived-timeline parquet files through a Trino-backed reader (#19732)\n\nHudiTrinoFileReaderFactory.newParquetFileReader threw UnsupportedOperationException, and\nhudi-common\u0027s ArchivedTimelineLoaderV2 obtains its reader for the LSM history files under\n.hoodie/timeline/history from that factory. Since #14019 the archived timeline is loaded\nlazily, so the throw surfaced whenever the completion time of an archived instant was needed,\ne.g. file slicing of a MOR table whose log file belongs to an archived delta commit.\n\nAdd TrinoParquetFileReader, a HoodieAvroFileReader over Trino\u0027s ParquetReader, and return it\nfrom the factory. The constructor reads the footer once, taking the schema from the file\u0027s\nparquet.avro.schema property and the record count from its row groups. Each\ngetIndexedRecordIterator opens its own ParquetReader over the projection the loader requests,\nwith column handles typed by HudiUtil.toColumnHandle, and turns every page into IndexedRecords\nthrough HudiAvroSerializer; VARBINARY values become the ByteBuffer that ArchivedTimelineV2\ncasts the metadata and plan columns to. The iterator remembers exhaustion, since on Trino 484\na nextPage() past the last row group throws instead of returning null again, and closing the\nreader releases every iterator it handed out, rethrowing the first close failure with the\nrest attached as suppressed. Failures at open, page read (checked or unchecked) and close go\nthrough the same handleException as Hive\u0027s ParquetPageSource: a ParquetCorruptionException is\nHUDI_BAD_DATA, anything else HUDI_CURSOR_ERROR, each with its cause. Record-key, key-prefix and\nrow-key lookups, the bloom filter and the min/max record keys are unsupported: a timeline\nfile carries none of the data-file footer metadata those rely on, and no Trino path asks for\nthem. Only LoadMode.TIME is reachable from the connector today.\n\nPorted from onehouseinc/trino#72 and adapted to the HoodieSchema-based reader contract, the\nPrefilledColumnValues serializer and Trino 484\u0027s ParquetReader API.\n\nTests. TestTrinoParquetFileReader reads archived_timeline.parquet, the first history file of\na four-instant tv8 COW table written with Hudi 1.0.2 (create script in\nhudi_cow_archived_timeline.md): the full read, the TIME, METADATA, PLAN and FULL projections\n(FULL orders plan before metadata, the reverse of the file, so it pins name-based mapping and\nthe ByteBuffer conversion), a drained iterator staying drained, close() releasing an open\niterator, a corrupt footer failing with HUDI_BAD_DATA, a corrupt data page failing with\nHUDI_CURSOR_ERROR and the ParquetDecodingException as its cause, and the unsupported footer\nlookups. TestHudiSmokeTest.testReadTableWithArchivedTimeline reads a tv8 MOR fixture\n(hudi_mor_archived_timeline.zip, script in hudi_mor_archived_timeline.md) whose oldest log\nfile belongs to archived delta commit 20250918122106595, read-optimized and real-time; both\ncases fail on master.\n\nFixes #13994" }, { "commit": "d43fa7fb63199a11bc0f88707b70694b5021b7a2", "tree": "1215d68e27c13ac68d8695e4f40fcfba4668f196", "parents": [ "18ae8c349058140f25a4f7247ff4a4bc8fc56dec" ], "author": { "name": "Lokesh Jain", "email": "ljain@apache.org", "time": "Thu Aug 27 15:30:42 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Aug 27 18:00:42 2026 +0800" }, "message": "fix(debezium): backfill Postgres TOAST columns on a copy so the merge result survives (#19749)\n\nOn a table-version-6 MOR table using PostgresDebeziumAvroPayload, an unchanged Postgres\nTOAST column (emitted as __debezium_unavailable_value) leaked to readers instead of being\nbackfilled from the prior value.\n\nThe payload backfilled the sentinel by mutating the incoming Avro record in place and\nreturning the same reference. HoodieAvroRecordMerger.merge has an identity shortcut\n(updatedRecord \u003d\u003d newerAvroRecord returns the engine-native newer record), so the merge\nresult was skipped and the backfill dropped.\n\nBackfill onto a copy instead: mergeToastedValuesIfPresent returns a new record when a\nTOASTed column is filled, and the same reference otherwise. The copy is allocated only\nwhen a sentinel is found, so records without one keep the merger fast-path. The merger is\nleft unchanged, so other payloads and the HoodieOperation marker it preserves are\nunaffected.\n\nAdd TestPostgresDebeziumToastV6ReadMerge, red without the fix and green with it." }, { "commit": "18ae8c349058140f25a4f7247ff4a4bc8fc56dec", "tree": "7a2b1ce6eba5c026778deef413f23c6679364d83", "parents": [ "3ddf6285d52e69ce842ac9197509dee854ca1acf" ], "author": { "name": "Peter Huang", "email": "huangzhenqiu0825@gmail.com", "time": "Wed Aug 26 20:01:04 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Aug 27 11:01:04 2026 +0800" }, "message": "feat(flink): Extend RocksDBDAO with generic column-family lifecycle operations (#19744)\n\nAdd columnFamilyExists and listColumnFamilies alongside the existing\naddColumnFamily/dropColumnFamily so callers can check existence and list\nmanaged column families; partition-aware naming and completeness tracking\nbelong in the Flink backend instead. Unit tests only; not wired into any\noperator yet." }, { "commit": "3ddf6285d52e69ce842ac9197509dee854ca1acf", "tree": "67214d6ab98b96eaf69175a2a14d01db68f6a7d4", "parents": [ "b5263e1a2e63eb8d1414b884a3b9138a9a5460f8" ], "author": { "name": "Lokesh Jain", "email": "ljain@apache.org", "time": "Thu Aug 27 05:55:12 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 26 17:25:12 2026 -0700" }, "message": "fix(timeline-service): support response charset for Timeline Server interaction (#19753)\n\nCo-authored-by: Y Ethan Guo \u003cethan.guoyihua@gmail.com\u003e" }, { "commit": "b5263e1a2e63eb8d1414b884a3b9138a9a5460f8", "tree": "3f12566ea53c3562db72ffbb6587627b6f9ce591", "parents": [ "ad221c449ecf30967d21ff352b1862e1e3f8f4ef" ], "author": { "name": "Lokesh Jain", "email": "ljain@apache.org", "time": "Thu Aug 27 05:10:34 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 26 16:40:34 2026 -0700" }, "message": "fix(client): create a new log file version instead of reopening the latest on pre-v8 writes (#19755)" }, { "commit": "ad221c449ecf30967d21ff352b1862e1e3f8f4ef", "tree": "fea13a8f936c8c3bcd7f9e362b643eceb789edbd", "parents": [ "34b4850421eb4c27d63679204e0014137bb1ab20" ], "author": { "name": "Lokesh Jain", "email": "ljain@apache.org", "time": "Thu Aug 27 04:49:55 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 26 16:19:55 2026 -0700" }, "message": "fix(common): escape dots in single-level hive-style partition column names (#19751)" }, { "commit": "34b4850421eb4c27d63679204e0014137bb1ab20", "tree": "861d6aea2546abcdcc9f060d6ecbc4e36d755f19", "parents": [ "c28a87d8987ca7f8b27904438430c5158c864b62" ], "author": { "name": "Lokesh Jain", "email": "ljain@apache.org", "time": "Thu Aug 27 04:00:14 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 26 15:30:14 2026 -0700" }, "message": "fix(streamer): route configured write table version into sample-writes flow (#19746)" }, { "commit": "c28a87d8987ca7f8b27904438430c5158c864b62", "tree": "e4079e6590a700d51265c31f5ee1dd4fdfedcabd", "parents": [ "bcaf2d48e6b77c5b4bb56346adefdad0df25e550" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Wed Aug 26 09:06:23 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 26 09:06:23 2026 -0700" }, "message": "fix(storage): resolve HoodieStorage from a path instead of a default local URI (#19737)" }, { "commit": "bcaf2d48e6b77c5b4bb56346adefdad0df25e550", "tree": "18f8e57d42410cfd533a363edbe4834091170d55", "parents": [ "a27828d8de756d2c871f7f5d9d255e73c261aa1d" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Wed Aug 26 18:18:42 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 26 10:18:42 2026 +0000" }, "message": "chore(docker): address PR #18520 review comments for Spark 4.0.1 stack (#18524)\n\n* chore(docker): address PR #18520 review comments for Spark 4.0.1 stack\n\n* fix(docker): match hadoop-aws and the AWS SDK bundle to the Hadoop that Spark bundles\n\nThe hadoop-aws jar lands on Spark\u0027s classpath next to the hadoop-client the\nSpark distribution bundles (3.3.4 for Spark 3.5.x, 3.4.1 for 4.0.x, 3.4.2 for\n4.1.x), so key the selection off SPARK_VERSION rather than the cluster\nHADOOP_VERSION. hadoop-aws 3.4.x is built against software.amazon.awssdk:bundle\n(SDK v2), not com.amazonaws:aws-java-sdk-bundle, so spark_base now picks the\nbundle artifact from the SDK major version and Spark 4 builds pull the v2\nversion each hadoop-aws release declares.\n\n* fix(docker): add the Spark 4.2 hadoop-aws pairing and fix the spark401 compose files\n\n- build_docker_images.sh: explicit 4.1 and 4.2 arms (Spark 4.2.x bundles Hadoop 3.5.0,\n SDK bundle 2.35.4); unmapped 4.x lines now warn before falling back to the newest pairing\n- drop the 19888 mapping: run_history.sh only starts the YARN timeline server on 8188\n- arm64: same apache/kafka:3.7.2 KRaft broker as amd64 so kafkabroker:29092 has a listener\n\n* fix(docker): key the hadoop-aws fallback off the Spark major and move the mapred-site guard to base_java11\n\n- a 5.x Spark line fell past 4.* into the Spark 3 pairing while the base image had already\n switched to java17; unmapped 4+ lines now warn and take the newest pairing\n- Hadoop 3.x ships mapred-site.xml directly (MAPREDUCE-6875), so the guard was dead in\n base_java17 and missing in base_java11, where --hadoop-version 3.3.4 failed on the bare cp\n\n* fix(docker): ship analyticsaccelerator-s3 alongside hadoop-aws 3.4.2+\n\nhadoop-aws 3.5.0 made the analytics input stream the default\n(StreamIntegration.DEFAULT_STREAM_TYPE), so the Spark 4.2 arm and the 4+\nfallback failed S3A init without the jar. spark_base now takes\nANALYTICS_ACCELERATOR_VERSION and fetches the jar when it is set; the script\npins 1.2.1 for 3.4.2 (still opt-in there) and 1.3.1 for 3.5.0, the versions\neach hadoop-project pom declares. Spark 4.0.x (hadoop-aws 3.4.1) and 3.x leave\nit empty since the stream type does not exist there. The jar declares no\ndependencies of its own." }, { "commit": "a27828d8de756d2c871f7f5d9d255e73c261aa1d", "tree": "680b3372759297619957720226febf71ff8ece07", "parents": [ "29b7a52564f4a3fe3979994c97605098733d6997" ], "author": { "name": "Ranga Reddy", "email": "rangareddy.avula@gmail.com", "time": "Wed Aug 26 15:32:09 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 26 10:02:09 2026 +0000" }, "message": "test(io): drive the HFile writer tests through byte arrays instead of a file (#19618)\n\n* test(io): drive the HFile writer tests through byte arrays instead of a file\n\nTestHFileWriter wrote every fixture to \"test.hfile\", a fixed relative path in the module\nworking directory, then read it back through FileChannel.map. HFileWriterImpl already\naccepts any OutputStream and HFileReaderImpl already reads from a\nByteArraySeekableDataInputStream - three sibling tests in this package build readers that\nway - so the file was never needed.\n\nWrites now go to a ByteArrayOutputStream and reads go through a new openReader(byte[])\nhelper, matching the idiom already used by TestHFileMultiBlockScan,\nTestHFileCompatibility and TestHFileReadCompatibility. No production code changes.\n\nWhat this removes beyond the file itself:\n\n - the fixed relative path shared by four test methods, which was only safe because they\n never run concurrently;\n - a hidden coupling where assertSingleBlockBytesMatchGoldenAndHBase re-read the file that\n assertSingleBlockBytesMatchHBase had left behind. The latter now returns the bytes and\n the caller passes them on;\n - the @AfterEach that existed solely to delete the file, and the FileChannel /\n StandardOpenOption imports;\n - three leaked readers: the old code closed the FileChannel but not the HFileReaderImpl\n opened inside it. Every reader now sits in try-with-resources.\n\nRenamed for what the helpers actually do now that no path is involved: writeNativeFile -\u003e\nwriteNativeHFile (returns the bytes), writeTestFile -\u003e writeThreeRecordHFile, and the\nvalidate* helpers take the bytes rather than reading a file behind the caller\u0027s back.\n\nThe HBase writer fixture keeps its temp file: that writer needs a Hadoop Path, and it is\ncreated with Files.createTempFile rather than a fixed name.\n\nhudi-io: 125 tests pass, checkstyle and apache-rat clean.\n\nCloses #17413\n\n* test(io): finish the byte-array sweep in TestHFileWriter\n\n- Drive the HBase fixture through an in-memory FSDataOutputStream via\n HFile.WriterFactory.withOutputStream. The writer never needed a Path,\n and the createTempFile it used leaked four files per run.\n- Drop the DataOutputStream wrapper around ByteArrayOutputStream at the\n three native write sites; HFileWriterImpl takes a plain OutputStream\n and closes it itself. The catch-and-rethrow-as-RuntimeException blocks\n went with it since both tests already declare IOException.\n- Route the remaining String.getBytes() calls through bytes() so the\n whole file encodes as UTF-8.\n- Reuse CONTEXT and hoist the shared three-record fixture; narrow\n writeThreeRecordHFile to throws IOException.\n\n---------\n\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "29b7a52564f4a3fe3979994c97605098733d6997", "tree": "1cbc644969351c698bdfbf72c3ee16044c2d65cd", "parents": [ "39c510d682e9014f7c776937379b23bd459e5ceb" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Wed Aug 26 15:18:50 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 26 16:18:50 2026 +0800" }, "message": "fix(trino): skip predicate pushdown on type-evolved parquet columns (#19467)\n\n* fix(trino): skip predicate pushdown on type-evolved parquet columns\n\n* test(trino): remove the dead INT96 branch in the parquet statistics test helper\n\n* fix(trino): drop domains the parquet bloom filter and decimal rescale would misread\n\n* test(trino): name the test that actually pins the bloom-width mechanism\n\nThe type-table comment on the int -\u003e long / long -\u003e int rows pointed at\ntestABigintLookupNeverFindsAnInt32ColumnsBloomHashes, which does not exist.\nThe test that pins the bloom miss on a real read is\nTestHudiEvolvedColumnPredicates.testEqualityOnAnIntColumnEvolvedToBigintStillFindsItsRows,\nwhich is what the class Javadoc and ParquetStatisticsDomains already name.\n\n---------\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e\nCo-authored-by: voon \u003cvoonhousu@gmail.com\u003e" }, { "commit": "39c510d682e9014f7c776937379b23bd459e5ceb", "tree": "dfca27873df5aa5e723f3b46d53975abe87b8472", "parents": [ "fe716f1f5b585ac79a027811f7a8a2ca6944e234" ], "author": { "name": "Venkateswarlu Boggavarapu", "email": "mailtoboggavarapu@gmail.com", "time": "Wed Aug 26 00:42:35 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 26 12:42:35 2026 +0800" }, "message": "fix: ensure AmazonDynamoDBLockClient is always closed in DynamoDBBasedLockProviderBase.close() (#19742)\n\n* fix: ensure AmazonDynamoDBLockClient is always closed in DynamoDBBasedLockProviderBase.close()" }, { "commit": "fe716f1f5b585ac79a027811f7a8a2ca6944e234", "tree": "40cc4d76b9366d31d003b945bd7e9a84e19fdd72", "parents": [ "e3ee9600fc5be950f3332fd9ffe2413697485fe0" ], "author": { "name": "Venkateswarlu Boggavarapu", "email": "mailtoboggavarapu@gmail.com", "time": "Wed Aug 26 00:13:13 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 26 12:13:13 2026 +0800" }, "message": "fix: ensure CuratorFramework is always closed in BaseZookeeperBasedLockProvider.close() (#19741)\n\nEnsure curatorFrameworkClient is closed in the finally block to prevent resource leaks." }, { "commit": "e3ee9600fc5be950f3332fd9ffe2413697485fe0", "tree": "92d6d62ff6fb4c3e1b5b7d10ab57f0c2dacdda39", "parents": [ "6cd1196795d875adcc935a956e2790a769d39bb0" ], "author": { "name": "Peter Huang", "email": "huangzhenqiu0825@gmail.com", "time": "Tue Aug 25 20:01:51 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 26 11:01:51 2026 +0800" }, "message": "feat(flink): add RocksDB partitioned RLI Cache Flink Options (#19641)\n\n* feat(flink): add RocksDB partitioned RLI Cache Flink Options\n\nExposes the 7 configuration properties defined by RFC-107 (Support data\npartition aware RocksDB RecordIndexBackend) in\nFlinkOptions\n\n* replace opt in config to index.rli.backend.type" }, { "commit": "6cd1196795d875adcc935a956e2790a769d39bb0", "tree": "8fbf3fc6df4dc7d1e92004a365d964816982f419", "parents": [ "6aa6a86f3dad86bcf9de03e46447aede8d327ba6" ], "author": { "name": "Sepuri Sai Krishna", "email": "saik20533@gmail.com", "time": "Wed Aug 26 07:55:28 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 26 10:25:28 2026 +0800" }, "message": "fix(spark): make partition DDL commands honor slash separated date partitioning (#19703)\n\n* fix(spark): make partition DDL commands honor slash separated date partitioning\n\nHoodieSqlCommonUtils#makePartitionPath read hive-style partitioning and URL\nencoding from the table config but never getSlashSeparatedDatePartitioning, so\non a slash-partitioned table the DDL commands computed 2026-01-05 while the\nwriter had laid the partition out as 2026/01/05. ADD PARTITION created a stray\ndashed directory (and its existence check never saw the real partition), while\nDROP PARTITION and TRUNCATE PARTITION targeted a directory that does not exist\nand silently removed nothing.\n\nApply the same substitution the write path performs, confined to a single\npartition field and skipped for hive-style partitioning, with a leading dash\nleft alone so the partition path never becomes absolute.\n\nCloses #19702\n\n* fix(spark): reuse the write path\u0027s path-breaking dash guard in partition DDL\n\nThe dash-to-slash substitution these commands perform carried its own guard,\nwhich only refused a leading dash. #19648 widened the write-side rule to reject\nany dash-delimited token that is empty, \".\" or \"..\", after finding that \"..-a\"\nbecame \"../a\" and resolved outside the table base path.\n\nDuplicating the rule is what let the two drift, so ask KeyGenUtils rather than\nrestate it: the commands have to name the directory the writer created, which\nmeans agreeing on every value, not just on the well formed dates.\n\nCovers the three token classes at the DDL level, where the stray directory is\nobservable: \"..-a\" must not create a sibling of the table, \"2026-\" must not lose\nits trailing dash to a normalized-away slash, and \"a--b\" must not split into\nnested directories." }, { "commit": "6aa6a86f3dad86bcf9de03e46447aede8d327ba6", "tree": "d5c22af55e5ce92f0258ca8c236954afdac6e11d", "parents": [ "1c573f84f6e08d3942ad0dc1fe47cc6f1ea0eff7" ], "author": { "name": "Vinish Reddy", "email": "vinishreddygunner17@gmail.com", "time": "Wed Aug 26 04:11:22 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Aug 25 15:41:22 2026 -0700" }, "message": "fix(core): resolve rollback storage from the partition path, not the default URI (#19735)" }, { "commit": "1c573f84f6e08d3942ad0dc1fe47cc6f1ea0eff7", "tree": "9398e75cfb86f3aea863794b7c727f5216843a88", "parents": [ "1ca682ca13b545a2268631a81e0a377edcc9b0b6" ], "author": { "name": "wangxianghu", "email": "wxhjsxz@126.com", "time": "Tue Aug 25 20:02:29 2026 +0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 26 00:02:29 2026 +0800" }, "message": "fix(utilities): use endOffsets when no offset is greater than the checkpoint timestamp (#19658)\n\n* fix(utilities) Use endOffsets when there is no offsets greater than timestamp checkpoint\n\n* address comments" }, { "commit": "1ca682ca13b545a2268631a81e0a377edcc9b0b6", "tree": "dfd33b46f8c16afad846b8f99e67b7b32da36b20", "parents": [ "ef07f0f948c9260eeaf07d5652ab08cb5401f6e1" ], "author": { "name": "Ranga Reddy", "email": "rangareddy.avula@gmail.com", "time": "Tue Aug 25 16:36:03 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Aug 25 19:06:03 2026 +0800" }, "message": "docs(hudi-io): fill in the HFile format details the doc was missing (#19721)\n\nCloses #17030 (HUDI-9454).\n\nhfile_format.md describes the HFile subset Hudi uses. Four things it left out or got\nwrong, all checked against hudi-io\u0027s reader and writer:\n\nBlock magics. The list named 4 of the 12 types in HFileBlockType, and TRABLK\"$ appeared\nin the Trailer section without ever being listed as a magic. Split the list into what\nHudi writes and what its reader additionally accepts so it can read HBase-written files:\nthe leaf/intermediate index magics, HBase\u0027s three bloom block types, the encoded data\nblock, and the v1 index block.\n\nChecksum Type. The field was described only as \"type of checksum used\" with no values.\nChecksumType defines NULL \u003d 0, CRC32 \u003d 1, CRC32C \u003d 2.\n\nMulti-level data block index. num_data_index_levels is listed in the trailer fields but\nnothing described what more than one level means, even though HFileLeafIndexBlock and\nHFileIntermediateIndexBlock exist and HFileReaderImpl walks down the levels. Added a\nsection covering it, including that Hudi\u0027s writer always emits a single root level and\nsets num_data_index_levels to 1, so the multi-level path exists only for reading HBase\nfiles.\n\nFile Info keys. The list had two entries, one of which was wrong: the doc gave\nhfile.MAX_MEMSTORE_TS_KEY, but HFileInfo declares MAX_MEMSTORE_TS_KEY and\nKEY_VALUE_VERSION without the reserved hfile. prefix, which LASTKEY, CREATE_TIME_TS,\nAVG_KEY_LEN and AVG_VALUE_LEN do carry. Corrected that, added the four missing keys, and\ncalled the inconsistency out so a reader does not go looking for a prefixed name that\ndoes not exist.\n\nAlso names the meta block key Hudi stores its bloom filter under, \"bloomFilter\". The\ndocument mentioned bloom filters in the section overview but never said how to find one,\nand it is easy to confuse with HBase\u0027s own bloom blocks, which Hudi never writes.\n\nDocumentation only, no code change." }, { "commit": "ef07f0f948c9260eeaf07d5652ab08cb5401f6e1", "tree": "3f663545bdff63ba77935d94c7753db63d18dee0", "parents": [ "3affaf7dd232fc1d1a5eeb035ca33a04b032b969" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Tue Aug 25 18:15:30 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Aug 25 18:15:30 2026 +0800" }, "message": "feat(flink): support Flink 2.2 (#19655)\n\n* feat(flink): support Flink 2.2" }, { "commit": "3affaf7dd232fc1d1a5eeb035ca33a04b032b969", "tree": "888fbd56ac065b96be9fd97c260720c81bc5218b", "parents": [ "476b335c1f31a4d378ebeb254df7e04ec6823154" ], "author": { "name": "Sepuri Sai Krishna", "email": "saik20533@gmail.com", "time": "Tue Aug 25 15:45:00 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Aug 25 18:15:00 2026 +0800" }, "message": "fix(hive-sync): pass the default partition through the slash-encoded value extractors (#19710)\n\nSlashEncodedDayPartitionValueExtractor requires a partition path of exactly\nthree segments, but a null partition value is written to the single-segment\n__HIVE_DEFAULT_PARTITION__ directory. Since that extractor is auto-inferred for\ntables using slash.separated.date.partitioning, hive sync of any such table\nthat has seen a null partition value fails outright.\n\nReturn the default partition marker as-is, which is what Hive uses for a null\npartition value, before the segment-count check. Apply the same to\nSlashEncodedHourPartitionValueExtractor, which carries the identical defect\nagainst its four-segment check.\n\nCloses #19668" }, { "commit": "476b335c1f31a4d378ebeb254df7e04ec6823154", "tree": "c8479cbc933e836340e72a8ecda2d64f31ed776a", "parents": [ "231545998a12fd1943e2393065f76cbdc0602eac" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Tue Aug 25 16:52:01 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Aug 25 16:52:01 2026 +0800" }, "message": "feat(variant): auto-infer per-file variant shredding schemas on shredding inference (#18961)\n\n* feat(variant): auto-infer per-file variant shredding schemas\n\nToday typed_value comes only from an explicit table schema or the test-only\nforce-shredding DDL, so production tables never shred variants. This adds\nper-file inference of the shredding schema from the first records of each\nbase file, for both HoodieRecordType paths and the bulk-insert row writer,\nreusing Spark 4.1\u0027s InferVariantShreddingSchema heuristics verbatim\n(SPARK-53659). Gated by hoodie.parquet.variant.shredding.schema.inference.enabled\n(default off); on Spark 4.0/Flink/Java the inferrer is absent from the\nclasspath and writes silently stay unshredded.\n\nA buffering HoodieFileWriter decorator (mirroring Spark\u0027s\nParquetOutputWriterWithVariantShredding: 4096 records / 64MB, infer once,\nreplay in order) defers the parquet writer until the schema is known. The\nAVRO factory splices the inferred typed_value into the schema argument; the\nSPARK and row-writer factories splice a copied config because the row write\nsupport resolves its schema from hoodie.write.schema/hoodie.avro.schema.\nInference failures decline to unshredded (a throwing inference must not fail\ncompaction); writer-creation/replay failures latch and rethrow through\nclose() so buffered records cannot be dropped silently. The Spark 4.1\ninferrer batches all variant columns into one call (global width budget) and\ndrops avro-illegal object keys, which legally fall back to the residual\nvalue column.\n\nAlso fixes latent issues this feature would trip: recursive\nVariant.getPlainTypedValueSchema (depth\u003e\u003d2 objects, arrays, value-only\nwrappers), avro field-reuse (Field already used) in stripVariantShredding\nand VariantReconstruction, and the table-schema footer fallback now strips\ntyped_value by shape so per-file layouts never leak into the resolved\ntable schema.\n\nStacked on #18938 (read-side reconstruction); part of #18937.\n\n* fix(variant): repair shredded-base reads and inference splice on multi-variant tables\n\nFixes surfaced by the auto-inference COW test, which is the first to read a\nshredded base file end to end:\n\n- VariantReconstruction never engaged on real files: the reader\u0027s file schema\n comes from converting the parquet footer MessageType, which loses the\n variant logical type, so the shredded group was projected down to\n {metadata, value} and typed_value was silently dropped on the AVRO\n read-then-rewrite path. Detect the on-disk side by shape, anchored on the\n requested column being a variant.\n- The inferred-shredding config splice aliased every variant column: columns\n share one record type named \u0027variant\u0027, which Avro serializes as name\n references after the first occurrence, so replacing one column\u0027s record\n with a same-named shredded definition shredded all of them on re-parse.\n Spliced records now get a per-column unique name.\n- getPlainTypedValueSchema named every nesting level \u0027\u003cname\u003e_plain\u0027 with a\n null namespace; for nested objects (every spec level is named typed_value)\n the inner record became an Avro self-reference of its ancestor, which\n Spark rejects as recursion. Names now chain the field path.\n- HoodieRowParquetWriteSupport warned \u0027no corresponding HoodieSchema\u0027 for\n every nullable variant column because the top-level check did not unwrap\n the field\u0027s nullable union; shredding still happened via the nested\n fallthrough, so this only silenced a misleading warning.\n- Test fixes: cast(variant as string) on a string-typed variant extracts the\n raw string (not its JSON form), and the decline column now uses per-row\n empty objects: inference is per file and a multi-row insert can fan out to\n one file per row, so cross-row type conflicts never reach one inference\n call and cannot decline deterministically.\n\n* perf(variant): skip schema parse in the row-writer factory when inference is off\n\nresolveConfigSchema parses the avro schema string; gate it behind the\ninference flag so the default-off path adds no per-file cost.\n\n* test(variant): gate inference tests on inferrer availability, not Spark version\n\nPR CI runs on the merge ref against current master, whose spark4.2 profile\nsatisfies gteqSpark4_1 but builds only hudi-spark4.2.x, which has no\nshredding-schema inferrer yet; the three inference tests then ran without\none and failed their typed_value assertions on silently-unshredded files.\nGate them on VariantShreddingRuntime.lookupInferrer() instead so any\nprofile without an inferrer cancels rather than fails, and starts running\nagain as soon as that version module ships its inferrer.\n\n* review(variant): drop detectProviderClassOnClasspath superseded by VariantShreddingRuntime\n\nThe inference work centralizes engine-component classpath detection in\nVariantShreddingRuntime (provider + schema inferrer, memoized). The merge\nconverged both call sites on it, leaving the interface helper added earlier in\nthis stack unused; remove it and its CLASSPATH_CANDIDATES constant.\n\n* fix(variant): use inlined GroupType.containsField in inference footer assertions\n\nThe inference tests cherry-picked here called the groupContainsField test helper,\nwhich the merged #18065 inlined to GroupType.containsField. The helper no longer\nexists, so the calls failed to compile; switch them to the inlined form.\n\n* review(variant): avoid double delegate.close() in inference writer close()\n\nOn the success path the try already called delegate.close(); if it threw, the\ncatch closed it again, relying on delegate close() being idempotent. Track a\ndelegateClosed flag (set before the try-path close) so the catch only cleans up\na delegate that materialize() created but never closed. Applied to both\nVariantShreddingInferenceFileWriter and its InternalRow sibling.\n\n* review(variant): read AvroVariantRow typed_value via a typedValue() helper\n\nRoute the scalar/struct/array getters through a private typedValue()\nhelper instead of repeating record.get(VARIANT_TYPED_VALUE_FIELD). The\nhelper name makes it self-evident at every call site that the getters\nignore ordinal (Spark only invokes them for ordinal \u003d\u003d typedIdx), and\ncentralizes the rationale in one comment.\n\n* review(variant): hoist SIZE_ESTIMATE_INTERVAL above the instance fields\n\nMove the constant to the top of the class, before the nested interface\nand instance fields, matching the sibling VariantShreddingInferenceFileWriter.\n\n* fix(variant): tag shredded row-writer variant parquet group with VARIANT logical type\n\nA shredded variant column written via the InternalRow row-writer path was\nemitted as a plain parquet group, missing the VARIANT(1) logical type that\nvanilla Spark 4.1 stamps on the variant boundary. Hudi reads round-tripped\nbecause they identify variants from the table HoodieSchema, but external\nreaders (vanilla Spark / Trino) rely on the parquet annotation and would\nsee a plain struct.\n\nThe shredded variant reaches convertField as a marked struct (the\nVariantType was replaced by its {metadata, value, typed_value} schema), so\nit never hit the VariantType branch that already annotates. Promote\napplyVariantLogicalType onto the SparkAdapter trait (public, no-op default)\nand apply it in convertField when isVariantShreddingStruct is true. Stays a\nno-op on Spark 4.0 (parquet 1.15.2 lacks the annotation) and Spark 3.x\n(never shreds); Spark 4.1/4.2 stamp it. Assert it in assertInferredFooters.\n\nNote: the Avro/HoodieRecord write path (AvroSchemaConverterWithTimestampNTZ)\nhas the same gap and is tracked separately.\n\n* fix(variant): tag shredded variant parquet group with VARIANT logical type on the Avro path\n\nCompanion to the row-writer fix: the Avro/HoodieRecord write path\n(AvroSchemaConverterWithTimestampNTZ) also emitted shredded variant groups\nas plain parquet groups, missing the VARIANT(1) logical type that vanilla\nSpark 4.1 stamps on the variant boundary.\n\nThis module is engine-agnostic and builds against parquet 1.15.2 on Spark\n4.0 where LogicalTypeAnnotation.variantType does not exist, so the\nannotation is resolved reflectively (null on \u003c 1.16). The group is built via\nthe Types builder because the annotation-bearing GroupType constructor is\nnot public outside org.apache.parquet.schema. A version-aware test asserts\nthe annotation is present on parquet 1.16+ and absent otherwise.\n\n* review(variant): rename finalBuilder to taggedBuilder in convertField\n\nIn Parquet\u0027s builder API .named() is the finalizing call that closes the\nchain, so finalBuilder read as if already completed. taggedBuilder better\nconveys \u0027the builder we will eventually call .named() on\u0027.\n\n* review(variant): rename detectProviderClass to getProviderClass\n\nThe method returns the pre-computed PROVIDER_CLASS field rather than\nscanning the classpath on each call; getProviderClass makes the memoized\nsemantics obvious without chasing the class-level Javadoc.\n\n* fix(variant): ship a Spark 4.2 shredding-schema inferrer and harden runtime probing\n\nINFERRER_CANDIDATES listed only the spark4.1 module\u0027s inferrer, so a Spark 4.2\nruntime silently wrote unshredded with the flag on (the config doc promised\n\"Spark 4.1+\"), and every CI lane skipped the three functional inference tests:\nthe spark4.1 lanes are trimmed from bot.yml and the active Spark 4 lane is 4.2.\nSpark 4.2\u0027s InferVariantShreddingSchema has the same public API, so the 4.2\nmodule gets a copy of the 4.1 class (the same precedent as the adapters) and\nthe candidate list gains it, most recent first.\n\nWhile here, make the static probes safe against anything but \"absent\": they run\nfrom the class\u0027s static initializer, so a LinkageError escaping probe() would\nfail \u003cclinit\u003e and every later getProviderClass() call on the main Avro write\npath would see a bare \"Could not initialize class\" with the cause lost. Load\ncandidates without initialization and degrade every LinkageError to absent;\nnarrow loadInferrer\u0027s catch (Throwable) to Exception | LinkageError.\n\n* test(variant): make TestAvroSchemaConverter variant expectations version-aware\n\nThe four variant tests compared exact MessageType strings without an\nannotation; AvroSchemaConverterWithTimestampNTZ now stamps VARIANT(1) on\nvariant groups on parquet 1.16+, so they failed under the spark4.1/spark4.2\nprofiles (CI hides it: hudi-hadoop-common UTs only run on spark3.5).\nMessageTypeParser cannot parse the annotation, so compare the converter\u0027s\ntoString() directly against a literal that carries \" (VARIANT(1))\" exactly\nwhen the running parquet has LogicalTypeAnnotation.variantType.\n\n* perf(variant): materialize payload-backed records once on the Avro sampling path\n\nAvroVariantSampleExtractor called record.toIndexedRecord per buffered record to\nsample, and HoodieAvroFileWriter.write/writeWithMetadata called it again at\nreplay, so payload-backed HoodieAvroRecords deserialized twice for the first\n4096 records of every file (and a MERGE INTO expression payload evaluated its\nassignments twice). Give VariantSampleExtractor a prepare() hook that returns\nthe record to buffer; the Avro extractor returns the materialized\nHoodieAvroIndexedRecord, whose toIndexedRecord is a pass-through, so sampling\nand replay both see it once. Records without data (delete payloads) are\nbuffered as they are.\n\n* review(variant): detect on the config schema in the Spark factory; namespace generated record names\n\nHoodieSparkFileWriterFactory detected inferable columns on its schema argument\nbut spliced into the config keys HoodieRowParquetWriteSupport resolves\n(hoodie.write.schema / hoodie.avro.schema); when the two differ the buffering\ncost is paid for a splice that never applies. Detect on the config schema in\nboth Spark factories through VariantSchemaUtils.getInferableVariantColumnsFromConfig\n(which parses the schema only once the cheap config gates pass), replacing the\nrow factory\u0027s private resolveConfigSchema.\n\nThe record names applyInferredShredding and getPlainTypedValueSchema generate\nwere unique within their own sets but could collide with a user-declared type\n(\u003ccol\u003e_variant) or with each other (typed_value_a_b_c_plain for both a/b_c and\na_b/c). Put them in Hudi-owned namespaces: hoodie.variant.inferred for the\nspliced variant records, and hoodie.variant.plain.\u003cfield path\u003e for the plain\ntyped_value records, where \u0027.\u0027 separates the path segments so distinct paths\ncannot concatenate to one name.\n\nAlso document that stripVariantShreddingByShape has no requested-side anchor\n(the footer fallback runs precisely when no table schema exists to anchor on,\nso a same-shaped user struct is stripped too) and is top-level only, matching\ngetInferableVariantColumns.\n\n* docs(variant): inference applies to native parquet log files too; note the buffer memory\n\nThe config doc claimed MOR log files are always written unshredded, which is\nfalse on the default table version (10): native log data files are\n\u003cfileId\u003e_\u003ctoken\u003e_\u003cinstant\u003e_\u003cv\u003e.log.parquet and go through the same wrapped\nnewParquetFileWriter(path, ...), so each log file infers its own layout (as\nforced-DDL shredding already does on master; the read path handles it). Say\nso, and note the per-writer 4096-record / 64MB buffer so executor memory can\nbe sized for concurrently open handles.\n\n* test(variant): cover the row-writer decorator, extractors, the no-inferrer fallback and schema-utils legs\n\n- TestVariantShreddingInferenceInternalRowFileWriter: the row-writer decorator\n had no tests although its module\u0027s UTs run in CI; replay order across both\n writeRow overloads, row copies, record and UnsafeRow byte caps, inferrer\n failure, zero-row close, latched factory and replay failures, canWrite.\n- TestAvroVariantSampleExtractor: ByteBuffer vs byte[] payloads as defensive\n copies, absent column, null variant, delete payload, prepare() materializing\n once and passing indexed records through.\n- TestHoodieAvroFileWriterFactoryVariantInference: the flag with no inferrer\n on the classpath hands out the plain writer and writes unshredded, which is\n the only path most CI lanes (and Flink/Java/Spark 4.0) exercise.\n- TestVariantShreddingInferenceFileWriter: close exactly once (and not retried\n when the delegate\u0027s close throws), replay failure latched, getFileFormatMetadata\n before/after materialization and after close, prepared record sampled and\n replayed; the footer-metadata test now asserts forwarding on an open writer.\n- TestVariantSchemaUtils: two same-named variant columns stay apart through the\n config round trip (pins the 5f413b9c fix), every schema key is spliced,\n non-nullable splice, top-level-only detection, config-schema detection, and\n the documented missing anchor of the shape strip; fixtures via the new\n HoodieSchemaTestUtils.createPlainShreddedVariantRecord.\n- TestHoodieSchema uses HoodieSchema.createShreddedFieldStruct instead of a\n hand-rolled wrapper; TestHoodieAvroWriteSupportShredding shares its fixture\n and drops an assertion the sibling test already makes.\n- TestVariantDataType: the inference tests assert through assertVariantLayout\n and a new assertInferredTypedValue helper; the MOR test splits base from\n native log files so the base-file leg cannot pass on log files alone; the\n older shredding tests scope their session settings with withSQLConf instead\n of leaking them for later tests to unset.\n\n* review(variant): budget the byte cap on record payload, gate the Spark factory on the schema argument\n\n- The buffering byte cap estimated each record with a deep object-size walk,\n which charges the shared Avro Schema graph (or the Spark StructType) once per\n buffered record, the HUDI-9499 class of over-estimate; with prepare()\n materializing payload-backed records that now applied to every Avro record,\n shrinking the inference sample well below the intended 4096 rows. Add\n VariantSampleExtractor.sharedSizeEstimate (the Avro extractor measures the\n schema once per schema instance, the Spark extractor the StructType once) and\n subtract it from each estimate.\n- HoodieSparkFileWriterFactory checks the already-parsed schema argument before\n parsing the config schema, and intersects the two: the native log, delete and\n CDC writers share this factory with schemas that have no top-level variant and\n no longer pay a config-schema parse per file.\n- getInferableVariantColumnsFromConfig no longer re-evaluates the config gates.\n- SparkVariantSampleExtractor reuses VariantShreddingInferenceInternalRowFileWriter.resolveOrdinals\n instead of its own copy of the loop.\n- inferTypedValues in both decorators also degrades a LinkageError from the\n inferrer to unshredded instead of failing the write.\n- AvroSchemaConverterWithTimestampNTZ.isVariantLogicalTypeSupported() replaces\n the reflection probe the tests repeated.\n\n* test(variant): make the round-1 tests discriminate, and pin the round-2 changes\n\n- TestHoodieAvroFileWriterFactoryVariantInference names a shredding provider so\n the inferrer gate, not the provider gate, is the one under test.\n- TestHoodieAvroWriteSupportShredding keeps the shredded-group assertion: on\n every CI lane (parquet \u003c 1.16) it was the only one that could fail.\n- TestVariantDataType: the MOR test asserts the native-log leg instead of\n guarding on it and narrows it to *.log.parquet data logs; the bulk-insert test\n pins the BULK_INSERT operation so a silent fallback to the record writers is\n caught; an un-gated test asserts an inferrer is registered on Spark 4.1+ (the\n gap that let the spark4.2 module ship without one); helper tidy-ups.\n- New tests for the shared-size subtraction (extractors and both decorators\u0027\n byte-cap accumulation through the estimator), the row decorator\u0027s key copy,\n SparkVariantSampleExtractor\u0027s adapter-free branches, the namespace scheme\n keeping concatenating paths apart, and the detect/splice key asymmetry.\n- TestAvroSchemaConverter drops testVariantToParquetConversion (a duplicate of\n testUnshreddedVariantType) and uses the converter\u0027s own version probe.\n\n* test(variant): import getLastCommitMetadata from the HoodieSparkSqlTestBase companion\n\nThe BULK_INSERT assertion added in 2ce0b7d7 referenced getLastCommitMetadata\nunqualified; it is defined on the HoodieSparkSqlTestBase companion object, not\nthe base class, so the scala test compile failed on Azure.\n\n* test(variant): size the estimator-accumulation rows so the rounding slack stays under one row\n\ntestByteCapAccumulatesThroughTheEstimator allowed a 100-byte slack for the\nmoving average\u0027s long truncation, but a GenericInternalRow with one Long is\nabout 56 bytes, so the cap landed a row early on Azure. Use kilobyte rows and\nrecords in both decorator tests and assert that size up front.\n\n* review(variant): drop the stray MarkerCreationStatus enum\n\nZero references anywhere; a timeline-server marker artifact that slipped in with 09b011b7880b.\n\n* review(variant): null-default the synthesized value field and scope two javadoc claims\n\nThe value field synthesized by stripVariantShreddingByShape now carries NULL_VALUE like\nevery other optional field on the footer path, so the two- and three-field file layouts\nresolve to equal schemas. The same method\u0027s javadoc scopes its top-level claim to the AVRO\nwrite support (the row writer can force-shred at depth), and the VariantShreddingRuntime\njavadoc no longer claims loadInferrer skips candidate initialization.\n\n* review(variant): harden the inference decorators\n\n- cleanup-close failures now suppress onto the primary via CloseableUtils (both decorators)\n- an Error mid-replay latches like an exception, so close() cannot finish the file without\n the un-replayed records; latch tests added to both suites\n- the byte cap rescales already-buffered records when the size estimate is revised (the row\n writer rescales only estimated rows, keeping the exact UnsafeRow charges)\n- SparkVariantSampleExtractor.prepare() detaches the buffered record from the handle\u0027s own\n wrapper, closing the deflate window the Avro extractor already avoided\n- the row suite covers a throwing delegate close; the Spark shared-size test asserts against\n ObjectSizeCalculator instead of itself\n\n* test(variant): make the VARIANT logical-type expectation an independent oracle\n\nThe version-aware tests branched on the converter\u0027s own isVariantLogicalTypeSupported, so a\nrotted production probe would flip expectation and behavior together. Both suites now\nreflect on LogicalTypeAnnotation directly; the production helper lost its last caller and\nis deleted.\n\n* fix(variant): resolve shredding caps for compaction outside a SQL execution scope\n\nSpark\u0027s InferVariantShreddingSchema reads maxSchemaWidth/maxSchemaDepth from SQLConf.get in\nits constructor. Offline and async compaction dispatch via plain RDDs with no SQL conf\npropagation, so user overrides silently fell back to Spark\u0027s defaults there. Both inferrers\nnow resolve the caps SQLConf-first with a SparkConf fallback (the\nresolveSessionLocalTimeZone pattern) and pin them via SQLConf.withExistingConf.\n\n* review(variant): scope three comments to what the writers actually shred\n\nThe native-log data writer does receive a top-level variant through this\nfactory, so it drops out of the no-variant list. The by-shape strip\u0027s\njavadoc and the resolver comment now say the row writer shreds at any\ndepth its write schema asks it to, forced DDL or not, and that a nested\nshredded column can therefore still surface from the footer fallback.\n\n* review(variant): document the internal-schema gate on the inference config\n\n* review(variant): close the delegate when materialization dies with an Error inside close\n\nmaterialize() latches and rethrows Errors, but the close() catch listed\nonly IOException and RuntimeException, so an Error thrown by the first\nmaterialization inside close() left the just-created delegate open with\nthe closed flag already set. Both decorators now catch Error too, with a\ntest each pinning the cleanup.\n\n* test(variant): pin non-null typed_value values via block column statistics\n\nThe inference assertions read only the footer schema, so a file whose\nrows all fell back to the residual value with typed_value written\nall-null passed every test. Each pinned file must now show a non-null\ncount on a typed_value leaf in the block column statistics.\n\n* test(variant): pin the byte-cap rescale and the UnsafeRow charges, count typed leaves only\n\nRound-3 review follow-ups:\n- Reshape both byte-cap tests so the 100th record is much bigger than the\n first 99: the rescale on re-estimation is what meets a one-big-record\n cap, and both tests fail without it (same-sized records tripped the cap\n with or without the rescale).\n- Pin the row writer\u0027s incremental rescale with an UnsafeRow interleaved\n among estimated rows; assigning the estimated total would drop its\n exact charge.\n- Count only typed leaves in typedValueNonNullCount; the residual value\n leaves hold the fallbacks.\n- Re-parse the serialized plain schema before asserting the two leaf\n names differ; Avro aliases a repeated name silently.\n- Scope the writeRow comment: no production caller is wrapped today.\n\n* test(variant): pin the 64MB ceiling, the suppressed close failure and the per-schema memo\n\nRound-4 review follow-ups:\n- Both decorators cap the buffer at min(64MB, maxFileSize); no test passed\n a file size above 64MB, so the min was unpinned. One row (UnsafeRow) or\n record (64MB payload) at the ceiling now materializes with the file size\n limit at Long.MAX_VALUE, on both twins.\n- The replay-failure tests also fail the delegate\u0027s close and assert the\n close failure arrives as suppressed on the replay failure, so a bare\n swallow in the catch path no longer passes.\n- The shared-size memo test asks about a second schema and the first one\n again, pinning the identity-keyed invalidation.\n- The inference config doc now says data blocks inside Avro-format log\n files stay unshredded whether Avro or parquet: the inline parquet block\n writer goes through the OutputStream overload, which is never wrapped.\n- The writeRow comment names the two factories that build the decorator\n rather than enumerating callers; the Flink row-data handle is a third." }, { "commit": "231545998a12fd1943e2393065f76cbdc0602eac", "tree": "e19b52e80ed807f2e40c1e507d4735c3d3189a91", "parents": [ "e0fe585bbd4c5eab8a6bdbb93972e19fc454956d" ], "author": { "name": "Lin Liu", "email": "141371752+linliu-code@users.noreply.github.com", "time": "Tue Aug 25 01:34:30 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Aug 25 16:34:30 2026 +0800" }, "message": "fix(spark-sql): resolve a partition path without validating the record key (#19709)\n\n* fix(spark-sql): resolve a partition path without validating the record key\n\nSqlKeyGenerator#getPartitionPath(GenericRecord) delegated to BaseKeyGenerator#getKey,\nwhich builds the whole HoodieKey and so computes AND validates the record key. Asking\nfor a partition path therefore failed on a record whose record key was unset, even with\nthe partition column fully populated.\n\nThat is reachable on MOR with a global bloom or global simple index. The tagging stage\nmerges the incoming record with its existing version -- mayContainDuplicateLookup is\ntableType \u003d\u003d MERGE_ON_READ in HoodieGlobalBloomIndex and HoodieGlobalSimpleIndex, and\nrequiresMergingWithOlderRecordVersion in\nHoodieIndexUtils#tagGlobalLocationBackToRecords is true on MOR whereas CoW defers the\nmerge to the file rewrite -- and then asks the key generator for the merged record\u0027s\npartition path via HoodieIndexUtils#inferPartitionPath. Under MOR partial updates that\nmerged record is materialised against WRITE_PARTIAL_UPDATE_SCHEMA, carrying only the\ncolumns named in UPDATE SET, so a MERGE INTO that does not assign the record key -- the\nordinary partial-update shape -- died with\n\n HoodieKeyException: recordKey value: \"null\" for field: \"id\" cannot be null or empty\n\nResolve the partition path directly where the delegate exposes it. Narrowing on\nBaseKeyGenerator rather than casting keeps behaviour identical for a key generator that\nimplements only SparkKeyGeneratorInterface, which declares just the Row and InternalRow\naccessors; its only Avro-facing method, inherited from KeyGeneratorInterface, is getKey.\ngetPartitionPathFields in this same class already uses that construct.\n\nScoped to the GenericRecord overload. getRecordKey(GenericRecord) still routes through\ngetKey and still validates -- it wants the record key -- and getKey itself is untouched,\nso callers needing a fully validated HoodieKey keep getting one. The change is strictly\nmore permissive: getKey is\nnew HoodieKey(getRecordKey(record), getPartitionPath(record)), so this returns the same\nexpression for the partition half and merely stops computing the record-key half.\n\nTestSqlKeyGenerator pins all three edges: partition resolution with the record key unset\nnow succeeds (this case fails on the parent commit with the exception above), record-key\nresolution still rejects a missing key, and a complete record is unaffected.\n\nThis unmasks a separate, previously unreachable defect on the same path, where the\npartial-update merged record is serialized against a mismatched schema and raises\nUnresolvedUnionException from BaseAvroPayload#getRecordBytes. That is a distinct root\ncause in the payload path and is left for a follow-up so this change stays reviewable;\nend-to-end coverage of the shape is held with it.\n\n* Cover the sql partition schema and the default-partition substitution\n\nThree cases added after review. One sets hoodie.sql.partition.schema, which Spark SQL always\nsupplies, so convertPartitionPathToSqlType is actually exercised rather than short-circuited.\n\nThe other two pin what happens when the partition field itself cannot be resolved from the\nrecord. KeyGenUtils.getPartitionPath substitutes the default partition for a null or absent\nvalue, and that predates this change: getKey called the same getPartitionPath, so the\nsubstitution already applied whenever the record key resolved. Confirmed on the parent commit,\nwhere the record-key-present case passes and only the both-missing case fails. What this change\nwidens is therefore narrow, and now stated: with the record key also unresolvable, the record-key\nexception no longer pre-empts the substitution.\n\nResolving a partition path is not a validity check on the record, so a caller that needs the\npartition to be genuinely present has to assert that itself.\n\n* Cover the timestamp arms and the auto-record-key delegate, and mirror getRecordKey\n\nAddresses review feedback.\n\nThe sql-partition-schema case pinned nothing: a `dt string` schema takes the identity arm of\nconvertPartitionPathToSqlType, so deleting that call entirely left every test green. Replaced\nwith a `dt timestamp` schema over a microsecond value, which drives the TimestampType arm and\nescapePathName. The expected value was captured from a run rather than hand-derived, and the\ndefault DateTimeZone is pinned and restored because the output is formatted in it.\n\nAdded a TimestampBasedKeyGenerator delegate case. That generator resolves an absent partition\nfield to the formatted epoch rather than HUDI_DEFAULT_PARTITION_PATH, so the guard in\nconvertPartitionPathToSqlType never fires for it. This change makes that reachable, where the\nrecord-key exception previously pre-empted it, so the behaviour is pinned rather than implied.\n\nAdded a case that omits the record-key config. That makes isAutoGeneratedRecordKeysEnabled true\nand exercises the AutoRecordGenWrapperKeyGenerator, which this change does alter: as a\nBaseKeyGenerator it now takes the direct arm, so getPartitionPath no longer consumes a generated\nsequence id as a side effect of building a HoodieKey. Uniqueness does not depend on that stride,\nbut it is now a pinned decision rather than an unexercised side effect.\n\ngetRecordKey(GenericRecord) takes the mirror arm, so neither accessor pays for the other half.\nPreviously it computed the partition path and discarded it on every Avro record, and a\npartition-side failure could surface from a record-key lookup: TimestampBasedAvroKeyGenerator\nraises independently of the key.\n\nAnnotated the default-partition case with a pointer to TestSimpleKeyGenerator\u0027s \"TODO this\nshould throw as well\" on the Avro path, so both expectations are unwound together. Dropped the\nstring-interpolation prefixes that interpolate nothing and derived the two partial schemas from\nthe base schema instead of restating them.\n\n* Set the partition schema in the fixtures, guard the MergeIntoKeyGenerator ordinals, drop the getRecordKey mirror\n\nAddresses review feedback. Three things, the first of which invalidated most of the previous\nround\u0027s coverage.\n\nThe fixtures defaulted the partition schema to None, but production sets\nhoodie.sql.partition.schema at every construction site (ProvidesHoodieConfig and both\nMergeIntoHoodieTableCommand copies). With it unset, convertPartitionPathToSqlType returns its\ninput immediately, so seven of the eight cases never reached the default-partition guard, the\nfragment-count early-out or hive-style handling, including the two named after default-partition\nbehaviour. It now defaults to a real schema, with one explicit None case for the non-partitioned\ntable, paired against the timestamp case on the same value so the conversion is shown to be\nskipped rather than merely absent. The auto-record-key case was likewise red on the parent commit\nonly because the fixture omitted a property that HoodieCreateRecordUtils always sets, so it failed\non the harness rather than on the behaviour it claimed to pin; it now sets both properties and\nasserts the sequence id, which reads 100_1_0 here and 100_1_2 on the parent. The timestamp\ndelegate is pinned under all three partition-schema spellings, because they diverge: a timestamp\ncolumn turns the epoch string into a bare NumberFormatException, a string column silently accepts\nthe epoch partition.\n\nMergeIntoKeyGenerator reads the record key and partition path by meta-field ordinal, and a record\nmaterialised against WRITE_PARTIAL_UPDATE_SCHEMA can be shorter than the ordinal, so the read\nraised ArrayIndexOutOfBoundsException from inside the key generator. Both Avro arms now check the\nlength first and fall through to the existing fallback. This is a bounds check, not a proof that\nthe field at the ordinal is a meta field: a long record whose schema is not meta-prefixed still\nreads data there, as before. The new test covers both short-record arms plus two meta-prefixed\nregression cases, without which the guard could disable meta-field resolution outright and the\nshort-record cases would still pass.\n\ngetRecordKey(GenericRecord) goes back to the getKey form. The mirror is the right end state but it\nis a second behaviour change, unrelated to the reported issue, and this PR is already gated on\nmerge order; it belongs in a follow-up with the case that discriminates it.\n\nOf the 15 cases, 5 fail on the parent commit and 10 pass there. The 10 pin behaviour rather than\nguarding a regression, so they are coverage, not proof of the fix.\n\n* Set the partition id property as a string so both lines compile\n\nThe Int literal boxes to Object on this base but not on the 1.x line, where scalac rejects the\nimplicit conversion. Both TypedProperties variants render the value through\nString.valueOf on read, so the string form is equivalent at runtime and compiles on either." }, { "commit": "e0fe585bbd4c5eab8a6bdbb93972e19fc454956d", "tree": "8e871e5cf685fa1c874e57e3f24d7cd3ed1d0569", "parents": [ "81fd4992b8dc0687b8ccfa6190d0570682e55ee1" ], "author": { "name": "Shihuan Liu", "email": "skywalker0618@gmail.com", "time": "Mon Aug 24 21:05:42 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Aug 25 12:05:42 2026 +0800" }, "message": "fix(hive-sync): call Driver.destroy() so HiveQL sync stops leaking Drivers into ShutdownHookManager (#19718)\n\n* fix(hive-sync): call Driver.destroy() so HiveQL sync stops leaking Drivers into ShutdownHookManager\n\n* Make the Driver hook removal survive a failing close()\n\nDriver.close() is not documented to be exception free, and its internal\nrelease steps only swallow exceptions individually, so a close() that does\nthrow would skip destroy() and leave exactly the hook we are trying to\nremove. Attempt destroy() independently of close() at all three sites.\n\ndestroy() reports rather than rethrows its own failure: it ends up in\nShutdownHookManager.removeShutdownHook, which throws once JVM shutdown has\nbegun, and at that point the hook set no longer matters. On the constructor\npath this also stops a failing teardown from masking the construction error." }, { "commit": "81fd4992b8dc0687b8ccfa6190d0570682e55ee1", "tree": "034803a3d14dd70a26d1e01b6dd1ee2c668dce7a", "parents": [ "8adb095386e34480cf6ac2bcb5289536551bbae8" ], "author": { "name": "fhan", "email": "aaron.han.1986@gmail.com", "time": "Tue Aug 25 10:45:15 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Aug 25 10:45:15 2026 +0800" }, "message": "fix(flink): prevent reusing diverged buffers on memory exhaustion (#19692)\n\n* fix(flink): prevent reusing diverged buffers on memory exhaustion\n\n---------\n\nCo-authored-by: fhan \u003cyfhanfei@jd.com\u003e" }, { "commit": "8adb095386e34480cf6ac2bcb5289536551bbae8", "tree": "d0af48149c0bea27fcc8391efca8ec34a681905c", "parents": [ "739ab7696979150304faf221e3344fe4949e6adc" ], "author": { "name": "Rahil C", "email": "32500120+rahil-c@users.noreply.github.com", "time": "Mon Aug 24 17:00:40 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 24 17:00:40 2026 -0700" }, "message": "chore(api): declare the unstructured ingestion SPIs evolving (#19701)\n\nDocumentParser and EmbeddingProvider are both user-implementable through\nparser.class and provider.class, but neither declared a stability level,\nunlike Source and Transformer alongside them. Mark both EVOLVING so the\nsignature changes they still need do not read as breaking ones." } ], "next": "739ab7696979150304faf221e3344fe4949e6adc" }