)]}' { "log": [ { "commit": "3483d602d3b6b24abca0d8fa8fdabcddaa63d3bc", "tree": "1e27cd34d892519abf4788ff6aafafc605fd145b", "parents": [ "db2c7da30d6ef76c195dac6248df2ed38cf59cbd" ], "author": { "name": "Sivabalan Narayanan", "email": "n.siva.b@gmail.com", "time": "Wed Oct 28 19:46:01 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 29 10:46:01 2026 +0800" }, "message": "fix(client): do not block the writer on a running async table service when switching write clients (#20444)\n\nAsyncCompactService and AsyncClusteringService held the same monitor for a\nwhole compaction/clustering run and for updateWriteClient, so switching the\nwrite client waited for the running run to complete. The caller is the\nwriter\u0027s driver thread (HoodieStreamer on write client re-creation, the Spark\nstreaming sink inside addBatch on a schema change), and a compaction or\nclustering can run for an hour or more, e.g. while waiting for cluster\nresources; ingestion must not stall for that long.\n\nRestore the non-blocking switch: later runs use the new client, and a running\nrun keeps the client it started with. Make the switched client fields\nvolatile, as the service thread reads them while the writer switches them." }, { "commit": "db2c7da30d6ef76c195dac6248df2ed38cf59cbd", "tree": "39888931a80750240d42f199350e44517366a5fd", "parents": [ "b3d45629581df9dfbbee5994ee438954e343825f" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Thu Oct 29 10:15:07 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 28 19:15:07 2026 -0700" }, "message": "feat(lakehouse): add a Helm-deployable Spark Connect endpoint (#20243)\n\nAdd a hudi-spark-connect Helm chart and image (Spark 4.1.3 with the Hudi\n bundle) that serve Hudi tables over Spark Connect. The server uses the \n lakehouse Hive Metastore and object storage with Hudi\u0027s SQL extensions\n and catalog, so PySpark Connect and ADBC clients can create, read and\n write Hudi tables with SQL, and Trino reads the same tables back.\n\n - Cluster-wide Hudi writer defaults through hudi.defaults\n (HUDI_CONF_DIR); table and session properties still take precedence.\n - Runs as a single local[N] driver by default. Kubernetes executors are\n opt-in and not yet validated; spark:// masters are rejected.\n - Local-dev quickstart deploys the endpoint; test-spark-connect.sh runs\n smoke, SQL matrix (MERGE, MoR compaction, partitioned tables, CALL,\n time travel, incremental), external-writer freshness, Trino readback\n and restart checks. Offline chart tests run in CI.\n - README documents observed disconnect, retry and cancel behavior,\n REFRESH TABLE after external schema changes, the Foundry ADBC 4 MiB\n receive limit, and that the endpoint has no authentication or TLS and\n must stay on a trusted network.\n\n Closes #20231." }, { "commit": "b3d45629581df9dfbbee5994ee438954e343825f", "tree": "5bc8657dd00fd9e48abc3ffe354fbd478326ebf3", "parents": [ "ec36124f5e54af94ffecbf7f94edefb134113cf3" ], "author": { "name": "Danny Chan", "email": "yuzhao.cyz@gmail.com", "time": "Thu Oct 29 04:30:41 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 28 13:30:41 2026 -0700" }, "message": "fix(spark)!: forward session settings to streaming sinks (#20455)\n\nStreaming sinks now honor the session\u0027s spark.hoodie.* settings, as\n batch DataFrame writes already do. Settings are captured once when the\n sink is created and drive multi-writer validation, checkpoint\n deduplication and every micro-batch write. Precedence: explicit writer\n options, then session spark.hoodie.*, then defaults. Bare hoodie.*\n session keys are not forwarded. Table identity is validated against an\n existing table when the sink is created, including on replay.\n\n Also fixes explicit spark.hoodie.* options being overridden by the\n session value in collectSparkHoodieConfs.\n\n BREAKING CHANGE: existing writeStream applications pick up ambient\n spark.hoodie.* settings (e.g. --conf spark.hoodie.datasource.write.operation).\n Unset them or pass explicit writer options to keep the previous behavior.\n\n Closes #20403." }, { "commit": "ec36124f5e54af94ffecbf7f94edefb134113cf3", "tree": "84373f3e8e9d687cf2baac71001042ee2939ab23", "parents": [ "d1ec4d3b043d19487b6b8a2ab4bb44665863bc5a" ], "author": { "name": "Danny Chan", "email": "yuzhao.cyz@gmail.com", "time": "Thu Oct 29 04:28:45 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 28 13:28:45 2026 -0700" }, "message": "fix(spark): fail streaming queries instead of exiting the JVM (#20452)\n\n When a micro-batch exhausts its write retries, HoodieStreamingSink now\n throws a HoodieException carrying the batch id and the last failure,\n instead of calling System.exit(1). Only the failing query stops; other\n queries and Spark Connect sessions on the same driver keep running, and\n the uncommitted batch replays on restart from the same checkpoint.\n\n Closes #20370." }, { "commit": "d1ec4d3b043d19487b6b8a2ab4bb44665863bc5a", "tree": "564e7c3765d01f59d39ca6636778952888244092", "parents": [ "82d724b93b552b822a968ba4cbc7fdf3360c4ccb" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Wed Oct 28 16:32:16 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 28 16:32:16 2026 +0800" }, "message": "docs(rfc): claim RFC-116 for branch and tag support (#20456)" }, { "commit": "82d724b93b552b822a968ba4cbc7fdf3360c4ccb", "tree": "99901a827c316e6a2071a7996a08e72054c44748", "parents": [ "f2e0b661441e6fada9f54d40e0055c4ebc81059e" ], "author": { "name": "fhan", "email": "aaron.han.1986@gmail.com", "time": "Wed Oct 28 10:12:40 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 28 10:12:40 2026 +0800" }, "message": "feat(agent): add Flink catalog and metastore composition to Hudi Architect (#20419)\n\n* feat(agent): add Flink catalog and metastore composition to Hudi Architect\n* feat(agent): Harden Flink catalog composition validation and fixtures\n\n---------\n\nCo-authored-by: fhan \u003cyfhanfei@jd.com\u003e" }, { "commit": "f2e0b661441e6fada9f54d40e0055c4ebc81059e", "tree": "c42ce1f8b6c6b84c2ecd9f77f774cc0717119242", "parents": [ "c0ed52483fb22e57c57a9c7b211b9bcb4919cf5b" ], "author": { "name": "Srikanth Mamidala", "email": "92472671+smtwilio@users.noreply.github.com", "time": "Tue Oct 27 14:13:09 2026 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 27 12:13:09 2026 -0700" }, "message": "feat(utilities): add fail-fast continuous mode for HoodieMultiTableStreamer (#19811)\n\nIn continuous mode each table\u0027s sync() blocks until shutdown, so the\nexisting sequential loop would only ever sync the first table. Sync the\ntables concurrently (one thread per table) when continuous mode is on,\nwhile keeping the sequential path for run-once mode.\n\nAdd a --fail-fast-on-continuous flag: when enabled, the first table\nfailure shuts down the sibling streamers and throws a HoodieException so\nthe job exits non-zero; when disabled (default), each table is synced\nindependently and a single failure does not affect the others.\n\nUse thread-safe collections for the success/failed table tracking now\nthat syncs run in parallel." }, { "commit": "c0ed52483fb22e57c57a9c7b211b9bcb4919cf5b", "tree": "492cbc567b1790e91fa789b276b21b1d9377b9b9", "parents": [ "f39f99713f9f8327d403947f4e38aad4259eb55a" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Tue Oct 27 23:26:34 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 27 15:26:34 2026 +0000" }, "message": "fix(spark): read shredded variant leaves like Spark 4 (#20332)\n\n* fix(spark): match Spark 4 shredded variant reads\n\nHudi rebuilt shredded variants from the footer\u0027s Avro conversion, which\ndrops DECIMAL on INT32/INT64, the unsigned flag, JSON and NANOS. Files\nwritten by other engines therefore read DECIMAL on INT32/INT64 as the\nunscaled integer, failed on plain FLBA and TIMESTAMP(MILLIS), and read\nTIME and TIMESTAMP(NANOS) as integers where Spark 4 rejects the file.\n\nRead each typed_value leaf from its Parquet type instead, as Spark 4\u0027s\nParquetToSparkSchemaConverter and SparkShreddingUtils.buildVariantSchema\ndo:\n- HoodieAvroParquetReader passes the file\u0027s MessageType to\n HoodieVariantReconstruction, which prepares one rebuilder per shredded\n variant column at file open (records, LIST encodings, map values).\n- VariantShreddingProvider.rebuildVariantRecord becomes prepareRebuild,\n so the layout is validated once per column instead of per record.\n- Spark4VariantShreddingProvider maps leaves like Spark (unsigned ints,\n JSON as string, millis scaled to micros, unscaled decimals) and\n rejects TIME, TIMESTAMP(NANOS), FLOAT16, UUID, INTERVAL and INT96 at\n open. Code made dead by the new path is removed.\n\nAddresses the read side of #19442.\n\n* docs(spark): cross-reference variant list-element rules\n\nThe planner and the Spark 4 provider each resolve a parquet LIST\nelement under a different converter\u0027s rules and on different\ngroups. Say so on both so a change to one points at the other." }, { "commit": "f39f99713f9f8327d403947f4e38aad4259eb55a", "tree": "ec00e50aa373cfb5fb95703a8c2b3821c107916a", "parents": [ "53d0b14389bddd3d753c38d7635c09c21043491d" ], "author": { "name": "Sivabalan Narayanan", "email": "n.siva.b@gmail.com", "time": "Mon Oct 26 22:39:30 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 27 13:39:30 2026 +0800" }, "message": "fix(spark): create Hudi tables via the DSv2 Column API and accept unchanged properties in HoodieCatalog (#20426)\n\n* fix(spark): create Hudi tables via the DSv2 Column API and accept unchanged properties in HoodieCatalog\n\nSpark 4.1\u0027s TableCatalog.createTable(Identifier, TableInfo), used by Spark\nDeclarative Pipelines, defaults to the Column[]-based createTable overload,\nwhich DelegatingCatalogExtension forwards to the session catalog: the table is\nregistered without hoodie.properties. Route that overload to HoodieCatalog\u0027s\nStructType-based createTable for Hudi tables. The Column API exists from Spark\n3.4 on, so the override lives in a trait with a Spark 3.4+ and an empty Spark\n3.3 version, added from a source directory chosen by the Spark profile.\n\nDeclarative Pipelines also re-sends every table property as SetProperty on\neach run, which HoodieCatalog.alterTable rejected. Accept SetProperty and\nRemoveProperty: changes that keep the current value are no-ops, changes to\nproperties persisted in hoodie.properties are rejected, and other properties\nare applied to the catalog table.\n\n* fix(spark): validate and preserve catalog property changes\n\n---------\n\nCo-authored-by: danny0405 \u003cyuzhao.cyz@gmail.com\u003e" }, { "commit": "53d0b14389bddd3d753c38d7635c09c21043491d", "tree": "787b500c98963c2915dffb10acf6ba3c569ce5a7", "parents": [ "6ac51eb518073d6a6fdccc200d38ffed45755925" ], "author": { "name": "Sivabalan Narayanan", "email": "n.siva.b@gmail.com", "time": "Mon Oct 26 19:05:05 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 27 10:05:05 2026 +0800" }, "message": "fix(spark): re-create the streaming sink write client when the writer schema changes (#20415)\n\n* fix(spark): re-create the streaming sink write client when the writer schema changes\n\nThe streaming sink reuses one write client across micro-batches, so the\nclient\u0027s write config keeps the schema of the micro-batch that created it.\nWhen the writer schema changes under a running query (the table schema\nevolves and set-null-for-missing-columns / reconcile is on, a type is\nwidened, or a delete stream sees a new table schema), later writes and their\ncommit metadata would use the stale schema.\n\nLike HoodieStreamer, re-create the write client when the schema it was created\nwith no longer matches the write\u0027s schema, for upserts, deletes and delete\npartitions. The comparison uses a snapshot taken at client creation rather\nthan the live write config (compaction updates its schema) and ignores the\ninternal schema version id. As in HoodieStreamer, the sink hands the new\nclient to the async table services and closes the old one right away; the\nsink-owned timeline server keeps running across the swap.\n\n* fix(client): hand off async write clients between completed runs\n\n---------\n\nCo-authored-by: danny0405 \u003cyuzhao.cyz@gmail.com\u003e" }, { "commit": "6ac51eb518073d6a6fdccc200d38ffed45755925", "tree": "baf2c224ae79224e2ed78ffe05183e0ae6d94c39", "parents": [ "e4c9512bee94096dd2bcc141ba34436676fe3a0d" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Mon Oct 26 14:40:00 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 26 14:40:00 2026 -0700" }, "message": "fix(core): write the key generator class for tables below version 8 (#20276)" }, { "commit": "e4c9512bee94096dd2bcc141ba34436676fe3a0d", "tree": "d5f6a97928ce3c051b8a6bf4c2602c1fa1a5eb6f", "parents": [ "9a43cc1fd322786d94b758298a35a4468367670e" ], "author": { "name": "Sivabalan Narayanan", "email": "n.siva.b@gmail.com", "time": "Mon Oct 26 03:18:04 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 26 18:18:04 2026 +0800" }, "message": "feat(spark): stop forcing direct markers in the streaming sink (#20413)\n\n* feat(spark): stop forcing direct markers in the streaming sink\n\nThe sink forced DIRECT markers because the embedded timeline server used to be\nstopped after the first micro-batch. The sink now keeps one timeline server for\nthe whole query, so streaming writes can use the configured marker type\n(timeline-server-based by default; WriteMarkersFactory still falls back to\ndirect markers when the timeline server is disabled).\n\n* Address review: update marker type docs and test the streaming marker path\n\n- Drop the MarkerType.TIMELINE_SERVER_BASED note that structured streaming\n always uses direct markers.\n- Assert that a micro-batch failing mid-write leaves timeline-server-based\n markers, so a reintroduced override is caught.\n\n* Address review: describe the remote-view exception for timeline-server markers" }, { "commit": "9a43cc1fd322786d94b758298a35a4468367670e", "tree": "5bcdebec9568f23b9da86505bbaafbc499c22b29", "parents": [ "2f835a59e625d042ff35efda4ed7ec13d835b0d0" ], "author": { "name": "Sivabalan Narayanan", "email": "n.siva.b@gmail.com", "time": "Mon Oct 26 00:33:11 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 26 15:33:11 2026 +0800" }, "message": "fix(spark): reuse one embedded timeline server across streaming sink micro-batches (#20412)\n\n* fix(spark): reuse one embedded timeline server across streaming sink micro-batches\n\nHoodieStreamingSink now owns a single embedded timeline server per streaming\nquery, hands it to the write client (non-owning, as StreamSync does), keeps the\nclient open across micro-batches, and releases both when the query terminates\nvia a StreamingQueryListener. Drops the HUDI-3636 override that disabled the\ntimeline server for streaming writes.\n\nFixes #20400\n\n* Address review: release failed-write heartbeats and harden sink cleanup\n\n- Stop the heartbeat of a failed write\u0027s instant when the caller owns the\n write client, so the instant can be rolled back under the lazy\n failed-writes policy while the client is reused.\n- Match the termination listener on the run id too, and release resources on\n a separate thread with a graceful shutdown of async table services.\n- Proceed without the embedded timeline server if it fails to start, as a\n client starting its own server does.\n\n* Fix streaming sink hang between reset and async service shutdown callbacks\n\nreset() waited for the async compaction/clustering services while holding the\nsink lock, and the services\u0027 shutdown callbacks called reset() and blocked on\nthat lock, so the service executor never terminated. This hung the query when\na micro-batch exhausted its retries with an async service running (also on\nmaster), and the graceful release on query termination.\n\nreset() now detaches a service before stopping it; a callback does nothing for\na detached service and resets the sink on a separate thread when the service\nstopped on its own.\n\n* Address review: release failed-instant heartbeats on all write paths\n\n- Record the instant started by each write (upsert/insert, delete, delete\n partition, bulk insert as row) and stop its heartbeat on any failure when\n the caller owns the write client.\n- Register the sink\u0027s termination listener before acquiring the timeline\n server, so it is released even if the client fails to build.\n- Test the reset/async-callback deadlock by driving reset directly with an\n async service running, and add a failing delete case to the heartbeat tests.\n\n* fix(client): stop the heartbeat when commit startup fails\n\n---------\n\nCo-authored-by: danny0405 \u003cyuzhao.cyz@gmail.com\u003e" }, { "commit": "2f835a59e625d042ff35efda4ed7ec13d835b0d0", "tree": "77f6180251f361523585069283e0448a27e21fb8", "parents": [ "74c6125a04d923985a94b4e45c1d6343855bc8ee" ], "author": { "name": "Sivabalan Narayanan", "email": "n.siva.b@gmail.com", "time": "Sun Oct 25 22:16:50 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Oct 25 22:16:50 2026 -0700" }, "message": "feat(agent): add Hudi optimizer catalogs and Agent (#20197)\n\nA recommendation Agent under hudi-ai-operator/skills/hudi-optimizer for a deployment that works\n but costs more than it should. It takes writer properties, a Spark event log and read access to\n the table, runs the health checker, layout analyzer and event log analyzer itself, and reports\n an observation only when at least two inputs agree, stating confidence from how many did.\n\n observation-catalog.md: thirty \"works, but could be better\" rows built on derived expectations,\n configured intent against measured reality, each naming the JSON fields that corroborate it,\n the config key and direction, a risk tier and a verification step. capability-catalog.md: seven\n \"adopt this feature\" rows with signals for and against, sizing and adoption cost. Durable\n changes such as index type, table type or key generator are refused and handed to a design\n decision rather than recommended as tuning. Tools run through spark-submit; exit codes and the\n analyzers\u0027 attribution caveats, including the simple index having no sub-phase, are documented.\n Part of #19263." }, { "commit": "74c6125a04d923985a94b4e45c1d6343855bc8ee", "tree": "23bb45758e1f883b78877d79c51adb2b5530d31e", "parents": [ "c010ad7a8764a6c2413bbf47298968340df4f04b" ], "author": { "name": "Sivabalan Narayanan", "email": "n.siva.b@gmail.com", "time": "Sun Oct 25 22:15:18 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Oct 25 22:15:18 2026 -0700" }, "message": "feat(agent): add Hudi config catalog generator and config-consultant skill (#20195)\n\n* feat(agent): add Hudi config catalog generator and config-consultant skill\n\nscripts/generate_config_catalog.py parses every hoodie.* ConfigProperty in the Java source, the\n Scala datasource options and the Flink ConfigOptions that name a hoodie.* key, and emits\n config-catalog.json pinned to a commit: key, type, default, documentation, valid values,\n alternatives, since and deprecated versions, the advanced flag, the declaring file and line,\n and three things the published docs cannot give: where the value is read (file:line), the\n conditions that gate each read, and what is read alongside it. Engine-effective defaults are\n resolved from the builder overrides, so a Spark default is reported as Spark gets it, not as\n declared. Configs with no consumer outside their own class are flagged.\n\n hudi-ai-operator/skills/hudi-config-consultant answers from that catalog and never from memory:\n it states the catalog\u0027s version, leads with the consequence rather than the condition, labels\n claim strength (enforced, warn-only, co-occurrence), works thresholds through in the user\u0027s\n units, and refuses to invent a key. Scope is the hoodie.* namespace; Flink\u0027s unprefixed\n FlinkOptions keys are out of scope and the skill says so. Part of #19262." }, { "commit": "c010ad7a8764a6c2413bbf47298968340df4f04b", "tree": "59cc533d598952ddfe8d79a93c6e5e728ad91417", "parents": [ "62bae026d55fa926d66dade6b5f4f44ebdb43d78" ], "author": { "name": "fhan", "email": "aaron.han.1986@gmail.com", "time": "Mon Oct 26 11:38:49 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 26 11:38:49 2026 +0800" }, "message": "fix(agent): treat existing-table identity mismatch as evidence conflict (#20410)\n\nCo-authored-by: fhan \u003cyfhanfei@jd.com\u003e" }, { "commit": "62bae026d55fa926d66dade6b5f4f44ebdb43d78", "tree": "35386bc7ed7769b96d2143dbcf7f0c28eb7a4794", "parents": [ "67b39465f405280bf785f95818388d8347884704" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Sat Oct 24 11:28:39 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Oct 24 11:28:39 2026 +0800" }, "message": "chore(ci): show pr_compliance.py self-test failures (#20422)\n\n* chore(ci): show pr_compliance.py self-test failures\n\nThe failure path echoed the whole test log behind one ::error::, so the\nannotation carried only the first log line (a PASSED test) and the\nrest landed in the step log unlabelled. Print the log plainly and put\nthe names of the failed tests in the annotation.\n\nFollow-up to review on #20421.\n\n* chore(ci): split failed self-test names on semicolons\n\nReview feedback. Test names such as \"Swapped sections: 0, 1\" contain\ncommas, so join them with \"; \". A crash before any test reports\nlists no names; the annotation then points at the step log.\n\n* chore(ci): name the offending commit or file in annotations\n\nReview feedback: the co-author, binary, copyright and RAT steps\nannotated only \"Process completed with exit code 1\", so the run\nsummary did not say what to fix. Each now emits an ::error:: naming\nthe commit or file; the source-check ones are file annotations, so\nthey also show inline in the PR diff. The release scripts are\nunchanged, since release managers run them too." }, { "commit": "67b39465f405280bf785f95818388d8347884704", "tree": "88c8c9b5794a11cc2d31d7f04fd48697f3b61b14", "parents": [ "00a49c5f10d172b44a3db51916806a3f64530ba4" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Fri Oct 23 16:48:31 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 23 16:48:31 2026 -0700" }, "message": "chore(ci): run bundle validation on Spark 3.5.9 instead of 3.5.1 (#20428)" }, { "commit": "00a49c5f10d172b44a3db51916806a3f64530ba4", "tree": "e31ee50ab5b94c2c917778cfbe5c6a33595b54c4", "parents": [ "97bdd5b6212853c37b421822a7739b1bbdab625f" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Fri Oct 23 08:04:41 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 23 08:04:41 2026 -0700" }, "message": "chore(ci): build the bundle validation base image on glibc (#20417)" }, { "commit": "97bdd5b6212853c37b421822a7739b1bbdab625f", "tree": "969df51e80ce034e416f61e39b694d6315a3bccc", "parents": [ "1c54bd382ad9b37a004ba7ec785ca3ebc3c70a84" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Thu Oct 22 22:17:54 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 22 22:17:54 2026 -0700" }, "message": "feat(spark): add hudi-spark-native-bundle packaging Apache DataFusion Comet (#19674)" }, { "commit": "1c54bd382ad9b37a004ba7ec785ca3ebc3c70a84", "tree": "cd4cc06a55e0f00cc5452e6efcabcb204b2be4d6", "parents": [ "2aae125aeac807be0c3c792c426cabab15207eb9" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Fri Oct 23 10:56:31 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 22 19:56:31 2026 -0700" }, "message": "chore(ci): run all validation in one validate-pr-and-source job (#20421)" }, { "commit": "2aae125aeac807be0c3c792c426cabab15207eb9", "tree": "f64c80cecfbc7b057d875297bb84cf277c926b0c", "parents": [ "617be49f0ab5497174dcab098e258c648491f4ee" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Fri Oct 23 10:13:18 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 22 19:13:18 2026 -0700" }, "message": "chore(ci): require only the validate-pr-and-source check (#20420)" }, { "commit": "617be49f0ab5497174dcab098e258c648491f4ee", "tree": "52d9909a15e3bf990416c21a0e918476462b8823", "parents": [ "59ad99f4402f570a89eb0bbe0270d1fc1b5f8cd0" ], "author": { "name": "Shihuan Liu", "email": "skywalker0618@gmail.com", "time": "Thu Oct 22 18:58:40 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 23 09:58:40 2026 +0800" }, "message": "fix(flink): apply the nested array reader fix to Flink 1.19 and later (#20399)\n\nPort #20337 from hudi-flink1.18.x to hudi-flink1.19.x, 1.20.x, 2.0.x, 2.1.x and 2.2.x,\nwhich have the same COW Parquet reader code:\n\n- createWritableColumnVector unwraps the Parquet LIST element of an ARRAY with the same\n backward-compatibility rules as getArrayElementColumn, now shared through\n isThreeLevelListRepeatedType, so reading array\u003crow\u003c..., array\u003crow\u003c...\u003e\u003e\u003e\u003e no longer fails\n with a NullPointerException in NestedColumnReader.readRow, and arrays of rows read their\n field values instead of nulls.\n- INT32/INT64-encoded decimals are allocated with int/long vectors.\n- NestedColumnReader grows the null placeholders of absent fields to the row\u0027s position\n count, and NestedPositionUtil.calculateRowOffsets returns the row\u0027s own position count\n when it has no nulls.\n\nThe change and its tests are the same as in hudi-flink1.18.x. The 2.1.x and 2.2.x modules\nalso support VARIANT, which the change leaves as is." }, { "commit": "59ad99f4402f570a89eb0bbe0270d1fc1b5f8cd0", "tree": "67d01c315ab315f7cdea55aa980587c190edd18d", "parents": [ "42913d08459c53edad0aea674234358bed6be9d3" ], "author": { "name": "Shihuan Liu", "email": "skywalker0618@gmail.com", "time": "Thu Oct 22 18:57:32 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 23 09:57:32 2026 +0800" }, "message": "feat(flink): report the steps of a Hive sync in the write coordinator metrics (#20165)\n\n* feat(flink): report the steps of a Hive sync in the write coordinator metrics\n\nThe Hive sync metrics on the write coordinator say whether a sync\nsucceeded and how long it took, but not where that time went or what\nthe sync changed. HiveSyncTool now records per sync what it did, in a\nHiveSyncStats that callers read through getSyncStats(): time reading\nthe schema from storage, time finding partitions to sync, the remaining\nmetastore time, partitions added, and whether the schema evolved.\n\nThe coordinator reports them next to the existing metrics:\n\n- hiveSyncInitDurationMs (histogram, building the sync tool)\n- hiveSyncSchemaReadDurationMs (histogram)\n- hiveSyncPartitionScanDurationMs (histogram)\n- hiveSyncMetastoreDurationMs (histogram)\n- hiveSyncPartitionsAddedCount (counter)\n- hiveSyncSchemaEvolvedCount (counter)\n\n* Record sync stats in doSync() so GlobalHiveSyncTool reports them too\n\nGlobalHiveSyncTool overrides syncHoodieTable() with a bare doSync(), so it\nskipped the stats reset and the total timer. Also explain why\nrecordPartitionsAdded keeps the max rather than the sum.\n\n* Explain why the sync stats are created in two places\n\n* Initialize the sync duration histogram like the other histograms\n\n* Rename the metastore time to the remaining time\n\nIt covers the local work between the metastore calls too, such as\ncomparing the partitions and schemas found, so hiveSyncMetastoreDurationMs\nand getMetastoreMs() become hiveSyncRemainingDurationMs and\ngetRemainingMs().\n\n* Count the partitions added batch by batch\n\nThe partitions were counted only after every batch was added, so when a\nlater batch failed, the batches already added were never counted: the\nretry does not add them again. HoodieMetaSyncOperations and DDLExecutor\nget an addPartitionsToTable overload that reports each batch as the\nmetastore takes it, which the HMS, HiveQL and JDBC executors implement,\nincluding on the HiveQL driver pool.\n\n* Record the init time when building the sync tool fails\n\nBuilding it fails when the metastore cannot be reached, which is when\nthe time spent there matters most.\n\n* Return the sync tool after recording its init time\n\n* Drop the overloads that only passed a no-op callback\n\nrunSQLs, HiveDriverPool.dispatchAll and DDLExecutor.addPartitionsToTable\neach keep one method taking the callback, and callers that do not count\npass a no-op. HoodieMetaSyncOperations keeps both addPartitionsToTable\nmethods, since sync clients outside hudi-hive-sync implement the one\nwithout a callback. Also rename the callback to onStatementSucceeded,\nsince it is only called for statements that succeed.\n\n* Say that the last batch of partitions can be smaller\n\n* Add an option to enable the Hive sync stats, off by default\n\n* perf(flink): skip disabled Hive sync histograms and rename stats\n\n---------\n\nCo-authored-by: danny0405 \u003cyuzhao.cyz@gmail.com\u003e" }, { "commit": "42913d08459c53edad0aea674234358bed6be9d3", "tree": "d7d3c41c5204274bf651581ae109b65380a02a98", "parents": [ "1d5462b8eea1f645696193059e9a6610eac2366e" ], "author": { "name": "Raghuramchadalawada", "email": "66631155+Raghuramchadalawada@users.noreply.github.com", "time": "Thu Oct 22 18:44:07 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 23 08:44:07 2026 +0700" }, "message": "fix(utilities): Re-add _corrupt_record when a transformer drops it in ErrorTableAwareChainedTransformer (#20336)\n\n* fix(utilities): Re-add _corrupt_record when a transformer drops it in ErrorTableAwareChainedTransformer\n\nCustom transformers that project columns drop _corrupt_record because they\nare unaware of the error-table contract, and ErrorTableAwareChainedTransformer\nfailed the pipeline with HoodieValidationException after any such transformer.\n\nRe-add the column as null when a transformer drops it while it holds no\nvalues. It is known to be all null until the source supplies it or a\ntransformer keeps it; otherwise one Spark job checks the transformer input.\nIf the dropped column held values, still fail, since those error records\ncannot be realigned with the transformer output and re-adding null would\nwrite them to the target table.\n\nCloses #18915\n\n* Simplify ErrorTableAwareChainedTransformer loop and extract dropped-column validation\n\n* Simplify corrupt record flag handling and extract test helpers in ErrorTableAwareChainedTransformer" }, { "commit": "1d5462b8eea1f645696193059e9a6610eac2366e", "tree": "6ada4a00395669cb4cda3252b8f227b0055a8bef", "parents": [ "2b6d8b2d6859ffa2ddf6f34b516ebcb239a6a25e" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Fri Oct 23 08:38:15 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 22 17:38:15 2026 -0700" }, "message": "chore(ci): pin ubuntu-latest runners to ubuntu-24.04 (#20418)" }, { "commit": "2b6d8b2d6859ffa2ddf6f34b516ebcb239a6a25e", "tree": "036228142f4b773f3a3e10a14e295e0279d91494", "parents": [ "a974f447089eaea3d9366ee495e88d5b5a349d58" ], "author": { "name": "Sivabalan Narayanan", "email": "n.siva.b@gmail.com", "time": "Thu Oct 22 08:31:17 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 22 15:31:17 2026 +0000" }, "message": "refactor(agent): move the Hudi Architect skill under hudi-ai-operator (#20223)\n\nThe agent gateway serves people consuming data from Hudi tables: Trino\nSQL tools, a chat UI, an MCP endpoint for querying. The Architect skill\nserves a different audience — people designing, deploying and operating\nthe tables themselves — and sat under the gateway only because that was\nthe first place skills lived.\n\nMoves it to a new top-level hudi-ai-operator/skills/, which is where\noperator-facing tooling will live. No content change beyond path\nreferences: the skill, its references and its three validation scripts\nare unchanged." }, { "commit": "a974f447089eaea3d9366ee495e88d5b5a349d58", "tree": "1a69b260066df623fc9345af453f3d91ae3742e4", "parents": [ "fe8627a8f42c08c42784de5a0461ef642e5d878f" ], "author": { "name": "Sivabalan Narayanan", "email": "n.siva.b@gmail.com", "time": "Thu Oct 22 08:16:57 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 22 15:16:57 2026 +0000" }, "message": "feat(utilities): add record-level index sizing check to HoodieTableHealthChecker (#20153)\n\n* feat(utilities): add record-level index sizing check to HoodieTableHealthChecker\n\nAdds RecordIndexSizingCheck (--checks record-index). The record index\u0027s file-group count is\n fixed at initialization, so the check re-runs the writer\u0027s own sizing estimate against the index\n as it is now and compares the result with the file groups it has, globally or per data partition\n for a partitioned index. It also flags any file group past the configured maximum size.\n\n The record estimate is taken from base-file bytes, falling back to log bytes before the first\n metadata compaction. A floor-bound ideal (min \u003d\u003d max, or an index below one file group\u0027s worth of\n records) is reported as such and never called undersized. The worst data partition is the one\n with a real shortfall, not the lowest ratio. The report echoes the effective sizing, the\n estimate basis and the ideal\u0027s source so a verdict can be audited.\n\n The remedy is a re-bootstrap with corrected sizing; for a partitioned index the check says why\n HoodieIndexer cannot do that today (#20343) rather than naming a command that would build the\n wrong index. Part of #19262." }, { "commit": "fe8627a8f42c08c42784de5a0461ef642e5d878f", "tree": "a7e9fa646d5c61e3cfa656f3603eee6baf26cdca", "parents": [ "5cce63e83c482e8c0dcd686919d00a1527b5ec9e" ], "author": { "name": "Sivabalan Narayanan", "email": "n.siva.b@gmail.com", "time": "Thu Oct 22 02:00:46 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 22 02:00:46 2026 -0700" }, "message": "docs(agent): correct the HoodieTableHealthChecker exit codes in the hudi-table-health skill (#20385)\n\n* docs(agent): correct the HoodieTableHealthChecker exit codes documented in the hudi-table-health skill\n\nThe skill documented the exit codes as 0 healthy, 1 unhealthy, 2 could not run. The tool returns\n 0 (every check healthy or skipped), 2 (bad arguments or an unreadable table), 3 (at least one\n check UNHEALTHY) and 4 (no check unhealthy but at least one could not run), and never uses 1 for\n a verdict, since the JVM exits 1 on start-up failure. A rejected command line also exits 1\n today, because the argument parser runs before the tool\u0027s own error handling, so 1 always means\n \"the tool did not run\", never a finding about the table.\n \n Follow-up to #20151." }, { "commit": "5cce63e83c482e8c0dcd686919d00a1527b5ec9e", "tree": "f968da69a7ee8f7ce8ed7c8ae9da3992d0b90d23", "parents": [ "0f9f6b75464659540069e114a78534b382586be0" ], "author": { "name": "Sivabalan Narayanan", "email": "n.siva.b@gmail.com", "time": "Thu Oct 22 01:53:36 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 22 01:53:36 2026 -0700" }, "message": "feat(utilities): add metadata table sync and compaction lag checks to HoodieTableHealthChecker (#20154)\n\n* feat(utilities): add metadata table sync and compaction lag checks with hudi-mdt-health skill\n\nTwo new checks, mdt-sync and mdt-compaction, plus the hudi-mdt-health skill that interprets them.\n\n mdt-sync compares the metadata table\u0027s file listing against storage for the latest file slices\n and reports which direction they diverge, since a rebuild only fixes one of the two. It\n recommends the validator with a scheme-qualified base path.\n\n mdt-compaction measures delta commits since the metadata table\u0027s last compaction against the\n writer\u0027s configured trigger, and reports a metadata table that nothing maintains instead of\n telling the operator to wait out the lag.\n\n Both report SKIPPED, naming the property, when the governing writer configuration is not\n supplied. Part of #19262." }, { "commit": "0f9f6b75464659540069e114a78534b382586be0", "tree": "6078cf1404141169415ba7fdb870a90e15c4a65e", "parents": [ "bb42885118659a2598e2ce88a453bd4de7d44a14" ], "author": { "name": "Sivabalan Narayanan", "email": "n.siva.b@gmail.com", "time": "Thu Oct 22 01:53:19 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 22 01:53:19 2026 -0700" }, "message": "feat(agent): add Hudi RCA skill with failure catalog, mitigation ladders and workflow catalog (#20194)\n\nA diagnostic skill under hudi-ai-operator/skills/hudi-rca that classifies a failure on the\n innermost cause and first Hudi frame against 39 catalogued patterns, states which evidence rung\n it reasons from, and gives an ordered mitigation ladder with counter-intuitive steps flagged.\n\n References: failure-catalog.md (signatures verified per release line by replay, with\n disambiguation for ambiguous exceptions), mitigation-ladders.md (14 procedures, including the\n file-not-found triage and the safe-deletion protocol), workflow-catalog.md (write-operation\n phases and a Spark-stage-to-Hudi-phase table) and test-corpus.md (public-issue regression cases).\n\n Replies are two-stage: a short family-level triage at rung 0, the full report once the evidence\n supports it. The skill never mutates a table and never scripts a deletion. Part of #19262." }, { "commit": "bb42885118659a2598e2ce88a453bd4de7d44a14", "tree": "091bb9ba1d6c5d47f198cdd612389e7a382c4615", "parents": [ "8c4a8360aad614d60287797372cc53eb6205efb0" ], "author": { "name": "Sivabalan Narayanan", "email": "n.siva.b@gmail.com", "time": "Thu Oct 22 01:52:59 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 22 01:52:59 2026 -0700" }, "message": "feat(utilities): add HoodieSparkEventLogAnalyzer for Hudi-aware Spark stage analysis (#20196)\n\n* feat(utilities): add HoodieSparkEventLogAnalyzer for Hudi-aware Spark stage analysis\n\nParses a Spark event log and attributes each stage to the Hudi operation and phase that\n produced it, using the job descriptions Hudi sets on the Spark context. Reports per-operation\n and per-phase shares of stage wall clock, per-stage task, shuffle, spill and skew metrics, and\n the caveats a reader needs: a phase names where work was forced rather than where it belongs,\n dedup and tagging share one bucket, the simple index has no sub-phase of its own, and the\n streaming metadata write leaves a share of stages unattributable.\n\n Read-only; no Spark session, but it needs Spark\u0027s jars on the classpath, so it runs via\n spark-submit. Part of #19262." }, { "commit": "8c4a8360aad614d60287797372cc53eb6205efb0", "tree": "f9126b684a8fbee68e370a45339ba9818dc6d4d5", "parents": [ "483bf7187c476ba894cc60ef272de2267d65e864" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Thu Oct 22 14:35:32 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 22 13:35:32 2026 +0700" }, "message": "chore(ci): flatten the Java CI job graph (#19890)\n\nPart of #19524.\n\nRemove the changes job and every needs and step-level if that\ndepended on it; the docs-only skip is a trigger-level paths filter,\nevaluated before any job exists. validate-source and\nvalidate-ci-baseline move verbatim to validate_source.yml, which is\nnever path filtered because both are required status checks. The\nFlink, bundle-validation and integration-test jobs move verbatim to\njava_ci_engines.yml so each workflow stays under the ASF target of\n15 concurrent jobs. The concurrency group carries the workflow name\nso the three workflows cancel only their own superseded runs.\n\nReview follow-ups: .github/** added to the positive paths list, a\nnote in .asf.yaml on why only always-on workflows may contribute\nrequired contexts, and a shorter Codecov range comment." }, { "commit": "483bf7187c476ba894cc60ef272de2267d65e864", "tree": "37d26beaef1ae1384bf77874966b31e58d5e5e2c", "parents": [ "31f4a5ef2de3023ef3374a42133d6679b06747e5" ], "author": { "name": "fhan", "email": "aaron.han.1986@gmail.com", "time": "Thu Oct 22 10:15:41 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 22 10:15:41 2026 +0800" }, "message": "feat(agent): add existing-table reconciliation to Hudi Architect (#20391)\n\n* feat: add existing-table reconciliation to Hudi Architec\n* fix: tighten existing-table evidence and layout classification\n\n---------\n\nCo-authored-by: fhan \u003cyfhanfei@jd.com\u003e" }, { "commit": "31f4a5ef2de3023ef3374a42133d6679b06747e5", "tree": "face18cba0b5946e874c0a4c6aea8839e6e03db4", "parents": [ "59561a8d8c1fa28ad7fb3da632c0779d9417053d" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Wed Oct 21 09:56:39 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 21 10:56:39 2026 +0800" }, "message": "fix(common): write ordering values to native delete logs in their stored form (#20373)\n\n* fix(common): write ordering values to native delete logs in their stored form\n\n---------\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e" }, { "commit": "59561a8d8c1fa28ad7fb3da632c0779d9417053d", "tree": "3269e36eaaef1e93da73ee5f65e98cdf280d8d7e", "parents": [ "5b8323f88891be103668ce098a0aafcf38e0b208" ], "author": { "name": "databird07", "email": "databird01@gmail.com", "time": "Tue Oct 20 19:41:55 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 21 10:41:55 2026 +0800" }, "message": "test(common): cover array fields in parquet-avro GenericData reader test (#20190)\n\nSwitching Hudi\u0027s parquet-avro readers to the GenericData model changes the\nruntime type of Avro array fields: SpecificData resolves an ARRAY schema to\nList.class, so parquet-avro materializes java.util.ArrayList, while\nGenericData materializes org.apache.avro.generic.GenericData.Array. Both\nimplement java.util.List, and Hudi\u0027s array consumers (HoodieAvroUtils,\nConvertingGenericData, HoodieRealtimeRecordReaderUtils and the Spark\nAvroDeserializer) only rely on Collection/List.\n\nAdd an array\u003cstring\u003e field to TestAvroStringableClassResolution and assert,\nfor both the HoodieAvroParquetReaderBuilder path and the ParquetUtils path,\nthat it is read back as a java.util.List with the original elements." }, { "commit": "5b8323f88891be103668ce098a0aafcf38e0b208", "tree": "3d76887400eb6546d7438140e4c119b65a62147f", "parents": [ "b5b05d1cafdda4549ca15b9d068b847033097c4f" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Wed Oct 21 09:40:41 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 21 10:40:41 2026 +0800" }, "message": "fix(spark): resolve nested ordering fields for Spark deletes written to delete blocks (#20368)\n\n* fix(spark): resolve nested ordering fields for Spark deletes written to delete blocks\n\n* addressed review comments: drop the repeated schema and path lookups in getOrderingValueAsJava, rename the test\n\n* addressed review comments: add a NestedFieldPath overload of getFieldValueFromInternalRowInternal\n\n---------\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e" }, { "commit": "b5b05d1cafdda4549ca15b9d068b847033097c4f", "tree": "ad5904b513cfa80e5309de7d49825fec8cd776a0", "parents": [ "a0313d1983844332d376c76daad7aedf6a513122" ], "author": { "name": "Shihuan Liu", "email": "skywalker0618@gmail.com", "time": "Tue Oct 20 19:35:05 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 21 10:35:05 2026 +0800" }, "message": "fix(flink): unwrap the list element type when creating vectors for nested arrays (#20337)\n\n* fix(flink): unwrap the list element type when creating vectors for nested arrays\n\ncreateWritableColumnVector passed the LIST group itself as the physical type of an\narray\u0027s element. For an array of rows the element ROW then looked its fields up by\nname in the LIST wrapper, found none, and filled every slot with a constant\nplaceholder vector. Reading array\u003crow\u003c..., array\u003crow\u003c...\u003e\u003e\u003e\u003e then failed with a\nNullPointerException in NestedColumnReader.readRow, because the placeholder for the\ninner array has no child vector.\n\nLocate the element type with the same Parquet LIST backward-compatibility rules as\ngetArrayElementColumn (3-level, 2-level \"array\" / \"\u003cname\u003e_tuple\", multi-field and\nprimitive repeated types), so the vector tree lines up with the ParquetField tree.\nRow fields are still resolved by name, preserving schema evolution.\n\nInside an array, a row has one position per element, which can exceed the batch size\nthat the null placeholders of fields absent from the file were sized with. Grow those\nplaceholders to the position count in NestedColumnReader.readRow, so that reading an\nevolved array of rows doesn\u0027t fail with an ArrayIndexOutOfBoundsException.\n\n* Fix review comments\n\n* Address review comments: row position count, integer-encoded decimals, _tuple list elements\n\n* Share the three-level list check between the vector and ParquetField paths" }, { "commit": "a0313d1983844332d376c76daad7aedf6a513122", "tree": "265a0eb0c871641f4347ffba8150410d35966942", "parents": [ "a6b8744d7fb26c71fd22cb8982d32fdf1b990109" ], "author": { "name": "Sivabalan Narayanan", "email": "n.siva.b@gmail.com", "time": "Tue Oct 20 17:57:59 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 21 00:57:59 2026 +0000" }, "message": "feat(utilities): add HoodieTableLayoutAnalyzer for small-file, micro-partition and hot-partition detection (#20152)\n\nThe micro-partition size rule flagged any partition with 25+ base files\n averaging under 50 MB and recommended clustering. That is the small-file\n detector\u0027s signal and remedy, so the micro verdict added nothing.\n\n Replace it with a tiny-partition rule on total bytes per partition\n (--micro-partition-max-partition-bytes, 50 MB), judged by prevalence\n (--micro-partition-tiny-pct, default half of examined partitions) so a\n still-filling newest partition cannot flag the table alone. A partition\n that small can never fill a well-sized file however it is clustered, so\n the finding now recommends a coarser partitioning scheme. Count rule and\n age gate are unchanged; single-partition tables skip the rule." }, { "commit": "a6b8744d7fb26c71fd22cb8982d32fdf1b990109", "tree": "c08324a95c671a4e5b9fc205983b9199007ac68c", "parents": [ "6aeb9a27ae989f3f8b955447e2db8bd3acaabc1a" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Tue Oct 20 12:04:53 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 20 13:04:53 2026 +0800" }, "message": "fix(flink): defer instant creation until earlier checkpoints finish (#20389)\n\n* fix(flink): wait for earlier instants before starting a new one in blocking mode\n* fix(flink): rely on task polling for pending instant creation\n* fix(flink): defer instant creation only for earlier checkpoints\n\n---------\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e\nCo-authored-by: danny0405 \u003cyuzhao.cyz@gmail.com\u003e" }, { "commit": "6aeb9a27ae989f3f8b955447e2db8bd3acaabc1a", "tree": "0d93b595131bba54027f9e8a676b6f7425a534ec", "parents": [ "d62850ee908acf00877d21ea12d0207e126e00f5" ], "author": { "name": "Danny Chan", "email": "yuzhao.cyz@gmail.com", "time": "Tue Oct 20 09:55:45 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 20 09:55:45 2026 +0800" }, "message": "fix: escape partition values and locations in Hive sync DDL (#20390)\n\n* fix: escape partition values and locations in Hive sync DDL\n* fix: escape Hive CREATE TABLE base locations" }, { "commit": "d62850ee908acf00877d21ea12d0207e126e00f5", "tree": "5df444044d290da4601d0b8c5a6ebdda9a6ea964", "parents": [ "7ad42f1c7ad670a16683d771f946dd1afb0ae10f" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Mon Oct 19 21:50:45 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 19 21:50:45 2026 +0800" }, "message": "chore(ci): path-filter Trino CI on pull requests (#20376)\n\nEvery PR started detect-trino-changes plus a test-hudi-trino-plugin\njob whose steps all skipped. test-hudi-trino-plugin is no longer a\nrequired context, so the pull_request trigger can reuse the push\npaths filter and start no runner on PRs that miss it." }, { "commit": "7ad42f1c7ad670a16683d771f946dd1afb0ae10f", "tree": "523b2f971442aa43ac99e0a6896e2f5aec32ddae", "parents": [ "0f076155b7a52a9e348cf07b8bc76201c6da6aac" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Mon Oct 19 19:28:39 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 19 11:28:39 2026 +0000" }, "message": "chore(ci): run all PR metadata checks in one job (#20375)\n\npr_compliance, pr_title_validation, commit_coauthor_validation and\nupdate_pr_compliance each started a runner for a check that takes\nseconds. Run them as steps of a single validate-pr job in\npr_validation.yml. Every check step runs even if an earlier one fails,\nso a run still reports each problem.\n\nThe pr_compliance.py self-tests take about 30 ms, so they now run on\nevery PR instead of behind a paths filter." }, { "commit": "0f076155b7a52a9e348cf07b8bc76201c6da6aac", "tree": "70bb7a1351e953820b71daa3e414456e271582cc", "parents": [ "91ffb05e62cf9a2f58bd10a326d07e02f69ab081" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Mon Oct 19 16:56:08 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 19 16:56:08 2026 +0800" }, "message": "chore(ci): require only validate-pr for PR metadata (#20374)\n\nDrop validate-pr-title and validate-commit-coauthor from the required\ncontexts. The follow-up that bundles the PR metadata checks into one\nvalidate-pr job would otherwise be blocked by those two contexts, which\nno workflow reports once it lands." }, { "commit": "91ffb05e62cf9a2f58bd10a326d07e02f69ab081", "tree": "3d7f67e51f152ed23678e69cfd0976b570bd9778", "parents": [ "e95ecf581c5457cbca9708fad265df2a2db7925e" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Mon Oct 19 15:47:15 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 19 16:47:15 2026 +0800" }, "message": "fix(spark): stop treating lower/upper as order-preserving in data skipping (#20228)\n\n* fix(spark): stop treating lower/upper as order-preserving in data skipping\n\n* addressed review comments: cover NOT startswith over date_format and trim the lower() case\n\n---------\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e" }, { "commit": "e95ecf581c5457cbca9708fad265df2a2db7925e", "tree": "45501a036632503c5e0d6791dffa28f2c623951b", "parents": [ "9b7a7139bffb8956de906a2ef0f71609e7be77ba" ], "author": { "name": "Danny Chan", "email": "yuzhao.cyz@gmail.com", "time": "Mon Oct 19 11:09:19 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 19 11:09:19 2026 +0800" }, "message": "chore: update DOAP for 1.1.2 release (#20382)" }, { "commit": "9b7a7139bffb8956de906a2ef0f71609e7be77ba", "tree": "86a28081d1dc68bd77dc1e87be47300636c2a4d4", "parents": [ "a99e89ed28351cb1e2f78bed7b48fbbd9e0c37c6" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Sat Oct 17 10:02:08 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Oct 17 10:02:08 2026 +0800" }, "message": "fix(spark): preserve delete payloads during record reconstruction (#20341)" }, { "commit": "a99e89ed28351cb1e2f78bed7b48fbbd9e0c37c6", "tree": "5bb51f2f1f45ed2d4d1c2ffe7441fb39aca2c3c2", "parents": [ "1ba1edce0a3f40d4e251328dca181581a76488ef" ], "author": { "name": "fhan", "email": "aaron.han.1986@gmail.com", "time": "Sat Oct 17 09:50:28 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Oct 17 09:50:28 2026 +0800" }, "message": "feat(agent): Add first executable mutable Flink SQL MOR sink path to Hudi Architect (#20302)\n\n* feat(agent): Add the first executable mutable Flink SQL MOR sink path to Hudi Architect\n* fix(agent): harden Flink MOR contract validation\n\n---------\n\nCo-authored-by: fhan \u003cyfhanfei@jd.com\u003e" }, { "commit": "1ba1edce0a3f40d4e251328dca181581a76488ef", "tree": "aaa4d9a2f8385766e47b89f373ce6a51c97bda28", "parents": [ "5764e96b68dd8d76dcf1d1b74491f382184f067c" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Fri Oct 16 17:58:07 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 16 17:58:07 2026 +0800" }, "message": "fix(common): widen REQUIRED shredded variant reads (#20351)\n\nA COW upsert on the default AVRO record type failed with \"Invalid\ndefault for field v: null not a {record ...}\" when the base file\nstored a shredded variant column as REQUIRED and the table schema had\nit nullable. Spark 4.1 writes such files with default configs: a\nDataFrame insert of a non-null parse_json column, or a METADATA_ONLY\nbootstrap of a vanilla Spark parquet file.\n\ntoShreddedReadSchema swapped in the file\u0027s shredded schema wholesale,\nnullability included, and rebuilt the requested field around it with\nits null default, which Avro rejects on a non-null record.\n\nAt a shredded position the swap now stays nullable when the base side\nis nullable, at every depth it walks. The read schema keeps the\nrequested nullability (reading a REQUIRED column as nullable is a safe\nwidening), and alignShreddedVariants keeps a nullable file column\nnullable under a non-null request, which hit the same invalid default.\nA nullable side is never narrowed.\n\nCloses #20349." }, { "commit": "5764e96b68dd8d76dcf1d1b74491f382184f067c", "tree": "a1c9b7b85e9a1c48b27781c3a7b0a8fdfc7e54b0", "parents": [ "92771dbf5afea049a0797c100a0d07e8e01815e5" ], "author": { "name": "Sivabalan Narayanan", "email": "n.siva.b@gmail.com", "time": "Fri Oct 16 02:33:14 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 16 15:03:14 2026 +0530" }, "message": "feat(utilities): add HoodieTableHealthChecker for table service health (#20151)\n\n* feat(utilities): add HoodieTableHealthChecker for table service health\n\nAdds a read-only utility that reports whether a Hudi table\u0027s table\nservices are keeping up with ingestion, meant for a scheduled run whose\nexit code feeds alerting. Four timeline-only checks ship in this change:\ncompaction cadence, cleaner cadence, savepoints blocking archival, and\narchival cadence. A small framework (HealthCheck, HealthCheckResult,\nHealthCheckContext, HealthStatus) lets later checks plug in.\n\nVerdicts are relative to how the table is written, so writer properties\nare required via --props / --hoodie-conf. A check whose governing\nproperty was not supplied reports SKIPPED and names it rather than\nguessing against Hudi defaults; --apply-all-defaults opts into defaults\nexplicitly. Every check echoes the effective configuration it used.\n\nOutput is a human table or --output JSON, with per-check status,\nsummary, actionable findings, and effective configs, so an agent can\nconsume it. A companion Claude Code skill, hudi-ai-operator/skills/\nhudi-table-health, interprets the report and recommends next steps\nwithout acting on the table.\n\nNo new hoodie.* configs, no public API or storage format changes.\n\n* fix(utilities): model the retention the archiver actually enforces\n\nReview feedback on the health checker, with reproductions.\n\nhoodie.keep.max.commits is not on its own the bound archival enforces.\nArchivalUtils.getMinAndMaxInstantsToKeep raises both bounds whenever the\ncleaner retains more history than archival would, because archival must not\nmove an instant off the active timeline while the cleaner still considers its\nfiles live. The archival check compared against the raw configured maximum, so\na supported setup -- clean.commits.retained\u003d100 against keep.max.commits\u003d30 --\ncame out UNHEALTHY, and KEEP_LATEST_BY_HOURS with an hour of retention did the\nsame at 50 commits. The check now reproduces that calculation from the cleaner\nconfiguration and reports the effective bound it derived. It cannot call\nArchivalUtils directly: that needs a HoodieTable, and this tool reads only the\ntimeline.\n\nIt was also counting the wrong thing. getWriteTimeline() includes requested and\ninflight compaction and clustering, which archival has not had the chance to\nconsider; the archiver reasons over completed commits, and so does the check\nnow.\n\nAttribution is better where it can be. Archival never moves past the earliest\npending write instant, so when one is on the timeline the finding names it --\nthat is the thing an operator can act on. Compaction retention and the metadata\ntable\u0027s own compaction also hold archival back and are not visible from here,\nso the closing finding says so rather than implying a savepoint is the only\ncause.\n\nThe cleaner check blamed the wrong config. With keep.min.commits at or below\nclean.commits.retained, archival trims the timeline before the cleaner ever has\nenough commits to act on, so no clean is ever scheduled -- 60 file versions and\nzero cleans, with the finding pointing at hoodie.clean.automatic. That pairing\nis now detected and named, and the generic finding is kept for when the configs\ndo not explain it.\n\nExit codes: 1 is what the JVM returns when the process fails to start, so a\nclasspath mistake read as UNHEALTHY. Unhealthy moves to 3, a check that could\nnot run to 4, and unhealthy now outranks a failed check rather than the other\nway round -- the table\u0027s verdict is the more actionable of the two.\n\nThe documented java -cp invocation could not work: the bundle does not carry\nslf4j, Hadoop or Jackson. Replaced with spark-submit, including in the cron\nexample, which had the same problem and keyed off the old exit codes. Renamed\nHoodieTableHealthChecker.md to README.md to match the convention.\n\nFound while fixing the above: the cleaner\u0027s earliest-commit-to-retain was\nresolved by parsing the latest commit\u0027s instant time unconditionally, but only\nKEEP_LATEST_BY_HOURS needs it as wall-clock. An instant time that would not\nparse silently disabled the whole derivation and fell back to the configured\nbound.\n\n* fix(utilities): do not judge the cleaner on an append-only table\n\nA table that only ever receives inserts never produces a clean instant, and\nthat is correct: cleaning retires superseded file slices, and an append-only\nworkload never supersedes anything. Every file group has a single slice and\nnothing is replaced, so the cleaner runs and finds no work.\n\nThe check read an empty cleaner timeline as a stalled service, so such a table\nwas reported UNHEALTHY every day with no action available to the operator --\nthe kind of standing false alarm that teaches people to ignore the tool.\n\nAppend-only is now read from the commit metadata rather than from\nhoodie.datasource.write.operation, because what the table has actually\nreceived is the question: a writer configured for upsert that has only ever\ninserted still has nothing to clean, and a property file can be stale or\nabsent. One insert_overwrite, upsert or delete anywhere on the timeline means\nfile slices can be superseded and the normal cadence judgement applies again.\n\nConservative on failure. An unreadable commit, or an operation type the\nmetadata does not carry, means the exemption does not apply and the check\njudges as before -- suppressing a real finding is the worse error of the two,\nso the exemption has to be earned by every completed write on the timeline.\n\n* feat(utilities): give health checks an optional Spark context\n\nChecks so far read only the timeline, so the checker has never needed Spark.\nTwo things change that. The metadata-table sync check should delegate its\nlisting comparison to HoodieMetadataTableValidator rather than reimplement a\nnarrower one, and the validator takes a JavaSparkContext. And the compaction\nand cleaner checks walk file slices per file group and file versions per\npartition on the driver, which does not hold on a table with hundreds of\nthousands of file groups.\n\nHealthCheckContext now carries an Option\u003cJavaSparkContext\u003e, and the driver an\noptional one it passes through. Both keep their existing constructors, so a\ncaller that does not supply a context gets exactly what it got before, and the\nchecks that only read the timeline stay runnable without a cluster. A check\nthat needs a context and is not given one reports SKIPPED naming what it\nneeded, rather than failing the run -- the same contract the checker already\napplies to a missing writer property.\n\nNo check uses it yet; this is the plumbing.\n\n* feat(utilities): build the Spark context in the driver\n\nFollows the previous commit, which added the optional context but left main\npassing nothing, so a check needing one would always have found it absent.\n\nmain now builds it unconditionally and stops it in a finally. Building it only\nfor the checks that need one would have made the selected check set decide\nwhether the tool can start, which is a worse property than paying for a local\ncontext on a run that does not use it. --spark-master sets the master when the\njob does not already have one; under spark-submit the submitted master wins.\n\nThe README claimed no Spark session was used. That was true when every check\nread only the timeline and is about to stop being true, so it now says to run\nunder spark-submit and that listing comparisons scan the table while the\ntimeline checks do not." }, { "commit": "92771dbf5afea049a0797c100a0d07e8e01815e5", "tree": "b762b55f4290f8869bf7700ea22aa9d7c4de38b1", "parents": [ "1568cebd32dc2164e1087e83658f97311e9e5678" ], "author": { "name": "Sagar Sumit", "email": "sagarsumit09@gmail.com", "time": "Fri Oct 16 15:01:58 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 16 15:01:58 2026 +0530" }, "message": "feat(client): add an opt-in to skip inline metadata table services (#20350)\n\nAdds hoodie.metadata.inline.table.services.enabled (default true). When\nset to false, a data table writer does not run clean, compaction or\narchival on the metadata table inline:\n\n- Spark and Java: skipped when a write starts, under the table lock.\n- Flink: skipped when the coordinator initializes the metadata table and\n on each metadata table commit, the same places that already skip them\n when streaming writes to the metadata table are enabled.\n\nhoodie.table.services.enabled\u003dfalse does not stop these, and is not\nreused for this: writers that keep it off rely on these paths to compact\nthe metadata table, and data table archival cannot move past the latest\nmetadata table compaction.\n\nSigned-off-by: codope \u003csagarsumit09@gmail.com\u003e" }, { "commit": "1568cebd32dc2164e1087e83658f97311e9e5678", "tree": "e78e6ac57323ca2bfdcd4f13b199a47b68a75e44", "parents": [ "c82957df5f757c2cc3faaf8b971fb554251100c1" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Fri Oct 16 16:17:54 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 16 16:17:54 2026 +0700" }, "message": "fix(spark): stop treating the datediff start date as order-preserving in data skipping (#20230)\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e" }, { "commit": "c82957df5f757c2cc3faaf8b971fb554251100c1", "tree": "ea92e131e061fc051c48bbd7b21ef01c6078ac35", "parents": [ "c167f7b392441cfb86b1dd64bb6c2708878e80d6" ], "author": { "name": "Prashant Wason", "email": "pwason@uber.com", "time": "Thu Oct 15 22:49:27 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 22:49:27 2026 -0700" }, "message": "fix(gcp): skip \"null\" type fields when syncing the BigQuery schema (#20356)" }, { "commit": "c167f7b392441cfb86b1dd64bb6c2708878e80d6", "tree": "ca856d96a42f4c9bed98ea2fcf204f7fd52d6627", "parents": [ "2cfe6b96de7958a5b631f58fea063d9c87f5db6b" ], "author": { "name": "Prashant Wason", "email": "pwason@uber.com", "time": "Thu Oct 15 19:20:02 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 19:20:02 2026 -0700" }, "message": "fix(hive-sync): skip null type fields when syncing the table schema (#20354)\n\n* fix(hive-sync): skip null type fields when syncing the table schema\n\nA field declared with the Avro primitive type \"null\" (not a nullable\nunion) is valid Avro but always holds null. The parquet writer does not\nwrite such fields, and Hive has no equivalent type. Since meta sync moved\nfrom the parquet MessageType to HoodieSchema, HiveSchemaUtil and\nSparkSchemaUtils throw \"Cannot convert type NULL\" for these fields, so\nHive sync fails for any table whose schema contains one, at the top level\nor nested inside a record.\n\nSkip \"null\" type fields in both converters so the synced schema matches\nthe columns in the data files, as it did before the HoodieSchema migration.\n\n* Address review comments: inline the null type check and wrap the stream chain\n\n* Add an end-to-end Hive sync test for \"null\" type fields\n\n* Use lowercase field names in the null type field sync test" }, { "commit": "2cfe6b96de7958a5b631f58fea063d9c87f5db6b", "tree": "c19595000b05037bdc2d12d196c25600e23e3821", "parents": [ "b6119ca4e04d7a5fd72c4a77e50b366caf9c50e5" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Thu Oct 15 12:57:11 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 19:57:11 2026 +0000" }, "message": "fix(hfile): reject block sizes the HFile format cannot represent (#20321)" }, { "commit": "b6119ca4e04d7a5fd72c4a77e50b366caf9c50e5", "tree": "7f58390334f99d45d039adec9aa6721d733bfa4b", "parents": [ "5dee63857193d19e5a61110ed6bbbbffe7955094" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Thu Oct 15 12:09:12 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 19:09:12 2026 +0000" }, "message": "docs(hudi-io): document HFile offset and size widths and the non-root index block layout (#20319)" }, { "commit": "5dee63857193d19e5a61110ed6bbbbffe7955094", "tree": "adaabe6467453eff886e65f4c60dca561c00a9e8", "parents": [ "75fb5d5ecd922ed873bcd6bb31382b6630f36e67" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Thu Oct 15 19:29:53 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 19:29:53 2026 +0800" }, "message": "fix(spark): reject variant leaves Spark 4 cannot shred (#20326)\n\n* fix(spark): decline shredding into fixed leaves\n\nSpark4VariantShreddingProvider mapped an Avro fixed typed_value leaf\nto Spark\u0027s BinaryType, so shredding handed parquet-avro a ByteBuffer\nwhere a GenericFixed is required (ClassCastException at write) and the\nfixed size was never checked. A uuid logical type on fixed(16) mapped\nto UuidType, which converts to a String that a fixed cannot hold\n(UnresolvedUnionException).\n\nSpark\u0027s VariantSchema has no fixed-length binary type, so decline to\nshred both leaves: avroTypeToScalarType returns null and the value\nstays in the residual, as it already does for millisecond timestamps.\nDecimal on fixed is unaffected because the decimal logical-type check\nruns first.\n\nAddresses the write side of #19442; the read-side plain FLBA case\nis left open.\n\n* fix(spark): reject unshreddable variant leaves\n\nMatch Spark 4\u0027s own variant shredding on the write path. Spark\u0027s\nSparkShreddingUtils.buildVariantSchema only accepts Boolean, Byte,\nShort, Int, Long, Float, Double, String, Binary, Date, Timestamp,\nTimestampNTZ and Decimal scalars and throws\ninvalidVariantShreddingSchema for anything else; its inference keeps\nUUIDs in the residual.\n\nSpark4VariantShreddingProvider now throws a HoodieException when a\nscalar typed_value is not the Avro equivalent of one of those types:\nplain fixed, uuid on string or fixed, millisecond and nanosecond\ntimestamps, time types, enum and map. Previously these broke the write,\nwere silently declined, mapped time to an integer, or (uuid on string)\nread back as a plain string.\n\nThe check is write-only. avroTypeToScalarType goes back to its master\nmapping because the read path shares it, so read behavior is unchanged.\nHudi\u0027s shredding DDL and inference never produce these leaf types.\n\n* refactor(spark): hoist shreddable scalar type list\n\nKeep the error-message list of supported typed_value types in a\nconstant next to isSparkShreddableScalar so the two do not drift." }, { "commit": "75fb5d5ecd922ed873bcd6bb31382b6630f36e67", "tree": "a57bc03d989308d3bf0302eab3c4794315e27c28", "parents": [ "616f9ae867654d70b58674600887e4ff1205ccf2" ], "author": { "name": "Danny Chan", "email": "yuzhao.cyz@gmail.com", "time": "Thu Oct 15 17:38:48 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 17:38:48 2026 +0800" }, "message": "fix(common): avoid archive lookups for active clean and rollback instants (#20340)" }, { "commit": "616f9ae867654d70b58674600887e4ff1205ccf2", "tree": "37f586360eb520c49ac021e6c8f67df633c311ca", "parents": [ "0170359e59d30a5f09ccadd47e796763c653f395" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Thu Oct 15 16:04:59 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 16:04:59 2026 +0800" }, "message": "fix(common): dedupe repeated Avro fixed names (#20324)\n\n* fix(common): dedupe repeated Avro fixed names\n\nAvroSchemaConverterWithTimestampNTZ is Hudi\u0027s copy of parquet-java\n1.13.1\u0027s AvroSchemaConverter. In the Parquet-to-Avro direction it\nnames every fixed (including DECIMAL on fixed_len_byte_array) after\nits field with no namespace, while repeated record names already get\na dedup namespace. Shredded variants name every leaf typed_value, so a\nDECIMAL leaf collides with its enclosing typed_value record and the\nread schema no longer survives the toString/parse round trip in\nHoodieAvroParquetReader, failing COW upserts with\n\"Undefined schema: k102.typed_value\".\n\nPort PARQUET-2456 (apache/parquet-java#1304, released in 1.14.0):\nrecords and fixed types share one name counter, and repeated names get\na \u003cname\u003e\u003ccount\u003e namespace. Hudi builds DECIMAL fixed types in its\nlogical-type branch rather than in convertFIXED_LEN_BYTE_ARRAY, so\nthat branch gets the same treatment.\n\nAlso restore the INT96 doc string, which the HoodieSchema migration\npassed in the namespace slot (HoodieSchema.createFixed takes the\nnamespace second, Avro\u0027s Schema.createFixed takes the doc second).\n\nFixes #20307\n\n* refactor(common): rename dedup namespace helper\n\nRename namespace() to nextDedupNamespace(): it bumps the shared name\ncounter as a side effect, which the old name did not say. Addresses\nreview feedback on #20324." }, { "commit": "0170359e59d30a5f09ccadd47e796763c653f395", "tree": "777c91446a4738526954e738395e6d4509aa27da", "parents": [ "b0ea242c8a65cb841c874cd79aa2a6eed3404789" ], "author": { "name": "Danny Chan", "email": "yuzhao.cyz@gmail.com", "time": "Thu Oct 15 12:45:52 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 12:45:52 2026 +0800" }, "message": "fix(flink): resend recovered write metadata on task retries (#20323)\n\n* fix(flink): resend recovered write metadata on task retries\n\n* fix(flink): ignore late bootstrap events for completed instants\n\n* fix(flink): mark checkpointed write metadata as the last batch\n\n* fix(flink): preserve received bootstrap metadata on retries" }, { "commit": "b0ea242c8a65cb841c874cd79aa2a6eed3404789", "tree": "ee66fb88c86e54472e90e6da5dd1ebc2c9deb22b", "parents": [ "9086d9819a5d15ac4b0cf40d6a7bd39e58e0b894" ], "author": { "name": "fhan", "email": "aaron.han.1986@gmail.com", "time": "Thu Oct 15 11:05:12 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 11:05:12 2026 +0800" }, "message": "fix(kafka-connect): close AdminClient after fetching topic partitions (#20303)\n\nCo-authored-by: fhan \u003cyfhanfei@jd.com\u003e" }, { "commit": "9086d9819a5d15ac4b0cf40d6a7bd39e58e0b894", "tree": "82f13fcd77d9c57dd9daaf7bd27ff988ca2fc442", "parents": [ "bc78d06bd2179b4304208b6aae23ce0fe6c9ad72" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Thu Oct 15 10:04:45 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 11:04:45 2026 +0800" }, "message": "fix(kafka-connect): retry start commit when the topic partition lookup fails (#20312)\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e" }, { "commit": "bc78d06bd2179b4304208b6aae23ce0fe6c9ad72", "tree": "a1b8eda162c99969a615e96dc034725dba0d98aa", "parents": [ "6bbf798f88d468d1ca3ce774aae991145b0cd0a2" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Thu Oct 15 10:03:53 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 15 11:03:53 2026 +0800" }, "message": "fix(kafka-connect): treat a completed commit as successful when a post-commit action fails (#20316)\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e" }, { "commit": "6bbf798f88d468d1ca3ce774aae991145b0cd0a2", "tree": "0899e31bb72d658af9757c9df905b76f77eff18f", "parents": [ "61dde86311368a7d373e582f40b941db55aa60a4" ], "author": { "name": "Sagar Sumit", "email": "sagarsumit09@gmail.com", "time": "Wed Oct 14 17:02:44 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 14 17:02:44 2026 +0530" }, "message": "fix(spark): render decimal, date and timestamp values in CDC images (#20305)\n\nCDC before/after images rendered DECIMAL as {\"zero\":false} (Spark\u0027s\nDecimal serialized as a bean) or, for inline Avro CDC log files, as the\nraw unscaled bytes. DATE and TIMESTAMP came out as epoch days/micros.\n\nInternalRowToJsonStringConverter now renders DECIMAL as an exact JSON\nnumber, DATE as an ISO-8601 date, TIMESTAMP as an ISO-8601 UTC instant\nand TIMESTAMP_NTZ as an ISO-8601 local date-time, at full precision.\nImages from inline Avro CDC log files are deserialized to InternalRow\nand rendered by the same converter, so every CDC inference path\nproduces the same image. HoodieCDCUtils.recordToJson has no remaining\ncallers and is removed.\n\nFixes #20299" }, { "commit": "61dde86311368a7d373e582f40b941db55aa60a4", "tree": "fabd08b7672bc2556441938d0debd5d957487526", "parents": [ "276d120b839a3d7b6ec64bf32b9132fab9f88a6d" ], "author": { "name": "Vova Kolmakov", "email": "wombatukun@gmail.com", "time": "Wed Oct 14 14:25:11 2026 +0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 14 15:25:11 2026 +0800" }, "message": "fix(flink): make the bucket pruning function serializable for source V2 (#20318)\n\nCo-authored-by: Vova Kolmakov \u003cwombatukun@apache.org\u003e" }, { "commit": "276d120b839a3d7b6ec64bf32b9132fab9f88a6d", "tree": "b1263a121ccfc19b5a54463abb349423b0954124", "parents": [ "fe1c31d385bd5cd8ec57ea06757bbc461a1c03ae" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Wed Oct 14 12:57:32 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 14 12:57:32 2026 +0800" }, "message": "fix(flink): retain files with missing column statistics (#20295)\n\n* fix(flink): retain files with missing column statistics\n\nPreserve unknown column statistics during transposition and retain candidates with missing statistics before evaluating predicates. Add probe, index expansion, and SQL integration regression coverage.\n\n* test(flink): cover incomplete statistics in partition pruning\n\nDocument conservative file retention when another conjunct excludes the file, and verify partition candidates after expanding indexed columns." }, { "commit": "fe1c31d385bd5cd8ec57ea06757bbc461a1c03ae", "tree": "8557b6ba4035d6dde33219afb62476a6a30b7ddb", "parents": [ "9b39f0450787ac1265766eeed80be96f379e2c58" ], "author": { "name": "Prashant Wason", "email": "pwason@uber.com", "time": "Tue Oct 13 21:51:28 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 13 21:51:28 2026 -0700" }, "message": "fix(hfile): use long cursor offsets in native HFile reader for files larger than 2GB (#20309)\n\nHFileCursor stored the absolute position in the file as an int, and\nHFileReaderImpl cast block offsets to int. For HFiles larger than 2GB the\ncursor offset overflowed, so isAtFirstKeyOfBlock() never matched and a valid\nlookup in [fake first key, actual first key) after SEEK_TO_BEFORE_BLOCK_FIRST_KEY\nwas rejected as a backward seekTo with IllegalStateException.\n\n- Make HFileCursor offsets long and pass long block start offsets to\n HFileDataBlock.seekTo/next; offsets relative to a block stay int.\n- Remove the int casts on block offsets in HFileReaderImpl.\n- Position the cursor in seekTo() using the first data block\u0027s offset\n instead of the first key\u0027s offset within its byte array.\n- Add a test that runs the point and prefix lookups with data block offsets\n shifted beyond 2GB and 4GB.\n\nCloses #20308" }, { "commit": "9b39f0450787ac1265766eeed80be96f379e2c58", "tree": "7c270d66d4e7bb765cd633588410b1ab8b8dfb3c", "parents": [ "aabed46593960ff3553cdc1ace1a491987983702" ], "author": { "name": "Aditya Goenka", "email": "63430370+ad1happy2go@users.noreply.github.com", "time": "Wed Oct 14 10:09:54 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 13 21:39:54 2026 -0700" }, "message": "fix(common): correct incremental fallback reads when a stale non-write instant precedes the first active commit (#20273)\n\nCo-authored-by: Aditya Goenka \u003cadityagoenka@Adityas-MacBook-Pro-2.local\u003e" }, { "commit": "aabed46593960ff3553cdc1ace1a491987983702", "tree": "0d4a0e574b8ece5b88cba1885e891256f21ffe2a", "parents": [ "092e4c40c504723f6b534b994ca950b6df131bc0" ], "author": { "name": "Sagar Sumit", "email": "sagarsumit09@gmail.com", "time": "Wed Oct 14 09:24:30 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 14 09:24:30 2026 +0530" }, "message": "fix(spark): honor \u0027earliest\u0027 begin instant in DataFrame CDC reads (#20304)\n\nA DataFrame CDC read with hoodie.datasource.read.begin.instanttime\u003dearliest\nreturned no rows: HoodieCDCFileIndex passed the literal into InstantRange,\nand the string comparison against instant times excluded every commit.\nMap it to HoodieTimeline.INIT_INSTANT_TS, matching what hudi_table_changes\nalready does for \u0027earliest\u0027.\n\nAlso correct the begin.instanttime doc: the start bound is exclusive\n(completion_time \u003e START_COMMIT), not inclusive.\n\nFixes #20300\nFixes #20301" }, { "commit": "092e4c40c504723f6b534b994ca950b6df131bc0", "tree": "bbc2951ff1ff89965e1be5e59c4e2eca21b9f2fb", "parents": [ "b86a34b59e015b68796d88de8b18eea38c1739ae" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Wed Oct 14 10:53:40 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 14 10:53:40 2026 +0800" }, "message": "fix(flink): reclaim empty checkpoint caches in record index (#20241)\n\n* fix(flink): reclaim empty checkpoint caches in record index\n* refactor(flink): separate empty cache cleanup from memory eviction\n\n---------\n\nCo-authored-by: danny0405 \u003cyuzhao.cyz@gmail.com\u003e" }, { "commit": "b86a34b59e015b68796d88de8b18eea38c1739ae", "tree": "96947f33ad8745fd7bbd8a067e7cd2a16933a2f8", "parents": [ "e9da226db3a90747cc51d13fec56ebd1704c1699" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Wed Oct 14 10:47:04 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 14 10:47:04 2026 +0800" }, "message": "fix(flink): preserve event-time delete ordering during precombine (#20288)" }, { "commit": "e9da226db3a90747cc51d13fec56ebd1704c1699", "tree": "c71172b4643950c190e73e92cb6f3ae98974d7c3", "parents": [ "da78989d9377c6603b1ef63bf5c82256266c2325" ], "author": { "name": "Sumit Singh", "email": "sumitthakur1234@gmail.com", "time": "Tue Oct 13 17:36:08 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 13 17:36:08 2026 +0530" }, "message": "fix(spark): decide vectorized read per scan in HoodieFileGroupReaderBasedFileFormat (#20183)\n\n* fix(spark): decide vectorized read per scan in HoodieFileGroupReaderBasedFileFormat\n Spark caches the resolved relation per session, so one file format instance serves every statement on a table, and supportBatch is only called for scans narrower than spark.sql.codegen.maxFields. The vectorized decision was kept in a mutable field and written into spark.sql.parquet.enableVectorizedReader, so a wide scan could inherit a narrow scan\u0027s answer and fail with ClassCastException: ColumnarBatchRow cannot be cast to UnsafeRow. Evaluate canReadVectorized against each scan\u0027s own schema in both supportBatch and buildReaderWithPartitionValues, and stop mutating the session conf. Adds TestVectorizedReadDecisionPerScan covering COW and MOR.\n\n* Fix new bug which got surface due to fix.\n\n* Fix comments\n\n* Fix comments\n\n* test(spark): check fallback scans keep the session vectorized reader conf\n\nCarry the two regression tests from #20193 into this change. The new\nTestVectorizedReadDecisionPerScan checks the session conf after wide,\nnarrow, Vortex and timestamp-millis scans, but never after a scan that\nfalls back to row-based reads for a variant column or an incremental\nquery. Before this fix, buildReaderWithPartitionValues wrote that\nfallback into spark.sql.parquet.enableVectorizedReader, so every later\nread in the session lost vectorization.\n\n - TestVariantShreddingMixedLayouts: a variant read on Spark 4.1+ falls\n back to row-based reads and must leave the session conf at true.\n - TestIncrementalReadWithFileGroupReader: a COW incremental read, which\n is never vectorized, must leave the session conf at true. It saves\n and restores one key instead of a one-entry list, and also checks\n that the read returns the rows written." }, { "commit": "da78989d9377c6603b1ef63bf5c82256266c2325", "tree": "f0f7703693618a358ca2a610865376eab8119843", "parents": [ "790414fe8ce530a8ad7e604c5d7dc0492178fc73" ], "author": { "name": "fhan", "email": "aaron.han.1986@gmail.com", "time": "Tue Oct 13 17:19:05 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 13 17:19:05 2026 +0800" }, "message": "fix(kafka-connect): terminate transaction coordinator event loop on stop (#20271)\n\n* fix(kafka-connect): terminate transaction coordinator event loop on stop\n\n* fix(kafka-connect): clean up coordinator resources on shutdown\n\n* fix(kafka-connect): retry coordinator service cleanup on repeated stop\n\n---------\n\nCo-authored-by: fhan \u003cyfhanfei@jd.com\u003e" }, { "commit": "790414fe8ce530a8ad7e604c5d7dc0492178fc73", "tree": "f6b34667b48683f72dd45837aadb2eb3672cccd4", "parents": [ "0202ca7e24c1662ece6ea4b7400168bac7d5dee7" ], "author": { "name": "Nanjesh Ramesh", "email": "nanjeshramesh7@gmail.com", "time": "Tue Oct 13 00:23:17 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 13 15:23:17 2026 +0800" }, "message": "refactor: Rename dataFile/df variables to baseFile/bf (#19944)\n\n* [HUDI-541] Rename dataFile/df variables to baseFile/bf\n\nHoodieBaseFile was previously named HoodieDataFile, and many call sites\nacross the codebase still name variables, parameters, and lambda\narguments of this type dataFile or df rather than baseFile or bf,\nper the terminology convergence the cWiki design and arch page calls\nfor.\n\nThis renames every local variable, field, method parameter, and\nlambda argument whose declared or inferred type is HoodieBaseFile (or\na direct List/Stream/Option/Collection of it) from dataFile/df to\nbaseFile/bf, across 35 files in hudi-common, hudi-hadoop-common,\nhudi-hadoop-mr, hudi-sync/hudi-hive-sync, hudi-client-common,\nhudi-spark-client, hudi-java-client, hudi-flink-client, and\nhudi-spark-datasource.\n\nDeliberately out of scope:\n- Method names such as HoodieFileGroup#getLatestDataFile and\n TestHoodieBaseFile#newDataFile are unchanged. Renaming a method,\n especially a public one, is a materially different, higher-risk\n change than renaming a local variable or parameter, and the issue\u0027s\n own wording (\"all variables of this type\") does not ask for it.\n- Identifiers that are not of type HoodieBaseFile even though they\n contain \"dataFile\" in the name, for example dataFileName (String),\n skipCreatingDataFile (boolean), and the several FileSlice-typed\n orphan/inflightFileSliceWithDataFile locals in\n TestHoodieTableFileSystemView. Each was checked individually against\n its actual declared or inferred type before being included or\n excluded.\n- df/dataFile identifiers that are unrelated Spark DataFrames or other\n types entirely, such as StreamSync\u0027s transformed.map(df -\u003e ...) and\n IncrementalTimelineSyncFileSystemView\u0027s\n deltaFileGroups.stream().map(df -\u003e df.getTimeline()), the latter\n being a HoodieFileGroup, not a HoodieBaseFile, despite sitting one\n line away from two genuine HoodieBaseFile df usages that were\n renamed.\n\nAn earlier attempt at this same issue (#1650) was opened in 2022 and\nclosed unmerged after review stalled; a good number of the dataFile/df\nsites it touched have since been renamed independently in other\nchanges, which is why this diff is smaller in file count than that\none.\n\nPure rename, no behavior change: 281 insertions, 281 deletions.\nVerified by test-compiling every touched module (hudi-common,\nhudi-hadoop-common, hudi-hadoop-mr, hudi-hive-sync, hudi-client-common,\nhudi-spark-client, hudi-java-client, hudi-flink-client,\nhudi-spark-datasource/hudi-spark and its spark3.5 dependency chain)\nagainst JDK 11 with zero compile errors.\n\n* [HUDI-541] Use baseFile consistently instead of bf\n\nAddresses review nits: the rename previously used the abbreviated bf\nin a number of lambda parameters and local variables while using the\nfull baseFile elsewhere. Converts every remaining bf introduced by\nthis PR to baseFile so the whole change uses one convention.\n\nFiles touched: CompactionAdminClient, AbstractTableFileSystemView,\nIncrementalTimelineSyncFileSystemView, RocksDbBasedFileSystemView,\nTestCleanerInsertAndCleanByVersions, TestHoodieCompactionStrategy,\nTestHoodieTableFileSystemView, TestCompactionUtils.\n\nVerified by test-compiling hudi-common, hudi-hadoop-common,\nhudi-client-common, and hudi-spark-client against JDK 11 with no\nerrors, including no lambda-parameter shadowing conflicts.\n\n* [HUDI-541] Wrap a line pushed over the checkstyle limit by the bf -\u003e baseFile rename\n\ngetLatestBaseFile\u0027s filter/findFirst chain was 196 characters with the\nabbreviated bf; renaming both lambda parameters to baseFile pushed it\nto 208, over the 200-character checkstyle limit and failing every CI\njob that depends on hudi-common. Wraps the chain across three lines,\nmatching the wrapping style already used for the same filter pattern\nelsewhere in this file (e.g. getLatestBaseFilesBeforeOrOnFromCache).\n\nVerified with mvn checkstyle:check on hudi-common." }, { "commit": "0202ca7e24c1662ece6ea4b7400168bac7d5dee7", "tree": "55fadb2a214354ea72c0b9e0c918453aa1243d85", "parents": [ "a2d88d9229fd9bb20f33c9826ca75a138127a559" ], "author": { "name": "Danny Chan", "email": "yuzhao.cyz@gmail.com", "time": "Tue Oct 13 13:32:24 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 13 13:32:24 2026 +0800" }, "message": "perf(core): avoid copying table properties for each native log file (#20297)\n\n* perf(core): avoid copying table properties for each native log file" }, { "commit": "a2d88d9229fd9bb20f33c9826ca75a138127a559", "tree": "c6397123e6c3cf343b3774349c52edc28673fad4", "parents": [ "45cec1a3bc139d1e9d6db9bb75efa20451575cea" ], "author": { "name": "Shihuan Liu", "email": "skywalker0618@gmail.com", "time": "Mon Oct 12 22:14:18 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 13 13:14:18 2026 +0800" }, "message": "fix(hive-sync): keep the sync client\u0027s metastore client out of Hive\u0027s thread cache (#20219)\n\n* fix(hive-sync): keep the sync client\u0027s metastore client out of Hive\u0027s thread cache\n\nHoodieHiveSyncClient took its metastore client from the Hive instance cached\nfor the thread. In HiveQL mode, the Driver session\u0027s authorization setup\nchanges the metastore filter hook and asks Hive for an instance with that conf,\nso Hive replaced the cached instance and closed the sync client\u0027s metastore\nclient. Every sync then relied on RetryingMetaStoreClient\u0027s single reconnect.\n\nTake the instance out of the cache with Hive.set(null) once the client is\ncreated, so a Hive session on the thread builds an instance of its own.\n\n* fix(hive-sync): release the metastore client from Hive\u0027s cache lazily, close it on failed construction\n\n- Leave the sync client\u0027s metastore client in Hive\u0027s thread cache until the session\u0027s Driver is\n about to run its first statement, so a sync that runs none lets the session teardown reuse it\n instead of opening a connection of its own.\n- Close the metastore client and partition pools when HoodieHiveSyncClient fails to construct.\n- Make HiveQueryDDLExecutor\u0027s session teardown run once: HiveSyncTool is closed by\n syncHoodieTable() and again by try-with-resources callers, and closing the session a second\n time opened a metastore connection.\n\n* refactor(hive-sync): name the no-op session hook, rename threadCachingClient, explain closing the client on failed construction\n\n* docs(hive-sync): simplify closeClient() Javadoc\n\n* refactor(hive-sync): centralize constructor failure cleanup\n\n* refactor(hive-sync): detach cached client in the HiveQL executor\n\n* refactor(hive-sync): use Option for the cache-owning thread\n\n* refactor(hive-sync): leave failed-construction pool cleanup to the caller\n\n---------\n\nCo-authored-by: danny0405 \u003cyuzhao.cyz@gmail.com\u003e" }, { "commit": "45cec1a3bc139d1e9d6db9bb75efa20451575cea", "tree": "63f04bfa52cb66d16a83e4227c2edc8713085b0f", "parents": [ "2d7be922ceb4bb671847b5abc687dc30ba0b8112" ], "author": { "name": "Burak Kalaycı", "email": "kalayciburak1996@gmail.com", "time": "Tue Oct 13 06:02:57 2026 +0300" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 13 11:02:57 2026 +0800" }, "message": "fix(common): propagate stored disruptor consumer failures (#20026)\n\n* fix(common): propagate stored disruptor consumer failures\n\nDisruptor handlers catch consume exceptions and store them, so execute()\ncan return the finish() result while a fatal failure is still stored.\nCheck that stored failure after finish() so the caller sees it.\n\nCloses #20016\n\nSigned-off-by: Burak KALAYCI \u003ckalayciburak1996@gmail.com\u003e\n\n* test(common): prefer stored consume failure when finish throws\n\nAdd a companion case where consume() and finish() throw different\nexceptions, and assert the stored consume failure stays the direct cause.\n\nSigned-off-by: Burak KALAYCI \u003ckalayciburak1996@gmail.com\u003e\n\n---------\n\nSigned-off-by: Burak KALAYCI \u003ckalayciburak1996@gmail.com\u003e" }, { "commit": "2d7be922ceb4bb671847b5abc687dc30ba0b8112", "tree": "daf61a94f280a67cef75b860dcf3b4d77e98f3c3", "parents": [ "9a629fd8293ef62f5f036038c58d2eb5e8d1cb04" ], "author": { "name": "fhan", "email": "aaron.han.1986@gmail.com", "time": "Tue Oct 13 10:56:11 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 13 10:56:11 2026 +0800" }, "message": "fix(flink): prevent write rate limiter stalls and thread leaks (#20189)\n\n* fix(flink): prevent write rate limiter stalls and thread leaks\n\n---------\n\nCo-authored-by: fhan \u003cyfhanfei@jd.com\u003e" }, { "commit": "9a629fd8293ef62f5f036038c58d2eb5e8d1cb04", "tree": "4e0a88af6c23b598c20ce701b60dfdcec384495d", "parents": [ "a41f6eaf7df0de8c9aefdd80a3cbb03967623e96" ], "author": { "name": "fhan", "email": "aaron.han.1986@gmail.com", "time": "Tue Oct 13 10:44:50 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 13 10:44:50 2026 +0800" }, "message": "feat(agent): add first mutable Flink SQL COW sink path to Hudi Architect (#20281)\n\n* feat(agent): add first mutable Flink SQL COW sink path to Hudi Architect\n* fix(agent): align mutable Flink replay and ordering validation\n\n---------\n\nCo-authored-by: fhan \u003cyfhanfei@jd.com\u003e" }, { "commit": "a41f6eaf7df0de8c9aefdd80a3cbb03967623e96", "tree": "cf527864cfc6aa9f6ccf36e837b5259d7165e6b7", "parents": [ "3aa7fe2a9f70ef52064c977d28d28b4f4c95d00b" ], "author": { "name": "voonhous", "email": "voonhousu@gmail.com", "time": "Mon Oct 12 15:52:24 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 12 15:52:24 2026 +0800" }, "message": "fix(spark): keep variant guard under nested filter pushdown (#20201)\n\n* test(variant): pin nested filters on the projection\n\nWith shortName() back to the base file format (#20029), Spark pushes\nnested predicates into Hudi scans again. A predicate on a variant that\nPushVariantIntoScan rewrote is then pushed as a filter on the\nprojection struct\u0027s ordinal-named member, e.g. GreaterThan(v.`0`,5).\nNo file carries that name as a parquet leaf, so ParquetFilters drops it\nper file: it skips no row group and lands on no other column.\n\nPin that on the two existing variant_get filter tests: the\nPushedFilters of the scan per pushVariantIntoScan arm, top-level on COW\nand MOR and nested (s.inner), next to the row counts both arms already\nshare, plus an IS NULL count over files where the path is typed,\nresidual, type-conflicted and absent.\n\n* fix(spark): drop pushed filters on unknown names\n\nUnder schema-on-read the parquet readers re-spell each pushed filter\nonto the file\u0027s column names before the variant guard runs, and\nInternalSchemaUtils.reBuildFilterName throws on a name the query schema\ndoes not hold. With nested predicate pushdown restored (#20029), a\npredicate on a variant that PushVariantIntoScan rewrote arrives as a\nfilter on the projection struct\u0027s ordinal-named member (v.`0`), so the\nread died with \"cannot find filter col name\" instead of the guard\u0027s\nmessage that names the conf to turn off.\n\nCollapse such a filter to AlwaysTrue, as a filter on a column the file\nlacks already is. Spark evaluates pushed filters again above the scan,\nso dropping one is safe, and the read reaches\nvalidateNoShreddedVariants as before.\n\nTestParquetSchemaEvolutionUtils pins the new arm for every leaf filter\ntype; the schema-on-read fail-fast test gains a variant_get predicate\nleg that expects the guard." }, { "commit": "3aa7fe2a9f70ef52064c977d28d28b4f4c95d00b", "tree": "f0e2b4f2ab366f47ad9620eb61b965d422177fc7", "parents": [ "9794ece2d55f69fc190f823f99a21a9f3c0fb031" ], "author": { "name": "fhan", "email": "aaron.han.1986@gmail.com", "time": "Fri Oct 09 17:22:15 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 09 17:22:15 2026 +0800" }, "message": "feat(agent): add first executable Flink SQL sink path to Hudi Architect (#20198)\n\n* feat(agent): add first executable Flink SQL sink path to Hudi Architect\n\n---------\n\nCo-authored-by: fhan \u003cyfhanfei@jd.com\u003e" }, { "commit": "9794ece2d55f69fc190f823f99a21a9f3c0fb031", "tree": "314e0f40ced18176a24e44cee1b87b76e7e46617", "parents": [ "308f78e71deabf7a9eec4b0a2ea26b1360658030" ], "author": { "name": "Shuo Cheng", "email": "njucshuo@gmail.com", "time": "Fri Oct 09 17:14:48 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 09 17:14:48 2026 +0800" }, "message": "fix(flink): close bulk insert writer helper during cleanup (#20242)\n\n* fix(flink): close bulk insert writer helper during cleanup\n* fix(flink): preserve bulk insert cleanup exceptions" }, { "commit": "308f78e71deabf7a9eec4b0a2ea26b1360658030", "tree": "38e1e80a966c749cb4d5c61c8eea55aafc31ad78", "parents": [ "17e36507ab1c71e8b3561282f6a6f4798d296ca6" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Thu Oct 08 21:52:11 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 08 21:52:11 2026 -0700" }, "message": "fix(spark): skip the catalog lookup in HoodieFileIndex unless the read declares a bucket index (#20240)" }, { "commit": "17e36507ab1c71e8b3561282f6a6f4798d296ca6", "tree": "5e19056ac0b64ed10f77b15a46712969d0e974b6", "parents": [ "43904a9336345cb1a855a17f33e3173b9e74ed35" ], "author": { "name": "Goutam Adwant", "email": "8672451+goutamadwant@users.noreply.github.com", "time": "Thu Oct 08 21:18:35 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 09 12:18:35 2026 +0800" }, "message": "fix(timeline-service): skip redundant refreshes for trailing cleans (#19946)\n\n* perf(timeline-service): avoid refreshing exact timeline extensions\n* refactor(timeline-service): check trailing clean before view lag\n\n---------\n\nCo-authored-by: danny0405 \u003cyuzhao.cyz@gmail.com\u003e" }, { "commit": "43904a9336345cb1a855a17f33e3173b9e74ed35", "tree": "e1253ce160e8007e9a29ae78d470a65d8834f0c9", "parents": [ "4f4c3571bfdce29753d455fed2e93b0ac95f1334" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Thu Oct 08 19:20:35 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 08 19:20:35 2026 -0700" }, "message": "fix(build): resolve same-module Scala classes when generating javadoc (#20238)" }, { "commit": "4f4c3571bfdce29753d455fed2e93b0ac95f1334", "tree": "63b78424a5c48fd16ae9e7d3518cce4b8cbc4eb2", "parents": [ "4cfba4ca4079c4abd09348f0826374db78c8ab35" ], "author": { "name": "fhan", "email": "aaron.han.1986@gmail.com", "time": "Fri Oct 09 09:51:26 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 09 09:51:26 2026 +0800" }, "message": "fix(spark): match SQL option names exactly when ignoring case (#20144)\n\nCo-authored-by: fhan \u003cyfhanfei@jd.com\u003e" }, { "commit": "4cfba4ca4079c4abd09348f0826374db78c8ab35", "tree": "03bf10fb3675c55e36956da5da2ffd6fe80f6b17", "parents": [ "a11dae45c7dc2615033694d31e325d240f24c6c9" ], "author": { "name": "Aditya Goenka", "email": "63430370+ad1happy2go@users.noreply.github.com", "time": "Fri Oct 09 03:03:41 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 08 14:33:41 2026 -0700" }, "message": "fix(HUDI-9666): read the table\u0027s timeline layout in upgrade_table, hudi-cli upgrade and rollback_to_instant (#20063)" }, { "commit": "a11dae45c7dc2615033694d31e325d240f24c6c9", "tree": "e02587f636f3f59e0362644d33c607387ae09846", "parents": [ "7f35e7eedcc002a5a1dd5f68b530a0911dc74d90" ], "author": { "name": "Prashant Wason", "email": "pwason@uber.com", "time": "Thu Oct 08 12:20:12 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 08 12:20:12 2026 -0700" }, "message": "perf(spark): enable vectorized reading and file splitting for COW incremental reads (#19914)" }, { "commit": "7f35e7eedcc002a5a1dd5f68b530a0911dc74d90", "tree": "c9d98c5947b0dbb18b1a07d57aa887ce8e43a55b", "parents": [ "7e4c944348137fb3ebb4805b331d1c90a9683efe" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Tue Oct 06 23:07:45 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 07 14:07:45 2026 +0800" }, "message": "test(spark): guard against executor .hoodie access and task payload growth on the read path (#20112)\n\n* test(spark): guard against executor .hoodie access and task payload growth on the read path\n\n* Replace the meta folder access recorder with a general file system call recorder\n\n* Enable the partition listing rows now that #20080 is merged\n\n* Move the guards that wait on open fixes out of this change and read every query in one meta folder test now that #20080 is merged\n\n* Guard what every read task deserializes with per-read-shape size budgets and against per-file Hadoop default resource parsing\n\n* Check the task size budget on version 6 COPY_ON_WRITE reads too\n\n* Read tables with a history of 50 commits and check the task size budget on COPY_ON_WRITE incremental and time travel reads\n\n* Keep MERGE_ON_READ log files uncompacted with the history, forbid any timeline per task, check that data skipping reads the column stats index, and drop unused recorder methods\n\n* Key the task size budgets on columnar and row reads measured on Spark 3.5 and Spark 4.0\n\n* Set the task size budgets per Spark and Scala binary version at about 1.1x the sizes measured on each\n\n* Prune files with data skipping in strict mode, time travel to the first commit, and check the task size budget on every query of both table versions and types\n\n* Check MERGE_ON_READ time travel against the row read budget and that time travel sees only the first commit\n\n* Keep the once per executor scan state apart from per-task streams, forbid timeline state per task, and record file system calls of cached file systems and task-started threads" }, { "commit": "7e4c944348137fb3ebb4805b331d1c90a9683efe", "tree": "d18d241a78d74d664da4eb3e25d6c81b4b0ddd5b", "parents": [ "b6fe7c3fd8b86d3d62b8bd74fdfcc1dfa183f316" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Tue Oct 06 19:16:28 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 06 19:16:28 2026 -0700" }, "message": "chore: fix javadoc and compiler warnings flagged in CI (#20215)" }, { "commit": "b6fe7c3fd8b86d3d62b8bd74fdfcc1dfa183f316", "tree": "e2079ec87fb1e63bc6084f1d96ce3520364f2699", "parents": [ "211ccba18c0f4f1041bd2016e725ab4a9106accd" ], "author": { "name": "Prashant Wason", "email": "pwason@uber.com", "time": "Tue Oct 06 19:15:30 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 07 10:15:30 2026 +0800" }, "message": "fix(metrics): handle stale DistributedRegistry across SparkContext restarts (#19790)\n\nDistributedRegistry is a JVM-wide singleton cached in Registry.REGISTRY_MAP and a\nSpark AccumulatorV2. After a SparkContext restart in the same JVM, register()\nthrew \"Cannot register an Accumulator twice\". Detect the stale state explicitly,\nregister a fresh accumulator that shares the stale instance\u0027s counters, and swap\nit into REGISTRY_MAP and HoodieSparkEngineContext\u0027s DISTRIBUTED_REGISTRY_MAP." }, { "commit": "211ccba18c0f4f1041bd2016e725ab4a9106accd", "tree": "84d211cdbeb92cc7ef491c7115ab857351b93a0e", "parents": [ "296f5d8283b9d6c576308a2bb78ddb7847fd090a" ], "author": { "name": "Prashant Wason", "email": "pwason@uber.com", "time": "Tue Oct 06 18:49:00 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 06 18:49:00 2026 -0700" }, "message": "fix(spark): avoid full partition listing in incremental reads (#19898)" }, { "commit": "296f5d8283b9d6c576308a2bb78ddb7847fd090a", "tree": "219ec7d189a33f0aac6fc69d52afeb3e273eb36d", "parents": [ "379f566530d0a73e865864a2a2878c2b5b94da42" ], "author": { "name": "Prashant Wason", "email": "pwason@uber.com", "time": "Tue Oct 06 17:03:36 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 06 17:03:36 2026 -0700" }, "message": "fix(spark): fix incremental file index reporting sizeInBytes\u003d0 (#19912)" }, { "commit": "379f566530d0a73e865864a2a2878c2b5b94da42", "tree": "f276e61beb53981fd95cf0fbdcc859f99fd9ca63", "parents": [ "ec2831ec932d3da193b15fee0164724913d33da5" ], "author": { "name": "Prashant Wason", "email": "pwason@uber.com", "time": "Tue Oct 06 16:51:39 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 06 16:51:39 2026 -0700" }, "message": "fix(Spark): Fix nested struct predicate pushdown for HoodieFileGroupReaderBasedFileFormat (#20029)" }, { "commit": "ec2831ec932d3da193b15fee0164724913d33da5", "tree": "69a10458c2b6d6c0d157418097289a1011b85f63", "parents": [ "77f5e021b8e58cad583689cc2c9f7849c2c093e4" ], "author": { "name": "Prashant Wason", "email": "pwason@uber.com", "time": "Tue Oct 06 16:50:07 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 06 16:50:07 2026 -0700" }, "message": "perf(spark): check file existence in parallel in MOR incremental fallback scan (#20211)" }, { "commit": "77f5e021b8e58cad583689cc2c9f7849c2c093e4", "tree": "df9e7c78f5ff989150e2284f62b907905e8ebcf0", "parents": [ "3b5bb09b1d365f5e31cf30a8dfdd0a48ec96b2b0" ], "author": { "name": "Prashant Wason", "email": "pwason@uber.com", "time": "Tue Oct 06 13:10:29 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 06 13:10:29 2026 -0700" }, "message": "fix(build): remove duplicate dependency declarations rejected by Maven 3.10 (#20212)" }, { "commit": "3b5bb09b1d365f5e31cf30a8dfdd0a48ec96b2b0", "tree": "2c367aeb20bd865a1e401d2bfe5fcef48533c96b", "parents": [ "862f4b9a5c92b82047ed40c8880708e4351bc5f2" ], "author": { "name": "Sivabalan Narayanan", "email": "n.siva.b@gmail.com", "time": "Mon Oct 05 22:54:44 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 06 05:54:44 2026 +0000" }, "message": "fix(writer): reject directory-traversal partition paths across all writers (#19342)\n\nfix(writer): reject directory-traversal partition paths across all writers\n\n A partition-path field value containing \"..\" segments could escape the\n table base path and write Hudi-managed files into arbitrary directories\n reachable by the writer\u0027s credentials. URL-encoding of partition values\n is opt-in and disabled by default, and nothing else validated the value.\n\n PartitionPathEncodeUtils.validateNoPathTraversal rejects whole \"..\"\n segments regardless of the urlencode setting. It runs in the write\n handles (HoodieWriteHandle, HoodieRowCreateHandle,\n HoodieRowDataCreateHandle), once per partition rather than per record,\n right before the partition directory is created.\n\n Only a whole \"..\" segment can escape, so 2024-01-01, region\u003dus-west-2\n and hive-style rider\u003d../evil are still accepted. Encoding instead of\n throwing was rejected: escapePathName leaves \".\" untouched, and encoding\n here would desync the directory from the partition path already recorded\n in the HoodieKey, markers and WriteStatus." }, { "commit": "862f4b9a5c92b82047ed40c8880708e4351bc5f2", "tree": "43b1d6205ae4366132ec0a16e58dd7e8772ffbee", "parents": [ "438c0d372569bb40138cc3046069ed405f75bdbe" ], "author": { "name": "Vinish Reddy", "email": "vinishreddygunner17@gmail.com", "time": "Tue Oct 06 09:45:13 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 06 12:15:13 2026 +0800" }, "message": "fix(metadata): key RLI and SI write stats by file group, not file id (#20188)\n\n* fix(metadata): key RLI and SI write stats by file group, not file id\n\nThe record level index and the secondary index grouped a commit\u0027s write\nstats by file id alone. A file group is identified by its partition and\nfile id, and for files written outside Hudi the file id is only unique\nwithin a partition: one task of a Spark partitionBy write leaves the same\npart-\u003ctask\u003e-\u003cjob uuid\u003e name in every partition it writes. The secondary\nindex then saw several base files under one key and failed with \"Only one\nnew parquet file expected per file group per commit\".\n\nGroup the write stats by HoodieFileGroupId in both indexers, and match the\nreplaced file groups the secondary index skips by HoodieFileGroupId too.\n\n* Address review feedback" }, { "commit": "438c0d372569bb40138cc3046069ed405f75bdbe", "tree": "388f8ccd83a515b0eded7b7981e7b12c3bea3902", "parents": [ "1c30d058c57978dd77abf423c6209579797d7c66" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Mon Oct 05 20:55:37 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 05 20:55:37 2026 -0700" }, "message": "fix(spark): group file slices by partition in the file index listing (#20115)" }, { "commit": "1c30d058c57978dd77abf423c6209579797d7c66", "tree": "560e64afde6343a641f305b30b2e99d581d79c17", "parents": [ "76da56fd81deea57502f1a08f5642fdcb7d18b0d" ], "author": { "name": "Y Ethan Guo", "email": "ethan.guoyihua@gmail.com", "time": "Mon Oct 05 20:54:21 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 05 20:54:21 2026 -0700" }, "message": "perf(spark): stop per-file Hadoop configuration parsing and copies on the parquet read path (#20079)" } ], "next": "76da56fd81deea57502f1a08f5642fdcb7d18b0d" }