)]}'
{
  "log": [
    {
      "commit": "5684b974c2bcefbd000a2671fc7b2856f9579240",
      "tree": "99bf098df5fc53e89dda215e6fc0be3713966dca",
      "parents": [
        "f066153277d96aa4346a4749781da8a357d87b8e"
      ],
      "author": {
        "name": "yujun",
        "email": "yujun@selectdb.com",
        "time": "Mon Nov 09 15:25:21 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Mon Nov 09 15:25:21 2026 +0800"
      },
      "message": "[fix](fe) Reject static partition assignments for OLAP tables (#70287)\n\nDoris accepts both partition-name lists and static partition assignments\nin INSERT statements. Static assignments such as\n`PARTITION(dt\u003d\u00272026-11-04\u0027)` are supported by external-table write\npaths; OLAP tables use partition names, or `PARTITION(*)` for automatic\noverwrite partition detection.\n\n### What problem does this PR solve?\n\nIssue Number: None\nRelated PR: #58396 (introduced static partition overwrite syntax)\nPR Type: Bugfix\n\nProblem Summary:\n\nAn INSERT OVERWRITE on an OLAP table accepts\n`PARTITION(dt\u003d\u00272026-11-04\u0027)` but silently discards the static partition\nassignments when creating the table sink. The resulting empty\npartition-name list is treated as a whole-table overwrite, so data in\nunrelated partitions is removed. Reject static partition assignments\nbefore constructing an OLAP sink, for both INSERT INTO and INSERT\nOVERWRITE, including remote Doris targets. The error directs users to\npartition names or automatic overwrite partition detection.\n\n### Details\n\n- **Frequency:** Reproducible whenever an OLAP INSERT OVERWRITE\nspecifies static partition assignments; ordinary INSERT INTO silently\nignores the same assignments.\n- **Root cause:** The parser preserves assignments separately from\npartition names, but both OLAP sink creation entry points ignore the\nassignments.\n- **Why existing tests missed it:** Parser tests verify that assignments\nare accepted, and external-table tests exercise supported static\npartition writes. They do not check that OLAP targets reject the\nunsupported syntax or preserve unrelated partitions.\n- **Fix:** Share an admission check between both OLAP sink creation\nentry points and raise an analysis error before any write or partition\nreplacement.\n- **Other paths and branches:** External connector sinks retain their\nstatic partition assignments. Partition-name inserts, partition-name\noverwrites, automatic partition detection, and explicit whole-table\noverwrites retain their behavior."
    },
    {
      "commit": "f066153277d96aa4346a4749781da8a357d87b8e",
      "tree": "bbecef4c66db80caf2399abcb7f6529c9e63fa08",
      "parents": [
        "41cd2b473e9d9bb26be089f302cdfdfe6e107eaf"
      ],
      "author": {
        "name": "924060929",
        "email": "lanhuajian@selectdb.com",
        "time": "Mon Nov 09 13:41:54 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Mon Nov 09 13:41:54 2026 +0800"
      },
      "message": "[fix](cache) Invalidate query caches when replaying table changes (#63612)\n\n### What problem does this PR solve?\n\nProblem Summary:\n\nA query cached on a follower or observer can keep returning an old\nschema after a table change. For example, a heavy `ALTER TABLE ... DROP\nCOLUMN` switches the base index without changing the table\u0027s data\nvisible version. The master\u0027s ALTER path clears its local SQL cache,\nwhile another FE replays the schema change and retains the old cached\nresult.\n\nThis change invalidates derived query caches while applying existing\nmetadata journals. Ordinary DML publishes table and partition visible\nversions, which SQL cache admission and lookup validate without scanning\nthe cache under transaction table write locks. It replaces the earlier\nimplementation of this PR that emitted a separate metadata-change\njournal.\n\n### Implementation\n\n`Env.onTableChanged` is the shared entry point. Every cache invalidation\nruns in its own `try/catch`, logs a failure, and allows the remaining\ncache invalidations to run. Checkpoint replay skips these local derived\ncaches.\n\n| Change | Invalidation |\n|---|---|\n| Table metadata | SQL result contexts, sorted partition ranges,\ndependent MTMV rewrite plans, and exact simple-aggregate caches |\n| Visible DML | Shared admission/lookup validation rejects changed\ntable-wide or partition dependencies; unaffected partition dependencies\nremain reusable. No full-cache scan during transaction publication or\nreplay |\n| Database/catalog namespace | Matching SQL, sorted partition, and MTMV\nrewrite caches, including cold external objects |\n\nThe hook runs at the existing mutation boundary: schema-change/rollup\ncompletion, table drop/replacement, and truncate. Transaction visibility\nretains its original partition-before-table version publication; data\nvalidation is performed when admitting or looking up a cache entry.\nRemaining replay handlers notify it after applying their mutation.\nRenames and replacements notify after applying the name mapping,\npreserving the original names and table IDs under the existing locks.\nBatch partition changes notify each affected table once.\n\nReplay coverage includes table/view creation and removal, recovery,\ncolumn/table/partition/rollup renames, heavy/light schema and index\nchanges, partition changes/replacement, table\nreplacement/engine/properties/distribution changes, administrative\nversion changes, constraints, MTMV changes, branch/tag operations, and\ndatabase/catalog changes. External refreshes reuse their existing\ninvalidation protocol and retain the row-count cache\u0027s final\ninvalidation barrier.\n\nSQL cache publication and lookup share dependency validation: local\nmetadata revisions, table identity/schema/data versions, view\ndefinitions, and external catalog properties. Revisions are scoped by\ntable ID/name, database and catalog; each statement retains its own\ndependency snapshot, with no registry of executing contexts. Publication\nuses a fresh statement and connector scope, restores the executing\nstatement in finally, and skips cache admission if validation fails.\nLookup also rejects a late put that races invalidation. The BE cache key\nincludes the base index/schema identity, and MTMV rewrite invalidation\nreuses the existing generation barrier.\n\nPartition-version validation constructs temporary partition storage only\nafter the metadata and unchanged-version fast checks.\n\nNo journal opcode, extra journal record, persisted payload, or FE/BE\nprotocol field is added. Approximate statistics and transaction-scoped\nauto-partition routing retain their existing lifecycles.\n\n### Release note\n\nFix stale cached query results on follower and observer FEs after table\nmetadata changes."
    },
    {
      "commit": "41cd2b473e9d9bb26be089f302cdfdfe6e107eaf",
      "tree": "b2ca0170e2cea88a938f5f3f29b1b1d7d763dfec",
      "parents": [
        "f1bc8b3690c9398b5286cfc10e8248d0e79b7ced"
      ],
      "author": {
        "name": "feiniaofeiafei",
        "email": "moailing@selectdb.com",
        "time": "Sat Nov 07 23:27:38 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Sat Nov 07 23:27:38 2026 +0800"
      },
      "message": "[fix](eager-agg) Disable bilateral eager aggregation for decimal SUM (#69093)\n\n### What problem does this PR solve?\n\nRelated PR: #63690 \n\nProblem Summary:\n\nBilateral eager aggregation can change the result type and scale of a\ndecimal SUM because its compensation uses ordinary decimal\nmultiplication. This can cause an output type mismatch or silently lose\nprecision; for example, a valid result of `0.0000000004` can become\n`0.000000`.\n\nThis PR disables bilateral eager aggregation when any aggregate is a\nDecimalV3 SUM. Single-sided decimal SUM pushdown remains supported.\n\nThis causes either:\n- An output type mismatch when an upper projection still expects the\noriginal SUM type.\n- Silent precision loss when the reduced-scale result reaches the query\noutput.\nFor example, a valid SUM of `0.0000000004` could become `0.000000`.\n\nThis Pr fix this problem:\n- Introduce an internal decimal multiplication expression that preserves\nthe SUM result type.\n- Cast the compensation input to the original SUM type when necessary.\n- Preserve multiplication statistics estimation and prevent arithmetic\nsimplification or FE constant folding from reverting to ordinary decimal\nmultiplication semantics.\n- Reuse the existing BE multiplication and overflow checks. No BE\nchanges are required."
    },
    {
      "commit": "f1bc8b3690c9398b5286cfc10e8248d0e79b7ced",
      "tree": "7d1c0abd0c55c0f018c0555d4b985fc52d044e14",
      "parents": [
        "d6c5a8da68853d2265cd6afd0656f27ccf6382ca"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Sat Nov 07 05:43:03 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Sat Nov 07 05:43:03 2026 +0800"
      },
      "message": "[test](regression) Enable retryIfHttpError for long S3 stream loads (#70312)\n\n### Background\n\nThe regression framework\u0027s `streamLoad {}` reads its body from the\nsuite\u0027s `file`. For an S3 URL it streams the object from S3 while the BE\nconsumes the request body, so if the BE stalls, the S3 connection goes\nidle, and S3 closes idle connections after about two minutes (#54613).\n`retryIfHttpError true`, added by #54613, wraps that read in\n`ResumableHttpInputStream`: on an I/O error it reopens the object with\n`Range: bytes\u003d\u003coffset\u003e-` and the same stream load continues. It never\nresends the load.\n\n### What problem does this PR solve?\n\nIssue Number: None\nRelated PR: #65337 (the branch-4.1 change by @shuke987, ported here),\n#54613 (added `retryIfHttpError`)\nPR Type: Test\n\nProblem Summary:\n\nThirty-five p1, p2 and docker suites stream large TPC-H, TPC-DS and\nsimilar files from S3 without that option, so a single S3 disconnect\nfails the whole suite. Branch-4.1 added the option to all 35 after its\nP2 case `test_base_compaction_no_value` turned flaky (#65337). This PR\nadds the same 45 lines on master: 34 files become identical to\nbranch-4.2, and `test_read_from_peer` gets the line in master\u0027s version\nof its block.\n\n### Details\n\n- **Failure:** No public data for master: no PR pipeline runs these\nsuites (31 are p2, 2 p1, 2 docker), and the public P1 pipeline last ran\non September 11. #65337 names the flaky branch-4.1 P2 case\n`test_base_compaction_no_value`.\n- **Cause:** The framework reads an S3 input only as fast as the BE\nconsumes the load body. When the BE stalls, for example under high CPU\nin CI, the input connection sits idle until S3 closes it, and the stream\nload fails with an I/O error on its body.\n- **Stability evidence:** With a local server that closes the first\nconnection partway through the file, a load with the option resumes at\nthe byte where the connection broke and loads every row exactly once;\nwithout it, the load fails and nothing is committed (see Testing).\nReopening is bounded (`maxRetries \u003d 3` per read), so an S3 object that\nstays unreachable still fails the suite.\n\n### Testing\n\nA script that walks each added line to its enclosing block shows that\nall 45 sit inside a `streamLoad {}` block whose `file` is an S3 URL\nunder `${getS3Url()}/regression/`, and all 35 files pass an offline\nGroovy parse. None of the 35 suites ran: they need multi-GB TPC-H and\nTPC-DS inputs from S3, HDFS, or docker clusters.\n\nTo check the option on master\u0027s framework, a temporary suite ran on a\nlocal single-FE, single-BE cluster built from master. It stream-loads a\n200,000-row CSV from a local HTTP server that closes the first\nconnection after 1,000,000 of 2,777,790 bytes and serves `Range`\nrequests:\n\n| `streamLoad` | Result |\n|---|---|\n| with `retryIfHttpError true` | passes: the framework logs `Premature\nend of Content-Length delimited message body (expected: 2,777,790;\nreceived: 1,000,000)` and reopens at offset 1000000, the server answers\n`bytes\u003d1000000-` with a 206, and the table then has 200,000 rows,\n200,000 distinct keys and `sum(k)` \u003d 20000100000, matching the file |\n| without it | fails: `ClientProtocolException` caused by the same\n`ConnectionClosedException`; the table stays empty |\n\n### Notes for Reviewers\n\n- The framework never caches these inputs, so the option takes effect\nwherever these suites run. `Config.enableCacheData` is a primitive\n`boolean`, so the `\u003d\u003d null` check in `fillDefaultConfig` never sets it\nto true, and the `-ECD` and `-CD` options are built but never added to\nthe parser. NonConcurrent build 1094305, for example, streamed all 58 of\nits S3 inputs directly. This PR leaves the framework as it is.\n- As the comment on `ResumableHttpInputStream` says, resuming can hide a\nBE that stalls a stream load for minutes; branch-4.x accepted that for\nthese suites in #65337.\n\nCo-authored-by: shuke \u003cshuke@selectdb.com\u003e"
    },
    {
      "commit": "d6c5a8da68853d2265cd6afd0656f27ccf6382ca",
      "tree": "d5d8e3007b8e5e7b15af8cf599649c38b96455e0",
      "parents": [
        "cab48ddc68abd4d75d3d8729ea9b6e9c66699c9a"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Sat Nov 07 05:42:23 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Sat Nov 07 05:42:23 2026 +0800"
      },
      "message": "[test](regression) Stabilize update_after_create_rollup and add test_writer_profile (#70309)\n\n### Background\n\n`query_p0/update/update_after_create_rollup` checks partial `UPDATE` on\na merge-on-write table that has a rollup. `ALTER TABLE ... ADD ROLLUP`\nstarts an FE job that sends one alter task per tablet to the BEs, and\n`SHOW ALTER TABLE ROLLUP` reports the job\u0027s state. Separately, a load\nthrough the move-memtable sink (`enable_memtable_on_sink_node`, on by\ndefault) can add a `DeltaWriterV2 \u003ctablet\u003e` and a `MemTableWriter\n\u003ctablet\u003e` child to its profile for each tablet; since #64620 the BE\nattaches them only when the session\u0027s `profile_level` is at least 2, the\ndefault.\n\n### What problem does this PR solve?\n\nIssue Number: None\nRelated PR: #65310 (the branch-4.1 fix of the rollup suite by @shuke987,\nported here), #59863 (added test_writer_profile on branch-4.0, by\n@0AyanamiRei), #65232 (pins its first half to profile level 1 on\nbranch-4.1, by @shuke987), #64620 (the master gate that\ntest_writer_profile covers), #64378 (made profile level 2 the default),\n#61842 (moved the rollup suite from nereids_p0 to query_p0)\nPR Type: Test\n\nProblem Summary:\n\n`update_after_create_rollup` sleeps 10 s after `ADD ROLLUP` and then\nrequires EXPLAIN of an unrelated GROUP BY to pick `rollup1`, so it fails\nwhenever the rollup job takes longer; as in #65310, it now waits until\nthe job reports FINISHED. `test_writer_profile`, which branch-4.x has\nand master lacks, checks that a load\u0027s profile hides the writer details\nat profile level 1 and shows them at level 2; master has had that gate\nsince #64620, with no test. The rollup suite becomes identical to\nbranch-4.2; `test_writer_profile` also waits for `COMPLETE` profiles and\nhonors TLS.\n\n### Details\n\n- **Failure:** `update_after_create_rollup` has not failed on its own in\nmaster\u0027s public pipelines: since June 30, P0 ran it 7,257 times and\ncloud_p0 739 times, and all 55 failures were in builds where most suites\nfailed. A forced slow rollup job reproduces it (see Testing).\n- **Cause:** The rollup job converts all 54 tablets on the BEs, so how\nlong it takes depends on the BE\u0027s alter task queue, while the suite\nallows a fixed 10 s. The EXPLAIN check also depends on the optimizer\nchoosing the rollup, which suites such as `rollup_p0/test_rollup_agg`\ncover.\n- **Stability evidence:** `getAlterRollupFinalState` polls `SHOW ALTER\nTABLE ROLLUP` every second for up to 300 s and returns the final state,\nso a CANCELLED job still fails. With each tablet\u0027s alter task delayed by\n5 s, master\u0027s suite fails at the EXPLAIN check and the ported one passes\nafter 91 s.\n\n- **Scenarios:** In `test_writer_profile`, an `INSERT INTO ... SELECT`\nat `profile_level\u003d1`, whose profile must contain neither `DeltaWriterV2`\nnor `MemTableWriter`, then the same INSERT at `profile_level\u003d2`, whose\nprofile must contain both; both checks wait for a `COMPLETE` profile,\nwhich the BE reports after the INSERT returns; cloud mode is skipped.\n- **What existing tests miss:** The closest suite,\n`query_profile/test_load_channel_profile`, turns\n`enable_memtable_on_sink_node` off and checks only that level 3 shows\nwriter details; no suite checks the level-1 gate of the move-memtable\nsink.\n- **Fails when broken:** Without `set profile_level\u003d1`, the first half\nfails with `should not contain DeltaWriterV2`, as a missing gate would\nmake it; with the second INSERT at level 1, the second half fails with\n`should contain DeltaWriterV2`.\n- **External dependencies:** Both INSERTs read `load/data by line.json`\nfrom the regression bucket through the `S3` TVF.\n\n### Testing\n\nAll runs used `./run-regression-test.sh --run -d query_p0/update -s\nupdate_after_create_rollup` or `--run -d query_profile -s\ntest_writer_profile` against a local single-FE, single-BE cluster built\nfrom master.\n\n| Suite | Version | Result |\n|---|---|---|\n| `update_after_create_rollup` | master | passes |\n| `update_after_create_rollup` | this PR | passes; the job was FINISHED\nabout 1 s after `ADD ROLLUP` |\n| `update_after_create_rollup` | master, with BE debug point\n`SchemaChangeJob::process_alter_tablet.leave.sleep` (5 s per tablet) |\nfails: `Explain and check failed, expect contains \u0027rollup1\u0027`, with the\njob RUNNING at 9/54 tasks |\n| `update_after_create_rollup` | this PR, same debug point | passes\nafter waiting 91 s |\n| `update_after_create_rollup` | this PR, with `CANCEL ALTER TABLE\nROLLUP` right after `ADD ROLLUP` | fails as it should: `expected:\n\u003cFINISHED\u003e but was: \u003cCANCELLED\u003e` |\n| `test_writer_profile` | this PR | passes; each profile was `COMPLETE`\nat the first read, the level-2 one has a `DeltaWriterV2` and a\n`MemTableWriter` child, the level-1 one neither |\n| `test_writer_profile` | this PR without #65232\u0027s `set profile_level\u003d1`\n| fails: `should not contain DeltaWriterV2 \u003d\u003d\u003e expected: \u003cfalse\u003e but\nwas: \u003ctrue\u003e` |\n| `test_writer_profile` | this PR, second INSERT at level 1 | fails as\nit should: `should contain DeltaWriterV2 \u003d\u003d\u003e expected: \u003ctrue\u003e but was:\n\u003cfalse\u003e` |\n\nMy local regression config\u0027s OSS key is rejected (`InvalidAccessKeyId`),\nso the `test_writer_profile` runs used a copy whose INSERT reads\n`numbers(\"number\" \u003d \"2\")` instead of the `S3` TVF; nothing else differs.\nThe PR\u0027s NonConcurrent pipeline runs that suite as committed, and P0 and\ncloud_p0 run the rollup suite.\n\n### Notes for Reviewers\n\n- After the CI review, `test_writer_profile` finds each INSERT\u0027s profile\nby a marker through `ProfileAction`, where branch-4.2 reads the newest\nLOAD profile over plain HTTP 500 ms after the INSERT.\n- Two more suites from the same branch-4.1 batch need nothing on master:\nmaster\u0027s renamed `test_jdbc_refresh_catalog_schema_refresh_non_blocking`\n(#65126) already has #65335\u0027s fixes, and master #65696 replaced #65234\u0027s\nsuite with an FE unit test.\n\nCo-authored-by: shuke \u003cshuke@selectdb.com\u003e\nCo-authored-by: Refrain \u003c113875799+0AyanamiRei@users.noreply.github.com\u003e"
    },
    {
      "commit": "cab48ddc68abd4d75d3d8729ea9b6e9c66699c9a",
      "tree": "07cca2e1ee56cbca2468601dc99b9960f2a644d2",
      "parents": [
        "8fd5e503cc01c9d5c843380588bb048bb688f4dc"
      ],
      "author": {
        "name": "Jack",
        "email": "jiangkai@selectdb.com",
        "time": "Fri Nov 06 22:55:29 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 22:55:29 2026 +0800"
      },
      "message": "[refactor](be) Add the CLucene-free text analysis interface and its two CLucene adapters (#70261)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #70260 (depends on)\n\nProblem Summary: Add a CLucene-free text analysis interface\n(`TextAnalyzer`, `TextTokenStream`, `TokenBatch`) that hands out tokens\nin batches, and adapters in both directions so that consumers and\nanalyzers can move to it separately. Nothing in production uses it yet."
    },
    {
      "commit": "8fd5e503cc01c9d5c843380588bb048bb688f4dc",
      "tree": "9bcffd62fe2e5dacf4111777c15a6b567401e0c4",
      "parents": [
        "4c63661f60f96c694757000640bcebc832e8bb6d"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Fri Nov 06 21:44:52 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 21:44:52 2026 +0800"
      },
      "message": "[test](be) Size two SNII spill-threshold tests to cross the cap with libc++ (#70322)\n\n### Background\n\nSNII is an inverted index storage format whose writer collects postings\nin memory in a `SpimiTermBuffer`, which charges its resident bytes to a\n`MemoryReporter`. Under the spill-threshold policy that index ingestion\nuses, the reporter\u0027s cap only triggers spills of the posting arena and\nnever fails an allocation, because the persistent vocabulary (term\nstrings, the intern set, the slot index) may exceed the cap by design.\nTwo cases in `snii_streamed_session_test.cpp` check that finalizing the\nindex and draining the buffer work, and are accounted exactly, while the\nvocabulary alone is above the cap.\n\n### What problem does this PR solve?\n\nIssue Number: None\nRelated PR: #66052 (added the two cases)\nPR Type: Test\n\nProblem Summary:\n\n`FinalizesAfterPersistentVocabularyCrossesSpillThreshold` and\n`TerminalDrainReleasesLookupStateWithoutUnderAccountingIds` fail on\nevery run of a BE unit test build that uses libc++, such as the macOS\nbuild, before they reach what they test: their precondition that the\nvocabulary is above the 384 KiB cap does not hold. They insert a fixed\n3000 terms, which is enough only with libstdc++, because the buffer\ncharges each owned term string by `sizeof(std::string)`: 32 bytes with\nlibstdc++, 24 with libc++. This PR raises the count to 4000, which keeps\nthe vocabulary above the cap with both libraries without letting the\nin-memory case spill. The accounting is right on both libraries, so\nproduction code is unchanged.\n\n### Details\n\n- **Failure:** With libc++, `ASSERT_GT(reporter.current_bytes(),\nkReporterCap)` fails in the first case (`actual: 371332 vs 393216`) and\n`ASSERT_GT(before_drain, kReporterCap)` in the second (`actual: 371188\nvs 393216`), on every run, in the iteration that forces a spill. The BE\nUT pipeline builds with libstdc++ on Linux, and `be-ut-mac.yml` builds\nthe BE on macOS without building or running the unit tests, so no\npipeline has seen it since #66052.\n- **Cause:** The buffer charges its owned vocabulary as\n`owned_vocab_.capacity() * sizeof(std::string)` plus heap payloads,\nwhich these short terms don\u0027t have because they fit the small-string\nbuffer. The capacity grows by explicit doubling and is 4096 for 3000\nterms with either library, so libc++ accounts 8 × 4096 \u003d 32,768 bytes\nless for the same input. After the forced spill frees the posting arena,\n371,332 bytes are left with libc++, under the 393,216-byte cap;\nlibstdc++ accounts 404,100 for the same input. The iteration that does\nnot spill keeps its arena and is above the cap with both.\n- **Stability evidence:** The failure is deterministic, not timing\nrelated. With 4000 terms both cases pass with libc++, where the forced\niteration accounts 421,332 bytes; libstdc++ accounts 32,768 bytes more\nfor the same input. The posting arena, which decides whether the\nin-memory iteration spills, does not depend on the library: it reaches\n64 KiB, below the 96 KiB (cap / 4) that a spill needs, so that iteration\nstill never spills.\n\n### Testing\n\n- `run-be-ut.sh --run --filter\u003d\u0027SniiStreamedWriterSessionTest.*\u0027`,\nlocally on macOS arm64 with ASAN (libc++): all 41 cases pass with this\nchange, and the two cases fail without it.\n- Two mutations of `spimi_term_buffer.cpp`, run locally against the new\ncount: raising MEM_LIMIT_EXCEEDED once the reporter is over the cap,\nwhich treats the spill threshold as an allocation limit, fails both\ncases; keeping the intern set and the string ranks through the terminal\ndrain fails the second case at `during_drain \u003c before_drain` (519,296 vs\n503,296), and leaves 208,000 bytes charged at the end of both.\n- These ran on `42cf4e3e5d1` plus this change; the SNII sources and\ntests are the same on `bb3156d30e6`, where this PR is based. The\nlibstdc++ run is left to the BE UT pipeline.\n\n### Notes for Reviewers\n\n- Why 4000: the count must stay at or below 4096, where the vector\ncapacities, and with them the margins above, stay as measured. 3500\nterms would leave only about 3 KB above the cap with libc++ (396,332\nbytes)."
    },
    {
      "commit": "4c63661f60f96c694757000640bcebc832e8bb6d",
      "tree": "f9dd7a68c9515016473fcbc52ba1b3b74fd51797",
      "parents": [
        "f7396d7229a26b421b07dde19f3be1dede8a7256"
      ],
      "author": {
        "name": "Mryange",
        "email": "yanxuecheng@selectdb.com",
        "time": "Fri Nov 06 16:52:10 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 16:52:10 2026 +0800"
      },
      "message": "[fix](fe) Quote dictionary data load identifiers (#67052)\n\nThis change picks #66594 onto master and adapts it to the latest\ndictionary loading code. Dictionary refresh previously concatenated raw\ncatalog, database, table, and dictionary names into an internal `INSERT\n... SELECT` statement executed by the background loader. Identifier text\ncontaining SQL syntax could therefore alter the parsed query structure.\nThe fix quotes and escapes every identifier component with the existing\nNereids helper and adds a parser-level unit test to verify that\nhostile-looking names remain a single source relation.\n\nCo-authored-by: Doris-Breakwater \u003c356893142@qq.com\u003e"
    },
    {
      "commit": "f7396d7229a26b421b07dde19f3be1dede8a7256",
      "tree": "1c3bbed194c3bde678655155700d1060e8a36ef7",
      "parents": [
        "f5c993f2c94acc5937a8e67c9659151251bdc941"
      ],
      "author": {
        "name": "Jack",
        "email": "jiangkai@selectdb.com",
        "time": "Fri Nov 06 16:03:50 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 16:03:50 2026 +0800"
      },
      "message": "[improvement](be) Remove per-term contention from SNII index build memory accounting (#70007)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #66052, #67657\n\nProblem Summary: Stream load into a SNII table stops scaling once flush\nthreads compete. On a log table with ten inverted indexes and mostly\nunique terms, SNII loaded at 314 MB/s against 854 MB/s for V2, with 3.0\ntimes the CPU and 9.8 times the kernel CPU. The adaptive flush pool is\non by default and allows up to 4 threads per core, so this is the normal\ncase on a busy BE. With the pool fixed at 6 threads SNII was 23% slower\nthan V2, and at 30 threads 60% slower.\n\nTwo process-wide hot spots cause it, and both are reached per term:\n\n1. `SpimiTermBuffer::report_arena_delta()` calls\n`GlobalMemoryLimiter::report()` whenever the resident total changes,\nwhich is once per new term. `report()` takes a process-wide mutex, looks\nthe writer up and reads the memory signals. The limiter keeps only the\narena size, which grows one 32 KiB block at a time, so nearly every call\nrepeats the same number. At 30 threads the flush threads blocked about\n476k times per second against 21k for V2, and every futex wait sampled\nwas on one address. That was 13.6 cores of kernel CPU against 3.5.\n2. The Reservations that #67657 added to the term encoders change\nseveral times per term, and each change updates two process-wide atomics\nthrough `snii_build_consume_release()`. That lambda took 54% of the user\nCPU of the flush threads.\n\nBoth scale with new terms and with threads, which is why duplicate data\nand a small pool look mild.\n\n- The limiter is now told only when the arena size changes, or when the\nresident total has moved by 1 MiB since its last decision. Nothing it\nstores is lost. The step catches capacity doublings that move the\nresident total by MiBs inside one arena block, and it is the step in\nwhich the decision\u0027s input moves anyway. On 2M tokens with 600k new\nterms a buffer now makes 321 decisions; before, every new term was one.\n- A reporter now holds its changes back from the two process-wide\ncounters until they add up to 1 MiB, and publishes the remainder\nwhenever its live bytes are back at zero, and when it is destroyed. One\nterm swings tens of KiB plus a ZSTD context, so per-term reservations no\nlonger reach the shared cache lines. The zero rule matters because a\nfinished text writer\u0027s reporter is not destroyed with the writer: its\n`IndexFileWriter` keeps it until the rowset writer closes.\n\n**What this costs.** The reporter itself stays exact, so per-writer\nbudgets and spill thresholds do not change. The process-wide counters\ncan lag each writer that is still building by less than 1 MiB:\n\n- `SniiIndexBuild` and `snii_index_build_mem_consumption` no longer show\na writer that stays below 1 MiB while it builds. They are exact again as\nsoon as a writer is empty, so a finished segment leaves nothing behind.\n- The forced-spill decision reads the sum, so it can start late by the\nlag of the writers still building: at most 1 GiB with 100 flush threads\non a 10-index table, against a share of at least 4 *\n`inverted_index_ram_buffer_size` (2 GiB by default). A writer below 1\nMiB could not be a victim anyway, since forced spills only pick arenas\nabove `snii_forced_spill_min_arena_bytes` (64 MiB by default), and the\nprocess-level backstops read real memory.\n\nThe index bytes do not change.\n\n| Throughput, MB/s, default adaptive flush pool | V2 | SNII | SNII vs V2\n| SNII CPU / V2 CPU |\n|---|---|---|---|---|\n| Log data, before | 854 | 314 | -62.9% | 3.01 |\n| Log data, after | 852 | **770** | -7.7% | 1.11 |\n| Short logs, before | 997 | 389 | -61.0% | 2.84 |\n| Short logs, after | 1003 | **865** | -13.7% | 1.20 |\n\nBoth binaries were built from master and alternated in one window, 4\nrounds each with the order flipped, and the table shows medians. One BE\npinned to 32 cores, local disk, stream load of JSON lines with 16\nconcurrent 100 MB batches. Log data is expanded from a real sample of\nservice logs, where 1% of the rows carry base64 blobs and make up 68% of\nthe bytes. Short logs leave those rows out. V2 is unchanged, as it\nshould be, and kernel CPU falls from 516 s to 43 s on the log data.\n\nAn earlier prototype with the same two changes also held up with other\nshapes and pools: fixed 30 threads -59.6% to -8.7%, fixed 6 threads\n-23.1% to -14.8%, 1 GB batches -43.8% to -13.7%, and data with few\nunique terms -20.2% to -11.7%.\n\nForced spills were checked on a real load: with\n`snii_index_build_max_memory_limit_percent \u003d 1`,\n`snii_forced_spill_min_arena_bytes \u003d 1 MiB`, and 16 concurrent 1 GB\nbatches, the spill directory held up to 224 files at once and the\nlimiter logged its shortfall warning. It stayed empty with the default\nshare. Eight MATCH, phrase and equality queries returned the same counts\nas V2 on the 11M rows, and the tracker went back to 0."
    },
    {
      "commit": "f5c993f2c94acc5937a8e67c9659151251bdc941",
      "tree": "dd6c0dd2f03e42c393637b60763902dbf43f0c09",
      "parents": [
        "59c49656240451ed1caab1e69e69bbe0be8c61aa"
      ],
      "author": {
        "name": "Mryange",
        "email": "yanxuecheng@selectdb.com",
        "time": "Fri Nov 06 15:24:45 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 15:24:45 2026 +0800"
      },
      "message": "[fix](be) Check timezone resolver results (#69985)\n\n### What problem does this PR solve?\n\nRelated PR: #67174\n\nSome BE callers ignore `TimezoneUtils::find_cctz_time_zone()` failures\nand continue with an unresolved timezone. Mark the resolver\n`[[nodiscard]]`, reject invalid external timezone values, and check\ninternal timezone invariants.\n\n### Release note\n\nInvalid timezone values in affected BE conversion paths now return an\nerror.\n\n### Check List (For Author)\n\n- Test: clang-format 16, `git diff --check`, and `bash\nbuild-support/check-build-hygiene.sh` passed. Build, unit tests, and\nregression tests have not been run.\n- Behavior changed: Yes. Previously ignored timezone resolution failures\nnow return errors or fail invariant checks.\n- Does this need documentation: No.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label"
    },
    {
      "commit": "59c49656240451ed1caab1e69e69bbe0be8c61aa",
      "tree": "2c7f1ba42c27369a7c42e80b89786f1a64b44293",
      "parents": [
        "bb3156d30e678d05a770aa741016b18a85f218e3"
      ],
      "author": {
        "name": "yiguolei",
        "email": "guolei@selectdb.com",
        "time": "Fri Nov 06 12:37:44 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 12:37:44 2026 +0800"
      },
      "message": "[fix](show command) some show command should not be forward to master (#69677)\n\n### What problem does this PR solve?\n\nIssue Number: close #xxx\n\nRelated PR: #xxx\n\nProblem Summary:\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test \u003c!-- At least one of them must be included. --\u003e\n    - [ ] Regression test\n    - [ ] Unit Test\n    - [ ] Manual test (add detailed scripts or steps below)\n    - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n        - [ ] Previous test can cover this change.\n        - [ ] No code files have been changed.\n        - [ ] Other reason \u003c!-- Add your reason?  --\u003e\n\n- Behavior changed:\n    - [ ] No.\n    - [ ] Yes. \u003c!-- Explain the behavior change --\u003e\n\n- Does this need documentation?\n    - [ ] No.\n- [ ] Yes. \u003c!-- Add document PR link here. eg:\nhttps://github.com/apache/doris-website/pull/1214 --\u003e\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label \u003c!-- Add branch pick label that this PR\nshould merge into --\u003e"
    },
    {
      "commit": "bb3156d30e678d05a770aa741016b18a85f218e3",
      "tree": "757d6c5efa16cf4e6a399f524de4aedf2c390176",
      "parents": [
        "d9d60bee9fea41273206052e7eba9b331c7f9959"
      ],
      "author": {
        "name": "minghong",
        "email": "zhouminghong@selectdb.com",
        "time": "Fri Nov 06 11:53:53 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 11:53:53 2026 +0800"
      },
      "message": "[fix](topn) Guard TopN filters across Repeat and AssertNumRows (#70194)\n\n### What problem does this PR solve?\n\nIssue Number: N/A\n\nRelated PR: #70100\n\nProblem Summary:\n\nA TopN runtime filter above `UNION ALL` can reach scans below operators\nthat synthesize NULLs. Another UNION branch can publish a bound before\nthose scans finish, causing incorrect results:\n\n- **Grouping sets:** rows rejected on their original grouping key can\nstill contribute to a grouping set that replaces that key with NULL.\nFiltering them before Repeat produces incomplete aggregate counts or\nloses the grand-total row. For one million input rows, `ORDER BY k NULLS\nFIRST LIMIT 1` can return `(NULL, 1024)` instead of `(NULL, 1000000)`.\n- **Scalar subqueries:** filtering before AssertNumRows can remove the\nscalar\u0027s only row, making it synthesize a NULL row instead. A scalar\nresult of `20` unioned with `10` can consequently become NULL and win\n`ASC NULLS FIRST`; `COALESCE(..., -1)` also fails with `NULLS LAST`.\nFiltering can additionally remove multiple rows and suppress a required\ncardinality error.\n\nOnly push a TopN filter through Repeat when every probe input slot is\ncommon to all grouping sets, using the existing\ncommon-grouping-expression helper. Stop pushdown at AssertNumRows to\npreserve its input cardinality and empty-input behavior. Keep filtering\non common grouping keys and unaffected UNION branches.\n\n### Release note\n\nFix incorrect TopN results for grouping sets and scalar subqueries, and\npreserve scalar-subquery cardinality errors when TopN runtime filters\nare enabled.\n\n### Check List (For Author)\n\n- Test\n    - [x] Regression test\n    - [x] Unit Test\n    - [x] Manual test\n    - [ ] No need to test or manual test\n\nValidation:\n\n- `./build.sh --fe -j16` — successful; zero Checkstyle violations.\n- `./run-fe-ut.sh --run\norg.apache.doris.nereids.postprocess.TopNRuntimeFilterTest` — 12 tests\npassed. The new Repeat test covers 36\ngrouping-set/expression/sort/NULL-order combinations. The Assert test\ncovers eight combinations and directly visits the assertion, with a\npositive control on its child, so it does not depend on CBO\u0027s\nscalar-join orientation.\n- `./run-regression-test.sh --run -d nereids_p0 -s\ntopn_filter_grouping_sets,topn_filter_assert_num_rows` — two suites\npassed, with 35 result checks and eight expected-cardinality-error\nchecks. Covers GROUPING SETS/ROLLUP/CUBE, common grouping keys, scalar\none-row/empty/multi-row inputs, COALESCE, and both NULL orders. Expected\noutput was generated by the regression runner.\n- The new Assert regression plan assertion fails on the unpatched FE:\nthe filter has two scan targets instead of only the unaffected UNION\nscan.\n- Manual reproduction confirmed incomplete grouping-set counts,\nsynthetic scalar NULLs, and suppressed scalar-cardinality errors before\nthe corresponding fixes.\n\nValidation used a single-BE native-OLAP cluster. FE was built from this\nbranch; BE reused an existing ASAN master build (`7acfd9bfc27`).\n\n- Behavior changed:\n    - [ ] No.\n- [x] Yes. Prevent unsafe scan filtering across Repeat and AssertNumRows\nwhile retaining safe targets.\n\n- Does this need documentation?\n    - [x] No.\n    - [ ] Yes.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label"
    },
    {
      "commit": "d9d60bee9fea41273206052e7eba9b331c7f9959",
      "tree": "842a411c4eea8cf33963c890e4a27f8c379e748b",
      "parents": [
        "dcf82859fd0f9d660f62d04168df2945c7bfd9cf"
      ],
      "author": {
        "name": "Yixuan Wang",
        "email": "wangyixuan@selectdb.com",
        "time": "Fri Nov 06 11:46:49 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 11:46:49 2026 +0800"
      },
      "message": "[fix](cloud) Add configurable GCS object deletion (#70119)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: None\n\nProblem Summary: Recycler previously used the generic S3 accessor for\nGCS. Add a GCS accessor that deletes objects individually for file lists\nand prefixes while preserving expiration filtering and missing-object\nhandling.\n\nThe enable_gcs_batch_delete_object startup configuration selects the GCS\naccessor and is enabled by default. Disabling it retains S3Accessor.\n\n### Release note\n\nGCS recycling uses individual-object deletion by default. Set\nenable_gcs_batch_delete_object\u003dfalse to retain the generic S3 accessor.\n\n### Check List (For Author)\n\n- Test: Not run, per user instruction; no build, formatting checks, or\nunit tests\n- Behavior changed: Yes, GCS uses GcsAccessor by default with an opt-out\nconfiguration\n- Does this need documentation: No\n\n### What problem does this PR solve?\n\nIssue Number: close #xxx\n\nRelated PR: #xxx\n\nProblem Summary:\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test \u003c!-- At least one of them must be included. --\u003e\n    - [ ] Regression test\n    - [ ] Unit Test\n    - [ ] Manual test (add detailed scripts or steps below)\n    - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n        - [ ] Previous test can cover this change.\n        - [ ] No code files have been changed.\n        - [ ] Other reason \u003c!-- Add your reason?  --\u003e\n\n- Behavior changed:\n    - [ ] No.\n    - [ ] Yes. \u003c!-- Explain the behavior change --\u003e\n\n- Does this need documentation?\n    - [ ] No.\n- [ ] Yes. \u003c!-- Add document PR link here. eg:\nhttps://github.com/apache/doris-website/pull/1214 --\u003e\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label \u003c!-- Add branch pick label that this PR\nshould merge into --\u003e"
    },
    {
      "commit": "dcf82859fd0f9d660f62d04168df2945c7bfd9cf",
      "tree": "facbe9edc0a73ffe4894c3ee68c6f77a748fb408",
      "parents": [
        "87b462e4f58104140f098f5f4a8b5d58bec70149"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Fri Nov 06 11:43:27 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 11:43:27 2026 +0800"
      },
      "message": "[test](regression) Stabilize test_auto_new_recycle and test_sql_digest_generation (#70308)\n\n### Background\n\nAn auto-partition table with `partition.retention_count \u003d N` keeps only\nits N latest historical partitions: the FE\u0027s `DynamicPartitionScheduler`\ndrops older ones on every run, once per\n`dynamic_partition_check_interval_seconds` (600 by default).\n`partition_p0/auto_partition/test_auto_new_recycle` lowers that interval\nto 1 s and checks the partition count after each load or ALTER.\n`audit/test_sql_digest_generation` checks that the audit log gets an\n`sql_digest` only for queries slower than the FE config\n`sql_digest_generation_threshold_ms`, once with the threshold at 0 and\nonce at 100000. Both suites are in group `nonConcurrent`.\n\n### What problem does this PR solve?\n\nIssue Number: None\nRelated PR: #65375 and #65646 (the branch-4.1 recycle-suite fixes by\n@shuke987, ported here), #65343 (the branch-4.1 digest-suite fix by\n@shuke987, ported here), #65435 (master\u0027s earlier move of the recycle\nsuite\u0027s day-boundary guard)\nPR Type: Test\n\nProblem Summary:\n\nBoth suites rely on timing the product does not promise. The recycle\nsuite checks the partition count after a fixed `sleep(8000)` (5 s after\nan ALTER), while the drop happens asynchronously at an unbounded time\nafter the config change; branch-4.1\u0027s NonConcurrent pipeline failed at\nthe first such check repeatedly (#65375). The digest suite\u0027s threshold-0\ncheck needs the query\u0027s elapsed time to be above 0 ms. This PR gives\nboth suites their branch-4.1/4.2 form: the recycle suite polls the\npartition count every second, for at most 1200 s, and widens its\nday-boundary guard to cover the poll; the digest query adds `sleep(1)`.\nBoth files become identical to branch-4.2.\n\n### Details\n\n- **Failure:** On master\u0027s public NonConcurrent pipeline neither suite\nhas failed on its own since mid-July: of about 8,000 runs each, the\nrecycle suite failed 3 times and the digest suite 87 times, all in\nbuilds whose cluster had died (most suites failing). The failures that\nmotivated the fixes are on branch-4.1\u0027s pipeline (#65375); #65343 made\nthe digest query deterministic there.\n- **Cause:** `ADMIN SET FRONTEND CONFIG` sets the scheduler\u0027s new\ninterval at once, but the scheduler sleeps in 5-second slices and\nnotices the change only at the next slice. It then runs\n`executeDynamicPartition` over every registered table before the drop.\nSo the drop lands 0 to 5 s plus one full pass after the config change,\nand a pass longer than about 3 s fails the 8-second check. In the digest\nsuite, `AuditLogHelper` writes a digest only when `elapseMs \u003e\nsql_digest_generation_threshold_ms`, and nothing bounds the elapsed time\nof `select *` on a two-row table from below (9 to 15 ms locally). With\n`sleep(1)` on each row it takes about 2 s, still far below the second\ncheck\u0027s 100000 ms.\n- **Stability evidence:** The recycle checks now wait for the drop and\nfail only when the count misses the target for 1200 s. The poll runs on\nthe suite thread, because the framework sets\n`Awaitility.pollInSameThread()` (#68338), so it does not leak\nconnections. Shortening one of master\u0027s waits to 1 s reproduces the\nbranch-4.1 failure shape, and the ported suite passes; mutants show that\nboth suites still catch wrong results (see Testing).\n\n### Testing\n\nAll runs used `./run-regression-test.sh --run -d\npartition_p0/auto_partition -s test_auto_new_recycle` or `--run -d audit\n-s test_sql_digest_generation` against a local single-FE, single-BE\ncluster built from master.\n\n| Suite | Version | Result |\n|---|---|---|\n| `test_auto_new_recycle` | master | passes |\n| `test_auto_new_recycle` | this PR | passes; the polls reached the\ntarget after 1, 2, 1, 3 and 3 s |\n| `test_auto_new_recycle` | master, with the wait before the\n10-partition check cut to 1 s | fails as #65375 describes: `expected:\n\u003c101\u003e but was: \u003c10\u003e` (master\u0027s `assertEquals` has its arguments swapped,\nso 101 partitions were still there) |\n| `test_auto_new_recycle` | this PR, expecting 11 partitions, poll\ncapped at 20 s | fails as it should: `ConditionTimeoutException` |\n| `test_sql_digest_generation` | master | passes |\n| `test_sql_digest_generation` | this PR | passes; the query took 2,024\nand 2,025 ms |\n| `test_sql_digest_generation` | this PR, second threshold 1000 ms |\nfails as it should: `Count of sql_digest changed` |\n\nThe PR NonConcurrent pipeline runs both suites; cloud_p0 skips\n`nonConcurrent` suites.\n\n### Notes for Reviewers\n\n- The day-boundary guard before the final `now()`-based section goes\nfrom 100 s to 1300 s, as in #65646, so that it covers the 1200 s poll\nplus the insert and checks. It sleeps only when that section would\notherwise start within 1300 s of midnight.\n- Four more suites fixed in the same branch-4.1 batch need nothing on\nmaster: #65340 (master #68449 enables the publish debug point on all\nFEs, superseding the open port #65556), #65929 (master #68329 fixed the\nactual cause, a connection leak), #65885 (master #69723 removed the V1\nscanner whose message it also accepted), and #67363 (ported as #67402).\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Behavior changed: No\n- Does this need documentation: No\n- Pre-submission status: Ready\n\n### AI Disclosure\n\n- Tool: Claude Code (Claude Opus 5.5)\n- Generated: the port (both suites taken from branch-4.2), the local\nverification runs, and this description\n- Human review: not stated\n- Prompts: The author asked to continue moving branch-4.2\u0027s changes back\nto master, where six branch-4.1 test fixes were recorded as one task.\nAfter the check showed that master already covers four of them and has\nnot failed on the other two, the author chose to port those two exactly\nas branch-4.2 has them and to verify them on a local cluster.\n\nCo-authored-by: shuke \u003cshuke@selectdb.com\u003e\nCo-authored-by: Claude Opus 5.5 \u003cnoreply@anthropic.com\u003e"
    },
    {
      "commit": "87b462e4f58104140f098f5f4a8b5d58bec70149",
      "tree": "b1231ee6598049bc315433ba900daa780a8f48ce",
      "parents": [
        "edf13ed60fad86273a77c1fe80025d13747ef485"
      ],
      "author": {
        "name": "Jack",
        "email": "jiangkai@selectdb.com",
        "time": "Fri Nov 06 11:19:00 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 11:19:00 2026 +0800"
      },
      "message": "[test](be) Freeze the output of every text analyzer and ANN index file (#70260)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: None\n\nProblem Summary: Add golden tests that freeze the output of every text\nanalyzer, of the index files they write and of the ANN index files, and\na tokenizer benchmark (be/benchmark/benchmark_analyzer.hpp). Also fix\nthree analyzer bugs found on the way: word_delimiter failing on a NUL\nbyte with a catenate option, an IK tokenizer leak on undecodable values,\nand IK dictionary loading that could not be retried after a failure."
    },
    {
      "commit": "edf13ed60fad86273a77c1fe80025d13747ef485",
      "tree": "6c9f72e9eaf1cd18f83bb815d9d4d2d37e692162",
      "parents": [
        "6969d59c128763bbe86f83648df340127245a751"
      ],
      "author": {
        "name": "Jack",
        "email": "jiangkai@selectdb.com",
        "time": "Fri Nov 06 11:09:37 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 11:09:37 2026 +0800"
      },
      "message": "[improvement](compaction) Seal compaction index files through the rowset level close (#69976)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #65858\n\nProblem Summary:\n\n#65858 fixed non-vertical compaction losing string indexes, but the fix\nmade the close path depend on one feature:\n\n- `SegmentFlusher::close()` asked `has_index_compaction()` before\nsealing.\n- `BaseBetaRowsetWriter::_close_file_writers()` added a collection total\nthat is only meaningful for deferred writers, and relied on it being\nzero for everything else.\n- The sequence of sealing, adding the size and waiting already existed\nin `_close_inverted_index_file_writers()`, which the vertical writer\nuses.\n\nClosing does not need to know about index compaction. It only has to\nseal the index files that are still open.\n\nThis PR makes three changes:\n\n1. **Closing no longer checks for index compaction.**\n`_close_file_writers()` always calls\n`_close_inverted_index_file_writers()` after the segments are closed.\n`InvertedIndexFileCollection::begin_close()` skips writers sealed when\ntheir segment was flushed, and `SegmentFlusher::close()` only closes\nsegments. The one remaining `has_index_compaction()` check is at flush,\nwhere the flusher has to know that postings are still to come.\n2. **`IndexFileWriter::open()` rejects a closed writer.** The original\nbug stayed silent because `open()` accepted a closed writer and the\ndirectory it returned never reached the sealed file. A mistake of this\nkind now fails the compaction instead of committing a rowset without its\nindex.\n3. **End to end test.** `test_index_compaction_non_vertical` covers this\npath, including a table with both numeric and string indexes. The\nexisting BE unit test\n`HorizontalCompactionSealsIndexesAfterPostingsAreWritten` is unchanged\nand passes.\n\nIndex size accounting is unchanged:\n\n- A writer sealed at flush is counted in the segment statistics and\nskipped by the collection, so it is not added twice.\n- A deferred writer is added by the helper, the same `_total_index_size`\nupdate as before.\n\n`finish_close()` keeps its position after the segment files are closed\nand the preload is done, so loads and compactions without index\ncompaction close in the same order as before.\n\nBackports: #69933 (branch-4.1) and #69936 (branch-4.2) carry the fix\nwith a check in `_close_file_writers()`. They do not have this cleanup\nyet.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test\n- [x] Regression test: in non-cloud mode, all 7 suites in\n`inverted_index_p0/index_compaction` pass, including the new\n`test_index_compaction_non_vertical`.\n- [x] Unit Test: 120 tests pass, covering `IndexFileWriterTest` with the\nnew `OpenAfterCloseFails` and `CollectionClosesOnlyOpenWriters`,\n`SegmentFlusherTransformFormatTest`, segment compaction,\n`BetaRowsetTest` and cloud compaction.\n- [x] Manual test: full compaction with vertical off on V2, V3 and SNII,\nwith string indexes only and with numeric plus string indexes. Index\nsize is above zero, matches the previous commit, and no index downgrade\nhappens.\n    - [ ] No need to test or manual test.\n\nCloud mode was not run for this change.\n\n- Behavior changed:\n    - [ ] No.\n- [x] Yes. Opening an index on a closed `IndexFileWriter` now returns an\nerror. No such caller exists after this change.\n\n- Does this need documentation?\n    - [x] No.\n    - [ ] Yes.\n\n🤖 Generated with [Claude Code](https://claude.com/claude-code)\n\n---------\n\nCo-authored-by: martin.foo838 \u003cmartin.foo838@gmail.com\u003e\nCo-authored-by: Claude Opus 5.5 \u003cnoreply@anthropic.com\u003e"
    },
    {
      "commit": "6969d59c128763bbe86f83648df340127245a751",
      "tree": "82dc9d70f5deb978346c96ce3d4d7e81a6fa8b25",
      "parents": [
        "9c48a8067f399996c9a36145fabf61dbca7695b2"
      ],
      "author": {
        "name": "lihangyu",
        "email": "lihangyu@selectdb.com",
        "time": "Fri Nov 06 11:09:10 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 11:09:10 2026 +0800"
      },
      "message": "[fix](regression) Stabilize variant schema change and SNII cases (#70196)\n\n## What problem does this PR solve?\n\nTwo VARIANT regression cases can fail despite correct results:\n\n- The schema-change double-write case compares JSON booleans against\nnumeric values in its expected output. The input contains booleans,\nwhich are preserved before and after schema change. Regenerate the\nexpected output and add a tie breaker to the limited query.\n- The SNII range case checks a cumulative filtered-row counter during\neach segment\u0027s initialization. When a segment that filters no rows runs\nfirst, it fails before the segment containing the filtered row is\nprocessed. Check the completed query profile instead, and write the test\ndata in separate rowsets to cover this ordering.\n\n## Release note\n\nNone\n\n## Check List (For Author)\n\n- Test: ASAN BE and FE build; `double_write_schema_change_with_variant`\nregression (generated expected output and ordinary comparison);\n`regression_test_variant_var_index_snii` regression (five consecutive\nruns). The original schema-change mismatch and a multi-rowset counter\nfailure were reproduced before the fixes.\n- Behavior changed: No product behavior change.\n- Does this need documentation: No."
    },
    {
      "commit": "9c48a8067f399996c9a36145fabf61dbca7695b2",
      "tree": "ede6f671d6187e0a6f809f41da6f7251dbc8014b",
      "parents": [
        "115d5c831b9a7aa59fd1c82b00ca740db85c9590"
      ],
      "author": {
        "name": "morrySnow",
        "email": "zhangwenxin@selectdb.com",
        "time": "Fri Nov 06 11:08:10 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 11:08:10 2026 +0800"
      },
      "message": "[refactor](type coercion) Centralize temporal coercion and cast policies (#69833)\n\nTIMESTAMP_NS and TIMESTAMPTZ handling is scattered across signature\nsearch, implicit conversions, CAST and expression rewrites. This FE-only\nrefactor gives the rules explicit owners and reuses the existing\ncoercion framework.\n\n- `TypeCoercionUtils` owns temporal common-type selection alongside the\nexisting numeric/date/time rules. NS promotion, mixed-family detection\nand comparison checks are private helpers near their callers;\nmixed-family detection traverses nested types once.\n- `TypeCoercionUtils.implicitCast(DataType, DataType)` resolves\nconversions from types alone; primitive conversion remains a private\nimplementation behind the recursive entry point. Merge the primitive\nwrapper and internal implementation, remove the catalog `matchesType`\nfallback and swallowed exceptions, and consult the existing Nereids CAST\nlegality check for remaining conversions. NS restrictions are ordinary\nterminal type rules, with no separate bypass around the fallback.\nSame-family temporal wildcards retain concrete input precision.\n- `characterLiteralTypeCoercion(String, DataType)` parses character\nvalues into candidate literals and may infer a wider type.\n`getExactTimeStampNsLiteralCastType(Expression, DataType)` separately\nadmits exact NS literal conversions and returns only a concrete target\ntype. It evaluates enclosing CASTs through normal FE semantics, checks\nliteral exactness, resolves wildcard precision and verifies the target\nCAST. Argument insertion uses type-only conversion first, then this\nNS-specific check, and constructs the CAST around the original\nexpression tree. There is no expression-based `implicitCast` overload or\nshared literal dispatcher.\n- `SearchSignature` keeps contextual temporal admission in the existing\nmatching rounds. Character parsing supplies a candidate type; NS\nadmission only checks whether the exact-cast target is present. An exact\nNS literal remains NS for matching, preserving native NS overload\npreference, character-literal preference and timezone scoring.\n- `TemporalConversionProperties` describes range narrowing, literal\nexactness and nullable output layouts. It neither evaluates expressions\nnor selects types/cast modes. Common-type and mixed-argument CAST\ninsertion share one private range-check helper in `TypeCoercionUtils`;\neach path delegates ordinary conversion to its existing matching\nfunction. The range check runs before literal promotion or subquery\nannotation. Partition pruning uses the same literal exactness API.\nExplicit CAST legality remains in `CheckCast`, and rewrite-specific\nguards stay in their owning rules.\n- `DateUtils.convertTimeZone` centralizes session-style timezone parsing\nand conversion for literal CAST and `CONVERT_TZ`, reusing the existing\nDST resolver. DATETIMEV2-to-TIMESTAMPTZ CAST uses\n`TimestampTzLiteral.fromTimeZone`; NS CAST retains its round-first,\nfraction-preserving path.\n- `AdjustableScaleTimeType` distinguishes promotable precision from\nfixed-scale NS. Inference pairs source/target slots in one traversal;\nnested precision replacement also traverses once.\n\nRemoving the permissive fallback also requires handling rejection at its\ndirect callers: `dict_get` and `dict_get_many` now reject unsupported\nkey coercions before constructing their custom signatures, instead of\ntreating the query\u0027s original type as the dictionary schema type. Legal\nkey conversions retain their existing normalization.\n\nStrict CAST failures propagate with the original error;\nTRY_CAST/non-strict NULL, nonconstant, unsupported and BE-only casts do\nnot prove exactness. Existing scalar and nested NS type restrictions\nremain in place. Mixed NS/date indexed-ANY arguments still infer NS in\nboth coercion modes; incompatible groups report all actual input types.\n\n### Release note\n\nPreserve normal CAST errors and evaluate nested CAST results when\nresolving temporal overloads for NS constants. Report the complete\ninput-type list for failed indexed-ANY inference, and reject unsupported\ndictionary key coercions during FE analysis."
    },
    {
      "commit": "115d5c831b9a7aa59fd1c82b00ca740db85c9590",
      "tree": "71eb8978d2dfe1dee9ed14da7910a6b004fd9669",
      "parents": [
        "8442911b266b1a5d01e9cf1f2c932ea778a28b1f"
      ],
      "author": {
        "name": "lihangyu",
        "email": "lihangyu@selectdb.com",
        "time": "Fri Nov 06 11:07:59 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 11:07:59 2026 +0800"
      },
      "message": "[fix](variant) Read exact sparse cells during path promotion (#70212)"
    },
    {
      "commit": "8442911b266b1a5d01e9cf1f2c932ea778a28b1f",
      "tree": "de5a31929e46fc727bb7ae3fc443fa8ea730a373",
      "parents": [
        "4f40ac9067d9f9de98e7ebbd1a6052527ece5c2b"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Fri Nov 06 02:16:40 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 10:16:40 2026 +0800"
      },
      "message": "[fix](fe) Close the FileIO of an Iceberg HiveCatalog that fails to initialize (#70164)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #66913, #66914\n\nProblem Summary:\n\n**In short.** This ports from branch-4.x #66913 the cleanup of an\nIceberg `HiveCatalog` whose initialization fails after it has created\nits FileIO: on master, nothing closes that FileIO. With the FileIOs\nmaster configures, the leaked object holds no native resource today, so\nthis is hygiene that keeps a FileIO with real resources from leaking\nonce per failed catalog creation.\n\n**Background.** `DorisHiveCatalog` (fe-connector-iceberg, used for\nIceberg on HMS) extends Iceberg\u0027s `HiveCatalog` and owns the FileIO the\ncatalog creates (#66914). In Iceberg 1.12, `HiveCatalog#initialize`\ncreates the FileIO first, then the Hive client pool, and only then the\n`CloseableGroup` that closes the FileIO, the KMS client and the metrics\nreporter. #66913, the branch-4.1 port of #66914, closes the FileIO when\ninitialization fails; master\u0027s `DorisHiveCatalog` does not.\n\n**The problem, and what it cost.** When `CachedClientPool` fails (an\nunknown element in `client-pool-cache-keys`, a `clients` that is not a\nnumber) or the metrics reporter fails to load (`metrics-reporter-impl`),\nthe catalog has a FileIO but no group: `HiveCatalog#close` closes\nnothing, and `DorisHiveCatalog` has not captured the FileIO yet. The\ncatalog properties reach Iceberg as the user gave them, and\n`IcebergConnector` caches only a catalog that initialized, so each\nmetadata access on such a catalog creates and drops another FileIO.\nToday that is only garbage: master\u0027s HMS catalogs use `HadoopFileIO`,\nwhose `close` does nothing, and `ResolvingFileIO` and `S3FileIO` create\ntheir clients lazily.\n\n**How this PR fixes it.** When `HiveCatalog#initialize` throws,\n`DorisHiveCatalog#initialize` closes the FileIO the catalog created and\nrethrows; a failure to close is added to the initialization failure as\nsuppressed. The FileIO is null when loading the FileIO itself failed,\nand then there is nothing to close. Not ported from #66913: its\n`close()` changes, since Iceberg 1.12 (#69031) closes the FileIO in\n`HiveCatalog#close` itself, and its `DorisHadoopCatalog`. The hadoop\nflavor has the same order (`HadoopCatalog` creates the FileIO before the\nlock manager), but master uses Iceberg\u0027s `HadoopCatalog`, and a subclass\nwould change the catalog-impl mapping without freeing anything today.\n\n**Results.**\n\n| `client-pool-cache-keys` with an unknown element | Before | After |\n|---|---|---|\n| `DorisHiveCatalog#initialize` | throws `IllegalArgumentException`, the\nFileIO stays open | throws `IllegalArgumentException`, the FileIO is\nclosed |\n\nCo-authored-by: 924060929 \u003clanhuajian@selectdb.com\u003e"
    },
    {
      "commit": "4f40ac9067d9f9de98e7ebbd1a6052527ece5c2b",
      "tree": "fc232587ffd5b394af04fbbb52bccaf501ab707b",
      "parents": [
        "716f193dbd67148adca32bb56cc7ba761f4e1c50"
      ],
      "author": {
        "name": "Gabriel",
        "email": "liwenqiang@selectdb.com",
        "time": "Fri Nov 06 10:14:53 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 10:14:53 2026 +0800"
      },
      "message": "[chore](build) Add native MongoDB client dependencies (#70249)\n\n### Background\n\nDoris loads FE connectors in isolated plugin directories, while BE\nnative clients are built through the shared third-party toolchain. A\nnative MongoDB connector needs the synchronous Java client for FE\nmetadata access and the C driver/BSON libraries for BE reads.\n\n### What problem does this PR solve?\n\nIssue Number: None\nRelated PR: None\nPR Type: Release/Dependency\n\nProblem Summary:\n\nPrepare the client dependencies and build outputs needed for native\nMongoDB catalog development. Build libmongoc and libbson together,\nexpose explicit BE CMake targets, and package an isolated FE dependency\nbundle with aligned MongoDB jars and optional compression providers. The\nbundle is staged as an archive until a connector provider is\nimplemented; no catalog or scanner is registered by this change.\n\n### Details\n\n- **Change map:** 256 added lines of build configuration/dependency\ndeclarations, 189 lines of license metadata/notices, and 28 lines of\nREADME. No runtime query code, tests, or generated sources are changed.\n- **Versions:** MongoDB C Driver/libbson 2.5.8; Java\nsync/core/BSON/record codec 5.14.0; snappy-java 1.1.10.10 pinned in the\nMongoDB module; zstd-jni 1.5.7-25.\n- **Upstream changes:** Pin the official [C driver\nrelease](https://github.com/mongodb/mongo-c-driver/releases/tag/2.5.8)\nand [Java driver\nrelease](https://github.com/mongodb/mongo-java-driver/releases/tag/r5.14.0).\nC Driver 2.5.8 includes cursor-response, compressed-reply and GridFS\nsecurity fixes; snappy-java 1.1.10.10 includes allocation,\nbuffer-boundary and stream-handling fixes. Java compression providers\nare optional SDK dependencies and are included explicitly so configured\ncompressor URI options work for FE metadata access as well as BE reads.\n- **Compatibility:** Use Doris-managed OpenSSL, Cyrus SASL, Snappy, zstd\nand zlib; enable SRV and bundled utf8proc. Native archives are\nPIC/static. Checked initial-cache paths and restricted pkg-config lookup\nprevent host dependencies from being selected. The Java BOM is\nmodule-local, logging remains host-provided, and the BE targets stay\noutside `COMMON_THIRDPARTY` until consumed by a reader.\n- **Rollback:** Revert the dependency registration and bundle staging;\nthis change introduces no persisted catalog state.\n\n### Testing\n\nLocally on Linux x86_64:\n\n- `bash thirdparty/build-thirdparty.sh -j 8 mongo_c_driver`: builds and\ninstalls both static libraries successfully.\n- A standalone CMake/C++ consumer links the actual BE `mongoc2`/`bson2`\ntargets and runs BSON/ObjectId, client lifecycle and\nsnapshot-session-options smoke checks. Required TLS/SASL/SRV/compression\nfeature macros are enabled.\n- An empty managed dependency prefix fails configuration instead of\nselecting host OpenSSL.\n- `mvn -f fe/pom.xml -pl :fe-connector-mongodb -am package -DskipTests\n-Dskip.doc\u003dtrue -Dmaven.build.cache.enabled\u003dfalse`: succeeds, including\nFE Checkstyle and the connector import gate. No module unit tests are\nclaimed.\n- Inspect the produced archive: exactly the six intended runtime jars,\naligned MongoDB versions, and no logging binding. Run Java raw\nBSON/read-options smoke checks and Snappy/Zstd JNI compression round\ntrips against those jars.\n- Execute the added `build.sh` staging block and verify the copied\narchive hash and absence of an auto-loaded MongoDB plugin directory.\n- A local negative-input smoke check reproduces `OutOfMemoryError` with\nsnappy-java 1.1.10.5 using a five-byte input declaring an oversized\nuncompressed length; the same check against the packaged 1.1.10.10 jar\nrejects the input without that allocation. The JVM is capped at 64 MiB\nfor both checks.\n- Verify the C source archive against the official release asset\nSHA-256, its configured MD5, and the unchanged upstream third-party\nnotices.\n- Shell syntax and `git diff --check` pass.\n\nThe full `build.sh --fe` build was not completed: the existing\nthird-party installation lacks `libpaimon_c.a` and\n`paimon_rust/paimon.h`, so its preflight starts an unrelated full\nthird-party rebuild. That rebuild was stopped; the scoped validations\nabove completed. Other platform and full-build coverage remain for CI.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Behavior changed: No query behavior change; MongoDB client dependency\nartifacts are added to build outputs.\n- Does this need documentation: No user-facing documentation yet; the\ndependency module README explains the preparation stage.\n- Pre-submission status: Partially verified: dependency builds,\npackaging and smoke checks pass; full FE build and other platforms\nremain for CI.\n\n### AI Disclosure\n\n- Tool: OpenAI Codex.\n- Generated: Build configuration, dependency module, README, and this\ndescription. License notices are copied from upstream.\n- Human review: Pending maintainer review; agent self-review completed."
    },
    {
      "commit": "716f193dbd67148adca32bb56cc7ba761f4e1c50",
      "tree": "7c27c32db1d6a91c680f5cb3c9bae67e5b564121",
      "parents": [
        "32accf9301cb2db4f3197287f54971e2a45787dd"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Fri Nov 06 02:04:02 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 10:04:02 2026 +0800"
      },
      "message": "[test](regression) Fix p1 suites agg_sync_mv and test_left_anti_join_batch_size (#70306)\n\n### Background\n\n`nereids_syntax_p1/mv/aggregate/agg_sync_mv` creates one synchronous\nmaterialized view per aggregate function on the same base table and\nchecks that queries are rewritten to it. An unaliased aggregate in a\nsync MV gets a generated column name such as `__stddev_1`, and a new\nMV\u0027s column names must not repeat any column of the table\u0027s full schema,\nwhich includes the columns of its existing sync MVs.\n`tpch_unique_sql_zstd_bucket1_p1/sql/test_left_anti_join_batch_size.sql`\nruns a `NOT EXISTS` query over TPC-H `lineitem` with the hint\n`SET_VAR(batch_size\u003d3)`, so that the left anti join sees many small\nblocks; the suite\u0027s `load.groovy` creates the tables in the suite\ndatabase `regression_test_tpch_unique_sql_zstd_bucket1_p1`.\n\n### What problem does this PR solve?\n\nIssue Number: None\nRelated PR: #65336 and #65338 (the branch-4.1 fixes by @shuke987, ported\nhere), #69446 (the branch-4.2 follow-up by @mrhhsg, ported here), #56986\n(moved the TPC-H unique suite from p0 to p1), #65538 (fixed the suite\u0027s\nsecond load on master)\nPR Type: Test\n\nProblem Summary:\n\nNeither suite can pass on master. `agg_sync_mv` stops at `mv_sync41`\nwith `Duplicate column name \u0027__stddev_1\u0027`, because `stddev(kint)` and\n`stddev_pop(kint)` are one function and get one generated name;\n`variance` / `var_pop` and `variance_samp` / `var_samp` collide the same\nway. Past that point its last check compares `map_agg` output with a\nstale golden. `test_left_anti_join_batch_size` reads\n`regression_test_tpch_unique_sql_zstd_bucket1_p0.lineitem`, a database\nthat no suite creates since #56986, and its hint is misspelled\n`SEV_VAR`, which Doris ignores. This PR ports the branch-4.1 fixes\n(aliases, the suite\u0027s own `lineitem`, `SET_VAR`) and branch-4.2\u0027s\norder-independent rewrite of the last check.\n\n### Details\n\n- **Failure:** `agg_sync_mv` fails on every run at `mv_sync41` with\n`errCode \u003d 2, detailMessage \u003d Duplicate column name \u0027__stddev_1\u0027`. With\nthe aliases alone it fails at its last check, `qt_test`: the golden\nexpects `{}` for the `kbint \u003d NULL` group, master returns\n`{null:\"null\"}`, and the `kbint \u003d 1` group now holds two entries\n(`{null:\"string1\", 0:\"string1\"}`). `test_left_anti_join_batch_size`\nfails with `Database [regression_test_tpch_unique_sql_zstd_bucket1_p0]\ndoes not exist` on any cluster that has not kept a database from the old\np0 suite.\n- **Cause:** In `agg_sync_mv`, `stddev_pop`, `stddev`, and `std` bind to\nthe `Stddev` function, `var_pop` and `variance` to `Variance`, and\n`var_samp` and `variance_samp` to `VarianceSamp`, so the unaliased\noutputs of each pair get the same name, and `MaterializedViewHandler`\nrejects a name already in the table\u0027s full schema. The `qt_test` golden\npredates `map_agg` keeping NULL keys, and a map with two entries prints\nthem in no guaranteed order. In the anti-join case, the database name is\nleft over from when the suite lived in p0, and `SEV_VAR` is not a hint\nname, so `batch_size` keeps its default (`SELECT\n/*+SEV_VAR(batch_size\u003d3)*/ @@batch_size` returns 8160; with `SET_VAR` it\nreturns 3).\n- **Stability evidence:** The failures were deterministic, and the fixes\nremove their causes: the aliases make the generated names irrelevant,\nthe rewritten `qt_test` checks the map size, NULL-key presence, and the\nvalue for each key instead of the map\u0027s text, and the query reads the\ndatabase its own suite loads. Results of the local runs are under\nTesting.\n\n### Testing\n\nAll runs used `./run-regression-test.sh --run -d \u003cdir\u003e -s \u003csuite\u003e`\nagainst a local master-based cluster.\n\n| Suite | Version | Result |\n|---|---|---|\n| `agg_sync_mv` | master | fails at `mv_sync41`: `Duplicate column name\n\u0027__stddev_1\u0027` |\n| `agg_sync_mv` | with the aliases only | fails at `qt_test`: expected\n`{}`, got `{null:\"null\"}` |\n| `agg_sync_mv` | this PR | passes |\n| `test_left_anti_join_batch_size` | master | fails: database\n`regression_test_tpch_unique_sql_zstd_bucket1_p0` does not exist |\n| `test_left_anti_join_batch_size` | this PR | passes against the\nexisting `.out` |\n\nThe new `qt_test` block in `agg_sync_mv.out` was generated by the\nframework (`-forceGenOut`) on master and equals branch-4.2\u0027s block. The\nrest of the regenerated file differs from the committed one only in how\ndoubles print (`0` instead of `0.0`), which the framework compares by\nvalue, so those blocks are left unchanged, as #69446 did. For the\nanti-join suite, the full TPC-H fixture download was too slow locally,\nso `lineitem` held only the rows with `l_orderkey \u003c 10000`, which are\nall the query reads, taken from the start of the sf0.1 fixture file\n(sorted by `l_orderkey`).\n\n### Notes for Reviewers\n\n- The second commit carries branch-4.2\u0027s #69446; without it\n`agg_sync_mv` still fails on master at its last check. Apart from that\nblock, the `.out` stays as on master, including\n`select_group_array_intersect` for the NULL group (`\\N [1]`, from\n#66284), where branch-4.2 has `\\N []`.\n- `mv_sync42` (`stddev_samp`) has no synonym MV in the suite; #65336\naliased it too, and this PR keeps that for consistency.\n\nCo-authored-by: shuke \u003cshuke@selectdb.com\u003e\nCo-authored-by: Jerry Hu \u003chushenggang@selectdb.com\u003e"
    },
    {
      "commit": "32accf9301cb2db4f3197287f54971e2a45787dd",
      "tree": "4dfeaa8367a919c85a6dd7bdcded1f127251ab0d",
      "parents": [
        "7d5e00004e80a857e4770f9ba1062bd1d2efcae6"
      ],
      "author": {
        "name": "TengJianPing",
        "email": "tengjianping@selectdb.com",
        "time": "Fri Nov 06 09:55:52 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 09:55:52 2026 +0800"
      },
      "message": "[improvement](topn) Refresh TopN runtime filter page ranges (#70020)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nProblem Summary:\n\nSegment readers keep row ranges computed before a TopN runtime bound\narrives or tightens. Scan-key normalization can also remove a shared\nruntime predicate on a leading storage key. Readers then miss later\nbound updates.\n\nPublish versioned predicate snapshots and use page Zone Maps to refresh\nunread row ranges between batches. Apply the first bound immediately and\nthrottle later refreshes by the number of raw rows read. Preserve scan\nprogress, bitmap holes, and read direction. Refresh the bitmap cursor,\ncached cardinality, and filter counters together. Keep shared runtime\npredicates during static scan-key normalization while still removing\nstatic predicates covered by scan keys.\n\nIn the existing ascending TopN check with 1,048,576 rows and three sort\nkeys, ScanRows fell from 1,048,576 to 16,384 and TotalPagesNum fell from\n372 to 6. These figures compare filtering disabled and enabled on the\nsame patched build. The query results were identical.\n\n### Release note\n\nTopN scans can skip unread pages when runtime bounds arrive or tighten.\nRuntime predicates remain active after scan-key normalization.\n\n### Check List (For Author)\n\n- Test: Manual checks of commit boundaries, dependencies,\ngenerated-result preservation, whitespace, clang-format v16, and header\nhygiene. Include BE unit tests and two TopN regression suites. Builds\nand tests were not rerun for this commit split.\n- Behavior changed: Yes. TopN scans can read fewer rows and pages. This\ncommit retains inclusive runtime bounds.\n- Does this need documentation: No\n\n### What problem does this PR solve?\n\nIssue Number: close #xxx\n\nRelated PR: #xxx\n\nProblem Summary:\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test \u003c!-- At least one of them must be included. --\u003e\n    - [ ] Regression test\n    - [ ] Unit Test\n    - [ ] Manual test (add detailed scripts or steps below)\n    - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n        - [ ] Previous test can cover this change.\n        - [ ] No code files have been changed.\n        - [ ] Other reason \u003c!-- Add your reason?  --\u003e\n\n- Behavior changed:\n    - [ ] No.\n    - [ ] Yes. \u003c!-- Explain the behavior change --\u003e\n\n- Does this need documentation?\n    - [ ] No.\n- [ ] Yes. \u003c!-- Add document PR link here. eg:\nhttps://github.com/apache/doris-website/pull/1214 --\u003e\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label \u003c!-- Add branch pick label that this PR\nshould merge into --\u003e"
    },
    {
      "commit": "7d5e00004e80a857e4770f9ba1062bd1d2efcae6",
      "tree": "bb0f9ca86734c63012a80aaa1fc6e81018f7e1a8",
      "parents": [
        "0d1ae9ee0646a068851da51d32ed9f6bfd0d8e7d"
      ],
      "author": {
        "name": "Gabriel",
        "email": "liwenqiang@selectdb.com",
        "time": "Fri Nov 06 09:55:21 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 09:55:21 2026 +0800"
      },
      "message": "[feature](iceberg) Add native encrypted Parquet reads (#70248)\n\n### Background\n\nDoris has per-file Iceberg encryption parameters and a native Parquet\nreader, but that reader cannot consume encrypted footers or pages.\nParquet modular encryption authenticates each module with file AAD and\nrow-group, physical-column and data-page ordinals.\n\n### What problem does this PR solve?\n\nIssue Number: #70245\nRelated PR: #70087 (PME dependency build), #70192 (Iceberg metadata\nreads), #70213 (file-key wire contract), #70246 (per-file key\nextraction)\nPR Type: Feature\n\nProblem Summary:\n\nAdd the physical reader needed to read AES_GCM_V1 Parquet files with\nencrypted footers and footer-key columns. Authenticate and decrypt\nmodules through parquet-cpp before passing bytes into the existing\nnative metadata and value decoders. Java/Iceberg interoperability,\nbounded parsing and cache isolation require changes beyond the crypto\nwrapper. Enforce the native container budget independently of unverified\nserialized element types, so malformed authenticated metadata cannot\nundercount its allocation cost. Encrypted SQL admission remains closed\nwhile index, delete-file and final scan integration are completed\nseparately.\n\n### Details\n\n- **Change map:** 1300 production lines changed, 1420 test/generator\nlines, 57 generated result lines, 81 documentation lines, 9 scanner\nconfiguration lines and 4 small synthetic binary fixtures/twins/copies\n(about 10 KB total). Production changes comprise module crypto/parsing,\nfooter ownership, native page/column plumbing and sequential scan\nguards. The native column factory now takes four arguments through a\ncreation context and explicit read options, with separate complex-type\nhelpers and one scalar initialization path. Tests cover independent\nJava/Iceberg interoperability, validation/authentication boundaries and\nreal S3 SQL scans.\n- **Scope:** AES_GCM_V1 encrypted footers, encrypted page headers,\ndictionary pages and Data Page V1/V2; 128/192/256-bit footer keys;\nscalar and nested physical decoding. AES_GCM_CTR_V1, plaintext-footer\nencryption and separate column keys are explicitly unsupported.\n- **Design:** Keep physical decoding in format_v2. The crypto wrapper\nuses parquet-cpp AES and module-AAD helpers, retains an immutable\nSecureString file key, and creates mutable decryptors per physical\ncolumn reader. Track ciphertext disk sizes separately from decrypted\ncompressed sizes and count only data pages in AAD ordinals. Authenticate\nmetadata before accepting Parquet Java\u0027s zero-filled serialization\npadding. Bound encrypted modules and decompressed pages to 64 MiB and\nimpose Thrift container, recursion and aggregate allocation limits\nbefore decoding. Bypass shared decrypted metadata/page/condition caches\nand index/dictionary/bloom pruning for the sequential stage. Recursive\ncolumn construction shares immutable file/row-group inputs by reference,\nwhile keeping in_collection local to each node and mutable decryptors\nlocal to each physical leaf. Pass cache policy directly instead of\ncreating a temporary RuntimeState to override it.\n- **Metadata budget:** Generated Thrift readers use statically declared\nC++ vector elements without validating the serialized element type.\nCharge each list/set element and both sides of each map entry at the\nlargest reachable native element size. This is a conservative budget;\nvery large primitive/string lists may reach the limit sooner. No schema\nmirror or generated source change is needed.\n- **Scratch reclamation:** Include decrypted page capacity and active\nbytes in the existing scalar scratch policy. A page that grows to 32 MiB\ncan relinquish its unused capacity after ordinary small batches, while\nrelease waits for a page boundary so live payload and V2 level views\nstay valid. The existing idle thresholds remain unchanged. Page activity\nis tracked separately from vector size, preserving initialized bytes for\nsame-size reuse without an additional full-page initialization.\n- **GCC unity builds:** Keep the shared secure-key implementation type\nin a named namespace. The externally linked PImpl types must not contain\nan anonymous-namespace type when this source is included by a unity\ntranslation unit.\n- **Usage:** Controlled physical-reader callers can supply the existing\nTFileEncryptionInfo as an optional constructor parameter. No new engine\nconfiguration, Thrift field, capability negotiation or user SQL entry is\nadded. Encrypted indexes and delete-file integration remain follow-up\nwork.\n- **Default behavior:** Existing plaintext callers retain default\nparameters and their existing cache/pruning paths. FE encrypted-table\nadmission and BE scan dispatch guards remain closed.\n- **Compatibility:** Reuses the existing descriptor. PME requires the\ninternal headers and actual encryption implementation from the\npreviously enabled third-party build; older prebuilt bundles compile and\nexplicitly report PME_DEPENDENCY_UNAVAILABLE. No storage format or\ndependency version changes.\n- **Behavior change:** The native physical reader can decrypt the\nsupported layout when explicitly given file parameters; encrypted files\nwithout keys and unsupported layouts fail explicitly. This PR does not\nenable encrypted SQL queries.\n\n### Testing\n\n- Latest full local ASAN BE build through `build.sh --be -j 6` passed,\nincluding the GLIBC_2.17 compatibility gate.\n- Latest final run: 249 focused ASAN tests passed across ten\nParquet/native-decoder, page-cache and existing\nfile-encryption/scan-range guard suites, including all 24 encryption\ntests. Used `run-be-ut.sh` with a local CMake test-source filter;\nproduction libraries were built normally. This is not the full BE\nunit-test suite.\n- Decrypted-page scratch red/green coverage: the new real-reader test\nfails before the repair because a 32 MiB page is absent from retained\nscratch accounting. After the fix, Page V1/V2 x UNCOMPRESSED/SNAPPY all\npreserve active page data and NULL values, defer release while page\nviews remain live, and reduce retained scratch to at most 4 MiB after\nsmall-page transitions. SNAPPY cases assert that decompression actually\nruns.\n- GCC 15.1 unity compilation: a syntax check that includes the actual\nproduction source reproduces both subobject-linkage errors before the\nfix and passes after it. This is a focused compiler check, not a full\nGCC release build.\n- Allocation-budget red/green coverage: the new real-parser regression\nfails on the old implementation and passes after the fix for\nbyte-labeled LIST/SET/MAP containers. It uses sub-MiB inputs and\nskippable unknown fields to validate cumulative accounting without\nallocating large native vectors. The MAP threshold detects accounting\nfor only one side of an entry. Existing Java/Arrow fixtures cover\nsuccessful normal metadata decoding.\n- Factory refactor coverage: all four collection/offset-index\ncombinations read the selected middle-page rows after the temporary\ncreation context is destroyed. Explicit page-cache policy is verified\ncold/warm with no RuntimeState and with both session settings, including\nopposing settings. Both new tests also pass twice in one process.\nExisting nested projection and INT96 timezone tests continue to pass.\n- Positive physical-reader coverage: dictionary/plain encoding, Data\nPage V1/V2, three AES key sizes, multiple pages and row groups,\nnull/empty nested shapes, levels-only reads and the ParquetReader block\ninterface. The checked-in Iceberg 1.12.0 Java sample is compared\nfield-by-field with the independent parquet-cpp reader.\n- Boundary coverage: descriptor fields, unsupported ciphers/plaintext\nfooters/column keys, AAD policy and authentication,\nmodule/row/column/page AAD binding, ordinal/length limits,\ntampering/truncation, another key after cache warmup, independent\ndecryptor state and avoiding encrypted index reads. Errors do not expose\nkeys or AAD.\n- Added `test_iceberg_encryption_native_reader`: on a local cluster with\na freshly built PR ASAN BE and an existing compatible FE, generation and\na separate normal comparison run both passed; the normal comparison also\npassed against the refactored BE. Each run checks 21 expected errors and\nseven positive result sets/aggregates across page-cache and\nCOUNT-pushdown settings. A plaintext first file allows schema discovery;\nactual mixed-file SELECT/COUNT must still reject the later encrypted\nfile with MISSING_KEY. Repeated failures and a truncated input preserve\nsubsequent plaintext nested values/counts. The suite needs standard\nMinIO, without KMS or a new encryption-specific test flag, and deletes\nits unique bucket/objects. This SQL regression was verified earlier on\nthe PR and was not rerun for the metadata-budget, GCC linkage or\nscratch-reclamation updates.\n- Before implementation, the SQL missing-key assertion fails on the old\nBE with Invalid Parquet v2 footer magic. The initial missing-key unit\ncontract also failed before implementation.\n- Positive crypto tests use PME rebuilt with the Doris toolchain.\nOriginal prebuilt dependencies previously passed four explicit\ndependency/metadata contracts and rejected decryption.\n- Gitleaks 8.30.0 reproduced the reported three fixture false positives;\nthe complete PR commit history, including the budget fix, has zero\nfindings. The exact-path/exact-value exception still detects a different\nvalue in that file and all three original occurrences outside that path.\n- Fixture generator compilation, clang-format 16, changed-line\nclang-tidy, build hygiene and source whitespace checks passed. SQL\nresults are runner-generated; SQL binary copies byte-match their BE\noriginals.\n- Additional diagnostic: repeating the whole selection in one process\npasses the first round, then two unchanged cold-cache profile tests fail\nsecond-round counter assertions because the global page cache is already\nwarm (UnknownMtimeUsesPageCacheForImmutableFile and\nDictionaryPruningPublishesColdAndWarmNativePageProfile). The standard\nsingle-round selection passes; new cases use independent cache\nidentities and pass repeated execution.\n- Full BE CI and positive encrypted SQL end-to-end validation are not\nclaimed; encrypted SQL admission remains intentionally gated.\n\n### Notes for Reviewers\n\nStart with parquet_crypto, then encrypted footer parsing and native\npage_reader. The direct format_common include fixes a pre-existing\nunity-neighbor include dependency exposed by adding a source file. The\nfixture README includes its generator, public synthetic parameters and\ntest matrix. CI scans the full PR commit range; the Gitleaks exception\nrequires both the exact native test path and one of the two documented\npublic fixture values.\n\n### Release note\n\nNone. This prepares the native encrypted Parquet reader; encrypted SQL\nadmission remains disabled.\n\n### Check List (For Author)\n\n- Behavior changed: Yes (see Details).\n- Does this need documentation: No user-facing enablement; fixture and\nstage limitations documented alongside the tests.\n- Pre-submission status: Ready; full CI pending.\n\n### AI Disclosure\n\n- Tool: Codex (GPT-6).\n- Generated: Code, tests, synthetic fixture generator and this\ndescription.\n- Human review: Pending maintainer review; agent self-review completed."
    },
    {
      "commit": "0d1ae9ee0646a068851da51d32ed9f6bfd0d8e7d",
      "tree": "c1180dde1087c5eb15872b3a1e3ed11f8ab654d5",
      "parents": [
        "3bacce9b1aed1fa04057e29615b2cf9ea9f8f0fb"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Fri Nov 06 01:54:43 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 09:54:43 2026 +0800"
      },
      "message": "[feature](be) Read Lance datasets with FileScannerV2 (#69581)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #65730 #66608 #66805 #66841 #66999 #67005 #67011 #67110\n#67268 #67289 #67325 #67650 #67921 #68028 #68335 #68375 #68615 #68690\n#68813 #68815 #68707 #68649\n\nProblem Summary:\n\n**In short.** This ports the Lance BE read path of branch-4.1 /\nbranch-4.2 to master, in its branch-4.2 form as of `cabcf45c1af`, plus\nthe BE side of three 4.x changes merged after that snapshot: the search\nfilter `prefilter` flag and profile layout (#68813), distance bounds for\n`vector_search` (#68815), and the split-level index rule of #68707 that\n#68815 relies on. It gathers the BE parts of the PRs listed above.\nMaster links `liblance_c` but has no Lance reader, so every\n`FORMAT_LANCE` scan fails today. With this PR the BE reads Lance\ndatasets through FileScannerV2: plain scans, vector search and full-text\nsearch, predicate and runtime-filter pushdown, the schema fetch of a\nlocal Lance TVF, and the second phase of TopN lazy materialization. The\nPR is BE only. The FE connector plugin and the regression suites come in\nseparate PRs, so until then nothing on master plans a Lance scan, and\nthe BE unit tests carry the proof.\n\nOriginal authors: @zhangstar333 @Gabriel39 @wangzhaobo957-cloud @Jay-ju\n@FANNG1\n\n**Background.** Lance shipped in 4.1.4: the Lance catalog, `local()` /\n`s3()` with `format\u003dlance`, and the `vector_search` / `full_text_search`\nTVFs. On master the FE side is being rebuilt as a connector plugin\n(`fe-connector-lance`), and the BE needs the reader from 4.x. The thrift\nids are the branch-4.1 / 4.2 ones, including `TExternalSearchRequest.5`\n(`prefilter`) and `TVectorSearchParams.6/7` (distance bounds) added\nafter the snapshot, so a BE built from master also reads the Lance scan\nranges and search requests that a 4.x FE sends during a rolling upgrade.\nThe one exception is `TVectorSearchOptions` 6, branch-4.2\u0027s\n`approx_mode` (#69852, merged into branch-4.2 after the snapshot and not\nported): it is reserved in a comment, and a master BE ignores it, so\nduring a rolling upgrade a 4.2 FE that asks for `fast` or `accurate`\ngets Lance\u0027s default (`normal`).\n\n**The problem, and what it cost.**\n\n| | master today | with this PR |\n|---|---|---|\n| A `FORMAT_LANCE` scan | FileScannerV2 has no Lance reader, so the scan\nfails with \"FileScannerV2 does not support file format\" | read by\n`LanceTableReader` in FileScannerV2 |\n| `fetch_table_schema` for a local Lance TVF | not handled | returns the\nArrow schema mapped to Doris types; Null leaves keep `NULL_TYPE` |\n| The row location of a Lance row in TopN lazy materialization |\n`GlobalRowLoacationV2` only has a 32-bit row id, but a Lance row id is a\n64-bit id within the snapshot | a version 1 layout carries the 64-bit\nrow id; phase two reads the rows with the dataset\u0027s `take_rows` |\n\n**How this PR fixes it.**\n\n- Reader: `be/src/format_v2/lance/*` and\n`be/src/format_v2/table/lance_reader.*`, as on branch-4.2 except one\nlocal renamed for master\u0027s `-Wshadow`. The unit tests construct\n`RuntimeState(TQueryOptions(), globals)`, because master removed the\nglobals-only constructor.\n- FileScannerV2: the reader dispatch and `_to_file_format` handle\n`FORMAT_LANCE` / `\"lance\"`; `FileFormat::LANCE` is added to its two\nexhaustive switches. Since #69723 every file scan uses FileScannerV2, so\nnothing else routes Lance scans (on branch-4.2 a Lance query fails with\nNotSupported when `enable_file_scanner_v2` is off).\n- Row locations: `ROW_VERSION {FILE_LOCAL_ROW_ID \u003d 0,\nLANCE_DATASET_ROW_ID \u003d 1}` and a union layout for\n`GlobalRowLoacationV2`. Version 0 keeps its byte layout, which static\nasserts pin. `PRequestBlockDesc.row_id` becomes `uint64` (a proto2\nvarint, so the wire format is compatible). The materialization operator\ndecodes row locations by version, rejects an unknown version or a wrong\nsize, and stamps `row_location_version`. `RowIdStorageReader` validates\nthe counts and the version, range-checks internal row ids against\n`uint32`, and groups Lance rows by plan node, snapshot version and\ndataset URI, so each dataset is read with one `take_rows`. Master\u0027s\nsegment batching, scan-slot dedup and fetch concurrency stay as they\nare.\n- Thrift (`PlanNodes.thrift`): the external search request body\n(`TVectorElementType`, `TVectorMetric`, `TSearchVector`,\n`TVectorSearchParams` 1-7, `TFtsCoverageMode`, `TFtsQueryType`,\n`TFtsMatchOperator`, `TFullTextSearchParams`, `TSearchFilterFormat`,\n`TSearchFilter`, `TVectorSearchOptions`, `TExternalSearchQuery`),\n`TExternalSearchRequest` 2-5 and `TLanceFileDesc` 4-6, all with the\nbranch-4.1 / 4.2 ids. The enums spell out their 4.x values. No id is\nreused. On master, #70075 reserved `TExternalSearchRequest` 2-5 in\ncomments and added 6 (`paimon_options`); this PR replaces those comments\nwith the fields and leaves 6 as it is. `TVectorSearchOptions` 6 is\nreserved in a comment for branch-4.2\u0027s `approx_mode` (#69852).\n- After the snapshot (one commit each):\n- #68813: `prefilter` decides whether the TVF search filter runs before\n(true) or after (unset / false) candidate selection, for vector and\nfull-text search. The profile groups the Lance counters under\n`LanceTiming`, `LanceIO`, `LanceIndex`, `LanceVectorIndex`,\n`LanceScalarIndex`, `LanceExecutionPlan`, `LanceScan`, `LanceSearchPlan`\nand `LanceDataCache`, uses the 4.x counter names, and registers every\nscan metric before the statistics callback is installed, so the\ncallback, which Lance invokes from its execution streams, never adds\ncounters. #68813\u0027s FE tweak that hides `TUnit.NONE` labels in\n`AggCounter` belongs to the FE port.\n- #68707 (part): a vector split searches exactly the index segments FE\nplanned for it, and a split without segments is searched flat whatever\n`use_index` says; a split with segments under `use_index \u003d false` is\nrejected. 4.x FEs plan segments only when `use_index` allows them.\n- #68815: `distance_lower_bound` (inclusive) and `distance_upper_bound`\n(exclusive) are validated (finite FLOAT values, lower \u003c upper at FLOAT\nprecision, single-vector only) and installed with\n`lance_scanner_set_distance_range()`, which master\u0027s pinned lance-c\nprovides.\n- Review fixes:\n- A nested array goes to the Doris SerDes without a copy only when it\nand every descendant start at offset 0 and reference exactly their child\nranges. Lance\u0027s legacy (v1) decoder can return a struct or list that is\ntight at its own level over a Boolean child with a bit offset (a batch\nthat starts mid-byte in the bitmap), and the SerDes reject that under\nArrow input validation.\n- A zero-field `STRUCT\u003c\u003e`, which Lance decodes, is reported as an\nunsupported column at any nesting level, like the other unsupported\nLance types. It used to be typed as a STRUCT that `ColumnStruct` cannot\nhold (its row count comes from its first child).\n- A Lance Decimal256 field with precision \u003c\u003d 38 is narrowed to\nDecimal128 by a safe Arrow cast before decoding. Doris types it\nDECIMAL32/64/128I by precision, and those SerDes read only Decimal128\narrays; the BE used to crash on the first value, also for nested leaves.\nOnly values a reader can see are cast: before a nested leaf is narrowed,\nevery child slot that no visible row references (under a NULL list,\nfixed-size-list, map or struct row, or outside a sliced parent) is\nmarked NULL, so a hidden value does not fail the cast and a visible\nvalue outside the declared precision still does; map keys, which cannot\nbe NULL, stay checked.\n- FLOAT / DOUBLE `\u003e\u003d` runtime filters stay residual-only. Lance compares\nfloats by totalOrder, so a pushed `x \u003e\u003d c` dropped negative-NaN rows\nthat Doris keeps.\n- Runtime filters that mention INT64_MIN stay residual-only. The pinned\nLance SQL parser reads `-9223372036854775808` as the negation of a\nmagnitude that it parses as i64 first; the overflow turns the literal\ninto a Float64, which the planner cannot coerce to Int64 or Decimal, so\na pushed IN, `\u003e\u003d` or `\u003c\u003d` filter on an INT64 or scale-zero decimal\ncolumn failed planning and the query failed. Every other i64 is still\npushed.\n- The shared Arrow DATE32 decoding\n(`DataTypeDateV2SerDe::read_column_from_arrow`) rejects days outside\n0000-01-01..9999-12-31 instead of returning a non-null `0000-00-00`,\nskips null slots and adds the epoch in 64 bits: the DATE32 range part of\nbranch-4.1 / 4.2 #68649. It also changes ADBC, Arrow stream load and\npaimon-rust reads.\n- Child values hidden by a NULL list, fixed-size-list, map or struct row\nare not decoded: the array, map and struct SerDes give the element SerDe\na validity bitmap that also clears those slots\n(`mask_arrow_children_hidden_by_null_parents()`). Arrow leaves them\nunspecified, and Lance\u0027s v2.1 miniblock encoding keeps them, so a hidden\n10000-01-01 under a NULL fixed-size list used to fail the DATE32 range\ncheck above. Visible values are unchanged; this applies to ADBC, Arrow\nstream load and paimon-rust as well. The masks come from ExecEnv\u0027s\ntracked Arrow pool. A nested Lance array that is compacted before\ndecoding is masked the same way, at every level, before its\n`ValidateFull()`, so a hidden invalid UTF-8 string, out-of-precision\ndecimal, Date64 or time no longer fails the batch while a visible one\nstill does. Since #69697 the same masks keep hidden DATETIMEV2 /\nTIMESTAMPTZ values out of its range checks. A mask costs at most one\npass per batch: a NULL row that owns no child slots, or only NULL ones,\nallocates nothing, and Arrow stream load masks each retained IPC batch\nonce (`mask_arrow_descendants_not_visible()`) before it hands row ranges\nto the SerDes, so its per-block reads allocate nothing. The one\nper-block mask left is for map keys under a NULL map row that still owns\nentries, since a MapArray cannot hold NULL keys; Arrow\u0027s map builder\nwrites NULL map rows without entries.\n- Core: `to_protobuf` keeps `NULL_TYPE` for a NULL literal type, so a\nNull leaf in ARRAY / MAP / STRUCT no longer comes back as BOOLEAN. The\nnullable serde reads an Arrow Null array as NULLs without touching\nbuffers that the array does not have.\n- Configs, caches and metrics, with the 4.x names and defaults:\n\n  | Config | Default |\n  |---|---|\n  | `lance_index_cache_size_bytes` | 10 GiB |\n  | `lance_metadata_cache_size_bytes` | 1 GiB |\n  | `enable_lance_data_cache` | true |\n  | `lance_data_cache_path` | `${DORIS_HOME}/lance_data_cache` |\n  | `lance_data_cache_disk_capacity_bytes` | 100 GiB |\n  | `lance_data_cache_read_block_size_bytes` | 1 MiB |\n  | `lance_io_buffer_size_bytes` (mutable) | 2 GiB |\n  | `lance_batch_readahead` (mutable) | 5 |\n  | `lance_fragment_readahead` (mutable) | 5 |\n\nThe BE-wide Lance session (index cache, metadata cache and the optional\nFoyer disk cache) is created by the first Lance scan, so a BE that never\nreads Lance creates no cache directory. The first six configs take\neffect after a restart. A negative cache size (and, while the data cache\nis enabled, a negative disk capacity or read block size) fails the first\nLance scan with a message that names the setting, as does any data-cache\nvalue lance-c rejects; the rest of the BE keeps serving. Ten\n`lance_session_*` metrics report the two in-memory caches. With the\ndefaults, the disk tier keeps 51,200 sparse 2 MiB block files open from\nthe first Lance scan on (100 GiB in blocks of twice the read block\nsize), the same as on branch-4.1 / 4.2. Size\n`lance_data_cache_disk_capacity_bytes` and\n`lance_data_cache_read_block_size_bytes` against the BE\u0027s open-file\nlimit, or set `enable_lance_data_cache \u003d false`.\n\nThe index and metadata caches live in lance-c memory: they are not\ncharged to a MemTracker or registered with CacheManager, so memory GC,\nthe cache capacity adjustment under memory pressure and\n`/api/clear_cache/all` do not shrink them, and `enable_lance_data_cache\n\u003d false` does not turn them off. lance-c has no call to clear, evict or\nresize them, so size them with the two configs. The disk tier keys\nblocks by Lance\u0027s store prefix (`scheme$bucket` for S3, GCS and OSS,\nwithout the endpoint) and the object path\n(`thirdparty/patches/lance-c-foyer.patch`). Several endpoints that serve\nthe same bucket name and data-file path with different bytes (data files\nregistered under chosen names or replaced in place; Lance\u0027s writers use\nrandom file names) share entries, also across a BE restart; such a\ndeployment should set `enable_lance_data_cache \u003d false`. Both are as on\nbranch-4.1 / 4.2 and tracked as follow-ups.\n\nA Lance scan\u0027s native working set, that is the I/O buffered up to\n`lance_io_buffer_size_bytes` per scanner, the decode readahead and the\noperators of a search plan such as the top-k of a flat vector search, is\nallocated inside lance-c and is charged to neither the query nor its\nworkload group. It counts toward the BE process memory only, so memory\nGC, which ranks queries by tracked memory, does not attribute it to the\nLance query. lance-c accepts no memory budget, so on BEs with little\nmemory headroom size `lance_io_buffer_size_bytes`,\n`lance_batch_readahead` and `lance_fragment_readahead` for the number of\nconcurrent Lance scanners. As on branch-4.1 / 4.2; tracked as a\nfollow-up.\n- Build: macOS links `-framework IOKit`. This is the first time objects\nof `liblance_c.a` are pulled into `doris_be`, so the binary and the link\ntime grow. With this PR `util/rle_encoding.h` reaches 231 TUs and\n`util/threadpool.h` 589, the same as master `315c2e00331` alone (limits\n275 and 660 since #69717), and `gen_cpp/internal_service.pb.h` 562\n(master alone 559, limit 660); the reach budgets are unchanged.\n\nNot included: the Lance index metadata scanner (`LANCE_INDEX_ENTRIES`);\nthe FE side, including the rolling-upgrade guard that keeps Lance lazy\nmaterialization off while an old BE could decode a version 1 row\nlocation as version 0; the regression suites; the rest of #68707\n(searching historical versions, tags and branches, rejecting fragment\nids missing from the snapshot, naming the snapshot in errors and the\nprofile). The `InitReaderAvgTime` / `GetBlockAvgTime` profile fix that\n#67005 carried goes in a separate PR.\n\n**Results.**\n\n| | master | with this PR |\n|---|---|---|\n| A `FORMAT_LANCE` query | fails before reading anything | read by\n`LanceTableReader` in FileScannerV2 |\n| A Lance row in the second phase of TopN lazy materialization | cannot\nbe represented | a version 1 row location, read back with `take_rows` |\n| A row location of an unknown version or size in phase two | decoded as\nversion 0 | rejected with an error |\n| An Arrow DATE32 day outside 0000-01-01..9999-12-31 (ADBC, Arrow stream\nload, paimon-rust, Lance) | read as a non-null `0000-00-00` | the read\nfails with `InvalidArgument` |\n\nBE unit tests (ASAN, macOS arm64, against master\u0027s pinned lance-c), on\nthe head rebased onto master `378404eeb12`: 1029 cases in 54 suites run,\n1027 pass and 2 Parquet microbenchmarks skip by design. They cover the\nLance suites (107: `LanceTableReader*` 83, `LanceNestedNullTest` 10,\n`LanceRuntimeFilterHelperTest` 10, `LanceSessionManagerTest` 4); the\nrow-location protocol (`MaterializationSharedStateTest`,\n`RowIdStorageReaderTest`, and the new `RowIdFetchRequestValidationTest`:\ncount mismatch, unknown version, a Lance version on an internal mapping,\nan internal row id of 2^32); `FileScannerV2Test` (28); the ORC and\nParquet reader suites (`NewOrcReaderTest`, `NewParquetReaderTest`,\n`ParquetScanTest` and the other `Parquet*` suites), whose row ids now go\nthrough `file_local`; and the Arrow / protobuf / DATE serde suites with\nthe new `ArrowParentNullsTest`, plus the Iceberg and Parquet reader\nsuites that decode Arrow dates. Reverting any of the review fixes, the\n#68813 flag, the #68707 split rule or the #68815 bounds makes its new\ntests fail. The sixth review round adds 13 cases (`ArrowParentNullsTest`\n10, `LanceTableReaderTypeTest` 3) and extends one; with them the Arrow\nSerDe, Lance, ADBC and `IcebergV2ReaderTest` suites run 317 cases: 310\npass and the 7 ADBC SQLite cases skip without a local SQLite driver. The\nninth review round adds\n`LanceRuntimeFilterHelperTest.KeepsInt64MinFiltersResidualOnly` and\n`LanceTableReaderFilterTest.KeepsInt64MinRuntimeFiltersOutOfLanceScanner`:\nboth pass with the 106 Lance cases (`Lance*`) on the branch rebased onto\n`339e087b11f`, and restoring `parse_int64()` in either check makes\nexactly these two fail. On the head rebased onto master `315c2e00331`,\nwith the per-block mask fix (`42cf4e3e5d1`), 328 cases in 24 suites\npass: the Arrow SerDe and validation suites, `ArrowParentNullsTest`\n(15), `LoadArrowPipeTest` (47), the DATETIMEV2 / TIMESTAMPTZ SerDe\nsuites, `ArrowColumnToDorisColumnTest`, `ArrowBlockConvertorTest`,\n`AdbcReaderTest`, `RemoteDorisV2ReaderTest`, `PaimonRustTableReaderTest`\nand the Lance suites. Allocating at the first NULL row of each call\nagain fails the new `ArrowParentNullsTest` case and the four new\n`LoadArrowPipeTest` block cases, dropping the per-batch mask fails the\nLIST and STRUCT block cases, and returning the child unmasked fails the\nnew timestamp case, four `ArrowParentNullsTest` cases and\n`LanceTableReaderTypeTest.IgnoresDate32PayloadHiddenByNullParents`.\n\n### Release note\n\nAdd nine `lance_*` BE configs and `lance_session_*` BE metrics. They\ntake effect once the FE can plan Lance scans. Reading an Arrow DATE32\nvalue outside 0000-01-01..9999-12-31 through ADBC, Arrow stream load or\npaimon-rust now fails instead of returning 0000-00-00.\n\n### Check List (For Author)\n\n- Test \u003c!-- At least one of them must be included. --\u003e\n    - [ ] Regression test\n    - [x] Unit Test\n- `run-be-ut.sh --run --filter\u003d...` with the suites listed under\nResults: 420 of 420 pass on macOS arm64 (ASAN) on the content before the\nrebases, and the Lance and Arrow SerDe suites pass again with the\nnested-offset, zero-field struct, hidden-child, hidden-Decimal256 and\ncompaction-mask fixes (the ADBC SQLite cases skip without a local SQLite\ndriver). Rebased onto master `43f9ceb48d7`: #69668 conflicted in the\nscanner selection and in `fetch_table_schema`, which now hands every\nfile format to `format::fetch_file_schema()`, so the Lance branch runs\nahead of that call; #69646 added a TopN row-group test that reads the\nversion 0 row location fields, which now go through `file_local` (the\nearlier rebase onto `5caaeb3e3d4` resolved #69614 and #69601 in the\nscanner selection, the `FileFormat` enum, its test and the reach budget\ntable). The 95 TUs that this PR changes or that master changed since\n`5caaeb3e3d4` were syntax-checked against the rebased tree with the UT\nbuild flags, and CI runs the suites again. Rebased again onto master\n`8d88f1cc6f7`: the only conflict was the budget table, where #65858\nmoved the `util/rle_encoding.h` baseline to 828, and the 163 TUs that\nthis PR changes or that master changed since `43f9ceb48d7` pass the same\nsyntax check. Rebased again onto master `339e087b11f`: the only conflict\nwas the budget table again, where #67292 moved the `util/rle_encoding.h`\nbaseline to 834; #69872 added Arrow type checks ahead of the NULL-parent\nmasking in the map and struct readers and merged cleanly. Of the 114\nsources that this PR changes or that master changed since `8d88f1cc6f7`,\n111 pass the same syntax check. The other three are master\u0027s: two new\n#67292 files under `storage/segment` fail the same way on master alone,\nsince they compile only inside their unity batch, and this PR adds no\nsource to that target; the #69818 benchmark is not in the UT build and\nincludes only `util/simd` headers that this PR does not touch. Reverting\nany of the review fixes, the #68813 flag, the #68707 split rule or the\n#68815 bounds makes its new tests fail. Rebased again onto master\n`378404eeb12`, which removed the legacy file scanner (#69723): the\nconflicts were in the scanner-selection code and tests that #69723\ndeleted, so this PR\u0027s Lance lines there went with them (including its\n`LanceQueriesAlwaysUseScannerV2` test), in the profile keys of\n`materialization_opertor.cpp`, and in the row fetch of\n`rowid_fetcher.cpp`, whose non-Lance path is now only\n`FileScannerV2::read_by_rows`; the deleted V1 ORC / Parquet row id tests\ntook this PR\u0027s `file_local` rename with them, and the\n`util/rle_encoding.h` budget commit is dropped because master is no\nlonger near that limit. Of the 100 sources that this PR changes or that\nmaster changed since `339e087b11f`, 99 pass the same syntax check; the\nother, `format_v2/orc/orc_reader.cpp`, fails the same way on master\nalone. On the rebased head the suites listed under Results run 1029\ncases: 1027 pass and 2 microbenchmarks skip. The test-utility fix that\n`DataTypeDateTimeV2SerDeTest` needed on macOS (#69695) is now on master.\nRebased again onto master `315c2e00331`: the only conflict was\n`TExternalSearchRequest` in `PlanNodes.thrift`, where #70075 had\nreserved fields 2-5 in comments; the two commits that add the fields\nreplace those comments, and field 6 stays #70075\u0027s `paimon_options`.\n#69717 cut transitive includes from shared headers: of the 160 sources\nthat this PR changes or that master changed since `378404eeb12`, 156\npass the same syntax check; the two `jemalloc_hook` tests have no\ncompile command on macOS, `runtime/memory/jemalloc_hook.cpp` uses glibc\naliases that macOS cannot build (#69819), and\n`format_v2/table_reader.cpp` alone fails the same way on master alone\nwhile its unity batch compiles. The full `doris_be_test` builds and\nlinks on the rebased head; `ArrowParentNulls*` and `Lance*` (121 cases)\npass before the mask fix, and the 328 cases above with it.\n- `build-support/check-build-hygiene.sh` (on this branch rebased onto\nmaster `315c2e00331`), clang-format 16 and\n`build-support/run-clang-tidy.sh` on the changed files: all pass; on the\nchanged lines clang-tidy (llvm 22) reports only\n`readability-inconsistent-ifelse-braces` on the `DORIS_CHECK` lines of\nthe masking functions, the macro\u0027s expansion that master\u0027s existing\n`DORIS_CHECK`s report too.\n    - [ ] Manual test (add detailed scripts or steps below)\n    - [ ] No need to test or manual test. Explain why:\n- Behavior changed:\n    - [ ] No.\n- [x] Yes. `PRequestBlockDesc.row_id` is `uint64`; phase two of TopN\nlazy materialization rejects row locations of an unknown version or\nsize; `FORMAT_LANCE` scans run on FileScannerV2; an Arrow DATE32 day\noutside the Doris DATE range fails the read instead of becoming\n`0000-00-00`.\n- Does this need documentation?\n    - [x] No. Lance is documented with the FE connector.\n- [ ] Yes. \u003c!-- Add document PR link here. eg:\nhttps://github.com/apache/doris-website/pull/1214 --\u003e\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label \u003c!-- Add branch pick label that this PR\nshould merge into --\u003e\n\n---------\n\nCo-authored-by: zhangstar333 \u003czhangsida@selectdb.com\u003e\nCo-authored-by: Gabriel \u003cliwenqiang@selectdb.com\u003e\nCo-authored-by: zhaobo wang \u003cwangzhaobo957@gmail.com\u003e\nCo-authored-by: jay \u003cqingfengyouchen@163.com\u003e\nCo-authored-by: FANNG \u003cfanng@apache.org\u003e\nCo-authored-by: Claude Opus 5.5 (1M context) \u003cnoreply@anthropic.com\u003e\nCo-authored-by: zy-kkk \u003czhongyk10@gmail.com\u003e"
    },
    {
      "commit": "3bacce9b1aed1fa04057e29615b2cf9ea9f8f0fb",
      "tree": "5f8727f608d86236ab86434a699a9065eeace7fb",
      "parents": [
        "c79dc9ce0b7539a36360d38d284a962c08364dc3"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Fri Nov 06 01:44:58 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 09:44:58 2026 +0800"
      },
      "message": "[test](compaction) Fix test_time_series_compaction_level2 rowset counts with three replicas (#70304)\n\n### Background\n\nWith `compaction_policy \u003d time_series` and\n`time_series_compaction_level_threshold \u003d 2`, cumulative compaction\nfirst merges loaded rowsets into level-1 rowsets, then merges enough\nlevel-1 rowsets into one level-2 rowset.\n`compaction/test_time_series_compaction_level2` (group `nonConcurrent`)\nloads 10 rounds of 10 one-row inserts into a one-bucket table, compacts\nevery replica after each round, and checks the number of rowsets summed\nover all replicas that `SHOW TABLETS` lists. It derives `replicaNum`\nfrom `SHOW TABLETS` and accepts 1 or 3; an FE with\n`force_olap_table_replication_num \u003d 3` overrides the table\u0027s\n`replication_num \u003d 1`.\n\n### What problem does this PR solve?\n\nIssue Number: None\nRelated PR: #65342 (the branch-4.1 fix by @shuke987, ported here),\n#45412 (added the suite and these assertions)\nPR Type: Test\n\nProblem Summary:\n\nThe suite\u0027s three rowset-count assertions multiply only the 10 newly\nloaded rowsets by `replicaNum` and count the base rowset and the level-1\nand level-2 rowsets once, although each replica has its own. With one\nreplica, as in the PR NonConcurrent pipeline, the numbers happen to be\nright; with three replicas the first assertion expects 31 while every\nrun sees 33. This PR multiplies each expected per-replica count by\n`replicaNum`, the same change branch-4.1 made in #65342.\n\n### Details\n\n- **Failure:** With `force_olap_table_replication_num \u003d 3`, the suite\nfails on every run at its first check: `assert (rowsetCount \u003d\u003d 10 *\nreplicaNum + i + 1)` with `rowsetCount` 33 and the right side 31. #65342\nreported it from a branch-4.1 NonConcurrent build, and it reproduces on\nmaster locally. The PR NonConcurrent pipeline runs the suite with one\nreplica and passes.\n- **Cause:** A wrong expectation in the test. Every replica keeps its\nown base rowset `[0-1]`, level-1 rowsets, and level-2 rowset, because\nthe suite compacts and waits on every replica, and `get_rowset_count`\nadds them all up. Per replica the count is `11 + i` before round `i`\u0027s\ncompaction, 11 after the loop, and 2 after the level-2 merge, so the\ntotals are those numbers times `replicaNum`. The sibling suite\n`test_time_series_compaction_policy` already writes its expectations\nthis way (`26 * replicaNum`).\n- **Stability evidence:** The expectations stay exact counts, so the\nresult is deterministic. With `replicaNum \u003d 1` the new expressions equal\nthe old ones term by term. With three replicas the master version fails\nand this version passes, and the new final check still catches a missing\nlevel-2 merge (see Testing).\n\n### Testing\n\nAll runs used `./run-regression-test.sh --run -d compaction -s\ntest_time_series_compaction_level2` against a local master-based\ncluster.\n\n| Replicas | Suite version | Result |\n|---|---|---|\n| 1 | this PR | passes; `rowsetCount` 11 to 20 |\n| 3 (three BEs on one host, `force_olap_table_replication_num \u003d 3`) |\nmaster | fails at the first check: `rowsetCount` 33, expected 31 |\n| 3 | this PR | passes; `rowsetCount` 33 to 60 in steps of 3, then 33\nand 6 |\n| 3 | this PR with the level-2 debug point not enabled | fails as it\nshould: `expected 6 rowsets, got 33` |\n\n### Notes for Reviewers\n\nThe same branch-4.1 batch also fixed a Groovy scope bug in\n`fault_injection_p0/test_full_compaciton_run_status` (#65360); master\nalready fixed it in #58123, so this PR leaves that suite alone.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Behavior changed: No\n- Does this need documentation: No\n- Pre-submission status: Ready\n\n### AI Disclosure\n\n- Tool: Claude Code (Claude Opus 5.5)\n- Generated: the port of #65342\u0027s assertions, the local verification\nruns, and this description\n- Human review: not stated\n- Prompts: The author asked to continue moving branch-4.2\u0027s changes back\nto master, where branch-4.1 #65360 and #65342 were recorded as one task,\nand chose to verify the three-replica path on a local three-BE cluster\nbefore opening the PR.\n\nCo-authored-by: shuke \u003cshuke@selectdb.com\u003e\nCo-authored-by: Claude Opus 5.5 \u003cnoreply@anthropic.com\u003e"
    },
    {
      "commit": "c79dc9ce0b7539a36360d38d284a962c08364dc3",
      "tree": "1a4485598d921282dab24756053985b2c135ef91",
      "parents": [
        "439906a36a4933f15fdfc785e00f51ea16639510"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Fri Nov 06 01:15:21 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 09:15:21 2026 +0800"
      },
      "message": "[fix](fe) Forward four local exchange session variables to the master (#70197)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #69273, #70177\n\nProblem Summary:\n\n**In short.** Found while bringing branch-4.2 #66216 to the FE local\nshuffle planner (#70177): four session variables that decide the local\nexchanges of a plan are not forwarded to the master. #69273 made the\nplanner and executor variables travel with a statement that a follower\nor an observer forwards to the master,\n`enable_local_exchange_before_agg` among them, but not\n`enable_local_exchange_before_streaming_agg`,\n`enable_streaming_agg_hash_join_force_passthrough`,\n`enable_distinct_streaming_agg_force_passthrough` and\n`force_to_local_shuffle`. A session-level `SET` of one of them is\ndropped on that path, and the master plans the statement with its own\nglobal value. This PR forwards the four as well.\n\n**Background.** A statement that has to run on the master FE (`INSERT\nINTO ... SELECT`, `CREATE TABLE ... AS SELECT`, ...) and arrives at a\nfollower or an observer is forwarded: `FEOpExecutor` sends\n`SessionVariable#getForwardVariables()`, and `ConnectProcessor` on the\nmaster applies them with `setForwardedSessionVariables()` to a session\nthat starts from the master\u0027s global variables. Only variables whose\n`@VarAttr` sets `needForward` (or one of the `affectQueryResult*`\nattributes) are sent.\n\n**The problem, and what it cost.** These four do not set it:\n\n| Variable | Default | What it decides |\n|---|---|---|\n| `enable_local_exchange_before_streaming_agg` (#66222) | false |\nwhether a streaming pre-aggregation asks for a local hash exchange on\nits grouping keys: BE `StreamingAggOperatorX`, and the FE planner with\n#70177 |\n| `enable_streaming_agg_hash_join_force_passthrough` | true | whether a\nstreaming pre-aggregation right above a hash join asks for a passthrough\nlocal exchange: the FE planner (`AggregationNode`) and BE\n`StreamingAggOperatorX` |\n| `enable_distinct_streaming_agg_force_passthrough` | true | the same\nfor a distinct streaming pre-aggregation: the FE planner and BE\n`DistinctStreamingAggOperatorX` |\n| `force_to_local_shuffle` | false | makes every scan serial\n(`ScanNode#isSerialNode`), so the FE planner pools the scan and adds a\nlocal shuffle above it |\n\nSo a session that sets one of them gets that plan only for the\nstatements its own FE runs. A forwarded statement, such as an `INSERT\nINTO ... SELECT` or a CTAS issued through a follower or an observer, is\nplanned with the master\u0027s global value, and `EXPLAIN` on the follower\nshows a plan that the master does not run. Query results stay the same,\nsince these variables only decide how rows move between the pipeline\ntasks of a BE; what differs is the plan, and with it the speed of the\nstatement.\n\n**How this PR fixes it.** Set `needForward \u003d true` on the four, as\n#69273 does for the others, and add them to the list in\n`SessionVariablesTest#testForwardPlanAndExecutionKnobs`.\n\n**Results.**\n\n| Statement forwarded from a follower after `SET \u003cvariable\u003e \u003d\n\u003cnon-default value\u003e` | Before | After |\n|---|---|---|\n| Value the master plans it with | the master\u0027s global value | the\nsession\u0027s value |\n\nbranch-4.1 and branch-4.2 have the same gap, and #69273 is not on them\neither."
    },
    {
      "commit": "439906a36a4933f15fdfc785e00f51ea16639510",
      "tree": "450683439940c437b7f60559421249cc457e9cb4",
      "parents": [
        "52362a455cd0a584431648f2ff16d429ec3bb8d1"
      ],
      "author": {
        "name": "924060929",
        "email": "lanhuajian@selectdb.com",
        "time": "Fri Nov 06 09:11:21 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 09:11:21 2026 +0800"
      },
      "message": "[opt](hive) Decouple partition metadata batches from directory tasks (#70247)\n\n### Background\n\nPartitioned Hive Batch Mode fetches partition objects from Hive\nMetastore (HMS), lists directories and streams splits into bounded\nqueues. Small directory work units allow scans to start early and\nproducers to yield. HMS has a separate property limiting partition names\nper RPC, but the streaming path previously also used the small\ndirectory-task size for those RPCs.\n\n### What problem does this PR solve?\n\nIssue Number: None\nRelated PR: None\nPR Type: Performance\n\nProblem Summary:\n\nA cold scan of 1000 partitions could make 63 partition-object requests\nwith a directory batch size of 16. Separate bounded metadata acquisition\nfrom directory work: fetch a small first HMS batch, then larger bounded\nbatches, while readers continue claiming small directory slices. Deliver\nthe first slice without walking the next large cached batch, and resume\nsources through callbacks when metadata becomes available. Profiling\nalso exposed redundant per-split destination selection under the\nassignment lock when only one backend is eligible; that case now assigns\nremote batches directly to their only possible owner. Session defaults\nand BE code are unchanged.\n\n### Details\n\n- **Change map:** 636 changed production lines, 1159 test lines, 10\ndocumentation lines and 4 build/API-baseline lines; no generated files.\nHive loader/source integration owns shared batch cursors and\ncancellation; an HMS cache-only probe avoids remote reads when cached;\nan optional scan SPI and generic readiness handling release workers\nwhile metadata is pending. The common backend policy adds a 19-line\nsingle-destination fast path.\n- **Why it was slow:** small directory units caused excessive HMS round\ntrips. Preparing a subsequent large cached batch synchronously inside\nthe last-slice `take()` also delayed delivery of the first small slice.\nSeparately, single-backend streaming assignment still hashed every\nsplit, built candidate/alternative-host lists and updated weight maps\nper split under `SplitAssignment.assignBatch`\u0027s monitor.\n- **Optimization:** fetch `min(16,\nhive.hms_partitions_batch_size_per_rpc)` names first, then use the\nexisting positive RPC cap (default 5000). Share one ready result list;\nreaders claim at most the existing directory batch size. Admit the next\nlarge cache probe without waiting; an idle reader may help its own\nprobe, with `FutureTask` electing one executor. Only the existing\nMETADATA scheduler performs remote loads, and readiness callbacks resume\nreaders on the existing SPLIT scheduler. For one eligible backend and\nwholly remote batches, preserve split order and exact accumulated\nweights while skipping candidate selection. Local-only and\nmultiple-backend batches retain the general path.\n- **Results:** the cold single-query full scan fetches 1000 HMS\npartitions in two physical RPCs (16 and 984), compared with 63 in the\nparent streaming path. The controlled SQL comparison below isolates the\nsubsequent first-slice and assignment corrections; all three columns\nalready contain bounded HMS batching. A separate same-JVM experiment\nconfirms lower FE CPU/allocation and a LIMIT improvement; full-scan\nlatency still has substantial variance, so no fixed 22 ms penalty or\nstable full-scan speedup is claimed.\n- **Applicability and cost:** targets partitioned Hive scans with HMS\npartition-cache misses; a directory\u0027s file Listing retains its existing\nbehavior. Buffered metadata is bounded by one configured HMS result\nbatch plus slices held by bounded readers. A remote call already running\nat cancellation may finish; buffered/late results are discarded and\nsubsequent requests are prevented. Cache helpers do no remote I/O.\nSingle-backend assignment adds a linear remote-accessibility check and\nreplaces per-split hash/candidate/map work with one exact weight\naccumulation. No additional pool or cache is introduced.\n- **Behavior change:** metadata RPC grouping and source readiness\nchange. Split batch size, directory batch size, queue capacity, pruning,\ninvalidation, query-pinned handles, early stop, scheduler quanta and\nbackpressure retain their meanings. Connectors that do not opt in retain\ntheir path. Connector API changes from 29 to 30; deploy the matching\nconnector bundle with FE. No FE/BE wire-format change.\n\n#### Controlled complete-SQL comparison\n\nAll packages use parent `315c2e003317ef44e8a8fbb445a1ec8452d5647e`.\n“Initial batching” is `5e7b1c2ca33d3c20119e95eebaab9bd35d90e6c2`;\n“First-slice correction” is `c1e67745edb537a96109f6a2f43d2bd75a249417`;\n“Final” also includes the single-backend fast path. Initial/corrected\npackages differ only in the Hive loader classes. Corrected/final\nmeasurement packages differ in exactly `FederationBackendPolicy.class`,\ncompiled by the successful full FE build, to isolate the optimization.\n\n| Scenario | Initial batching (ms) | First-slice correction (ms) | Final\n(ms) |\n|---|---:|---:|---:|\n| Single query, cold, LIMIT 10 | 20.75 | 20.20 | 20.80 |\n| Single query, cold, full scan | 155.88 | 160.56 | 166.42 |\n| Single query, warm, LIMIT 10 | 8.14 | 7.26 | 7.32 |\n| Single query, warm, full scan | 86.38 | 81.79 | 84.69 |\n| Four-query wave, cold, LIMIT 10 | 46.43 | 47.03 | 45.83 |\n| Four-query wave, cold, full scan | 370.16 | 375.65 | 368.44 |\n| Four-query wave, warm, LIMIT 10 | 11.21 | 11.27 | 9.61 |\n| Four-query wave, warm, full scan | 235.75 | 232.23 | 236.50 |\n\nARM64 Docker, identical Release BE SHA256\n`f1ac268df25c8f9a6f8095e3dda20f6b0dbc591cb4609b03612dd5a913bfd0b1`, one\nengine at a time, 6 CPUs and 14 GiB, FE heap and BE memory limit 4 GiB\neach. MinIO retains its CPU set and memory limit with CFS quota removed;\nevery timed wave verifies zero throttling. User VMs remain stopped. The\nfixture has 1000 partitions, ten small Parquet files per partition and\nfive rows per file: 10000 files, 50000 rows.\n\nSettings: `enable_external_table_batch_mode\u003dtrue`,\n`num_partitions_in_batch_mode\u003d1`, `split_batch_size\u003d8`,\n`split_queue_capacity\u003d1024`, `split_partition_batch_size\u003d16`,\n`split_producer_parallelism\u003d0`, `enable_sql_cache\u003dfalse`,\n`enable_stats\u003dfalse`. Cold/warm refer to FE metadata and file-list\ncaches; JVM and BE footer/data caches are warmed. Cache gates drain\nprevious tasks and retain connector identity. Each phase runs the same\n3000-SQL warmup. Order: initial/corrected/final/final/corrected/initial;\neach cell has 32 samples per variant. Four-query latency ends when all\nfour synchronized persistent clients complete. All 1920 formal SQL\nexecutions validate results. Profile, bytecode tracing and sampling are\ndisabled during formal timing. Across 240 independent query Profiles,\nall 120 full scans verify 10000 footer hits, zero footer reads, 20000\npage hits and zero page misses/writes.\n\nPhase effects are material: unchanged corrected code has warm four-query\nfull-scan medians of 221.36 and 240.81 ms across its two phases.\nSeparate randomized, paired diagnostics therefore keep the same JVM and\nBE process. Choosing first-return, inline-cache-probe or wait-for-probe\nordering yields full-scan medians of 209.61 / 210.33 / 208.47 ms, with\n45 waves per mode and twelve split-fetch RPCs per query in all modes;\nthis does not establish a fixed first-return penalty.\n\nWith only the new assignment fast path enabled/disabled in that\nsame-process style of diagnostic (60 randomized waves per mode),\nfull-scan medians are 226.74 / 237.40 ms. The median paired difference\nis -5.31 ms, bootstrap 95% interval [-9.18, +1.28] ms: stable full-scan\nlatency improvement is not established. LIMIT medians are 10.30 / 11.37\nms, paired difference -1.15 ms, interval [-1.60, -0.52] ms. Full-wave FE\nprocess CPU falls from 215 to 170 ms. This temporary diagnostic guard is\nabsent from production code and the formal comparison above.\n\nIndependent 240-query flamegraph windows in the same JVM reduce\nassignment CPU samples from 5.03 to 0.76 ms/query and sampled\nallocations from 16.77 to 1.23 MiB/query. Hash-ring samples disappear.\nThe assignment monitor\u0027s cumulative waiting falls from 676.63 to 39.77\nms/query; these are sums across overlapping producer waits, not SQL\nelapsed time. The first-slice helper remains around 0.36–0.43 ms\nCPU/query. The full scan continues to spend most elapsed time waiting\nfor scan results, so reduced producer work need not translate directly\ninto the same wall-time reduction.\n\n### Testing\n\n- Current head, local ARM64 Docker limited to 2 CPUs: `./build.sh --fe\n-j2`, including Checkstyle, and 130 focused tests all pass:\n`FederationBackendPolicyTest`, `FederationBackendPolicySpreadTest`,\n`SplitProducerTest`, `SplitSourceTest`,\n`PluginDrivenScanNodeHiveStreamingTest`.\n- The new single-backend legacy-hash test fails before the optimization\nwith 140 unnecessary split-identity reads and passes after it. Six\nselection settings, exact cumulative weights across empty/small/large\nbatches, local-only validation and overflow are covered; existing\nmultiple-backend tests pass.\n- The Hive/HMS/SPI implementation previously passed 578 Hive, 184 HMS,\n85 SPI and 71 focused core tests. Those broader connector suites precede\nthe isolated backend-policy addition; current-head core and complete-SQL\nvalidation are listed separately above.\n- `takingFirstDirectoryBatchDoesNotWalkTheNextLargeCachedBatch`\nreproduces the first-slice barrier. Deterministic coverage also includes\nhelper/worker execution exactly once, full scheduler queues, one-slot\nhandoff, cancellation during misses, queued cancellation, remote\nfailures/late results, bounded batches, dropped partitions and readiness\ncompletion races.\n- `git diff --check` passes. No BE change or rebuild. Runtime test\nservices and temporary quota changes are restored after testing.\n\n### Notes for Reviewers\n\nStart with `HivePartitionBatchLoader` and\n`HiveScanPlanProvider.streamPartitionSources`, then the generic adapter\nin `PluginDrivenScanNode` and readiness handling in `SplitProducer`. The\nloader monitor protects cursors/results/failure only; cache work, RPCs,\nscheduling and future completion run outside it. `FutureTask.get()` is\nused by `done()` after completion and never waits there. A reader\nretains only its slice. Backend-policy state keeps its existing\nassignment lock; the single-destination fast path reduces work inside\nthat lock.\n\n### Release note\n\nReduce cold Hive partition planning round trips while retaining early\nsplit delivery, cancellation and bounded queues. Reduce FE CPU and\ntemporary allocations for remote split assignment to one eligible\nbackend.\n\n### AI Disclosure\n\n- Tool: OpenAI Codex.\n- Generated: Implementation, tests, self-review and this description.\n- Human review: The author directed the design and investigation. Agent\nself-review is complete; human line-by-line review is pending."
    },
    {
      "commit": "52362a455cd0a584431648f2ff16d429ec3bb8d1",
      "tree": "f9dba36334bb370186cfaac9ae0a7ffebc98c5a6",
      "parents": [
        "9c9447fc96922cc51d5a8f9260ab0f3a9704fb88"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Thu Nov 05 17:26:44 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 01:26:44 2026 +0800"
      },
      "message": "[fix](local shuffle) Keep local exchanges before pre-aggregations over key-breaking children (#70296)\n\n### Background\n\nOn master the FE plans the local exchanges that redistribute rows among\nthe pipeline tasks of one BE (`enable_local_shuffle_planner`, on by\ndefault). Each plan node follows the rule of the BE operator it becomes,\n`required_data_distribution()`, which BE applies itself when the FE\nplanner is off. `enable_local_exchange_before_agg` (default true)\ncontrols the exchange before a LOCAL (first-phase) aggregation. Since\n#63766, BE skips it with the switch off only while the child keeps the\ngrouping-key distribution; a child breaks it when it is serial or asks\nfor a non-hash local exchange itself, such as the probe side of a nested\nloop join or a table function.\n\n### What problem does this PR solve?\n\nIssue Number: None\nRelated PR: #63366 (introduced the FE rule without the child check),\n#63766 (the BE rule), #70177 (the same fix for streaming aggregation;\nthis PR reuses its `childBreaksLocalKeyDistribution()`), #62438 (added\nthe switch)\nPR Type: Bugfix\n\nProblem Summary:\n\nWith `enable_local_exchange_before_agg\u003dfalse`, the FE planner dropped\nthe local exchange below a LOCAL `AggSink` and below a streaming\ndistinct pre-aggregation whatever their child was, while BE keeps it\nafter a child that breaks the key distribution. Queries planned by the\nFE therefore got fewer local exchanges than BE planning gives, for\nexample no hash exchange between a nested loop join and the\npre-aggregation above it. This PR adds BE\u0027s child check to both cases.\nResults do not change; most of the diff is tests.\n\n### Details\n\n- **Change map:** production code +15 / -7 in `AggregationNode` (two\nconditions and their comments) and a `PlanNode` javadoc; tests +345 /\n-34: cases in `LocalShuffleNodeCoverageTest` and\n`LocalExchangePlannerTest`, and the new regression suite\n`nereids_p0/local_shuffle/test_local_exchange_before_agg_breaking_child`\nwith its generated `.out` (+169).\n- **Frequency:** Always, when `enable_local_exchange_before_agg\u003dfalse`\n(not the default; P0 randomizes it) and a LOCAL aggregation sits on a\nchild that breaks the key distribution. Only plans and speed change: the\nFE always requires hash input at FIRST_MERGE and at the non-streaming\nDISTINCT dedup (a deliberate divergence from BE), so results stay right.\n- **Root cause:** `AggregationNode.enforceAndDeriveLocalExchange()`\nmirrors the BE operators, but its DistinctStreamingAgg and AggSink LOCAL\nbranches fall back to the base class on `!enableLeBeforeAgg` alone,\nwithout BE\u0027s `child_breaks_local_key_distribution()`. #63366 wrote them\nafter #63766 had merged, from the rule before it.\n- **Why existing tests missed it:** Unit tests: the two \"LE disabled\"\ncases in `LocalShuffleNodeCoverageTest` use a child that keeps the key\ndistribution, no case reached the DistinctStreamingAgg branch, and\n`LocalExchangePlannerTest` had no plan with the switch off. Regression:\n`query_p0/join/test_agg_after_nested_loop_join_local_exchange` (#63766\u0027s\nsuite) and\n`nereids_p0/local_shuffle/test_enable_local_exchange_before_agg` compare\nonly results, which were right, and no `.out` shows local exchanges.\n- **Fix:** Both branches fall back to the base class only when\n`childBreaksLocalKeyDistribution()` is false. After a key-breaking\nchild, the AggSink LOCAL phase goes on to its hash requirement, and the\nstreaming distinct pre-aggregation to\n`enable_distinct_streaming_agg_force_passthrough` (PASSTHROUGH, else the\nbase class), as on BE; it never asks for hash.\n- **Other paths and branches:** Streaming aggregation got the check in\n#70177; BE needs no change. One known difference from BE planning\nremains, with the switch on or off: the FE lets an ADAPTIVE_PASSTHROUGH\nor PASSTHROUGH child output satisfy a PASSTHROUGH requirement, where BE\n`Pipeline::need_to_local_exchange()` always adds one (last row of the\ntable in Testing). branch-4.1 and branch-4.2 have no FE local shuffle\nplanner and are not affected.\n\n### Testing\n\n- `LocalShuffleNodeCoverageTest` and `LocalExchangePlannerTest`, two new\ncases each: all four fail without the fix (for example, a serial child\ngot `PASSTHROUGH` instead of `LOCAL_EXECUTION_HASH_SHUFFLE`) and pass\nwith it. The 12 fe-core test classes that plan local exchanges or print\ndistributed plans pass (168 tests). Run locally.\n- A planner probe over 288 plans (18 query shapes × pooling on and off ×\n`agg_phase` 0 and 2 × `disable_streaming_preaggregations` on and off ×\nthe switch): only 32 with the switch off change. 30 of them now equal\nthe plan with the switch on; the other 2 keep the distribution of a\n`UNION ALL`, as the switch asks.\n- Local exchanges in the query profiles on a local 1 FE + 1 BE cluster\n(master `4530f761fa1`; `doris-fe.jar` from `7c4b4462dcc` with #70177,\nand for \"after\" with only `AggregationNode.class` replaced), switch off,\n`parallel_pipeline_task_num\u003d4`, `agg_phase\u003d2`. The `AggSink` rows set\n`disable_streaming_preaggregations\u003dtrue`; the distinct rows are a `GROUP\nBY` without aggregate functions:\n\n| Query | BE planner | FE planner before | FE planner after |\n|---|---|---|---|\n| LOCAL `AggSink` over a nested loop join | ADAPTIVE_PASSTHROUGH,\nLOCAL_HASH_SHUFFLE | ADAPTIVE_PASSTHROUGH | ADAPTIVE_PASSTHROUGH,\nLOCAL_HASH_SHUFFLE |\n| LOCAL `AggSink` over `explode` | PASSTHROUGH, LOCAL_HASH_SHUFFLE |\nPASSTHROUGH | PASSTHROUGH, LOCAL_HASH_SHUFFLE |\n| LOCAL `AggSink` over a one-tablet scan (serial) | PASSTHROUGH,\nLOCAL_HASH_SHUFFLE | PASSTHROUGH | PASSTHROUGH, LOCAL_HASH_SHUFFLE |\n| LOCAL `AggSink` over `UNION ALL` | none | none | none |\n| distinct pre-aggregation over `explode` | PASSTHROUGH ×2 | PASSTHROUGH\n| PASSTHROUGH ×2 |\n| distinct pre-aggregation over a nested loop join |\nADAPTIVE_PASSTHROUGH, PASSTHROUGH | ADAPTIVE_PASSTHROUGH |\nADAPTIVE_PASSTHROUGH |\n\n- The new regression suite fails on the FE without the fix at its first\nplan check (`nlj: [VNESTED]`) and passes with it; its 16 result blocks\n(4 queries × both planners × the switch on and off) match results\ncomputed independently from the inserted rows.\n`nereids_p0/local_shuffle` (11 suites),\n`query_p0/join/test_agg_after_nested_loop_join_local_exchange` and\n`nereids_syntax_p0/agg_4_phase` pass on the same cluster with the fix.\n\n### Notes for Reviewers\n\n- `baseClassRequire()` stays in both branches although only a non-serial\nchild reaches them now, since a serial child always breaks the key\ndistribution. It keeps the branches shaped like BE\u0027s call to the base\nclass, as #70177 does for streaming aggregation.\n- In `LocalShuffleNodeCoverageTest`, #70177\u0027s five key-breaking children\nmoved into `childrenThatBreakKeyDistribution()` and its aggregate\nbuilder into `buildLocalPhaseAgg()`, so the new cases reuse them;\n#70177\u0027s case is otherwise unchanged.\n- The `PlanNode#requiresShuffleForCorrectness` javadoc still named\n`enable_local_exchange_before_agg` for streaming aggregation, which\nreads `enable_local_exchange_before_streaming_agg` since #70177."
    },
    {
      "commit": "9c9447fc96922cc51d5a8f9260ab0f3a9704fb88",
      "tree": "00eb17886cc9475afdf99a5509683a5a80ae7d2e",
      "parents": [
        "b6c67510385391d0dd913589f1b323ec86ba7bd6"
      ],
      "author": {
        "name": "Lijia Liu",
        "email": "liutang123@yeah.net",
        "time": "Fri Nov 06 00:05:03 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Fri Nov 06 00:05:03 2026 +0800"
      },
      "message": "[opt](infer predicate) Infer IN predicate from OR when slot is shared by sibling conjuncts (#69921)\n\nOrToIn extract mode dropped \"slot \u003d c\" / \"slot IN (...)\" candidates\nwhose slot is also referenced by sibling conjuncts, e.g. an IN inferred\nfrom a kept inner OR. So \"spm IN (...)\" could not be inferred from\nnested multi-column ORs, and parquet lazy materialization decoded large\nJSON columns on nearly all rows.\n\nSkip the independence check for plain-slot candidates in extract mode\nonly. The inferred IN is implied by the OR and the OR is kept, so\nresults are unchanged. Add FE config\nenable_or_to_in_shared_slot_inference (default true, mutable) as a\nrollback switch covering only this change.\n\n---------\n\nCo-authored-by: 刘立家 \u003cliulijia1029@gmail.com\u003e\nCo-authored-by: liutang123 \u003cliulijia1029@google.com\u003e\nCo-authored-by: Claude Code \u003cnoreply@anthropic.com\u003e"
    },
    {
      "commit": "b6c67510385391d0dd913589f1b323ec86ba7bd6",
      "tree": "9c3c499821e332650d9eca8bcb3cefdd7d1ec386",
      "parents": [
        "65f8692f4e8c6b8e072f18e34bfe30fd471ead36"
      ],
      "author": {
        "name": "Jack",
        "email": "jiangkai@selectdb.com",
        "time": "Thu Nov 05 21:57:54 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 21:57:54 2026 +0800"
      },
      "message": "[improvement](be) Resolve a tokenizer\u0027s char filter once per value (#70259)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: None\n\nProblem Summary: DorisTokenizer did a `dynamic_cast` on its reader for\nevery token to correct offsets, which took about half of the basic\nparser\u0027s time. Do the cast once per value in `reset()`. The basic parser\nand custom basic / standard + lowercase analyzers get 1.5x to 1.9x\nfaster, with the same tokens."
    },
    {
      "commit": "65f8692f4e8c6b8e072f18e34bfe30fd471ead36",
      "tree": "ff1a921eb992dbe15eee5d85024a035663871089",
      "parents": [
        "117a6c69f92c0cc76f56c9c3ec3fd067e2e9907e"
      ],
      "author": {
        "name": "Calvin Kirs",
        "email": "guoqiang@selectdb.com",
        "time": "Thu Nov 05 21:20:48 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 21:20:48 2026 +0800"
      },
      "message": "Revert \"[fix](build) Migrate branch protection to GitHub rulesets\" (#70298)\n\nReverts apache/doris#70274\n\nAn error occurred while processing the github feature in .asf.yaml:\n\nBranch not protected: 404 {\"message\": \"Branch not protected\",\n\"documentation_url\":\n\"https://docs.github.com/rest/branches/branch-protection#delete-branch-protection\",\n\"status\": \"404\"}"
    },
    {
      "commit": "117a6c69f92c0cc76f56c9c3ec3fd067e2e9907e",
      "tree": "d2efc634fabdf9a5df335fbb92d35c3df58a03f3",
      "parents": [
        "17e561b04925ac2531c998bad7a3fa5207432a7c"
      ],
      "author": {
        "name": "Calvin Kirs",
        "email": "guoqiang@selectdb.com",
        "time": "Thu Nov 05 21:07:28 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 21:07:28 2026 +0800"
      },
      "message": "[fix](build) Migrate branch protection to GitHub rulesets (#70274)\n\n### Background\n\nThe repository uses .asf.yaml to manage required CI checks and pull\nrequest reviews for master and seven release branches. ASF Infra\ncurrently reconciles these settings through GitHub\u0027s classic branch\nprotection API.\n\n### What problem does this PR solve?\n\nIssue Number: None\nRelated PR: None\nPR Type: Bugfix\n\nProblem Summary:\n\nASF Infra reported that GitHub\u0027s classic branch protection API fails\nwith the number of required check contexts, without returning an\nactionable error. Migrate all eight protected branches to\ngithub.rulesets using the convenience syntax documented at\nhttps://github.com/apache/infrastructure-asfyaml/blob/main/README.md#rulesets.\nPreserve each branch\u0027s check names, non-strict status-check policy,\nstale review dismissal, approval count, and Code Owner requirement, with\nexplicit deletion and force-push restrictions.\n\n### Details\n\n- **Frequency:** When Infra reconciles branch protection settings\naffected by the GitHub API bug.\n- **Root cause:** The classic branch protection API cannot process the\nrequired check contexts reliably, as reported by Infra.\n- **Why existing tests missed it:** This failure occurs in GitHub\u0027s\nremote configuration API; Doris runtime tests do not exercise that API.\n- **Fix:** Replace protected_branches with eight named branch rulesets,\nusing status-check name lists and required_status_checks_strict: false.\nRemoving protected_branches lets ASF Infra remove the legacy rules\nduring reconciliation.\n- **Other paths and branches:** Cover master and branch-4.2, branch-4.1,\nbranch-4.0, branch-3.1, branch-3.0, branch-2.1, and branch-2.0. Leave\nrepository metadata, merge buttons, collaborators, and notifications\nunchanged.\n- **Behavior change:** Configure branch protection through active\nrulesets instead of classic protection. No bypass teams are configured.\n\n### Testing\n\n- ASF\u0027s asfyaml-validate --repo \u003cworktree\u003e --branch master passed\nlocally using apache/infrastructure-asfyaml at ebc4a47.\n- Parsed the GitHub configuration through the official schema and\nnormalized all eight rulesets through the official payload builder.\nVerified branch scope, active enforcement, deletion and force-push\nrestrictions, exact check names and any-source integration IDs,\nnon-strict policy, and review parameters against the original\nconfiguration.\n- git diff --check passed. No Doris build or runtime tests are needed\nfor this configuration-only change.\n\n### Notes for Reviewers\n\nLive GitHub reconciliation has not been performed; it occurs after the\nconfiguration is merged into the default branch. Rulesets have no\nimplicit administrator bypass, and this configuration does not add\nbypass actors.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Behavior changed: Yes (see Details)\n- Does this need documentation: No\n- Pre-submission status: Partially verified: local schema and payload\nchecks passed; live GitHub reconciliation has not been performed.\n\n### AI Disclosure\n\n- Tool: Codex (GPT-6)\n- Generated: .asf.yaml migration and this pull request description.\n- Human review: Pending."
    },
    {
      "commit": "17e561b04925ac2531c998bad7a3fa5207432a7c",
      "tree": "ff1a921eb992dbe15eee5d85024a035663871089",
      "parents": [
        "9cb2d8a5e9a96b65647107a1e9bfd9237e860660"
      ],
      "author": {
        "name": "Xin Liao",
        "email": "liaoxin@selectdb.com",
        "time": "Thu Nov 05 20:45:03 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 20:45:03 2026 +0800"
      },
      "message": "[fix](be) Replay schema-change WAL with duplicate column IDs (#70160)\n\n### What problem does this PR solve?\n\nHeavy ALTER operations can write base and shadow columns with the same\nID into Group Commit WALs. V2 rejects these valid headers as corruption.\nAccept duplicate IDs while preserving physical column order and the\nexisting first-column mapping.\n\nRelated PR: #66008\n\n### Release note\n\nFix Group Commit WAL replay failures during heavy schema changes with\nFileScannerV2."
    },
    {
      "commit": "9cb2d8a5e9a96b65647107a1e9bfd9237e860660",
      "tree": "bda35019edc3aea8d2984f26d79067df608153ae",
      "parents": [
        "1503e390275c7c2bcf814907776491b8cbf3ccc0"
      ],
      "author": {
        "name": "Gabriel",
        "email": "liwenqiang@selectdb.com",
        "time": "Thu Nov 05 19:28:41 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 19:28:41 2026 +0800"
      },
      "message": "[feature](iceberg) Propagate per-file encryption keys to scan splits (#70246)\n\n### Background\n\nIceberg decrypts encrypted manifests before native scan planning. Doris\nalready has a per-file encryption Thrift descriptor, but the Iceberg\nplanner does not populate its key and AAD fields.\n\n### What problem does this PR solve?\n\nIssue Number: #70245\nRelated PR: #70192 (metadata planning and fixture), #70213 (file-key\nwire contract)\nPR Type: Feature\n\nProblem Summary:\n\nExtract each data file\u0027s StandardKeyMetadata into every scan split so\nnative readers can receive the correct file key and AAD. Preserve\nplaintext files and reject malformed metadata without exposing binary\npayloads. Reject mixed-file metadata COUNT during bounded validation so\nunsupported encryption cannot trigger eager full-table task and range\nallocation before dispatch fails. Also fix the encrypted metadata\nfixture\u0027s first reset after KMS restart: retained HMS tables previously\ncaused purge to fail while reading manifests encrypted with the lost\nkey.\n\n### Details\n\n- **Change map:** 238 production lines changed, 325 test lines, 38\ndocumentation lines and 2 Checkstyle configuration lines; no generated\nfiles. Production changes add a bounded SDK metadata bridge, an\nimmutable per-file carrier and split population; tests cover SDK\ninteroperability, split ownership, bounded COUNT rejection and a real\nfixture restart.\n- **Scope:** Parquet data-file key transport. Encrypted SQL admission\nremains disabled; encrypted deletes and native reader integration are\nsubsequent stages. Reject independently encrypted deletes in\nmerge-on-read and native position_deletes metadata-table planning until\ndelete decryption is integrated.\n- **Design:** Validate metadata size and encoded field lengths before\nthe SDK codec can allocate. Parse one immutable snapshot, reuse the\ncarrier only within a file, and copy mutable Thrift buffers for each\nsplit. Null per-file metadata remains plaintext. Errors contain static\nclassifications without codec causes.\n- **COUNT guard:** Both metadata COUNT algorithms validate live file\nmetadata in O(files) with bounded retained heap, without reading data\nfiles. A per-file encryption marker rejects immediately with\n`COUNT_READ_UNAVAILABLE`, including when the table has no encryption\ndeclaration. Returning to ordinary scan planning would retain all tasks\nand splits after COUNT scheduling had disabled streaming. Deterministic\nrejection bypasses the manifest cache retry; actual cache failures\nretain the SDK retry. Encrypted delete files also prevent\nposition-delete netting from bypassing the guard.\n- **Usage:** No new configuration. Existing encrypted-read admission and\nBE dispatch guards remain in place.\n- **Default behavior:** Encrypted SQL remains rejected. Plaintext ranges\nretain their previous wire shape.\n- **Compatibility:** Reuses existing optional Thrift fields and the\nIceberg SDK\u0027s StandardKeyMetadata v1 codec; no protocol or plugin SPI\nchange.\n- **Behavior change:** Native scan splits now carry per-file encryption\nparameters; the test fixture can reset retained tables after its KMS\nloses keys. Encrypted SQL admission is unchanged.\n- **Fixture fix:** Drop only the two dedicated fixture HMS entries\nwithout purge, then delete and verify their fixed object prefixes.\nCleanup therefore needs no old KMS key and leaves no orphan objects in\nthose prefixes. The restart regression retains HMS/storage, restarts\nKMS/fixture, and requires the first reset to succeed without retrying\nit.\n\n### Testing\n\n- Final local verification: **295 FE tests, zero\nfailures/errors/skips**, covering `IcebergScanPlanProviderTest`,\n`IcebergScanRangeTest`, `IcebergFileEncryptionTest`,\n`IcebergEncryptionReadTest` and `IcebergEncryptedMetadataIOTest`. Maven\nreactor validation and FE Checkstyle passed.\n- Red/green proof: the initial four-case COUNT matrix failed before the\nfix. The final eight-case matrix passes across SDK/cache,\npresent/missing manifest row aggregates and manifest concurrency 1/8. A\nbatch threshold of 1 proves ordinary scanning would stream while COUNT\nstays synchronous; assertions verify early rejection, no second SDK\nplanning pass, SDK iterable closure, no cache-failure accounting, and\nstopping before the tail manifest in serial mode.\n- Existing plaintext COUNT collapse, delete handling and both real\ncache-failure SDK retries pass. Final verification uses\n`-XX:ActiveProcessorCount\u003d2 -XX:TieredStopAtLevel\u003d1`, matching the FE\nunit-test JVM setting. A diagnostic run without the processor cap hit\nthree existing cancellation-test fixture-count assertions caused by\nautomatic manifest merging; those test implementations were not changed.\n- Previously verified, unchanged fixture lifecycle: the first reset\nafter a disposable KMS restart failed before the fixture fix and passes\nafterward. SDK and Spark read historical snapshots `[1,2]` and\n`[1,2,3,4]`; missing KMS, invalid credentials, denied access and wrong\nkeys remain rejected.\n- Diff whitespace checks passed. Encrypted Doris SQL execution remains\nunavailable at this stage; this update validates the planner guard\nrather than enabling encrypted execution.\n\n### Release note\n\nNone. This prepares file-key transport while encrypted query execution\nremains disabled, and makes the external regression fixture restartable.\n\n### Check List (For Author)\n\n- Behavior changed: Yes (see Details: per-file transport and fixture\nreset)\n- Does this need documentation: No user-facing feature enablement;\ninternal contract and fixture instructions updated.\n- Pre-submission status: Ready\n\n### AI Disclosure\n\n- Tool: Codex (GPT-6)\n- Generated: Code, tests and this description.\n- Human review: Pending maintainer review; agent self-review completed."
    },
    {
      "commit": "1503e390275c7c2bcf814907776491b8cbf3ccc0",
      "tree": "c9b55e949aa21708ff9c9bf983e4a440de56ce3f",
      "parents": [
        "427a6400802cdc036f6264567490dfa073f1fa29"
      ],
      "author": {
        "name": "TengJianPing",
        "email": "tengjianping@selectdb.com",
        "time": "Thu Nov 05 17:31:07 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 17:31:07 2026 +0800"
      },
      "message": "[fix](be) Quote shell arguments in profiling and Python tools (#70207)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: None\n\nProblem Summary:\n\nBE passes configured paths to a shell when it runs profiling tools. The\nshell can treat parts of those paths as commands. For example,\n`profile$(printf injected).heap` becomes `profileinjected.heap` before\nthe profiling tool starts.\n\nThis change quotes each variable argument in the jeprof, pprof, and perf\ncommands. It also quotes the tool paths and the flame graph output path.\nThe same rule applies to Python version checks, environment creation,\nand package lists. Double quotes alone do not stop command substitution\nin these Python paths.\n\nThe bundled jeprof also checks for an existing local file before it\nexpands remote profile groups. For example, a local directory named\n`safe@1` keeps its original name. The tool\u0027s own shell encoder now\nescapes every apostrophe. These changes prevent a local profile path\nfrom entering the tool\u0027s remote command path through the group syntax.\n\nThe heap dump code copies `jeprofile_dir` under the configuration update\nlock. Directory creation and file creation then use the same copy. The\nshell command helper rejects NUL bytes before it calls `popen`. The\npprof content path removes the temporary file that it created.\n\nRelated paths checked:\n\n- `/jeheap/dump`: a mutable `jeprofile_dir` reaches the jeprof command.\n- `/pprof/heap?readable\u003d1` and `/pprof/profile?type\u003dtext`: configured\npaths reach pprof.\n- `/pprof/profile?type\u003dflamegraph`: configured paths reach perf, both\npipeline tools, and file redirection.\n- Python environment checks and `get_python_packages`: operator paths\nreach shell commands.\n- The HTTP `type addr2line` command has no variable argument.\n- The CDC launcher passes an argument array to `execv`.\n\nThe threat model classifies BE 8040 as an internal port. Network\nisolation remains required. Anonymous access to this port is out of\nmodel under sections 4.3 and 4.4. This change removes command\ninterpretation from path values. It does not change the HTTP\nauthentication default. The Python paths require operator configuration\nand Python UDF support.\n\n### Release note\n\nBE now quotes paths in profiling and Python environment commands. The BE\nshell command no longer interprets shell characters in those arguments.\nThe bundled jeprof preserves existing local profile paths that contain\n`@N` and quotes all apostrophes in its own commands.\n\n### Check List (For Author)\n\n- Test:\n  - ASAN BE build: `./build.sh --be -j32` passed.\n- Unit tests: 47 tests passed in `AgentUtilsTest`,\n`ProfilerCommandTest`, and `PythonEnvTest`. This includes the bundled\njeprof unit tests and a real-tool local-profile case.\n  - clang-format 16, clang-tidy, and BE header hygiene checks passed.\n- Manual test: the local BE accepted and returned the path unchanged.\nThe original configuration was restored.\n- Manual test: ASAN HTTP profiling handlers returned their expected\ndisabled responses.\n- Manual test: the installed jeprof read an empty profile with quotes\nand shell characters in its path. It returned `No nodes to print`.\n- Manual test: before/after comparison with a local `safe@1` directory.\nThe old tool selected its remote path. The fixed tool read the local\nprofile. A curl stub blocked network access in both runs.\n- Behavior changed: Yes. Shell characters in paths remain part of the\npath. Commands with NUL bytes fail before they start.\n- Does this need documentation: No. The API and configuration names are\nunchanged.\n\n### Minimal reproduction\n\nThe argument tests use small local scripts as profiling tools. They pass\npaths with spaces, repeated quotes, and `$(printf injected)` through the\nproduction code. The scripts must receive the original paths. A separate\nintegration case calls the bundled jeprof with a two-line profile that\nhas no samples. It uses a `safe@1` directory and blocks download\ncommands. The bundled tool also tests a shell round trip with `a\u0027b\u0027c`.\nThese tests need no tables or rows.\n\n```sh\n./run-be-ut.sh --run --filter\u003d\u0027AgentUtilsTest.*:ProfilerCommandTest.*:PythonEnvTest.*\u0027 -j32\n```\n\nASAN builds disable the HTTP sampling handlers. The tests call the\nproduction conversion functions directly to check the shell boundary."
    },
    {
      "commit": "427a6400802cdc036f6264567490dfa073f1fa29",
      "tree": "47fb3ff4b2b8c48691b9fb2dcd2f8e521613ab87",
      "parents": [
        "58f8cd61fec797084489f270319ba2b88203e74a"
      ],
      "author": {
        "name": "Gabriel",
        "email": "liwenqiang@selectdb.com",
        "time": "Thu Nov 05 17:16:21 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 17:16:21 2026 +0800"
      },
      "message": "[fix](nereids) Rewrite non_nullable MV outputs for nullable source columns (#70219)\n\n### What problem does this PR solve?\n\nProblem Summary:\n\nAn asynchronous MV defined with `non_nullable(vec) AS vec` cannot\nrewrite queries selecting the original nullable `vec`. This prevents\ntransparent use of NOT NULL materialized vector columns and their ANN\nindexes. The missing value mapping is in the shared async\nmaterialization context, so it affects external catalogs generally.\n\nKeep the existing `non_nullable(slot)` mapping and add one equivalent\nraw-slot mapping to a same-type NOT NULL MV output. Preserve query\nnullability and ExprIds through existing output alignment. Prefer direct\noutputs and avoid multiplying equivalent mapping alternatives. Also\nrecognize value-preserving nullable wrappers in ANN TopN virtual\nexpressions, while keeping surviving casts on the scalar path because\nANN requires a direct indexed slot. Request early MV rewriting only when\nthe actual sole order-key Alias directly computes approximate distance,\nso ANN pushdown runs after discovering the MV index under the default\nstrategy without adding a pre-optimizer for unrelated projections.\n\n### Release note\n\nAllow nullable source-column queries to use asynchronous MVs\nmaterialized with `non_nullable(column)`, including ANN TopN queries,\nwithout adding the assertion to the query.\n\n### Check List (For Author)\n\n- Test:\n- [x] Unit Test: all 31 focused FE tests pass with no skips. Five new\nassertion cases for cast fallback and order-key eligibility fail before\nthe guard fixes and pass afterward. Existing mapping, nullable ANN and\nstrategy controls remain covered.\n- [x] Regression test: the native suite passes after the guard fixes,\ncovering scalar/array projection, filters, ANN plan and execution\ncounter, cast-vector scalar fallback, outer-join NULLs, failed\nrefresh/fallback and partial partition UNION. The two cast queries\ncompare an indexed MV with the unrewritten scalar source and assert no\nANN SORT INFO. Loopback JDBC source comparison was validated for the\ninitial mapping change and was not rerun for this update.\n- Added gated external suites for Hive, Iceberg, Paimon, Hudi, JDBC,\nADBC, Elasticsearch, Fluss, plugin-based sources and MaxCompute. Hive\nORC/Parquet and Iceberg include array columns and ANN queries.\n- Quote reserved schema identifiers in the external MV regression source\npaths, including sources reused by metadata, DDL and vector queries.\nBoth affected suites pass SQL-generation checks covering 159 statements\nand seven shared-helper invocations; all 12 related Groovy scripts\ncompile. Full external fixture execution remains for CI.\n- Remote source suites require their respective fixtures and were not\nexecuted in the local environment. Local SQL validation uses the\nmodified FE with an earlier master ASAN BE build; this PR has no BE code\nchanges.\n    - FE build, focused incremental compilation and FE Checkstyle.\n- CI triage at `788935f3` found shared object-storage authorization\nfailures across regression builds; no new PR-suite failure was observed\nin the collected evidence. Shared access must be restored before a full\npipeline result is conclusive. The native MV/ANN suites passed in P0 and\ncloud, and the new Hive, Iceberg, Paimon and ADBC suites passed\nexternally at that head.\n- Behavior changed:\n- [x] Yes. More queries can use equivalent MV outputs. Source schema,\nNULL assertion failures and freshness checks are preserved.\n- Does this need documentation?\n    - [x] No."
    },
    {
      "commit": "58f8cd61fec797084489f270319ba2b88203e74a",
      "tree": "a8ccfb22af0161f352e1d34db22132aaac048969",
      "parents": [
        "bc9686578a5a3675b8d04b0a0b4fc9ff6a983d96"
      ],
      "author": {
        "name": "Jerry Hu",
        "email": "hushenggang@selectdb.com",
        "time": "Thu Nov 05 17:15:48 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 17:15:48 2026 +0800"
      },
      "message": "[fix](be) Preserve DATE type in min/max runtime filters (#70222)\n\n### What problem does this PR solve?\n\nIssue Number: N/A\n\nRelated PR: #69963\n\nProblem Summary: When a DATE min/max runtime filter is received through\nthe protobuf path, its bounds are decoded with the DATETIME target\nbecause `TYPE_DATE` and `TYPE_DATETIME` share one branch. This changes\nthe DATE bounds\u0027 internal type tags to DATETIME. The consumer then\ncreates DATETIME literals for DATE scan predicates, which can abort the\nBE on a Field type mismatch.\n\nDecode `TYPE_DATE` bounds with the DATE target in the shared min/max\nreceive path. Keep the existing `TYPE_DATETIME` decoding unchanged. The\nwire format and other runtime-filter types are unchanged.\n\n### Release note\n\nFix a BE crash when receiving DATEV1 min/max runtime filters.\n\n### Check List (For Author)\n\n- Test:\n- ASAN BE+FE build (`BUILD_TYPE\u003dASAN ./build.sh --be --fe -j 40`):\npassed, including FE checkstyle.\n- ASAN `RuntimeFilterWrapperTest.*`: 15 cases passed, including\nDATE/DATETIME type, min/max mode, and merge/publish request coverage.\n- `test_datev1_minmax_runtime_filter`: generated expected output and\npassed a separate verification run; `test_datev1_common`: passed.\n- C++ formatting, build hygiene, and changed-line clang-tidy checks:\npassed.\n- SQL regression ran on a single-BE cluster; cross-BE and mixed-version\nexecution were not exercised.\n- Behavior changed: Yes. Received DATE min/max bounds retain the DATE\ntype instead of being converted to DATETIME.\n- Does this need documentation: No"
    },
    {
      "commit": "bc9686578a5a3675b8d04b0a0b4fc9ff6a983d96",
      "tree": "d2c229557535c8897d9d284e6151c87c28b727c1",
      "parents": [
        "b69168cbe5a39ee77a88e5b126eacb4e0861c297"
      ],
      "author": {
        "name": "linrrarity",
        "email": "linzhenqi@selectdb.com",
        "time": "Thu Nov 05 17:11:40 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 17:11:40 2026 +0800"
      },
      "message": "[Fix](ai) Fix some AI Resours Issues (#68023)\n\n### What problem does this PR solve?\n\nIssue Number: close #xxx\n\nRelated PR: #xxx\n\nProblem Summary:\n\n- **Centralize property handling:** Move binding, defaults, validation,\nand Thrift conversion into `AIResourceProperties`, reusing the existing\n`ConnectorProperty` framework. Keep `AIProperties` as constants.\n- **Strengthen validation:** Reject blank values and invalid numeric\nranges, and apply standard string trimming.\n- **Ensure ALTER consistency:** Validate the complete configuration\nbefore replacement and safely publish updates to concurrent readers.\n- **Preserve metadata compatibility:** Keep the persisted map format and\ndefer validation of loaded metadata until use.\n- **Correct resource usage:** Update AI function resource checks and add\nregression coverage."
    },
    {
      "commit": "b69168cbe5a39ee77a88e5b126eacb4e0861c297",
      "tree": "b25668fbed08764ff72fef4452c8e82623ab1895",
      "parents": [
        "1c5e867d84958f2299d3d20fb2b7b5efbd8bc160"
      ],
      "author": {
        "name": "Socrates",
        "email": "suyiteng@selectdb.com",
        "time": "Thu Nov 05 16:51:56 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 16:51:56 2026 +0800"
      },
      "message": "[test](paimon) Verify Paimon null partition refresh and rewrite (#70201)\n\n### What problem does this PR solve?\n\nRelated PRs: #65904, #64304\n\nThe existing Paimon MTMV regression covers the initial refresh of a NULL\npartition. This test covers an out-of-band Spark append to that\npartition after the initial refresh. It verifies that the stale\npartition is ineligible for rewrite, automatic refresh touches only the\nNULL partition, and the refreshed MV is chosen again. The string\npartition \u0027null\u0027 is a control.\n\nThe suite compares live SQL results and checks the rewrite plan without\na golden output file. This PR adds regression coverage only.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test: `git diff --check` passed. Doris compilation and regression\nexecution were not run.\n- Behavior changed: No\n- Does this need documentation: No"
    },
    {
      "commit": "1c5e867d84958f2299d3d20fb2b7b5efbd8bc160",
      "tree": "ce289ad051acba6c4c2d6076037bca6cc23dc1a9",
      "parents": [
        "cb5b9cb69a1a1c4454e9439a6eabd8dd4a35b6c6"
      ],
      "author": {
        "name": "yujun",
        "email": "yujun@selectdb.com",
        "time": "Thu Nov 05 16:40:29 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 16:40:29 2026 +0800"
      },
      "message": "[refactor](stream) Consolidate OLAP table stream partition state (#70242)\n\nOLAP table streams track consumption separately for each base-table\npartition. Each partition can carry a consumed offset, its last\nconsumption time, and a historical snapshot TSO until initial rows are\nconsumed."
    },
    {
      "commit": "cb5b9cb69a1a1c4454e9439a6eabd8dd4a35b6c6",
      "tree": "3a286132c7265234e697546fdc68bb708b49a720",
      "parents": [
        "bff68004fda41c0649efc9de4c5d14df29069440"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Thu Nov 05 08:19:04 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 16:19:04 2026 +0800"
      },
      "message": "[fix](fe) Check the workload group ALTER ROUTINE LOAD sets, as CREATE does (#70130)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #69749, #50563, #49960, #53490\n\nProblem Summary:\n\n**In short.** `ALTER ROUTINE LOAD ... PROPERTIES(\"workload_group\" \u003d\n\"...\")` accepts any name. `CREATE ROUTINE LOAD` checks the workload\ngroup as the job\u0027s owner in the compute group the job runs in, and every\ntask of the job resolves it the same way, so a name that is missing\nthere, or that the owner may not use, is accepted by the ALTER and then\nfails every task. This PR checks it at ALTER time, as CREATE does.\n\n**Background.** A workload group belongs to a compute group in cloud\nmode, and to the owner\u0027s resource group outside it. Each task looks the\njob\u0027s workload group up as the owner in the job\u0027s compute group\n(`KafkaTaskInfo#rePlan`, `KinesisTaskInfo#rePlan`). Kafka jobs used to\ncheck an altered workload group (#50563, in the `AlterRoutineLoadStmt`\npath). The Nereids `AlterRoutineLoadCommand` path (#49960) did not carry\nthat check, and #53490 removed the old path. #69749 brought a check back\nonly for an ALTER that moves a Kafka job to another compute group. The\ncomment that delays the check (\"we can only get clusterId when alter\njob\") is no longer true: the ALTER has the job at hand. Found while\nreviewing #69749 (thread 4415694066).\n\n**The problem, and what it cost.** Measured on master `856194ece0e` with\nunit tests; each ALTER below goes straight to the PAUSED check, so on a\npaused job it is accepted and written to the edit log:\n\n| Job | ALTER | Result |\n|---|---|---|\n| Kafka, cloud mode, runs in `cg_a` | `\"workload_group\" \u003d \"wg_b\"`, which\nonly `cg_b` has | accepted |\n| Kafka, outside cloud mode | a workload group that does not exist |\naccepted |\n| Kinesis, outside cloud mode | a workload group that does not exist |\naccepted |\n\nAfter RESUME, the first task of each such job fails with `Can not find\nworkload group ...`, and the job pauses with `CREATE_TASKS_ERR`. That\ncode is retryable, so auto resume repeats this with backoff (20 s up to\n300 s) for as long as the job exists. The ALTER that caused it reported\nsuccess.\n\n**How this PR fixes it.**\n\n- `RoutineLoadJob#checkAlteredWorkloadGroup`, called by the Kafka and\nthe Kinesis job first thing under the job\u0027s write lock, before the\nPAUSED check, so that of two ALTERs of one job running at once the later\none checks what the earlier one left. The Kafka offset conversion stays\nbefore the lock, so an ALTER with Kafka offsets and a workload group it\ncannot use converts the offsets before it fails. When an ALTER sets a\nworkload group or declares a compute group, it checks the resulting\nworkload group in the resulting compute group, with\n`CreateRoutineLoadInfo.checkWorkloadGroup` as CREATE does. The resulting\nworkload group is the one the ALTER sets, else the one the job keeps;\nthe resulting compute group is the one the ALTER declares, else the\njob\u0027s. The check runs as the owner, because the tasks run as the owner.\nA rejected ALTER changes nothing and writes no edit log. The replay of\nan ALTER does not go through this path.\n- The Kafka job\u0027s check for a move to another compute group (#69749) is\nreplaced by this one; it checks the same thing for that ALTER.\n- The comments in `AlterRoutineLoadCommand` that pointed at the old\nplace, or at the old reason for not checking, are corrected.\n- `test_routine_load_alter` altered a job to `alter_routine_load_group`\nwithout creating the group: it worked only because the ALTER did not\ncheck, or because `test_routine_load` happened to create the group\nfirst. It now creates the group, the way `test_routine_load` does, and\nchecks that an ALTER to a missing group fails.\n\n**Results.**\n\n| ALTER | Before | After |\n|---|---|---|\n| a workload group the job\u0027s compute group has, and the owner may use |\naccepted | accepted |\n| a workload group the job\u0027s compute group lacks, or the owner may not\nuse | accepted, then every task fails and auto resume loops | rejected,\ne.g. `Can not find workload group wg_b in compute croup cg_a`; the job\nis unchanged |\n| a compute group, keeping a workload group the new group lacks (Kafka)\n| rejected (#69749) | rejected |\n| neither property | not checked | not checked |\n\nbranch-4.1 and branch-4.2 have the same gap."
    },
    {
      "commit": "bff68004fda41c0649efc9de4c5d14df29069440",
      "tree": "28edb89e831437223c0285123d1ec542797e2e91",
      "parents": [
        "35209ae701ccd255d4d22eaeac744919331d27e7"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Thu Nov 05 07:42:33 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 15:42:33 2026 +0800"
      },
      "message": "[fix](fe) Keep routine load job properties across replay and apply altered enclose and escape (#70121)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #70118\n\nProblem Summary:\n\n**In short.** When an FE rebuilds a routine load job from metadata (a\nfollower replaying the edit log, a restart, a new master), the job loses\n`max_filter_ratio`, `exec_mem_limit`, `send_batch_parallelism`,\n`load_to_single_tablet`, `enclose`, `escape` and the\n`enable_memtable_on_sink_node` it was created with, and runs with\ndefaults from then on. Separately, `ALTER ROUTINE LOAD` on `enclose` /\n`escape` never reaches the running job, and a multi-table CSV job never\napplies `enclose` / `escape` at all. This PR restores the values when\nthe job is rebuilt, applies the altered ones, and hands both to the\ntasks of multi-table jobs.\n\n**Background.** A routine load job is persisted with GSON, which writes\nonly the fields annotated with `@SerializedName`; these seven are plain\nfields. `gsonPostProcess` re-parses the stored CREATE statement, but\nonly to rebuild the `RoutineLoadDesc` (columns, filters, separators,\npartitions, merge type), and restores the partial update settings from\nthe persisted `jobProperties` map. CREATE records the other values in\n`jobProperties` as well (all but `exec_mem_limit`), and SHOW ROUTINE\nLOAD prints them from there, so the loss does not show. Found while\nmaking `SHOW CREATE ROUTINE LOAD` print these properties (#70118).\n\n**The problem, and what it cost.** Measured on master `856194ece0e` with\na unit test that creates jobs from SQL and rebuilds them with a GSON\nround trip on a thread without a session, as a follower or a restarting\nFE does:\n\n| | Job as created | Job rebuilt from metadata |\n|---|---|---|\n| `max_filter_ratio` \u003d 0.5 | 0.5 | 1.0: up to every row may be filtered\ninstead of the job pausing |\n| `exec_mem_limit` \u003d 4294967296 | 4 GB | 2 GB |\n| `send_batch_parallelism` \u003d 3, or 4 from the creating session | 3 / 4 |\n1 |\n| `load_to_single_tablet` \u003d true | true | false |\n| `enclose` `\"`, `escape` `\\` | `\"` / `\\` | none: quoted CSV fields are\nsplit at the separators inside the quotes |\n| `enable_memtable_on_sink_node` of the creating session (on by default)\n| true | false |\n| `ALTER ... (\"enclose\" \u003d \"\u0027\", \"escape\" \u003d \"\\|\")` | tasks keep `\"` / `\\`\n| SHOW ROUTINE LOAD shows the new values, tasks never use them |\n| `ALTER ... (\"enclose\" \u003d \"ab\")` | accepted, without effect | CREATE\nrejects it |\n\nThe CSV and filter ratio cases change what gets loaded, without any\nerror, after an event (a failover, a restart) that has nothing to do\nwith the job.\n\nA multi-table CSV job (one created without a target table) does not use\nits `enclose` and `escape` even before it is rebuilt. Each of its tasks\nis planned from the backend\u0027s per-table request, which does not carry\nthem, with the job\u0027s settings laid over it by\n`NereidsStreamLoadTask#setMultiTableBaseTaskInfo`, which copies the\nseparators and the JSON properties but not these two. So its quoted\nfields are split at the separators inside the quotes, whatever CREATE or\nALTER set.\n\n**How this PR fixes it.**\n\n- `gsonPostProcess` restores `max_filter_ratio`,\n`send_batch_parallelism`, `load_to_single_tablet`, `enclose` and\n`escape` from `jobProperties`, in the loop that already restores the\npartial update settings from there. CREATE has recorded these values\nsince each property was introduced, and ALTER records the new ones; a\njob created before one of them existed has no key and keeps the default,\nas today. `exec_mem_limit`, which no ALTER changes and `jobProperties`\ndoes not hold, is taken from the CREATE statement that `gsonPostProcess`\nalready re-parses.\n- CREATE records the creating session\u0027s `enable_memtable_on_sink_node`\nin the job\u0027s persisted session variables, next to `sql_mode`, and\n`gsonPostProcess` restores it. A job created before this has no record\nand keeps `false`, which is what it has run with after every restart so\nfar.\n- `ALTER ROUTINE LOAD` checks `enclose` and `escape` the way CREATE\ndoes, and `modifyCommonJobProperties`, which Kafka and Kinesis jobs and\nthe replay of an ALTER share, installs them on the job. The two rules\nmove into `CsvFileFormatProperties.parseEnclose` / `parseEscape`, which\n`analyzeFileFormatProperties` now calls without any change in behavior.\n- `NereidsStreamLoadTask#setMultiTableBaseTaskInfo` copies `enclose` and\n`escape` too, so the tasks of a multi-table job use the values CREATE or\nALTER set, as those of a single-table job do.\n- CREATE and ALTER ROUTINE LOAD take only an ASCII character as\n`enclose` or `escape` (`CreateRoutineLoadInfo#checkCsvControlValue`),\nwhich is what the documentation of both properties asks for: a\nsingle-byte character. No other value works for a routine load. Each\ntask is planned from the job\u0027s byte turned into text with `new\nString(new byte[]{b})` and parsed again (`NereidsDataDescription`, since\n#50882), which keeps every ASCII byte but turns every byte from 0x80 up\ninto 0xFD as enclose and 0xEF as escape. A job created with `\"escape\" \u003d\n\"é\"` (0xC3) has therefore run with escape 0xEF from its first task on,\nwhich drops the first byte of full-width characters such as `，` in every\nstring cell. And routine load validates its data as UTF-8 by default,\nwhere a single byte from 0x80 up is absent or part of a character\nanyway: 0xC3 as escape would cut the first byte out of every `é`. CREATE\nrecords the byte with `new String(new byte[]{b})` as it always has,\nwhich keeps every value it now takes.\n- Values that an older version accepted do not break the rebuild. The\nASCII check runs only when a statement is admitted, not when\n`gsonPostProcess` re-parses a stored CREATE statement, as the compute\ngroup and workload group checks beside it already do; failing there\nwould turn the job CANCELLED for good. A value from an older version\nthat CREATE and ALTER now reject, one that is not ASCII (an older CREATE\nrecorded every byte from 0x80 up as U+FFFD) or one that an older ALTER\nstored without any check, is ignored with a warning instead of failing\nthe replay, in the same way when the job is rebuilt and when the ALTER\nis replayed. The replay of such an ALTER leaves the job\u0027s value as it\nwas, and a rebuilt job keeps no enclose or escape, which is what it has\nrun with after every restart so far; ALTER ROUTINE LOAD can set an ASCII\nvalue again.\n\nNo new persisted field: the only addition is one key in the job\u0027s\nexisting session variable map, which older FEs ignore.\n\n**Results.**\n\n| | Before | After |\n|---|---|---|\n| declared properties above, job rebuilt from metadata | defaults | as\ndeclared, also in what tasks are built from |\n| `send_batch_parallelism` and `enable_memtable_on_sink_node` from the\ncreating session | 1 / false | the session\u0027s values |\n| ALTER of `enclose` / `escape` | shown, never used | used by the next\ntask, kept when the job is rebuilt |\n| `enclose` / `escape` of a multi-table CSV job, from CREATE or ALTER |\nnever used | used by its tasks |\n| ALTER with `\"enclose\" \u003d \"ab\"` | accepted | `enclose should not be\nlonger than one byte.` |\n| CREATE or ALTER with an `enclose` / `escape` that is not ASCII (`é`,\nU+FFFD, `，`) | CREATE takes a byte of it, which tasks get as 0xFD / 0xEF\nfrom 0x80 up; ALTER accepts it without effect | `enclose must be an\nASCII character` |\n| such a value, or an invalid one, stored by an older version | no\neffect | no effect, with a warning; the replay of the job and of the\nALTER goes on |\n\nbranch-4.1 and branch-4.2 have the same gaps."
    },
    {
      "commit": "35209ae701ccd255d4d22eaeac744919331d27e7",
      "tree": "cbed78d0206b71fd2b55dc1d88a0a010f5211fb0",
      "parents": [
        "3cf8cc628fbe2c1427ddaa94fcf29e8f40433172"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Thu Nov 05 07:28:23 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 15:28:23 2026 +0800"
      },
      "message": "[fix](fe) Let MV dry runs and EXPLAIN REFRESH borrow the session\u0027s compute group (#70125)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #69749, #62606\n\nProblem Summary:\n\n**In short.** In cloud mode, `REFRESH MATERIALIZED VIEW ... INCREMENTAL\nWITH DRY RUN` and `EXPLAIN REFRESH MATERIALIZED VIEW` of an MV that\ndeclares no `compute_group` plan (and the dry run executes) in whatever\ncompute group the `admin` user resolves to, not in the group of the\nsession that issues them, where a manual `REFRESH` of the same MV runs.\nThis PR binds both to the session\u0027s group, the way a manual REFRESH\nborrows it.\n\n**Background.** Both statements build an internal context with the admin\nidentity and the MV\u0027s session variables\n(`MTMVPlanUtil.createMTMVContext`). Its `cloud_cluster` is empty, so it\nresolves a group the way admin would: admin\u0027s `default_compute_group`,\nelse the first group admin may use. A manual REFRESH passes the\nsession\u0027s group to its task instead (`MTMVJobManager#refreshMTMV`), and\nthe task binds it when the MV declares none\n(`MTMVTask#applyComputeGroup`). #69749 made both statements honour a\ndeclared group; an MV that declares none, which is every MV created\nbefore #69749 and any MV created without the property, still goes to\nadmin\u0027s group. Found while reviewing #69749 (thread 4401555648).\n\n**The problem, and what it cost.** Measured on master `856194ece0e` with\na unit test in cloud mode: groups `cg_of_admin` and `cg_of_session`, the\nsession on `USE @cg_of_session`, an MV that declares no group.\n\n| | manual REFRESH | dry run | EXPLAIN REFRESH |\n|---|---|---|---|\n| compute group | `cg_of_session` | `cg_of_admin` | `cg_of_admin` |\n\n- If the MV\u0027s `workload_group` exists only in the session\u0027s group (a\nworkload group belongs to one compute group in cloud mode), the dry run\nfails with `Can not find workload group \u003cwg\u003e in compute croup\ncg_of_admin`, while the manual REFRESH succeeds.\n- Otherwise the dry run\u0027s delta query runs on the backends of another\ngroup, which the session\u0027s user may have no USAGE on, and starts that\ngroup if it is suspended. EXPLAIN REFRESH assigns backends there and can\nstart it as well. Results are not affected: the dry run only reads, and\nstorage is shared.\n\n**How this PR fixes it.**\n\n- The private lookup of the session\u0027s group in `MTMVJobManager` moves\nunchanged to `MTMVPlanUtil.getSessionComputeGroup()`, so a manual\nREFRESH, a dry run and EXPLAIN REFRESH resolve it with the same code.\n- `RefreshMTMVCommand` builds both internal contexts through one helper.\nIt reads the session\u0027s group first, while the session is still the\nthread-local context, then binds the declared group, else the session\u0027s\ngroup, as `MTMVTask#applyComputeGroup` does. With neither, the context\nstays unbound and the existing resolution applies; outside cloud mode\nnothing changes.\n- As in `MTMVTask`, the dry run re-checks only a declared group, since\nthe MV never chose the session\u0027s group. EXPLAIN REFRESH executes nothing\nand still checks nothing.\n\n**Results.**\n\n| MV | session | dry run / EXPLAIN REFRESH, before | after |\n|---|---|---|---|\n| declares no group | `USE @cg_of_session` | `cg_of_admin` |\n`cg_of_session`, as a manual REFRESH |\n| declares no group | resolves no group | admin\u0027s resolution | unchanged\n|\n| declares `cg_declared` | `USE @cg_of_session` | `cg_declared` |\nunchanged |\n| any, outside cloud mode | | no compute group | unchanged |\n\nbranch-4.1 and branch-4.2 have neither statement."
    },
    {
      "commit": "3cf8cc628fbe2c1427ddaa94fcf29e8f40433172",
      "tree": "0025b83a80b176c1856524233bbd4cc08ec16d86",
      "parents": [
        "d938708086e36dfb25b084014f5de4a4abfafabb"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Thu Nov 05 07:18:24 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 15:18:24 2026 +0800"
      },
      "message": "[fix](fe) Make SHOW CREATE ROUTINE LOAD output recreate the same job (#70118)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #69749\n\nProblem Summary:\n\n**In short.** `SHOW CREATE ROUTINE LOAD` prints the statement a job is\nrecreated from, but a job recreated from it was not the same job: eight\nproperties that CREATE accepts were never printed, a value with a quote\nor a backslash made the statement unparseable, `TEMPORARY` partitions\nturned into formal ones, and the comment was dropped. This PR makes the\nstatement carry all of them.\n\n**Background.** Users recreate a routine load job from `SHOW CREATE\nROUTINE LOAD` to rename it, to move it to another database, or to start\nit over after `STOP`; the statement prints the next Kafka offsets to\nconsume for exactly that purpose. #69749 made the statement carry the\n`compute_group` a job declares, and found the rest of the list\nincomplete: `RoutineLoadJob.getShowCreateInfo()` writes the properties\nby hand and covered 17 of the 25 keys in\n`CreateRoutineLoadInfo.PROPERTIES_SET`.\n\n**The problem, and what it cost.** Measured on master `1aa505cb0be` with\na unit test that creates jobs from SQL, recreates each one from its\n`SHOW CREATE` output and compares every property:\n\n| Job | Recreated from its SHOW CREATE output |\n|---|---|\n| CSV job with `enclose` `\"`, `escape` `\\`, `empty_field_as_null`,\n`send_batch_parallelism` \u003d 3, `load_to_single_tablet`, `workload_group`\n| all six back to their defaults; quoted fields get split at the\nseparators inside the quotes |\n| `unique_key_update_mode` \u003d `UPDATE_FLEXIBLE_COLUMNS` with\n`partial_update_new_key_behavior` \u003d `ERROR` | an `UPSERT` job with\n`APPEND` |\n| partial columns with `partial_update_new_key_behavior` \u003d `ERROR` |\n`APPEND`: rows with new keys get inserted instead of failing the load |\n| JSON job with `jsonpaths` such as `[\"$.k1\",\"$.v1\"]` | the statement\ndoes not parse (`mismatched input \u0027$\u0027`) |\n| Kafka property whose value holds a `\"`, such as a JAAS config | the\nstatement does not parse |\n| `TEMPORARY PARTITION(tp1)` | printed as `PARTITION(tp1)`: `Partition\ntp1 does not exist` |\n| `COMMENT \u0027...\u0027` | gone |\n\nThe jobs themselves keep running as declared; only the statement is\nwrong. The cost is a recreated job that silently loads differently (CSV\nparsing, partial update semantics, single-tablet loading, workload\ngroup, parallelism), or a statement that cannot be executed at all.\n\n**How this PR fixes it.**\n\n- `getShowCreateInfo()` prints the eight missing properties wherever\nleaving one out would give the recreated job a different value:\n`unique_key_update_mode` when it is not `UPSERT` (`partial_columns`\nalone cannot express `UPDATE_FLEXIBLE_COLUMNS`),\n`partial_update_new_key_behavior` when it is `ERROR` and a partial\nupdate mode is on (CREATE rejects the policy otherwise), `enclose` and\n`escape` when set, `empty_field_as_null` and `load_to_single_tablet`\nwhen true, and `workload_group` when set. `send_batch_parallelism` is\nalways printed, as the timezone is, because CREATE otherwise takes it\nfrom the session variable of whoever recreates the job. The values are\nthe ones the job runs with.\n- Keys and values are quoted with `SqlLiteralUtils.quoteStringLiteral`,\nwhich follows the session\u0027s `NO_BACKSLASH_ESCAPES` mode. A value without\na quote or a backslash prints exactly as before; the column separator is\nleft as it is, since it is kept as typed and already reads back.\n- `TEMPORARY PARTITION(...)` keeps `TEMPORARY`, and the job\u0027s comment is\nprinted as `COMMENT \"...\"`.\n- `CREATE ROUTINE LOAD` reads its `COMMENT` with\n`SqlLiteralUtils.parseStringLiteral`, as table, view and column comments\nare read, so that a printed comment with a quote reads back unchanged.\nIt used to strip the outer quotes only, so `COMMENT \u0027it\u0027\u0027s\u0027` was stored\nas `it\u0027\u0027s`.\n\nFor a job that uses none of these, the output gains a single line,\n`\"send_batch_parallelism\" \u003d \"\u003cthe job\u0027s value\u003e\"`.\n\n**Results.**\n\n| Job | Before | After |\n|---|---|---|\n| CSV job above | 6 properties and the comment lost | recreated with the\nsame definition |\n| flexible partial update with `ERROR` | `UPSERT` / `APPEND` | same\ndefinition |\n| JSON job with `jsonpaths` | statement does not parse | same definition\n|\n| Kafka property with `\"` and `\\` | statement does not parse | same\ndefinition |\n| temporary partition | `Partition tp1 does not exist` | same definition\n|\n| `NO_BACKSLASH_ESCAPES` session, `enclose` `\"`, `escape` `\\` | both\nlost | same definition |\n\nThe same test also found that a job\u0027s `max_filter_ratio`,\n`exec_mem_limit`, `send_batch_parallelism`, `load_to_single_tablet`,\n`enclose` and `escape` fall back to their defaults when a follower\nreplays the job or the FE restarts, and that `ALTER ROUTINE LOAD` on\n`enclose` / `escape` does not reach the running job. That is a separate\ndefect and will be fixed in its own PR; since this PR prints the values\nthe job runs with, the statement follows that fix. branch-4.1 and\nbranch-4.2 have the same gaps as master."
    },
    {
      "commit": "d938708086e36dfb25b084014f5de4a4abfafabb",
      "tree": "324225216d7224fa9c6c56d2bba82a0b937e6b69",
      "parents": [
        "8fa3c97115a96fe5924750c502b314f2b7949c16"
      ],
      "author": {
        "name": "Socrates",
        "email": "suyiteng@selectdb.com",
        "time": "Thu Nov 05 15:11:45 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 15:11:45 2026 +0800"
      },
      "message": "[fix](fe) Preserve JDBC predicate grouping and floating-point literals (#69987)\n\n### What problem does this PR solve?\n\nIssue Number: N/A\n\nRelated PR: None\n\nProblem Summary:\n\nJDBC predicate rendering loses expression boundaries when a compound\nexpression is used as an operand. For example, `(a AND b) \u003d c` becomes\n`a AND b \u003d c`, potentially discarding matching rows at the source.\n\nIntegral-looking doubles were also narrowed to `long`, changing `1e20`\ninto `9223372036854775807`. Simply emitting untyped floating-point text\nis insufficient: PostgreSQL interprets exponent literals as exact\nNUMERIC values, which can filter out a BIGINT row after CAST removal,\nwhile Oracle NUMBER literals cannot represent the full DOUBLE range.\n\nThe final change:\n- Groups compound operands across comparisons, IN, IS NULL, LIKE/REGEXP,\nBETWEEN, and arithmetic.\n- Uses explicit DOUBLE PRECISION literals for PostgreSQL and\nBINARY_DOUBLE literal suffixes for Oracle/OceanBase Oracle mode; retains\nfinite floating-point text for other dialects.\n- Keeps non-finite values and empty IN lists local and suppresses source\nLIMIT when filters cannot be translated.\n- Updates 23 EXPLAIN assertions in the MySQL, ClickHouse, and Oracle\nregression suites, including assertions after the original first\nfailures.\n- Adds unit coverage for grouping, remote literal types, BIGINT boundary\ncomparisons, floating-point extremes, signed zero, and local fallback.\n\n### Release note\n\nFix JDBC predicate pushdown that can miss matching rows because of\nexpression precedence changes or floating-point literal conversion.\nPreserve PostgreSQL floating-point comparison semantics and support\nOracle floating-point values outside the NUMBER range.\n\n### Check List (For Author)\n\n- Test\n    - [ ] Regression test\n    - [x] Unit Test\n- `bash run-fe-ut.sh --run\norg.apache.doris.connector.jdbc.JdbcQueryBuilderTest,org.apache.doris.connector.jdbc.JdbcTimestampProjectionTest`:\n66 tests passed, 0 failures/errors/skipped (52 query-builder tests and\n14 timestamp regression tests).\n- `bash run-regression-test.sh --compile` with Maven 3.9.12: passed; 50\nframework unit tests passed.\n- All three modified Groovy suites compiled successfully with the built\nframework.\n- PostgreSQL 18.6 (PGlite 0.5.14) SQL probe: 38 type, comparison,\nIN/BETWEEN, extreme-value and raw-bit assertions passed. The three\nBIGINT counterexamples fail with the previous untyped spelling and pass\nwith the typed literal.\n- Checkstyle: 0 violations; connector import boundary and `git diff\n--check` passed.\n- No Doris cluster end-to-end or live Oracle/OceanBase regression run\nwas performed locally. The standalone PostgreSQL probe does not claim\nDoris end-to-end coverage. Full distribution packaging remains outside\nthe local validation completed here.\n\n- Behavior changed:\n    - [ ] No.\n- [x] Yes. Preserve expression grouping, floating-point values and\nremote literal types; keep unsupported literal/list forms local.\n\n- Does this need documentation?\n    - [x] No.\n    - [ ] Yes.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label"
    },
    {
      "commit": "8fa3c97115a96fe5924750c502b314f2b7949c16",
      "tree": "56bb2c041a3d14700346518c95392b46ff2e4999",
      "parents": [
        "827973862a10224168c36b73cb02e12367313d76"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Thu Nov 05 07:11:36 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 15:11:36 2026 +0800"
      },
      "message": "[test](regression) Make test_http_tvf independent of HuggingFace and loopback hosts (#70232)\n\n### Background\n\nThe `http()` table-valued function reads CSV, JSON, Parquet, and ORC\nfiles from HTTP(S) URLs; for an `hf://` URI the FE expands the\nHuggingFace path into download URLs.\n`external_table_p0/tvf/test_http_tvf` (group `p0`, so every P0 run\nexecutes it) serves most of its fixtures from an HTTP server it starts\ninside the regression runner, and builds their URLs from `httpTvfHost`,\nelse `externalEnvIp`, which is `127.0.0.1` in every pipeline config. Its\nlast seven queries read public HuggingFace datasets through `hf://`.\n\n### What problem does this PR solve?\n\nIssue Number: None\nRelated PR: #65877, #66013, #66027 (branch-4.1 fixes by @hello-stephen,\nported here), #65299 (added the in-runner HTTP server and its host\nlookup), #60915 (moved the suite into P0 and relaxed sql15 / sql16 to\n`count \u003e 0`)\nPR Type: Test\n\nProblem Summary:\n\nOn master the suite still depends on huggingface.co and on HuggingFace\u0027s\nmutable `main` branches, and it advertises the runner as `127.0.0.1`,\nwhich a BE on another host resolves to itself. This PR ports the three\nbranch-4.1 fixes: a loopback or wildcard `externalEnvIp` falls back to\nthe runner\u0027s host address (an explicit `httpTvfHost` is still used as\ngiven), the seven `hf://` reads use byte-identical mirrors in the\nregression OSS bucket pinned to a dataset revision, and the mirrored\n`prompts.csv` skips its header line. With the mirror pinned, sql15 /\nsql16 assert the exact count 204 again.\n\n### Details\n\n- **Failure:** None on master CI today: P0 passed 66 of 66 runs on\n2026-11-04, though the HuggingFace queries took about 146 of the suite\u0027s\n149 seconds (build 1092587). On branch-4.1 the suite failed with\n`Connection refused` on `http://127.0.0.1:\u003cport\u003e` with remote BEs\n(#65877), with `java.net.SocketTimeoutException: Connect timed out` to\nHuggingFace (#66013), and then with sql15 getting 205 instead of 204 on\nthe mirror (#66027).\n- **Cause:** The suite assumed that every BE runs on the runner\u0027s host,\ntrue in P0 only, and that huggingface.co is reachable with fixed `main`\nbranches: `prompts.csv` on `fka/awesome-chatgpt-prompts` `main` now has\nfive columns and is over six times the 204-record file the golden came\nfrom. The mirrored CSV keeps a header line that `hf://` hid, hence\n`skip_lines \u003d 1`.\n- **Stability evidence:** No query reaches huggingface.co any more. The\nonly remote host left is\n`doris-regression-hk.oss-cn-hongkong.aliyuncs.com` (public reads), which\nP0 already uses in `query_p0/test_parquet_dict` (28 of 28 recent runs\npassed). Since master\u0027s P0 does not fail today, there is no\nbefore-and-after failure rate.\n\n### Testing\n\n- `-forceGenOut` run of the suite on a local 1 FE + 1 BE cluster from\nmaster `4530f761fa1` (the FE carries local patches that don\u0027t touch the\nHTTP TVF), with P0\u0027s `externalEnvIp \u003d 127.0.0.1` and no `httpTvfHost`:\nthe runner advertised `http://192.168.1.164:\u003cport\u003e`, and the 22 blocks\ngenerated for sql01 to sql18 and both `replaced_*` checks are\nbyte-identical to master\u0027s `.out`, with sql15 / sql16 now after\n`replaced_after`, where they run. sql19 then hit the 900 s query\ntimeout: this machine\u0027s BE reached the 20 MB Parquet mirror through a\nlocal proxy at about 10 KB/s (a direct GET from the same machine runs at\nabout 600 KB/s).\n- The suite without sql19 to sql21, in compare mode against the\ncommitted `.out`: passed. With `skip_lines` removed from sql15 it fails\nwith `Expect cell is: 204 But real is: 205`.\n- sql19 to sql21 did not run locally. Each mirrored Parquet file has the\nsame size as on HuggingFace and a byte-identical footer (schema, row\ngroups, statistics), and each footer\u0027s minimum `text` is the first row\nthe `.out` expects. This PR\u0027s P0 (build 1092831) then ran the whole\nsuite against the mirror: it passed in 8.9 s, with sql19, sql20 and the\ntwo sql21 queries taking 1.3 to 1.8 s each, and the runner advertised\n`http://172.16.0.146:\u003cport\u003e`; the 39 P0 runs before it, with the\nHuggingFace version, took 149 to 170 s.\n\n### Notes for Reviewers\n\n- Apart from a one-line comment on the loopback check, the suite now\nmatches branch-4.2\u0027s except for master\u0027s own parts: the `p0` label\n(#60915) and the `mutable/` cases (#68847).\n- `hf://` loses its only end-to-end coverage (real HuggingFace URLs,\ntree-API glob expansion, `@revision`); `HFUtilsTest` still covers\nparsing, URL conversion, glob matching, and one live `*` expansion\nagainst the HuggingFace tree API.\n- sql16, sql20, and the second sql21 now repeat sql15 / sql19, since\nplain HTTP has no globbing; they stay so that the goldens and branch-4.x\nline up.\n- In P0 the runner now advertises its host address instead of\n`127.0.0.1`; both work there, but branch-4.x labels the suite `p2`, so\nP0 runs this fallback for the first time.\n- `ai_p0/test_ai_profile` copied the old host lookup and has the same\nlatent loopback problem; it is not changed here.\n\nCo-authored-by: Dongyang Li \u003clidongyang@selectdb.com\u003e"
    },
    {
      "commit": "827973862a10224168c36b73cb02e12367313d76",
      "tree": "3d0e1f906886c3c4a7ac1e6ab1507b249d96a427",
      "parents": [
        "d03d410930e4e78bfe2730f55e6f7ebe200ab967"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Thu Nov 05 07:10:15 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 15:10:15 2026 +0800"
      },
      "message": "[test](fe) Port the tests branch-4.2 added for three master fixes (#70163)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #68206, #68636, #68368\n\nProblem Summary:\n\n**In short.** Three branch-4.2 backports of master fixes each added a\ntest: #68206 (of #61342), #68636 (of #57006) and #68368 (of #66354).\nMaster has the three fixes but none of the tests. This PR ports them.\nTwo guard a fix that has no test on master; the third keeps the\nClickBench `query1` shape case from failing whenever the exact row count\nof `hits` is cached. The PR changes tests only.\n\n**Background, and what each test covers.**\n\n- *Old primary key metadata (#61342, #68206).* A `PrimaryKeyConstraint`\nwritten before its foreign table collections existed has no `ft`, `ftn`\nor `ftni` in its JSON. Gson builds the object without its field\ninitializers, so these stay null unless `gsonPostProcess` fills them in,\nand an FE that replays the journal from the start (without an image, or\nwith `metadata_failure_recovery`) failed with a NullPointerException.\n#61342 added the null checks without a test; #68206 added one on\nbranch-4.2 for `ft`, the only collection there. The new\n`PrimaryKeyConstraintTest` removes all three from a serialized\nconstraint, loads it, and adds, renames and removes a foreign table on\nit.\n- *Shadow index columns (#57006, #68636).* During a schema change the\nnew schema names a changed column `__doris_shadow_\u003cname\u003e`.\n`Index.getColumnUniqueIds` compares the index\u0027s column names with the\nschema\u0027s after `Column.removeNamePrefix`, so the new tablet metadata\nkeeps the index column\u0027s unique id. #57006 has no test; #68636 added one\nto `IndexTest`, ported here to JUnit 5.\n- *ClickBench query1 shape (#66354, #68368).*\n`shape_check/clickbench/query1` checks the plan of `SELECT COUNT(*) FROM\nhits`. Once `SimpleAggCacheMgr` has cached the exact row count of\n`hits`, `REWRITE_SIMPLE_AGG_TO_CONSTANT` turns the count into a constant\nand the shape into `PhysicalOneRowRelation`, so the case fails whenever\nthat cache is warm when it runs. It did on P0 for #69830 (build 1087320:\nexpected `--hashAgg[GLOBAL]`, real `--PhysicalOneRowRelation`), a PR\nthat does not touch the planner. On branch-4.2, #68368 disables the rule\nin this case together with the cost model change it backports, which\nmaster has from #66354. This PR disables it the same way; the expected\noutput does not change.\n\n**Results.**\n\n| Change | Test | Result |\n|---|---|---|\n| none | `PrimaryKeyConstraintTest` (new), `IndexTest` (one new case),\n`ConstraintManagerTest`, `ConstraintPersistTest` | pass |\n| `gsonPostProcess` without its null checks | `PrimaryKeyConstraintTest`\n| fails: NullPointerException while loading |\n| `getColumnUniqueIds` without `removeNamePrefix` |\n`IndexTest#testGetColumnUniqueIdsForShadowColumn` | fails: `expected\n[107] but was []` |\n| `query1` as on master, row count of `hits` cached (local cluster) |\n`shape_check/clickbench/query1` | fails: `PhysicalOneRowRelation` |\n| `query1` with this PR, same cache | `shape_check/clickbench/query1` |\npasses |\n\nOn the local cluster, `explain shape plan SELECT COUNT(*) FROM hits`\ngives `hashAgg[GLOBAL]` the first time and `PhysicalOneRowRelation`\nthree seconds later, once the asynchronous row count load has finished;\nwith the rule disabled it stays `hashAgg[GLOBAL]`.\n\nCo-authored-by: minghong \u003czhouminghong@selectdb.com\u003e\nCo-authored-by: Jack \u003cjiangkai@selectdb.com\u003e\nCo-authored-by: feiniaofeiafei \u003cmoailing@selectdb.com\u003e"
    },
    {
      "commit": "d03d410930e4e78bfe2730f55e6f7ebe200ab967",
      "tree": "b860e251809dadb2bf2a6847b1e127768011936b",
      "parents": [
        "d5f624f29b3600fb0c11c44f42cf8c8051b4ecdb"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Thu Nov 05 07:09:52 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 15:09:52 2026 +0800"
      },
      "message": "[test](fe) Test forwarding prepared executions that reuse cached parameter types (#70150)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #68162, #69314\n\nProblem Summary:\n\n**In short.** This ports `MysqlPreparedStatementForwardingTest` from\nbranch-4.1 #68162 to master. Master already has the fix the test guards,\nthrough #69314. The port adds what master\u0027s own tests do not cover: a\nNULL bitmap of two bytes, and the forward request that `FEOpExecutor`\nbuilds and Thrift carries. The PR changes tests only.\n\n**Background.** A client that runs a server-side prepared statement\nagain may leave out the parameter types (`new_params_bind_flag` \u003d 0) and\nhave the server reuse the ones it cached. A follower forwards every\nCOM_STMT_EXECUTE to the master. The master prepares the statement again\nfor each forwarded execution, so it has no cached types and cannot bind\nthe parameters. #68162 fixed this on branch-4.1, together with the Lance\nprepared vector search it was written for. Master fixed it independently\nin #69314: when the client reuses cached types,\n`MysqlConnectProcessor.handleExecute` writes the forwarded packet again,\nwith the NULL bitmap, `new_params_bind_flag` \u003d 1, every type code\nincluding the unsigned flag, and the values. #69314 also has the prepare\nstage plan with the context of the re-parsed statement, which is the\nother generic part of #68162. Its tests in\n`MysqlConnectProcessorLongDataTest` cover cached types, unsigned flags,\nchanging NULLs, changing types, all-NULL executions and the prepare\ncontext.\n\n**What those tests leave out.** They use at most four parameters, so the\nNULL bitmap always fits in one byte. They also give the master the\nbuffer kept on the follower\u0027s context, rather than the request\n`FEOpExecutor.buildStmtForwardParams` builds.\n\n**What this PR adds.** The test runs a ten-parameter statement five\ntimes on a follower connection:\n\n- The parameter types are sent with the first and the fourth execution;\nthe other executions reuse the cached ones.\n- One execution has a NULL parameter in each byte of the NULL bitmap.\n- An unsigned INT parameter becomes a BIGINT in the fourth execution.\n\nAfter each execution, the test builds the forward request with\n`FEOpExecutor`, passes it through a Thrift round trip and executes it on\na new proxy context whose placeholders have no types. It then checks\nthree things: every parameter binds to the same value as on the\nfollower, the unsigned flag carries over, and the client\u0027s packet is\nread to its end and left unchanged. The code is the 4.1 test adapted to\nmaster\u0027s structure: master\u0027s `PrepareCommand`, query id and audit setup,\nas in `MysqlConnectProcessorLongDataTest`, and expressions compared\ndirectly. The 4.1 assertion that the follower keeps the client\u0027s buffer\nwas dropped, because master writes a separate buffer for cached types by\ndesign.\n\n**Results.**\n\n| `handleExecute` on master | `MysqlConnectProcessorLongDataTest` (14) |\n`MysqlPreparedStatementForwardingTest` (1) |\n|---|---|---|\n| as is | pass | pass |\n| no rewrite of the forwarded packet for cached types (the bug #68162\nfixed) | 3 fail | fails: the master binds untyped placeholders (`No\nvalue present`) |\n| the rewrite keeps only the first byte of the NULL bitmap | pass |\nfails: the master reads the packet one byte off |\n\nCo-authored-by: Gabriel \u003cliwenqiang@selectdb.com\u003e"
    },
    {
      "commit": "d5f624f29b3600fb0c11c44f42cf8c8051b4ecdb",
      "tree": "33b73ef44b471949f39a717f7415a20c10b84730",
      "parents": [
        "80994d8595fff86b102b0b73c88a613f823cf553"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Thu Nov 05 07:09:31 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 15:09:31 2026 +0800"
      },
      "message": "[fix](fe) Honor enable_local_exchange_before_streaming_agg in the FE local shuffle planner (#70177)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #66216, #66222, #63766\n\nProblem Summary:\n\n**In short.** This brings the FE local shuffle planner in line with\n`enable_local_exchange_before_streaming_agg`, which branch-4.1 / 4.2\n#66216 and master #66222 added on the BE side only. With the FE planner,\nwhich master uses by default, a streaming pre-aggregation still asked\nfor a local hash exchange on its grouping keys whatever the variable\nsaid. Master plans therefore had a `LOCAL_EXECUTION_HASH_SHUFFLE` local\nexchange that the BE planner, and branch-4.x, do not add, for example\nright above a `UNION ALL` or an external table scan. The FE now follows\nthe same rule as BE `StreamingAggOperatorX`.\n\n**Background.** #66222 added\n`enable_local_exchange_before_streaming_agg` (default false) so that a\nstreaming pre-aggregation keeps the distribution it inherits. It\nreplaced `enable_local_exchange_before_agg` for streaming aggregation.\nIn `StreamingAggOperatorX::required_data_distribution()`, the operator\nasks for a hash local exchange on its partition expressions only when\nthe variable is on, or when\n`OperatorBase::child_breaks_local_key_distribution()` (#63766) says the\nchild spreads rows without regard to any key: the child is serial, or\nthe child\u0027s own required distribution is a non-hash local exchange\n(PASSTHROUGH, ADAPTIVE_PASSTHROUGH, ...).\n\n**The problem, and what it cost.** On master the FE plans local\nexchanges (`enable_local_shuffle_planner`, default true) and BE does\nnot. `AggregationNode.enforceAndDeriveLocalExchange()` mirrors the BE\noperators, but its streaming aggregation branch still had the rule from\nbefore #66222. It never read the new variable. It skipped the hash\nrequirement only when `enable_local_exchange_before_agg` was off, and\nthen without the child check. With the default settings it required a\nhash distribution for every streaming pre-aggregation with grouping\nkeys, so:\n\n- Below a streaming pre-aggregation whose child keeps the key\ndistribution but does not report a hash distribution, the FE inserted a\n`LOCAL_EXECUTION_HASH_SHUFFLE` that BE planning does not. Examples are a\n`UNION ALL` and an external table scan that is not serial. Every row was\nhashed and moved once more before the pre-aggregation.\n- `enable_local_exchange_before_streaming_agg\u003dtrue` had no effect with\nthe FE planner.\n- With `enable_local_exchange_before_agg\u003dfalse`, the FE dropped the hash\nrequirement even after a child that breaks the key distribution, where\nBE keeps it.\n\nNone of this changes query results: a streaming pre-aggregation only\naffects performance, and the operators that need hash-distributed input\nfor correctness ask for it themselves.\n\n**How this PR fixes it.** The streaming aggregation branch of\n`AggregationNode.enforceAndDeriveLocalExchange()` now follows BE: after\nthe existing hash join passthrough rule, it keeps the base class\nrequirement (no exchange, or PASSTHROUGH over a serial child) unless\n`enable_local_exchange_before_streaming_agg` is on or the child breaks\nthe key distribution. `enable_local_exchange_before_agg` no longer\naffects streaming aggregation. The new\n`childBreaksLocalKeyDistribution()` mirrors\n`child_breaks_local_key_distribution()` with each BE operator\u0027s\n`required_data_distribution()`:\n\n| Child | BE operator\u0027s required distribution | Breaks the key\ndistribution |\n|---|---|---|\n| serial on BE (pooling scan, serial exchange, ...) | — | yes |\n| nested loop join | ADAPTIVE_PASSTHROUGH, NOOP for null-aware anti,\nright outer / semi / anti and full outer joins | yes, except those joins\n|\n| broadcast hash join | PASSTHROUGH when forced\n(`enable_broadcast_join_force_passthrough`) or when the probe child is\nserial | in those two cases |\n| table function, assert num rows | PASSTHROUGH | yes |\n| anything else (scan, exchange, union, partitioned join, ...) | NOOP or\na hash distribution | no |\n\n`SelectNode` and `RepeatNode` take the serial flag of their child, so\nthe serial check already covers the base class rule (PASSTHROUGH over a\nserial child) for them.\n\n**Results.** Local exchanges in the plans of three queries on a one-BE\ncluster, `parallel_pipeline_task_num\u003d4`, non-pooling scans, taken from\nthe query profiles:\n\n| Query | BE planner | FE planner before | FE planner after |\n|---|---|---|---|\n| `GROUP BY` over `UNION ALL` of two tables | none | LOCAL_HASH_SHUFFLE\n| none |\n| `GROUP BY` over a nested loop join | ADAPTIVE_PASSTHROUGH,\nLOCAL_HASH_SHUFFLE | same | same |\n| `GROUP BY` over an Iceberg table with 6 data files | none |\nLOCAL_HASH_SHUFFLE | none |\n\nA planner probe over 72 plans (8 query shapes × pooling / non-pooling\nscans × `agg_phase` 0 / 2 × the variable off / on) found only the `UNION\nALL` plans changed with the variable off. A scan of an OLAP table, a\nnested loop join, `explode`, broadcast and shuffle joins, a `LIMIT`\nsubquery and `GROUPING SETS` gave the same plans as before."
    },
    {
      "commit": "80994d8595fff86b102b0b73c88a613f823cf553",
      "tree": "34b1a3ae399bc5427fd354d8ae9c595563b0f4bb",
      "parents": [
        "bc5f8cd7c800245b925faefef0a7c01467fef6a9"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Thu Nov 05 07:09:13 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 15:09:13 2026 +0800"
      },
      "message": "[test](iceberg) Cover equality deletes on keys missing from older data files (#70226)\n\n### Background\n\nAn Iceberg equality delete removes the data rows whose key columns equal\none of its rows, its tombstones. A table can add a column after some\ndata files were written and then use it as a key; those files don\u0027t\nstore it, so Iceberg reads it there as the column\u0027s initial default, or\nNULL when it has none, and NULL equals only a NULL tombstone. The BE\u0027s\nV2 Iceberg reader builds such a missing key as a literal from the schema\nmetadata that the FE sends with the scan, and matches a data file\nwithout field ids, written before `migrate` or `add_files`, through the\ntable\u0027s name mapping.\n\n### What problem does this PR solve?\n\nIssue Number: None\nRelated PR: #70093 (an earlier version of these tests, built on old-FE\nplans; closed)\nPR Type: Test\n\nProblem Summary:\n\nA missing key without an initial default must read as NULL and match\nNULL tombstones only; master tests that on Parquet alone, with a single\nkey, almost always in files with field ids. This PR adds two matrices,\nfor a top-level and a nested missing key, on the scan plan the current\nFE sends, across ORC and Parquet, files with and without field ids, and\ncomposite and single-key deletes. It also fixes the ORC fixture writer,\nwhich stored child values under NULL parents and so shifted later rows.\nOnly test code changes.\n\n### Details\n\n- **Change map:** 338 added lines, all in\n`be/test/format_v2/table/iceberg_reader_test.cpp`: the two matrices\n(218), helpers that build the current FE\u0027s scan plan and name mappings\n(41), a writer for the top-level key\u0027s delete files (73), and the ORC\nfixture fix (6).\n- **Scenarios:** `IcebergMissingEqualityKeyMatchesOnlyNullTombstones`,\n24 cases: a file with `id` \u003d 1, 2, 3, written before the table added the\noptional key `added_key` (INT or STRING, no initial default), in Parquet\nor ORC, with or without field ids. The delete (id, added_key) \u003d (2,\nNULL), (3, 0 or \u0027\u0027) removes row 2 only; a single NULL tombstone removes\nevery row, with a predicate of NULL literals alone; a single 0 or \u0027\u0027\ntombstone removes nothing.\n`IcebergMissingNestedEqualityKeyMatchesOnlyNullTombstones`, 8 cases: the\nkey `payload.k`, added after a file that stores `id` and a nullable\n`payload {existing}`, NULL in row 2, in Parquet or ORC, with or without\nfield ids, the query reading `id` only or `payload` too. The tombstones\n(1, NULL payload) and (3, 0) leave rows 2 and 3, and a projected\n`payload` keeps its values.\n- **What existing tests miss:** The closest tests use the missing key as\nthe only key, against a NULL tombstone on Parquet only\n(`IcebergEqualityDeleteMatchesNullForMissingDataColumn`,\n`IcebergEqualityDeleteMatchesNullMissingKeyUnderUnknownTableType`) or\nagainst tombstones far from a NULL slot\u0027s value, such as 7\n(`IcebergEqualityDeleteMissingKeyDoesNotReadUnsupportedUnprojectedCarrier`,\n`IcebergMissingWholeStructEqualityKeyMaterializesDefault`). None matches\na NULL tombstone in ORC, uses a STRING key, pairs the missing key with a\npresent one, or deletes with a tombstone equal to a NULL slot\u0027s value; a\nkey missing under a present ancestor is tested only with a non-NULL\ndefault (`IcebergMissingNestedEqualityKeyPreservesNullableParent`).\n- **Fails when broken:** Two mutations, each run against all of\n`IcebergV2ReaderTest`. Reading a missing key without a default as its\ntype\u0027s default, 0 or \u0027\u0027, instead of NULL\n(`build_missing_equality_delete_key_expr`) fails all 32 new cases.\nLetting a NULL data key also match, and hash as, the value its nested\ncolumn holds (`EqualityDeletePredicate`) fails the 24 new cases with a\nnon-NULL tombstone, while the 8 single NULL-tombstone cases pass, as\nthey should. The two mutations also fail 3 and 5 existing tests.\n\n### Testing\n\n- `run-be-ut.sh --run\n--filter\u003d\u0027IcebergV2ReaderTest.*:IcebergTableReaderTest.*\u0027`, locally on\nmacOS (arm64, ASAN build) with `be/src/format_v2` and\n`be/test/format_v2` as on this PR\u0027s head: 129 tests pass, the 2 new ones\nincluded.\n\n### Notes for Reviewers\n\n- `write_nested_equality_orc_file` now marks a child slot NULL under a\nNULL parent. The ORC writer stores a child\u0027s PRESENT bits only for rows\nwhose parent is present but its values for every row the child batch\nmarks present, so `existing` \u003d {10, 20, 30} under a NULL row 2 read back\nas 10, NULL, 20. The projected-`payload` case exposed it; the helper\u0027s\nother callers never read a child value after a NULL parent and pass\nunchanged.\n- The cases without field ids carry an authoritative name mapping on\nevery field, as `migrate` and `add_files` set it and the current FE\nsends it: under scan semantics v2 such a file is read by name only\nthrough one (`IcebergTableReader::_mapping_mode`)."
    },
    {
      "commit": "bc5f8cd7c800245b925faefef0a7c01467fef6a9",
      "tree": "2dde4bc73fb0535b2375043b715ed173ed2bf81f",
      "parents": [
        "897bb25498454ad9f4b6f2386167dc8422b337c1"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Thu Nov 05 07:08:02 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 15:08:02 2026 +0800"
      },
      "message": "[fix](fe) Skip HMS database events that include/exclude_database_list filters out (#70165)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #67417, #68196\n\nProblem Summary:\n\n**In short.** This ports from branch-4.x #67417 and #68196 the check\nthat keeps a metastore event from adding a database that\n`include_database_list` or `exclude_database_list` keeps out of an HMS\ncatalog. On master, a CREATE DATABASE event for such a database puts it\ninto the catalog\u0027s database list on every FE that applies the event:\nSHOW DATABASES lists it and queries can reach it until the list is\nreloaded.\n\n**Background.** A catalog lists its databases through\n`ExternalCatalog#getFilteredDatabaseNames`, which drops a remote\ndatabase that `exclude_database_list` names or, when\n`include_database_list` is set, one that it does not name. On an HMS\ncatalog with `hive.enable_hms_events_incremental_sync`,\n`MetastoreEventSyncDriver` applies the metastore\u0027s events on every FE. A\nCREATE DATABASE event, and the new name of an ALTER DATABASE rename,\nreach `PluginDrivenExternalCatalog#registerDatabaseFromEvent`, which\nadds the database to the cached database names. branch-4.1 and\nbranch-4.2 have checked the two lists for these events since #67417.\n\n**The problem, and what it cost.** `registerDatabaseFromEvent` adds the\ndatabase without the check. A database that the catalog is configured to\nhide then shows in SHOW DATABASES, `USE` and queries reach it\n(privileges still apply), and table events register tables in it. It\ngoes away only when the cached names are reloaded: after\n`external_cache_refresh_time_minutes` (10 by default), on a lookup of a\nname that is not cached, on REFRESH CATALOG, or after a restart.\n\n**How this PR fixes it.** The check moves out of\n`getFilteredDatabaseNames` into `ExternalCatalog#isDatabaseListed`,\nwhich the listing and `registerDatabaseFromEvent` share; an event for a\ndatabase the lists keep out returns before it touches any cache. Like\nthe listing, the check compares the remote name exactly. Not ported from\nbranch-4.x: skipping the table, partition and refresh events of such\ndatabases, a performance item (on master they find no database and\nreturn, after one reload of the names), and comparing names ignoring\ncase when `lower_case_database_names` is not 0, which would change the\nlisting as well.\n\n**Results.**\n\n| Event, with the database names cached | Before | After |\n|---|---|---|\n| CREATE DATABASE `archived`, `exclude_database_list \u003d archived` |\n`archived` listed | not listed |\n| CREATE DATABASE `other`, `include_database_list \u003d sales,sales2` |\n`other` listed | not listed |\n| CREATE DATABASE `sales2`, allowed by either list | listed | listed |\n\nOriginal author: @924060929\n\nCo-authored-by: 924060929 \u003clanhuajian@selectdb.com\u003e"
    },
    {
      "commit": "897bb25498454ad9f4b6f2386167dc8422b337c1",
      "tree": "d6618c4d1b601eb4cac76d3c54c3f71884c47d47",
      "parents": [
        "dac77e06562bdb35690b49c02475822e45dd0d43"
      ],
      "author": {
        "name": "daidai",
        "email": "changyuwei@selectdb.com",
        "time": "Thu Nov 05 14:42:29 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 14:42:29 2026 +0800"
      },
      "message": "[fix](iceberg) Validate Parquet compression before file creation (#69376)\n\n### What problem does this PR solve?\n\nIssue Number: N/A\n\nRelated PR: #68854, #69338\n\nProblem Summary:\nThe Iceberg data, position-delete, and Hive Parquet writers created\ntheir output file before validating the requested Parquet compression\ncodec. For an unsupported codec, the Iceberg data writer then attempted\nto delete an open file, while the other writers returned after creating\nthe file. This made error cleanup depend on filesystem behavior and\ncould leave an orphan file. Resolve the codec before calling\n`create_file` so invalid codecs fail without creating any output file,\nand add a regression test for the failure path.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test\n    - [ ] Regression test\n- [x] Unit Test: added `UnsupportedParquetCodecDoesNotCreateFile`; CI\nwill run it.\n    - [ ] Manual test (add detailed scripts or steps below)\n    - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n        - [ ] Previous test can cover this change.\n        - [ ] No code files have been changed.\n        - [ ] Other reason\n\n- Behavior changed:\n    - [x] No.\n    - [ ] Yes.\n\n- Does this need documentation?\n    - [x] No.\n    - [ ] Yes.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label"
    },
    {
      "commit": "dac77e06562bdb35690b49c02475822e45dd0d43",
      "tree": "fc1e623ab9ec70348ef1a07afcf5307e65012f53",
      "parents": [
        "040cdca6fcb99c6415f51c026eb9a45d6c6d40db"
      ],
      "author": {
        "name": "Jerry Hu",
        "email": "hushenggang@selectdb.com",
        "time": "Thu Nov 05 14:39:26 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 14:39:26 2026 +0800"
      },
      "message": "[fix](be) Report sequence pattern iteration exhaustion as an error (#69834)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: None\n\nProblem Summary: For time-constrained `sequence_match` and\n`sequence_count`, both the deterministic prefilter and the backtracking\nsearch returned `false` when their one-million-iteration budget was\nexceeded. A valid event can still appear later in the sequence, so a\ncompleted query could incorrectly return `false`, `0`, or a partial\ncount. Report `INVALID_ARGUMENT` from both exhaustion paths instead of\nreturning an incomplete result. The existing iteration limit, comparison\nboundary, and results for searches that complete within the budget are\nunchanged.\n\n### Release note\n\nTime-constrained `sequence_match` and `sequence_count` now return an\nexplicit error when pattern evaluation exceeds one million iterations\ninstead of silently returning an incomplete result.\n\n### Check List (For Author)\n\n- Test:\n- Unit Test: `./run-be-ut.sh --run --filter\u003d\u0027VSequenceMatchTest.*\u0027 -j\n16` (13/13 passed).\n- Regression test: `test_sequence_match_iteration_limit` in\n`query_p0/sql_functions/aggregate_functions`; generated expected output\nand reran the suite (1/1 passed). The suite checks both SQL errors and\nnormal match/no-match results.\n- Build and checks: ASAN BE+FE build with FE Checkstyle; BE\nclang-format, check-format, build hygiene, clang-tidy, and\ncommitted-diff checks passed.\n- Behavior changed: Yes. Iteration-budget exhaustion now fails the query\nwith `INVALID_ARGUMENT` rather than returning an incomplete result.\n- Does this need documentation: No"
    },
    {
      "commit": "040cdca6fcb99c6415f51c026eb9a45d6c6d40db",
      "tree": "59db14f91fb23efb265857f51908b362502bcabf",
      "parents": [
        "913532ce1673595cd552bd6f582e5b2640f45dd9"
      ],
      "author": {
        "name": "Mryange",
        "email": "yanxuecheng@selectdb.com",
        "time": "Thu Nov 05 14:34:47 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 14:34:47 2026 +0800"
      },
      "message": "[fix](function) Preserve TIMESTAMPTZ in microsecond interval arithmetic (#70215)\n\n### What problem does this PR solve?\n\nMicrosecond interval arithmetic can convert low-precision TIMESTAMPTZ\ninputs to DATETIMEV2. Some composite interval functions also overwrite\nthe TIMESTAMPTZ signature during precision inference.\n\nUse wildcard family signatures and preserve TIMESTAMPTZ while promoting\narguments and results to scale 6. This covers composite microsecond\nintervals and microsecond/millisecond addition and subtraction. Existing\nBE implementations require no changes.\n\n### Release note\n\nFix TIMESTAMPTZ type and microsecond precision preservation in interval\naddition and subtraction.\n\n### Check List (For Author)\n\n- Test:\n- [x] Regression test: `test_timestamptz_microsecond_interval` passed\nwith `-forceGenOut` and a normal rerun; covers 36 expressions and result\ncolumn types.\n- [x] Unit Test: focused FE signature test passed, covering 7 input\nscales across 12 functions.\n    - FE build (`build.sh --fe -j48`) and Checkstyle passed.\n- Behavior changed:\n- [x] Yes. Affected TIMESTAMPTZ expressions preserve their type and\nreturn scale 6.\n- Does this need documentation?\n    - [x] No.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label"
    },
    {
      "commit": "913532ce1673595cd552bd6f582e5b2640f45dd9",
      "tree": "5895818c42fdd996a0d80baf697368459c2bfe93",
      "parents": [
        "c5bfd6819f393bde37b73147abddc2f9a44641cb"
      ],
      "author": {
        "name": "Gabriel",
        "email": "liwenqiang@selectdb.com",
        "time": "Thu Nov 05 12:03:04 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 12:03:04 2026 +0800"
      },
      "message": "[feature](fe) Add Iceberg encrypted scan wire contract (#70213)\n\n### What problem does this PR solve?\n\nRelated PRs: #70159, #70192\n\nPrepare the file metadata and diagnostic protection required for native\nIceberg encrypted reads. Data files and each delete file need\nindependent binary keys and AAD. Generated Thrift strings and JSON\ndiagnostics must not expose those values.\n\n- Define `TFileEncryptionInfo` directly in `PlanNodes.thrift`, alongside\nits data/delete consumers. The optional carrier contains a payload\nversion, binary file key and exact AAD bytes; malformed payloads are\nrejected. There is no separate encryption IDL file.\n- Assume FE and BE are deployed together. This PR adds no encryption\nheartbeat fields, receiver capability negotiation, capability\npersistence or downgrade/replay handling.\n- Validate assigned fragment parameters/ranges and every lazy split\nresponse. FE and BE uniformly reject encrypted execution with\n`READ_UNAVAILABLE` before readers, caches or metadata COUNT shortcuts. A\ndenied final split batch stops producers and remains an error on\nrepeated fetches.\n- Keep the BE file-encryption validator inside\n`file_scan_range_utils.h`, alongside its only production caller, with no\nseparate validator header.\n- Redact generated Java/C++ strings, nested output, ThriftDebugString,\nGson and Thrift JSON diagnostics. Binary/Compact retain the real\npayload. The generation transform targets only the secret carrier inside\n`PlanNodes_types.cpp`, leaves other plan diagnostics unchanged, is\nidempotent and fails on compiler-shape drift. FE, regression-framework\nand C++ generation use the same protection.\n- Keep encrypted SQL reads disabled until native decryption and\nauthenticated, confidential transport for both fragment dispatch and\nlazy split responses are implemented. The Iceberg option cannot bypass\nthese requirements.\n\nThe read and wire contracts document this scope. File-key extraction and\nnative decryption are subsequent roadmap work.\n\n### Validation\n\n- Full FE build and FE Checkstyle passed with Maven build cache\ndisabled.\n- Official focused FE tests: 55 passed (10 protocol tests and 45 FE\ntests), zero failures/errors/skips. Covers independent data/delete keys,\ndiagnostics, malformed payloads, plain scans, direct/lazy dispatch and\nterminal last-batch retries.\n- SQL normal comparisons passed: the wire guard suite executed 80\nexpected rejections with plaintext rows/counts; the existing D1\nadmission suite executed 96 expected rejections with plaintext\ncomparisons. Covers v2/v3, actual lazy planning, COUNT pushdown,\nrepeated queries, malformed versions and key lengths. Existing generated\nresult files remain valid and unchanged.\n- Compiled the actual regenerated `PlanNodes_types.cpp` with the\nexisting C++ contract suite: 4 tests passed. Generator self-tests: 4\npassed, including idempotence, compiler drift and preservation of other\nplan diagnostics. Regression-framework generation also passed with the\npinned Thrift compiler.\n- Clang-format 16, clang-tidy and BE build hygiene passed on affected\nfiles. Packaged heartbeat classes no longer contain either removed field\nor the removed backend state class.\n- The updated guard expectations failed against the old implementation,\nthen passed after simplification.\n- Reproduced all three `PluginDrivenScanNodePartitionBatchTest`\nfailures: its fixture omitted the required `scan_range`, so lazy\ndispatch validation dereferenced a missing range. Completed the fixture\nwith the same plaintext file-scan envelope used by production and\nneighboring streaming tests, preserving split identities and all\nexisting assertions. The official expanded FE run then passed 383 tests\nacross 46 classes, with zero failures/errors/skips, including all nine\npartition-batch tests, plugin scan, split\nproducer/consumer/assignment/compaction/concurrency, and encryption\nprotocol/guard coverage. FE Checkstyle passed with zero violations. This\nfollow-up changes only the unit-test fixture.\n\n- Inlined the unchanged BE validator and documented build-time redaction\nversus runtime wire/diagnostic serialization. Four standalone ASAN C++\ncontract tests passed against freshly regenerated `PlanNodes` code, and\nall four Python generator tests passed. Three relevant BE test\ntranslation units passed syntax checks using the existing ASAN compile\ndatabase. Clang-format 16, clang-tidy and header hygiene passed. Python\ntransform logic and Thrift field definitions are unchanged.\n\nThe SQL tests use the updated FE and an existing ASAN BE. Native\nencrypted SQL interoperability is outside this protocol-preparation\nstage. Full BE ASAN unit execution was not rerun in this update: the\nearlier attempt stopped in the unchanged baseline\n`vparquet_writer_test.cpp` because its `WriteInfo` initializer still\nuses removed path members. The standalone C++ tests do not replace a\nfull ASAN run.\n\n### Release note\n\nPrepare per-file Iceberg encryption metadata and redacted diagnostics.\nEncrypted SQL reads remain disabled.\n\n### Check List (For Author)\n\n- Test\n    - [x] Regression test\n    - [x] Unit Test\n    - [x] Manual test (full FE build and local external stack)\n- Behavior changed:\n- [x] Yes. Encrypted file payloads are validated and uniformly rejected\nuntil execution is implemented.\n- Does this need documentation?\n    - [x] Yes. Read and wire contracts are included.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label"
    },
    {
      "commit": "c5bfd6819f393bde37b73147abddc2f9a44641cb",
      "tree": "eb5c7ba87af7315dd2b824adabba673f4a3e6f1d",
      "parents": [
        "0d0d56eced63288ac565e33c256cbcf2701a18e5"
      ],
      "author": {
        "name": "Mryange",
        "email": "yanxuecheng@selectdb.com",
        "time": "Thu Nov 05 11:40:48 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 11:40:48 2026 +0800"
      },
      "message": "[opt](expr) reuse IFNULL input batches without nulls (#70110)\n\n### What problem does this PR solve?\n\nIssue Number: N/A\n\nProblem Summary:\n\nIFNULL rebuilds its result row by row when its first argument is a\nnullable column, even if the current batch contains no NULLs. This\ncauses unnecessary allocations and string copies.\n\nInspect the null-map state and reuse the first argument for batches\nwithout NULLs, preserving the result type\u0027s nullable wrapper. Keep the\nexisting all-NULL and mixed-NULL paths.\n\n### Release note\n\nImprove IFNULL performance for nullable input batches without NULLs.\n\n### Check List (For Author)\n\n- Test\n    - [ ] Regression test\n    - [ ] Unit Test\n    - [x] Manual test\n\nThe same modified source was built and tested on an ARM Release\nenvironment. All 25 SQL checks matched before and after the change,\ncovering all-NULL, no-NULL and mixed batches, nullable/nonnullable\nresults, numeric and string inputs, empty input, and shared input\ncolumns. The target checkout was not rebuilt; full BE UT and regression\nsuites were not run. clang-format 16, build hygiene, and git diff\n--check passed locally.\n\n- Behavior changed:\n    - [x] No.\n    - [ ] Yes.\n\n- Does this need documentation?\n    - [x] No.\n    - [ ] Yes.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label"
    },
    {
      "commit": "0d0d56eced63288ac565e33c256cbcf2701a18e5",
      "tree": "ec3153a13a40262741453c01fb120d2e3111630d",
      "parents": [
        "315c2e003317ef44e8a8fbb445a1ec8452d5647e"
      ],
      "author": {
        "name": "yiguolei",
        "email": "guolei@selectdb.com",
        "time": "Thu Nov 05 11:30:19 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 11:30:19 2026 +0800"
      },
      "message": "[chore](be) Update core and common code owners and remove stale paths (#70241)\n\n### Background\n\nGitHub uses `.github/CODEOWNERS` to select code owners for changed\npaths. The file has no ownership rules for BE core and common, and still\nlists two paths that no longer exist.\n\n### What problem does this PR solve?\n\nIssue Number: None\nRelated PR: None\nPR Type: Build/CI/Chore\n\nProblem Summary:\n\nAssign `@yiguolei` and `@Mryange` as code owners for `be/src/core` and\n`be/src/common`. Remove the stale FE `fsv2` and BE\n`olap/rowset/segment_v2/variant` rules.\n\n### Details\n\n- **Affected:** `.github/CODEOWNERS` only; two rules added and two\nremoved.\n- **Developer and release impact:** Updates ownership for BE core/common\nchanges and removes rules for nonexistent paths. No Doris runtime\nbehavior changes.\n- **License and third-party:** None.\n- **Rollback:** Revert the CODEOWNERS changes.\n\n### Testing\n\n- `git diff --check` passed locally.\n- Validated all 148 remaining rules against Git-tracked files; every\nrule matches.\n- Verified the effective core/common rules list exactly `@yiguolei\n@Mryange`.\n- No build or runtime tests run because this only changes GitHub\nownership metadata.\n\n### Release note\n\nNone.\n\n### Check List (For Author)\n\n- Behavior changed: No\n- Does this need documentation: No\n- Pre-submission status: Ready\n\n### AI Disclosure\n\n- Tool: OpenAI Codex.\n- Generated: CODEOWNERS changes and this description.\n- Human review: Pending."
    },
    {
      "commit": "315c2e003317ef44e8a8fbb445a1ec8452d5647e",
      "tree": "025fa2d7706e5281e948250c8fed627bbcb67e2b",
      "parents": [
        "28eb75d0d3dbf481c7e00c7e4e0c2ec9b1307347"
      ],
      "author": {
        "name": "Gabriel",
        "email": "liwenqiang@selectdb.com",
        "time": "Thu Nov 05 10:09:00 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 10:09:00 2026 +0800"
      },
      "message": "[feature](fe) Prepare encrypted Iceberg HMS metadata reads (#70192)\n\n### What problem does this PR solve?\n\nRelated PR: #70159\n\nThe authenticated Iceberg FileIO wrapper inherits typed defaults that\nreject encrypted data, delete, manifest and manifest-list descriptors\nbefore the SDK encryption delegate can handle them. Forward all four\ndescriptors inside the authentication scope, preserving their key\nmetadata and manifest-list key IDs.\n\nThis also provides the reproducible encrypted HMS metadata fixture\nneeded for the read roadmap:\n\n- Disposable Moto KMS, a separate HMS and a dedicated MinIO bucket in\nthe standard Iceberg stack. Reference Iceberg 1.12.0 artifacts are\nisolated, checksum pinned and readable by the non-root fixture user on\nboth fresh downloads and cache hits.\n- Real v3 encrypted Parquet, manifests and manifest lists, with two\nsnapshots and distinct file key metadata. Both the SDK generic reader\nand Spark SQL verify historical rows `[1, 2]` and `[1, 2, 3, 4]`.\n- Actual metadata-planning failures for missing KMS, injected AWS\ninvalid-credential/access-denied responses, and authenticated-decryption\nfailure with different KMS-wrapped key material. The injected responses\ntest propagation; they do not claim live AWS IAM validation.\n- SQL regression verifies plaintext results and 48 encrypted admission\nrejections across SELECT, COUNT, DESC and system tables, caching on/off,\ncount pushdown on/off and repeated reads. Required fixtures fail the\nenabled suite if unavailable. There is no additional test-mode switch.\n\nThe existing plugin already packages AWS KMS dependencies; this PR\nverifies that closure rather than adding a second dependency. Encrypted\nSQL execution remains blocked by the existing admission guard. Native\nencrypted file execution is subsequent work.\n\n### Validation\n\n- Reproduced the original reference-jar permission failure in the Spark\n4.0.0 container. After the fix, the actual fixture compiles as the\ndefault non-root Spark user with freshly downloaded pinned jars,\nrepaired mode-0600 cached jars, and mode-0644 cached jars owned by\nanother user. Both cache checks used an unavailable download source; the\nalready-readable shared cache retained its permissions and ownership\nwithout metadata changes. Shell syntax and diff checks passed.\n- Before the fix, all three encrypted descriptor tests failed at the\ninherited FileIO defaults. After the fix, the focused 16-test suite\npassed.\n- Full connector reactor package: 1,913 passed, 14 existing skips, zero\nfailures/errors.\n- Full FE build and FE Checkstyle passed.\n- `test_iceberg_encrypted_hms_metadata`: generated results using the\nstandard runner, then normal comparisons passed twice, including the\nfinal fixture source. Each run executed 48 expected encrypted rejections\nand verified plaintext rows/counts.\n- Real encrypted fixture verification passed with the SDK, Spark SQL and\nall four KMS/key failure checks.\n- Reproduced the external-CI endpoint failure: its unanchored `hms_port`\nsubstitution also rewrites the encrypted-fixture port, causing the suite\nto query the wrong metastore and report a missing database. Renamed the\nregression key to `iceberg_encryption_metastore_port` in both\nconfiguration entry points, the suite and the fixture documentation; the\nDocker mapping is unchanged. Verified the exact substitution with both\nlegacy port values on both configurations, then passed the full enabled\nsuite after applying it: 1 suite, 0 failures, 0 skips, including all 48\nexpected rejections and SDK/Spark/KMS/plaintext checks. This config-only\nrerun reused existing local FE/BE binaries.\n\n### Release note\n\nPreserve SDK encryption metadata through authenticated Iceberg FileIO.\nEncrypted SQL execution remains guarded."
    },
    {
      "commit": "28eb75d0d3dbf481c7e00c7e4e0c2ec9b1307347",
      "tree": "100652f3adc8d3a47199ae8301fc116c2713e6a7",
      "parents": [
        "19dfb8711ae3dbb8f56cc945f5dc7af20a844607"
      ],
      "author": {
        "name": "Xin Liao",
        "email": "liaoxin@selectdb.com",
        "time": "Thu Nov 05 10:02:15 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Thu Nov 05 10:02:15 2026 +0800"
      },
      "message": "[fix](cloud) Skip global packed index probe when opening files of committed rowsets (#69980)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #64995\n\nProblem Summary:\n\nA cloud BE logs a burst of WARN lines with a full stack trace like\n\nmeet error status: [NOT_FOUND]File not found in global packed index:\ndata/\u003ctablet\u003e/\u003crowset\u003e_0.idx\n\nwhile compaction reads its input rowsets. In one instance a 4-hour BE\nlog had 42k\nsuch lines over 21k paths in 158 tablets, yet every compaction\nsucceeded.\n\nReproduce: write a rowset whose segment file is smaller than\nsmall_file_threshold_bytes (so it is packed) but whose inverted index\nfile is\nlarger (so it is uploaded as a regular object), then compact or query\nit. Each\nopen of the regular .idx file prints the WARN with a stack, twice per\ncompaction\n(eligibility check and the merge itself).\n\nRoot cause: #64995 made PackedFileSystem::open_file_impl fall back to\nPackedFileManager::get_packed_slice_location when a path is not in\n_index_map,\nso that the file cache preload can read back a segment of the rowset\nbeing\nwritten, whose rowset meta does not exist yet. The fallback was added\nunconditionally, so it also runs for a PackedFileSystem built from a\nrowset\nmeta. That index map already holds every packed slice of the rowset, so\na miss\nmeans a regular file, and the probe can only fail. The failure is a\nStatus::NotFound, and NOT_FOUND records a stack trace and logs a WARN as\nsoon as\nit is constructed, before the caller falls back to opening the regular\nfile.\n\nFix:\n- Only consult the manager when the index map is not initialized, i.e.\non the\nwrite path, the same split exists_impl and file_size_impl already make.\n- Return NOT_FOUND without a stack trace from get_packed_slice_location:\non the\n  write path a miss is also expected, for files too large to be packed.\n\nAfter the fix, reading committed rowsets no longer takes the global\nindex lock\nor logs anything for regular files; the write-path preload still finds\npacked\nslices through the manager.\n\n### Release note\n\nFix noisy \"File not found in global packed index\" WARN logs with stack\ntraces\nwhen reading rowsets that mix packed and regular files in cloud mode."
    },
    {
      "commit": "19dfb8711ae3dbb8f56cc945f5dc7af20a844607",
      "tree": "0173edfa5c187b7439abdc880946357e9e8060c1",
      "parents": [
        "8e499c0a555d3382f1ca5cbff4e42012e5beab62"
      ],
      "author": {
        "name": "TengJianPing",
        "email": "tengjianping@selectdb.com",
        "time": "Wed Nov 04 21:02:46 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 21:02:46 2026 +0800"
      },
      "message": "[fix](partition prunning) Correct partition pruning for calendar arithmetic (#69828)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: None\n\nProblem Summary: Calendar month, quarter, year and YEAR_MONTH additions\nand subtractions clamp dates at month end while preserving the time of\nday. Consecutive input dates can therefore produce decreasing datetime\nvalues. Treating these functions as globally monotonic lets RANGE\npartition pruning remove partitions containing matching rows, such as\nJanuary 29 filtered by months_add(datetime_column, 1) \u003c \u00272023-02-28\n10:00:00\u0027. Share a calendar arithmetic monotonicity trait that retains\nthe optimization for DATE and single-day ranges and keeps other datetime\nranges conservatively. This also covers DATE_ADD, DATE_SUB,\nTIMESTAMPADD, INTERVAL expressions and runtime-filter classification.\n\nMONTHS_DIFF, QUARTERS_DIFF and YEARS_DIFF also fold some month-end\ndatetime arguments differently from BE because LocalDateTime.until\nadjusts the end date before counting calendar units. Count date units\nfirst, then compare time parts only when the remaining date fields\nmatch, preserving BE\u0027s truncation semantics for both argument orders and\nall supported date-like literal types.\n\n### Release note\n\nFix missing query rows caused by calendar arithmetic partition pruning\nand align month, quarter and year difference constant folding with\nruntime results.\n\n### Check List (For Author)\n\n- Test: FE build with Checkstyle; 94 FE unit tests; regression tests\n- Unit Test: CalendarAddSubMonotonicTest,\nDateTimeExtractAndTransformTest, PartitionPrunerTest,\nRuntimeFilterPruneClassifierTest, TimestampNsMonotonicityTest and\nInferPredicateFromMonotonicFunctionTest\n- Regression test: test_calendar_arithmetic_prune (73 generated result\nchecks) and test_date_function_prune; verify the new suite fails on the\noriginal FE\n- Behavior changed: Yes, retain matching rows and correct calendar\ndifference constants\n- Does this need documentation: No\n\n### What problem does this PR solve?\n\nIssue Number: close #xxx\n\nRelated PR: #xxx\n\nProblem Summary:\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test \u003c!-- At least one of them must be included. --\u003e\n    - [ ] Regression test\n    - [ ] Unit Test\n    - [ ] Manual test (add detailed scripts or steps below)\n    - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n        - [ ] Previous test can cover this change.\n        - [ ] No code files have been changed.\n        - [ ] Other reason \u003c!-- Add your reason?  --\u003e\n\n- Behavior changed:\n    - [ ] No.\n    - [ ] Yes. \u003c!-- Explain the behavior change --\u003e\n\n- Does this need documentation?\n    - [ ] No.\n- [ ] Yes. \u003c!-- Add document PR link here. eg:\nhttps://github.com/apache/doris-website/pull/1214 --\u003e\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label \u003c!-- Add branch pick label that this PR\nshould merge into --\u003e"
    },
    {
      "commit": "8e499c0a555d3382f1ca5cbff4e42012e5beab62",
      "tree": "a2be497ff218e90dd7cf54df3da32c64cb05339c",
      "parents": [
        "380912cc7344e92375c3de58ec37c99602cad013"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Wed Nov 04 12:17:57 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 20:17:57 2026 +0800"
      },
      "message": "[fix](test) Build vparquet_writer_test against the location-based Iceberg WriteInfo (#70209)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #69392, #69971\n\nProblem Summary:\n\n**In short.** `doris_be_test` does not compile on master since #69971\nwas merged, so the BE UT of every PR fails with `BE BUILD FAIL`. This\nfixes the one test case that breaks it.\n\n**The problem.** #69392 replaced the `write_path`, `original_write_path`\nand `target_path` fields of `IPartitionWriterBase::WriteInfo` with one\n`IcebergOutputLocation location`, and updated the callers it knew about.\n#69971, merged after it, added\n`VParquetWriterTest.HiveAndIcebergParquetPageIndexOptions`, which still\nsets the old fields:\n\n```\nbe/test/format/transformer/vparquet_writer_test.cpp:380:41: error: no matching constructor for initialization of \u0027VIcebergPartitionWriter\u0027\n```\n\nEach PR built alone, so both were green. TeamCity\u0027s BE UT has failed\nthis way for PRs merged with master `0eb5131db62` (for example build\n1092024 of #69605).\n\n**How this PR fixes it.** The case builds its `WriteInfo` as the other\nIceberg case in the same file does, `.location \u003d\nIcebergOutputLocation(\".\", \".\")`. Nothing else changes."
    },
    {
      "commit": "380912cc7344e92375c3de58ec37c99602cad013",
      "tree": "c42991ab006320fd2fcf04937be68c7ac48c5bef",
      "parents": [
        "b1720bdaee9c26c2b68c6955837257dd901bd736"
      ],
      "author": {
        "name": "Jack",
        "email": "jiangkai@selectdb.com",
        "time": "Wed Nov 04 18:57:44 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 18:57:44 2026 +0800"
      },
      "message": "[fix](regression) Pin the VARIANT subcolumn limit in test_match_keyword_score (#70151)"
    },
    {
      "commit": "b1720bdaee9c26c2b68c6955837257dd901bd736",
      "tree": "9860f1e5358f908d15159f3ae9e4e5c3d707ccf3",
      "parents": [
        "7ff8c69da8ffb2907a9d3a9adcbe5564fd13308d"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Wed Nov 04 10:30:03 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 18:30:03 2026 +0800"
      },
      "message": "[chore](pr-template) Adopt a per-type PR template and use it for commit messages (#70175)\n\n### Background\n\nEvery PR against `master` starts from\n`.github/PULL_REQUEST_TEMPLATE.md`, and the root `AGENTS.md` tells\ncoding agents to write commit messages in the same layout. The current\ntemplate asks for a problem summary, a release note, and checkboxes for\ntests, behavior changes, and documentation. Doris merges PRs by\nsquashing, so the description becomes the commit message on `master`.\n\n### What problem does this PR solve?\n\nIssue Number: None\nRelated PR: #42999 (introduced the current template), #70123 (a\ndescription written in this layout)\nPR Type: Build/CI/Chore\n\nProblem Summary:\n\nThe current template leaves out what reviewers most often have to ask\nfor: the context of the module, why the change is needed and why this\napproach, why existing tests missed a bug, and what was verified. Many\nPRs leave its checkboxes unticked. This PR replaces it with a layout\nthat opens with Background, has required fields for each PR type, asks a\ndiff of more than about 300 changed lines to open Details with a Change\nmap, and adds Testing, optional Notes for Reviewers, a pre-submission\nstatus, and an AI disclosure. Commit messages written by `AGENTS.md`\nrules follow the same layout.\n\n### Details\n\n- **Affected:** `.github/PULL_REQUEST_TEMPLATE.md`, which every new PR\ndescription starts from, and the commit and worktree rules in the root\n`AGENTS.md`. No build scripts, CI jobs, or workflows change;\n`title-checker.yml` accepts the same `[type](scope)` titles as before.\n- **Developer and release impact:** Authors fill in more sections, with\nthe required fields of each PR type listed in a comment under Details;\nthe reviewer checklist is gone. Coding agents write commit messages in\nthe new layout, and from a worktree they may keep PR handoff files in\n`${DORIS_REPO}/.git/pr-handoff/`. Releases are unaffected.\n- **License and third-party:** None.\n- **Rollback:** Revert this PR; nothing else depends on the template.\n\n### Testing\n\n- `check_pr.py`, the checker of the `doris-pr-description` skill (not\npart of this PR; see Notes for Reviewers), reports the unfilled template\nas not ready, with 17 errors that include the empty Background, Problem\nSummary, Testing, and Release note and every blank answer line. Its\neleven bundled examples, one or two per PR type and written in this\nlayout, pass it, and so does this description.\n- The description of #70123, written in this layout, passes the same\nchecker, with one warning that its prose is longer than the limit.\n\n### Notes for Reviewers\n\n- Two commits: the template and the `AGENTS.md` rules, then guidance for\nthe Change map and three refined comments. With the HTML comments\nremoved, the second commit changes nothing.\n- `AGENTS.md` also drops the second copy of the paragraph on which files\na commit may contain.\n- The new `AGENTS.md` text names the `develop-with-evidence` and\n`doris-pr-description` skills, which are not in this repository yet.\nThey are to be added to\n[apache/doris-skills](https://github.com/apache/doris-skills), which a\nlater PR includes here as a git submodule.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Behavior changed: No\n- Does this need documentation: No\n- Pre-submission status: Ready\n\n### AI Disclosure\n\n- Tool: Claude Code (Claude Opus 5.5)\n- Generated: the template, the `AGENTS.md` text, the commit messages,\nand this description\n- Human review: The author reviewed all of the changes line by line."
    },
    {
      "commit": "7ff8c69da8ffb2907a9d3a9adcbe5564fd13308d",
      "tree": "b5e18513beb73bfbabc3e1f20a42677cdfdcfdc8",
      "parents": [
        "0eb5131db62cd7939788508377fc148df6ac1ef1"
      ],
      "author": {
        "name": "Chenyang Sun",
        "email": "sunchenyang@selectdb.com",
        "time": "Wed Nov 04 17:31:26 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 17:31:26 2026 +0800"
      },
      "message": "[fix](fe) Do not use short-circuit point query when reading hidden columns  (#70162)\n\nProblem Summary:\n\nA singleton rowset stores only a placeholder for `__DORIS_VERSION_COL__`\nand `__DORIS_COMMIT_TSO_COL__`; the real value comes from the rowset\nmetadata at read time. The short-circuit point query reads the JSONB row\nstore, which holds that placeholder, and the BE fast path deliberately\ndoes not resolve these columns (see `point_query_executor.cpp`). So on a\nMOW table with `store_row_column \u003d true`:\n\n```sql\nCREATE TABLE b (id INT NOT NULL, v INT) UNIQUE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1\nPROPERTIES (\u0027replication_num\u0027\u003d\u00271\u0027, \u0027enable_unique_key_merge_on_write\u0027\u003d\u0027true\u0027,\n            \u0027store_row_column\u0027\u003d\u0027true\u0027, \u0027changelog.enable\u0027\u003d\u0027true\u0027);\nINSERT INTO b VALUES (1, 7), (2, 8);\nSELECT __DORIS_VERSION_COL__, __DORIS_COMMIT_TSO_COL__, v FROM b WHERE id \u003d 1;\n-- before: 0, 0, 7          after: 2, \u003ccommit tso\u003e, 7\n```"
    },
    {
      "commit": "0eb5131db62cd7939788508377fc148df6ac1ef1",
      "tree": "a19acfd5758b5dc76ec3d87f42c88312b8b2b296",
      "parents": [
        "56dfbb1c273050aa3cc0f32d20a242be24af4900"
      ],
      "author": {
        "name": "daidai",
        "email": "changyuwei@selectdb.com",
        "time": "Wed Nov 04 15:20:10 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 15:20:10 2026 +0800"
      },
      "message": "[fix](iceberg) Honor standard row-level isolation properties (#69702)\n\n### What problem does this PR solve?\n\nIssue Number: N/A\n\nRelated PR: #69551\n\nProblem Summary:\nDoris Iceberg DELETE, UPDATE, and MERGE all used the non-standard\n`delete_isolation_level` property. As a result, the standard\nper-operation table properties `write.delete.isolation-level`,\n`write.update.isolation-level`, and `write.merge.isolation-level` were\nignored. A table configured for `snapshot` isolation therefore still\nperformed serializable data-file conflict validation and rejected normal\nconcurrent appends.\n\nThis change selects the Iceberg `TableProperties` key based on\n`WriteOperation` and parses it with `IsolationLevel.fromName()`. The\nexisting delete-file, base-snapshot, conflict-filter, and\nreferenced-data-file validations remain enabled; only\n`validateNoConflictingDataFiles` is gated on serializable isolation.\nInvalid isolation-level values now fail explicitly instead of being\ntreated as snapshot isolation.\n\nThe three disabled regressions from #69551 are enabled. Unit coverage\nalso verifies per-operation key selection, serializable defaults, and\ninvalid values.\n\n### Release note\n\nIceberg DELETE, UPDATE, and MERGE now honor their standard per-operation\nisolation-level table properties.\n\n### Check List (For Author)\n\n- Test\n    - [ ] Regression test\n- [x] Unit Test: `IcebergConnectorTransactionTest` and\n`IcebergWriteKnownBugsTest` pass, including all three concurrent-append\nsnapshot-isolation regressions.\n    - [ ] Manual test (add detailed scripts or steps below)\n    - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n        - [ ] Previous test can cover this change.\n        - [ ] No code files have been changed.\n        - [ ] Other reason\n\n- Behavior changed:\n    - [ ] No.\n- [x] Yes. Snapshot isolation now permits concurrent data-file appends\nwhile retaining the other row-delta validations.\n\n- Does this need documentation?\n    - [x] No.\n    - [ ] Yes.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label"
    },
    {
      "commit": "56dfbb1c273050aa3cc0f32d20a242be24af4900",
      "tree": "df28f46a4487ed2545ea7715ddd197ef89359852",
      "parents": [
        "b0fd4955745146f4aac82b0bb8035d8ff8569b02"
      ],
      "author": {
        "name": "Gabriel",
        "email": "liwenqiang@selectdb.com",
        "time": "Wed Nov 04 15:15:15 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 15:15:15 2026 +0800"
      },
      "message": "[feature](fe) Add encrypted Iceberg read admission checks (#70159)\n\n### What problem does this PR solve?\n\nRelated PR: #70087\n\nProblem Summary:\n\nEnabling Parquet modular encryption in Arrow does not implement native\nencrypted Iceberg reads in Doris. Scan planning currently has no shared\nadmission check for an encrypted table. Ordinary scans can emit ranges\nwithout establishing encryption support, while metadata-only COUNT and\nsystem-table routes can bypass the native data reader altogether.\n\nFor example, a v3 table whose SDK metadata contains `encryption.key-id`\ncan reach scan planning and COUNT pushdown without any proof that the\nselected backends understand encrypted Iceberg tasks. This PR\nestablishes the first-stage read contract before native execution is\nenabled.\n\n- Add `enable_encryption_read` to the Iceberg plugin\u0027s `iceberg.conf`,\ndefaulting to `false`. This is deployment configuration requiring an FE\nrestart, with no catalog/session override or fe.conf fallback.\n- Share admission checks across table metadata and snapshot binding,\nordinary scans, COUNT eligibility/pushdown, streaming scans, system\ntables and explicitly empty snapshot pins.\n- Detect encryption from the table key property, retained v3 encryption\nkeys, or a non-plaintext SDK manager. Removing the current write key\nmust not admit old encrypted snapshots as plaintext.\n- Return stable classifications for disabled reads, unsupported\ncatalogs/table formats/custom managers, and unavailable backend\ncapabilities. Error text excludes key identifiers and encryption\nobjects.\n- Even with the switch on, reject encrypted execution until backend\ncapability negotiation and native readers exist. A declared encrypted\ntable is rejected before KMS manager initialization or manifest I/O.\nPlaintext scans and exact metadata counts retain their behavior.\n\nThe included developer contract defines the initial HMS / v3 / native\nParquet `AES_GCM_V1` target and the requirements for explicit BE\nadvertisement, placement, retries, key ownership and redaction. This PR\ndeliberately adds no SPI/wire fields and grants no encrypted-write\ncapability. Actual per-file format/algorithm validation and undeclared\nexternally encrypted files remain part of the subsequent planner/reader\nwork.\n\n### Validation\n\n```sh\nJAVA_TOOL_OPTIONS\u003d\u0027-XX:ActiveProcessorCount\u003d8\u0027 mvn -f fe/pom.xml -pl :fe-connector-iceberg -am package     -Dmaven.build.cache.enabled\u003dfalse\nmvn -f fe/pom.xml -pl :fe-connector-iceberg checkstyle:check\n```\n\n- The complete reactor package succeeded, including connector packaging\nand upstream module tests. Iceberg connector results: **1923 tests\nreported, 1909 passed, 14 existing skips, 0 failures, 0 errors**.\n- Added 8 contract tests and 1 configuration test. Real SDK metadata\nfixtures cover empty/populated tables, direct/cached acquisition,\nstatement scopes, metadata binding, COUNT, streaming, system tables,\nretained keys, unsupported catalogs/v2/custom managers, error redaction\nand rejection before KMS initialization or manifest/data I/O.\n- Before the implementation, five initial contract tests failed because\nencrypted reads were admitted; the plaintext compatibility test passed.\nThe final added contract/configuration tests all pass.\n- FE Checkstyle: **0 violations**. Verified that the plugin zip includes\nthe admission class and the updated configuration template.\n- Local HTTP mock servers were run without proxy interception. The test\nJVM uses 8 processors so the existing cancellation fixtures do not\ntrigger the SDK\u0027s automatic manifest-merge threshold. No existing test\nwas changed or excluded for this.\n\nSQL regression coverage is included in\n`test_iceberg_encryption_read_guard`:\n\n```sh\nbash run-regression-test.sh --run -d external_table_p0/iceberg \\\n    -s test_iceberg_encryption_read_guard\n```\n\n- Built the full FE from this PR using `build.sh --fe`, and ran the\nsuite with an isolated FE and an existing ASAN BE against the standard\nREST/MinIO fixture.\n- The SQL suite directly asserts the production default,\n`READ_DISABLED`, without a separate regression option for the expected\nFE switch state. It also sets a catalog-level\n`enable_encryption_read\u003dtrue` to verify that a catalog cannot bypass the\ndeployment-level setting.\n- Default plugin configuration: passed normal expected-output\ncomparison; **96 admission-error assertions**, with both table-cache\nsettings, both COUNT pushdown settings, populated/empty declarations,\nrepeated queries, SELECT, COUNT, DESC and `$files` / `$snapshots` /\n`$position_deletes`. Plaintext rows and counts remain correct.\n- Connector unit tests explicitly construct both enabled and disabled\nplugin contexts; enabled REST contexts reject with\n`UNSUPPORTED_CATALOG`, and eligible HMS contexts reject with\n`BACKEND_CAPABILITY_UNAVAILABLE`. The ordinary SQL regression does not\nrequire an enabled FE deployment.\n- Mutation control: in a temporary runtime-only artifact, replaced the\nadmission helper with a no-op. The suite failed at the first\ndeclared-table SELECT because the required `READ_DISABLED` error was\nabsent. Restored and hash-verified the unmodified production artifact\nbefore the passing runs; no test hook is included in the source change.\n- The expected `.out` file was generated by the standard regression\nrunner and then used for normal comparison. The suite was rerun against\nits existing remote tables, exercising repeatable fixture cleanup. FE\nCheckstyle was rechecked: 0 violations.\n\nThese SQL tests use real v3 tables and plaintext data files with\nsynthetic encryption declarations to isolate admission from decryption\nfailures. They validate the FE/SQL guard, not native ciphertext\ndecoding. Live KMS decryption, mixed-version cluster integration and\nfull encrypted-query execution remain subsequent work.\n\n### Release note\n\nDeclared encrypted Iceberg tables now fail with clear read-admission\nerrors. The plugin read switch defaults off, and enabling it cannot\nbypass unavailable backend support or enable encrypted writes.\n\n### Check List (For Author)\n\n- Test\n    - [x] Unit Test\n    - [x] Regression test\n    - [ ] Manual test\n- Behavior changed:\n- [x] Yes. Declared encrypted tables are rejected consistently before\nscan execution or metadata-count shortcuts.\n- Does this need documentation?\n- [x] Yes. Developer contract and configuration comments are included in\nthis PR; public query documentation follows when encrypted execution is\nimplemented.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label"
    },
    {
      "commit": "b0fd4955745146f4aac82b0bb8035d8ff8569b02",
      "tree": "94ef0a41015a3d915e625c40183284ecfa102c02",
      "parents": [
        "58d7b8153346aa4cf2c5f85971e8c3b75fb7dc5c"
      ],
      "author": {
        "name": "924060929",
        "email": "lanhuajian@selectdb.com",
        "time": "Wed Nov 04 14:48:23 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 14:48:23 2026 +0800"
      },
      "message": "[refactor](coordinator) Unify query execution in Coordinator (#69830)\n\n### What problem does this PR solve?\n\nProblem Summary:\n\nFE still carries two execution coordinators: the legacy `Coordinator`\nassigns scan ranges and fragment instances itself, while\n`NereidsCoordinator` executes Nereids distributed plans but inherits the\nlegacy coordinator\u0027s state and initialization. `CloudCoordinator` adds\nanother legacy-only factory branch. This duplicates scheduling and\nlifecycle ownership and retains a fallback that newer execution features\ndo not support.\n\nReplace the legacy implementation with the standalone Nereids\nimplementation, renamed to `Coordinator`, and remove\n`NereidsCoordinator` and `CloudCoordinator`. SQL, INSERT and background\nload creation now use the same factory. Cloud compute-group selection,\nprivilege validation and backend membership remain in\n`BackendDistributedPlanWorkerManager` during distributed planning.\n\nPreserve the remaining shared capabilities explicitly: HTTP Stream Load\ngenerates parameters with `ThriftPlansBuilder` and retains one fragment\ninstance on the receiving BE, matching its single completion callback;\nArrow Flight retains scans that must outlive dispatch; streaming INSERT\nretains scan backend IDs; cancellation uses the runtime task state,\nwithout treating remote-catalog workers as local backends. Group-commit\nbackend selection remains in distributed planning.\n\nKeep set-operation bucket-shuffle scheduling markers aligned with the\nselected child distribution, including when local shuffle is disabled,\nso pruned scans retain all required bucket receivers.\n\nPreserve backend-restart cancellation when a fragment captured an\nuninitialized epoch (-1 or 0), while retaining the current-zero\nreplay/upgrade exception. Stabilize the nested full-outer-join EXPLAIN\nwith a statement-scoped bucket-shuffle downgrade setting, keeping the\nouter-join elimination checks independent of actual backend count and\npipeline parallelism.\n\nMigrate join-distribution assertions to effective statement-scoped\nsettings: cover both partitioned and bucket-shuffle plans, retain\nexchange-count checks, and make pruning checks exercise the bucket path\nwithout an early skip. Keep the removed setting where it still provides\ncompatibility coverage; do not mechanically rewrite unrelated test\nexpectations.\n\nRetire `enable_nereids_distribute_planner` using the existing\nremoved-variable handling and remove its legacy-only optimizer branches.\nMigrate unit fixtures to the distributed planner\u0027s query identity,\nworker and local-exchange contracts. Regenerate EXPLAIN-shape\nexpectations with the Nereids distribution/property path while\npreserving query-result checks.\n\n### Release note\n\nUse Nereids distributed planning for all queries and loads.\n`enable_nereids_distribute_planner` no longer selects a planner;\nexisting SET statements for this removed variable remain accepted."
    },
    {
      "commit": "58d7b8153346aa4cf2c5f85971e8c3b75fb7dc5c",
      "tree": "554cec376db11e431e41c5510a506a7dff2244af",
      "parents": [
        "4bde8d0fb28452c9bb8314febbe05c4d63dde93c"
      ],
      "author": {
        "name": "daidai",
        "email": "changyuwei@selectdb.com",
        "time": "Wed Nov 04 14:06:33 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 14:06:33 2026 +0800"
      },
      "message": "[fix](be) Open children of Iceberg nested struct expressions (#69832)\n\n### What problem does this PR solve?\n\nIssue Number: #69782\n\nRelated PR: #69711\n\nProblem Summary:\n`NestedStructFieldExpr::open()` reuses the prepare guard and skips\nopening its child. Use the normal open lifecycle, completing the nested\ntraversal fix after #69711.\n\n### Release note\n\nFix initialization of nested struct expressions in FileScannerV2 Iceberg\nequality deletes.\n\n### Check List (For Author)\n\n- Test\n- [x] Unit Test: added coverage for cast defaults, nullable ancestors,\nand cloned contexts.\n- Behavior changed:\n    - [x] Yes. Open nested expression children before execution.\n- Does this need documentation?\n    - [x] No.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label"
    },
    {
      "commit": "4bde8d0fb28452c9bb8314febbe05c4d63dde93c",
      "tree": "83a841366eaa1210ac0744ae34182ae28c477498",
      "parents": [
        "75ab5ba108293af01da96e84f612e0b05bdce9db"
      ],
      "author": {
        "name": "Gabriel",
        "email": "liwenqiang@selectdb.com",
        "time": "Wed Nov 04 14:00:35 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 14:00:35 2026 +0800"
      },
      "message": "[fix](be) Preserve JSON and WAL rows with empty file projections (#70154)\n\n### What problem does this PR solve?\n\nIssue Number: close #69926\n\nProblem Summary:\n\nHive JSON queries that only need partition columns can return zero rows\neven when the selected files contain records. For example, `SELECT * ...\nWHERE dt \u003d \u0027p1\u0027` returns data while `SELECT COUNT(*) ... WHERE dt \u003d\n\u0027p1\u0027` returns zero. The V2 JSON reader derived its row count from the\noutput block, which has no columns after physical field pruning.\nWriter-backed reader tests reproduce the same row-count loss for empty\nWAL projections.\n\nTrack accepted JSON records independently of materialized columns,\nincluding batch progress, array expansion, error rollback, and\nignore-malformed NULL rows. Preserve JSON-path record selection for\nempty projections, and let the existing TableReader materialize\npartition values using the returned row count.\n\nFor WAL, save the deserialized source row count before materialization\ntransfers column ownership, then pass it through the existing filter\npipeline. This preserves the row count when no file columns are\nrequested. The empty-projection reproduction is at the reader interface;\nproduction replay reachability of an empty projection remains\nunverified.\n\nAdd ten JSON BE tests covering empty file schemas and projections,\narrays, JSON roots/paths, malformed input, batch boundaries, EOF, and\nactual TableReader partition materialization. Add eight writer-backed\nWAL cases covering empty, single-column, and reordered full projections\nacross multiple blocks, with and without schema discovery, plus repeated\nEOF reads. Add a WAL SQL regression that forces two original async-load\ntransactions to abort, then verifies row counts, NULLs, aggregates,\nfilters, and actual values after replay. Add a Hive SQL regression using\nthe existing partitioned JSON fixture for COUNT, partition projections,\ngrouping, and file-column filters with external COUNT pushdown enabled\nand disabled.\n\n### Release note\n\nFix incorrect zero counts and missing rows when querying only partition\ncolumns of Hive JSON tables. Preserve logical row counts for empty WAL\nreader projections.\n\n### Check List (For Author)\n\n- Test\n- [x] Unit Test: 166 ASAN tests passed, including JSON reader,\nTableReader, WAL V2 reader, WAL reader/writer, and CSV/Text\nempty-projection controls. Before the JSON fix, nine of ten new tests\nfailed; before the WAL fix, both empty-projection cases failed while six\nprojection controls passed.\n- [x] WAL SQL regression: `test_group_commit_wal_row_count` passed twice\non isolated Doris 4.1.5 with scanner V2 enabled. Generated the `.out`\nfile with `run-regression-test.sh`, then reran for result comparison.\nExecution on master is pending CI.\n- [x] Hive SQL regression: added `test_hive_json_partition_only`; Groovy\ncompilation passed. Full execution against the external Hive fixture is\npending CI.\n- Behavior changed:\n- [x] Yes. Empty JSON and WAL file projections preserve logical row\ncounts. Hive JSON partition-only queries produce the correct rows and\naggregates.\n- Does this need documentation?\n    - [x] No."
    },
    {
      "commit": "75ab5ba108293af01da96e84f612e0b05bdce9db",
      "tree": "afd6b190160dd38dccdbfdcd263c3de6fe7d1ac4",
      "parents": [
        "ec7f806fa3907f9be936cbaa789eb0a20476d41c"
      ],
      "author": {
        "name": "Gabriel",
        "email": "liwenqiang@selectdb.com",
        "time": "Wed Nov 04 13:57:35 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 13:57:35 2026 +0800"
      },
      "message": "[fix](be) Fix ORC condition-cache seeks past filtered stripe suffixes (#70079)\n\n### What problem does this PR solve?\n\nRelated PR: https://github.com/apache/doris-thirdparty/pull/431 (merge\nthe ORC dependency fix first)\n\nA condition-cache granule can start in a stripe\u0027s SARG-filtered suffix\nwhile containing a matching row in the next stripe. On an initialized\nstripe, ORC\u0027s `seekToRow()` uses the stripe-end sentinel as a row-group\nindex. This can access past the index when the stripe ends on an index\nboundary, or prematurely return EOF when the last row group is partial.\n\nThe same-stripe fast path originates in [Apache ORC\nORC-1170](https://github.com/apache/orc/pull/1114); the same missing\nstripe-end handling remains in the current upstream C++ reader.\n\nPin the ORC fix that initializes the next selected stripe before\npositioning its column readers. Preserve SARG pruning, lazy reads and\nsplit boundaries.\n\nAdd a Doris reader regression and a 1.5 KB synthetic ORC fixture with\nits generator. The fixture has two 20,000-row stripes, a 10,000-row\nindex stride, and one matching row at the start of each stripe. A cache\nhit seeks to row 18,432 in the first stripe\u0027s filtered suffix. The SQL\nregression compares cache-disabled and cached results with lazy reads\nand min/max pruning enabled or disabled, and checks the profile for\nactual cache hits and skipped rows.\n\n### Release note\n\nFix potential crashes and missing rows when ORC condition-cache seeks\nreach a SARG-filtered stripe suffix.\n\n### Check List (For Author)\n\n- Test:\n- [x] Unit Test: 247 ORC-related BE tests passed under AddressSanitizer,\nincluding the new Doris condition-cache case and 32 dependency boundary\ncases.\n- [x] Unit Test: 56 focused standalone ORC tests passed under\nAddressSanitizer. Reverting the fix makes all 16 initialized-stripe\nboundary cases fail.\n- [ ] Regression test: SQL suite added; Groovy compilation checked. The\nexternal SQL suite has not been executed locally.\n  - [x] clang-format 16 and BE build-hygiene checks passed.\n- [ ] Local full-file clang-tidy analysis did not complete within ten\nminutes; no completed result is available.\n- Behavior changed:\n  - [x] Yes. Safely advance to the next matching stripe or EOF.\n- Does this need documentation?\n  - [x] No."
    },
    {
      "commit": "ec7f806fa3907f9be936cbaa789eb0a20476d41c",
      "tree": "8b2b05b2457ca65a4516add7de15cacf868dc776",
      "parents": [
        "035a843648df12bf814ff946c96f8d1eff09eb87"
      ],
      "author": {
        "name": "Gabriel",
        "email": "liwenqiang@selectdb.com",
        "time": "Wed Nov 04 13:48:36 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 13:48:36 2026 +0800"
      },
      "message": "[fix](regression) Repair ES source predicate regression cases (#70155)\n\n### What problem does this PR solve?\n\nIssue Number: N/A\n\nRelated PR: #69672, #69723\n\nExternal ES regression suites fail during setup or retain expectations\nfrom the previous indexed-value predicate behavior:\n\n- Read the deprecated scanner option through `SELECT\n@@enable_file_scanner_v2`, whose compatibility name remains valid when\nSHOW prefixes deprecated names.\n- On ES 8.3.3, fall back from the specifically unsupported\n`_source.mode` mapping syntax to `enabled:false`, retaining the\ndisabled-source row and alias checks. Other HTTP failures remain fatal;\nsupporting ES8 versions still exercise the mode syntax.\n- Compare LIKE, REGEXP, NULL and `level` predicates against unfiltered\nrows materialized into internal tables, then update only the affected\nrunner-generated golden blocks.\n- Make the date-string equality test explicitly use source reads, test\nmidnight as the matching value, and retain 08:00 as a negative control.\n- Check that a residual predicate prevents early termination while a\nsafe numeric docvalue predicate still permits LIMIT pushdown.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test\n  - [x] Regression test\n  - [ ] Unit Test\n  - [x] Manual test\n\nValidation uses fresh FE and ASAN BE artifacts built from this PR commit\n(`6386f5f83a51b548c86ab3cff83d53cb48fb18df`) with the standard\n`./build.sh --fe --be -j64` entry point. The complete build exited\nsuccessfully, and both finished artifact versions match that commit. No\nproduction source is changed by this PR.\n\n- On the same fresh binaries and reset ES 5.6.16, 6.8.23, 7.17.5 and\n8.3.3 fixtures, the parent suites/golden files reproduced all ten\nfailures; the final PR suites passed all ten in normal comparison, with\nzero failed, fatal or skipped suites. The passing run executed 2,154\ngolden checks and 1,798 independent query comparisons, without\nforce-generation or further golden edits.\n- The complete `test_es_query` passed all four ES versions and both\nscroll settings, including its unchanged complex-array golden blocks\n(276 result checks and 20 independent comparisons).\n- Independent SQLite/Python evaluation of the unchanged raw fixture JSON\npassed 38 comparisons and covered all 17 changed golden blocks. Real ES\nrequests also confirmed that `ignore_above` omits matching index values\nwhile retaining source strings, and that the constant-keyword index\nmatches documents whose source `level` is absent.\n- Captured ES search requests verified the date modes: the original\nfour-column projection reads source, returns midnight values and rejects\nthe 08:00 predicate. A date-only docvalue projection correctly returns\n08:00 in the +08:00 session. The revised test explicitly fixes the\nsource contract and retains the 08:00 negative control.\n- The complete fallback suite additionally passed normal comparison on\nES 8.11.3: 304 golden checks and 304 independent comparisons. The actual\nmappings and document responses confirmed absent source for both the ES\n8.3.3 `enabled:false` fallback and the supported ES 8.11.3\n`mode:disabled` branch.\n- The existing `test_es_file_scanner_v2` compatibility suite also passed\non the fresh build across all four fixture versions. The deprecated\noption is accepted; this does not claim two distinct physical reader\nimplementations on current master.\n- Direct variable probes confirmed the old exact SHOW name returns no\nrows, the wildcard SHOW name exposes its deprecated prefix, and the\nscalar compatibility name remains readable.\n- `git diff --check` passed. ES9 compatibility is not claimed.\n\n- Behavior changed:\n- [x] No production behavior change; regression fixtures and\nexpectations are corrected.\n  - [ ] Yes.\n\n- Does this need documentation?\n  - [x] No.\n  - [ ] Yes.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label"
    },
    {
      "commit": "035a843648df12bf814ff946c96f8d1eff09eb87",
      "tree": "07a9e28b3a550e10aee0f8b3b382eb82f55e0fcb",
      "parents": [
        "c0274bfb4accd3f013b320704d55919f7ae46c2e"
      ],
      "author": {
        "name": "Gabriel",
        "email": "liwenqiang@selectdb.com",
        "time": "Wed Nov 04 13:06:31 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 13:06:31 2026 +0800"
      },
      "message": "[feature](be) Add session controls for Parquet page indexes (#69971)\n\n### What problem does this PR solve?\n\nParquet output relies on implicit page-index and data-page-size\ndefaults, so users cannot explicitly omit indexes or tune the target\npage size.\n\nAdd two validated session variables shared by OUTFILE, EXPORT, Hive and\nIceberg Parquet writers:\n\n- `enable_parquet_write_page_index`: write ColumnIndex/OffsetIndex\nmetadata; defaults to `true`, preserving the current master behavior.\n- `parquet_write_data_page_size`: target uncompressed data-page size in\nbytes; defaults to `1048576`, accepts positive 32-bit integers, and\napplies with indexes enabled or disabled.\n\nKeep the existing 128 KiB target cap for Hadoop-framed LZ4. Other codecs\nuse the requested target. Row-group size, dictionary-page size, and\ndata-page/format versions are unchanged. Optional query fields preserve\nexisting defaults when omitted by older FEs.\n\n### Release note\n\nAdd session controls for Parquet page-index output and target data-page\nsize.\n\n### Validation\n\n- 16 BE ASAN tests passed, including real file metadata/readback, index\non/off, dictionary on/off, null/all-null columns, Parquet versions, Data\nPage V1/V2, page offset bounds, Hadoop-framed LZ4 limits, and\nHive/Iceberg partition writers.\n- 9 FE tests passed for defaults, variable forwarding, Thrift\nserialization, valid bounds, and invalid input.\n- The new tests failed for the missing settings before implementation.\n- FE Checkstyle, clang-format 16, build hygiene, and clang-tidy passed.\n- No cluster SQL run or performance benchmark was performed locally; CI\nvalidation is pending.\n\n### Check List (For Author)\n\n- Test:\n  - [x] Unit Test\n- Behavior changed:\n- [x] Yes. Explicit controls are available; current defaults are\npreserved.\n- Does this need documentation?\n- [x] Yes. English and Chinese documentation:\nhttps://github.com/apache/doris-website/pull/4581\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label"
    },
    {
      "commit": "c0274bfb4accd3f013b320704d55919f7ae46c2e",
      "tree": "98554fd40f73f812eb680dbdc6247ecf185b190c",
      "parents": [
        "736649fcb41544ef977bb7677a1e6a5df5b7996b"
      ],
      "author": {
        "name": "wudi",
        "email": "wudi@selectdb.com",
        "time": "Wed Nov 04 11:08:18 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 11:08:18 2026 +0800"
      },
      "message": "[improvement](fe) Improve streaming job schema change error messages (#69725)\n\n### What problem does this PR solve?\n\nWhen an unsupported upstream DDL pauses a streaming job, the existing\n`ErrorMsg` contains the unsupported-change reason, source table, and\nobserved added/removed columns, but **does not contain the original\nupstream SQL or identify the mapped Doris target table**. For MySQL, the\noriginal DDL is available in the CDC logs only. Users therefore need to\nfind that statement and resolve the target mapping before deciding how\nto adjust the Doris schema.\n\nThis PR puts that diagnostic context directly in `ErrorMsg` and makes\nthe recovery guidance consistent between the initial failure and a\nfailed RESUME.\n\n| Information | Before | After |\n| --- | --- | --- |\n| Original MySQL DDL | Written to CDC logs; absent from `ErrorMsg` |\nComplete statement appended as `Source DDL: [...]`, including all\nclauses of a mixed DDL |\n| Table identification | Source table only | `source_table -\u003e\ndoris_db.actual_mapped_table` |\n| Same-name MODIFY/CHANGE | `(amount -\u003e amount)` | `(amount)`; real\nrenames retain both names |\n| PostgreSQL change details | Observed types/OIDs and column differences\n| Those details are retained, with the mapped target table added; no\noriginal SQL is available from the change event |\n| RESUME guidance | Confirmation explained on the initial failure, but\nomitted from the resumed failure hint | Both failure paths explain\nschema review and RESUME confirmation |\n| Target validation failure | Concrete error such as `missing columns\n[city]` | Retained alongside the diagnostic context and common recovery\nhint |\n "
    },
    {
      "commit": "736649fcb41544ef977bb7677a1e6a5df5b7996b",
      "tree": "d1756567e6d4ce0cb29e2846edf669b2d1217203",
      "parents": [
        "1adeb3a6801a984695fb3f3ec86b699685daaf74"
      ],
      "author": {
        "name": "Gavin Chou",
        "email": "gavin@selectdb.com",
        "time": "Wed Nov 04 10:44:51 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 10:44:51 2026 +0800"
      },
      "message": "[fix](regression) Isolate eliminate_distinct_constant database (#70105)"
    },
    {
      "commit": "1adeb3a6801a984695fb3f3ec86b699685daaf74",
      "tree": "70fac56c6ef88796083ff0688b89cc0ea6656106",
      "parents": [
        "30ce25dc1f9579130cba38f5a9baed7c96187e12"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Wed Nov 04 02:22:17 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 10:22:17 2026 +0800"
      },
      "message": "[fix](flight-sql) Return analyzed result schemas from Prepare and GetSchema (#69867)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #68631, #68768\n\nProblem Summary:\n\n**In short.** This ports branch-4.1 #68631 (also on branch-4.2) to\nmaster. A Flight SQL `CreatePreparedStatement` now returns the analyzed\nresult schema of the statement, `GetSchema` is implemented for both\nstatements and prepared statements, and a prepared execution is checked\nagainst the schema that Prepare advertised. Before this PR, Prepare\nalways returned a fixed one-column placeholder, and `GetSchema` answered\nUNIMPLEMENTED. The analysis runs in the FE only and does not plan or\nexecute the query.\n\nOriginal author: @Gabriel39\n\n**Background.** Flight SQL clients learn the shape of a result before\nthey fetch it. The Arrow Flight SQL JDBC driver prepares every statement\nand builds `ResultSetMetaData` from the dataset schema that\n`CreatePreparedStatement` returns. ADBC exposes the same schema as\n`execute_schema`, and `GetSchema` is the RPC that asks for it without\nrunning anything.\n\n**The problem, and what it cost.** On master:\n\n| | master today | with this PR |\n|---|---|---|\n| `CreatePreparedStatement` dataset schema | always `ResultMeta: Utf8`,\nwhatever the statement | the statement\u0027s columns, names and Arrow types\nas BE / FE will send them; statements whose schema cannot be told before\nthey run, and results with an untyped NULL column, keep the placeholder\n(see below) |\n| `GetSchema` for a statement or a prepared statement | UNIMPLEMENTED |\nthe analyzed schema, or UNIMPLEMENTED for a statement that keeps the\nplaceholder |\n| Invalid SQL | fails only at execution | fails at Prepare with\nINVALID_ARGUMENT |\n| A query whose privilege check fails | fails only at execution | still\nfails only at execution: Prepare keeps the placeholder, because only the\ncheck on the executed plan decides (see below), and GetSchema answers\nUNIMPLEMENTED without a schema |\n| Prepared handle used after the session\u0027s catalog / database changed,\nor after the result schema changed (e.g. another session\u0027s DDL) | runs\nthe unqualified SQL in whatever namespace the session has now, with\nwhatever columns come back | refused with NOT_FOUND \"prepare again\", and\nthe result left by that execution is dropped |\n| Prepared handle presented by another session | resolved by the\nidentity written in the handle | must start with the caller\u0027s own\nidentity (from #68768) |\n| TIMEV2 / VARBINARY columns in GetTables and Prepare schemas | `Null`,\nwhile BE sends `float64` / `binary` | `Float64` / `Binary`, as BE sends\nthem |\n\nA client that trusts the advertised schema saw one string column for\nevery query, and could not ask for a schema at all without executing.\n\n**How this PR fixes it.**\n\n- `arrowflight/FlightSqlQuerySchema` analyzes the statement on a clone\nof the session variables: a USE / SWITCH prefix is resolved in a\ntemporary namespace, the analysis `StatementContext` is closed on the\nsame thread, and the connector statement scope it may have opened is\nclosed explicitly. Each field gets the same `doris_type` markers and\nnative Variant layout the execution side uses\n(`FlightSqlSchemaHelper.withDorisTypeMetadata` / `nativeVariantField`),\nand TIMESTAMPTZ carries the canonical session time zone that BE stamps\non its batches.\n- Privileges are checked on the analyzed plan after the CTEs that\nexecution inlines have been inlined (`PullUpCteAnchor`, `CTEInline`, as\nthe rewriter does before its own check). A CTE used once is therefore\nchecked only on the columns the query reads, and a CTE used more than\nonce on every column it selects, which is also how execution checks\nthem. The rewriter normalizes the plan further before it checks\nprivileges. For example, it drops an ORDER BY under a derived table, so\nthe analyzed plan can read a column that the executed plan does not.\nWhen the analyzed plan is denied, the statement keeps the placeholder\nand execution decides. No schema is disclosed in either case.\n- FE-side commands whose result header does not depend on running them\nnow report that header without running: HELP, SHOW SNAPSHOT, SHOW CREATE\nTABLE (one header selector shared with `doRun`, including master\u0027s\nstream, plugin-view and system-table cases), SHOW PROC `/` and\n`/current_queries`, SHOW CONFIG, and EXPLAIN PLAN PROCESS (Rule / Before\n/ After).\n- `ShowStoragePolicyCommand.getMetaData()` returned the four-column\nUSING header for every form, while plain SHOW STORAGE POLICY answers the\nseven columns of `StoragePolicy.STORAGE_META_DATA`. It now returns the\nheader of the form that runs. This also corrects the column definitions\nthat MySQL `COM_STMT_PREPARE` reports for the statement. No other SHOW\ncommand declares a header that differs from the one it returns.\n- WARM UP SELECT keeps the placeholder. It answers OK instead of its\nstatistics whenever planning proves its source empty: a predicate that\nfolds to false, no partition left after pruning, or partitions without\ndata.\n- A result with an untyped NULL column keeps the placeholder too, at any\ndepth (`SELECT NULL`, `SELECT id, NULL FROM t`, `ARRAY(NULL)`).\n`DorisArrowTypeMapping` gives NULL_TYPE Arrow `Null`, but BE builds such\na column as a `UInt8` flagged as a NULL literal and sends it as nullable\n`Bool`, so an advertised `Null` would expire the handle at every\nexecution. A typed NULL (`CAST(NULL AS VARCHAR(20))`) is described as\nits type.\n- The prepared query kept in `FlightProtocolAdapter` records the SQL,\nthe namespace and the advertised schema. A prepared execution\nre-analyzes the statement and compares the result with the advertised\nschema before it runs, and compares the schema it actually produced\nbefore it publishes a DoGet ticket. It is the session\u0027s next request\nwhether or not these checks pass: before them, it drops what the\nprevious request left, as a statement\u0027s execution does, while Prepare\nand GetSchema are metadata requests and leave it. On a mismatch, the\nhandle is removed, the execution is cancelled, and the deferred\ncoordinator, the channel result and the endpoints are dropped. A handle\nwhose statement can no longer be described when it is re-analyzed, for\nexample after a column grant was revoked, expires the same way, and the\nstatement prepared again keeps the placeholder. A USE / SWITCH inside\nthe statement rebinds only that handle.\n- `DorisArrowTypeMapping` maps TIMEV2 to `Float64` and VARBINARY to\n`Binary`, the types BE sends. TIMESTAMPTZ keeps the fixed `\"UTC\"` in\nGetTables, and AGG_STATE keeps `Null` there.\n- `FeServiceClient` rejects a remote Doris metadata timeout whose\nmilliseconds do not fit in an int, instead of letting it wrap around.\n\nWhere master differs from branch-4.x:\n\n- branch-4.x answers UNIMPLEMENTED at Prepare for statements whose\nschema it cannot tell before they run: DESC, SHOW DATA / PARTITIONS /\nQUERY STATS / PYTHON PACKAGES, other PROC nodes, COPY INTO, WARM UP\nCLUSTER, multi-statement requests that do not start with USE / SWITCH,\n`?` placeholders, and result types the FE cannot describe. Python ADBC\nfalls back to executing without a prepared statement, but the Arrow\nFlight SQL JDBC driver prepares every statement and has no fallback, so\non master these statements would fail over JDBC.\n`test_arrow_flight_session_lifecycle` runs DESC and `SET @multi \u003d 2;\nSELECT ...` through JDBC. Here, these statements keep today\u0027s\nplaceholder schema, and their handles are not checked at execution.\n- branch-4.x checks privileges on the analyzed plan as it is, so a CTE\nused once is checked on every column it selects, and any denial fails\nPrepare. Here, CTEs are inlined first, and a remaining denial keeps the\nplaceholder.\n- branch-4.x advertises the statistics header for WARM UP SELECT and\nthen rejects, as a schema change, an execution whose source turned out\nempty. Here, WARM UP SELECT keeps the placeholder.\n- branch-4.x advertises Arrow `Null` for an untyped NULL column, which\nBE sends as `Bool`, so every prepared execution of such a query is\nrejected as a schema change. Here, such a result keeps the placeholder.\n- The analysis and the checks run under the session\u0027s command lock\n(`FlightProtocolAdapter.runCommand` / `callCommand`) instead of\n`synchronized (connectContext)`, and the namespace is saved and restored\nwith `getNameSpaceContext()` / `restoreNameSpaceContext()`.\n- EXPLAIN PLAN PROCESS has a result over Flight on master, so it gets a\nstatic header instead of UNIMPLEMENTED. PLAN REPLAYER and SHOW CREATE\nPROCEDURE do not exist on master and are not carried over.\n- SWITCH resolves the default database through the connector provider\n(`Env.defaultDatabaseOnUse`, now shared with `Env.changeCatalog`)\ninstead of `EsExternalCatalog.DEFAULT_DB`.\n- TIMESTAMPTZ fields carry the canonical session time zone that BE\nstamps, where branch-4.x uses the raw session value.\n- Only the two handle checks from #68768 that do not depend on parameter\nbinding are taken: the caller-prefix check and rebinding only the handle\nthat ran the USE / SWITCH. Parameter binding itself is not part of this\nPR.\n- `GetTableTypes`, the other half of #68631, is implemented by #67899.\n\nCost: every Prepare, every prepared execution and every `GetSchema` runs\none more FE analysis of the statement.\n\n**Results.** On a local cluster built from this branch, every suite\nunder `arrow_flight_sql_p0` passes: the new `test_schema_only` and\n`test_get_tables_schema`, plus the seven suites that run through Flight\nJDBC, `test_arrow_flight_session_lifecycle` among them.\n`datatype_p0/uuid/test_uuid_arrow_flight` passes too. `test_schema_only`\nchecks the advertised schema of each kind of statement against what its\nexecution actually returns. That covers queries, the side-effect-free\nfrontend-side commands executed as prepared statements (SHOW STORAGE\nPOLICY among them), a WARM UP SELECT on an empty table (executed in\ncloud mode, the only mode that warms up internal tables), and a\n`group_concat_state` AGG_STATE, which BE sends as Utf8.\n\nCo-authored-by: Gabriel \u003cliwenqiang@selectdb.com\u003e"
    },
    {
      "commit": "30ce25dc1f9579130cba38f5a9baed7c96187e12",
      "tree": "108a57026ab38cc74302614918deb7d2a2d4ae29",
      "parents": [
        "28cbcfc233e81218e5a3e38b83676637fa53772e"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Wed Nov 04 02:15:45 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 10:15:45 2026 +0800"
      },
      "message": "[fix](be) Cancel Arrow Flight queries on the result BE when a result stream closes early (#69945)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #68630, #67644\n\nProblem Summary:\n\n#### In short\n\nThis ports the BE half of branch-4.x\u0027s #68630 to master. When an Arrow\nFlight SQL client cancels or closes a result stream before its end,\nmaster keeps the query running: the BE thread serving the stream waits\non the result buffer without ever checking whether the client is still\nthere, and the unread result blocks stay in memory, holding back the\nquery, until the query times out or the delayed buffer cleanup runs\nminutes later. With this PR the BE notices the cancelled stream within\n20 ms, and when the stream is served by the BE that holds its result\n(the usual case), that BE cancels the query and frees all of the query\u0027s\nFlight result buffers there. The FE route that also reaches the query\u0027s\nother BEs (`cancelFlightQuery`, which changes `FrontendService.thrift`)\ncomes in a follow-up PR.\n\n#### Background\n\n- A Flight SQL query leaves its result on the BEs. `GetFlightInfo`\nreturns one endpoint per result buffer, and the client reads each\nendpoint with a `DoGet` stream. Since #67504 the FE closes the\ncoordinator as soon as it has returned the `FlightInfo`, so it no longer\ncancels anything on behalf of an open stream.\n- `DoGet` is served by `ArrowFlightBatchLocalReader` when the result\nbuffer is on the same BE, or by `ArrowFlightBatchRemoteReader`, which\npulls the blocks from the result BE with the `fetch_arrow_data` BRPC.\n- The result buffer (`ArrowFlightResultBlockBuffer`) is keyed by the\nquery id when parallel result sink is on (the default) and by the\nfragment instance id when it is off. When it is full, the result sink\nstops and the query waits for the reader.\n- `ResultBufferMgr` drops a buffer that nobody removed at\n`execution_timeout + 5 s`, or `result_buffer_cancelled_interval_time`\n(300 s by default) after the fragments writing it have finished,\nwhichever comes first. Finishing with an error does not clear the unread\nblocks. `cancel_plan_fragment` is the BRPC the FE uses to cancel a query\non a BE.\n\n#### The problem, and what it cost\n\n- **The local reader never looks at the client.** `get_arrow_batch`\nloops on 20 ms waits until a block arrives, the buffer is closed or it\nfails. It has no access to `ServerCallContext::is_cancelled()`, so a\n`DoGet` cancelled while the buffer is empty keeps a Flight server thread\nwaiting until the query produces a block or ends.\n- **Closing a stream cancels nothing.** The readers do not override\n`RecordBatchReader::Close()`, and their destructor only logs. A client\nthat stops reading (a `LIMIT` reached on the client side, a cancelled\ncursor, a dropped connection) leaves the query running and its unread\nblocks in the buffer. When the result does not fit in the buffer, the\nfull buffer back-pressures the result sink and the fragments, with their\nmemory, wait until the query times out; the unread blocks go only when\nthe buffer is dropped.\n- **The remote reader cannot be interrupted.** It waits for\n`fetch_arrow_data` with `brpc::Join`, which returns only when the result\nBE answers or `arrow_flight_reader_brpc_controller_timeout_ms` (300 s by\ndefault) passes.\n- **An FE cancel leaves the results behind.** `cancel_plan_fragment`\ncancels the fragments but not the query\u0027s Flight buffers, so their\nunread blocks stay until the delayed cleanup 300 s later.\n- **A fetch for a removed buffer never completes.** When\n`fetch_arrow_data` cannot find the buffer, it logs and returns without\nrunning `done`, so the caller waits for its BRPC timeout instead of\ngetting an error. #67644 fixes the same line.\n\n#### How this PR fixes it\n\n- **Bounded waits, cancellation checks in the reader.**\n`get_arrow_batch(result, eos)` waits at most 20 ms and then returns OK\nwith no block and `eos \u003d false`; `eos` now marks the real end. The local\nreader retries in a loop and checks `is_cancelled()` before each wait.\n`DoGetStatement` passes `context.is_cancelled()` to both readers.\n- **The reader closes itself.** `ArrowFlightBatchReaderBase` now\nimplements `ReadNext` (which calls each reader\u0027s `ReadNextImpl`) and\n`Close()`. A stream counts as aborted when it is cancelled, closed\nbefore EOF, fails while reading, or is destroyed before EOF; a stream\nthat reached EOF is never treated as aborted.\n- **Aborting a local stream cancels the query on this BE.**\n`ResultBufferMgr` keeps an index from query id to the query\u0027s Flight\nbuffers, maintained under the buffer-map lock together with the map.\n`cancel_arrow_flight_buffers` removes every Flight buffer of the query\nfrom the map, then, outside the lock, fails each buffer (which wakes its\nwaiting fetches and releases the back-pressure) and cancels the\n`QueryContext` it holds a weak reference to. Sibling buffers of the same\nquery are cleared even after their `QueryContext` has expired.\n- **Remote waits can be interrupted.** The remote reader waits for its\nBRPC with a callback that checks `is_cancelled()` every 20 ms and calls\n`brpc::StartCancel` when the stream is gone.\n- **`cancel_plan_fragment` drops the Flight buffers of an aborted\nquery.** It calls `cancel_arrow_flight_buffers` unless the reason is\n`LIMIT_REACH` or `FINISHED`, which can leave valid unread results that\nthe client still has to read.\n- **`fetch_arrow_data` answers when the buffer is gone.** It completes\nthe RPC with the lookup error. The line is identical to the one #67644\nadds, so the two PRs merge in either order.\n\nDifferences from branch-4.x: the two consistency checks between the\nindex and the buffer map are `DORIS_CHECK` (an `if` and a `DCHECK` in\n4.x), a null check on the reader\u0027s memory tracker that both constructors\nmake unnecessary is not carried over, and the\n`ArrowFlightResultBlockBuffer` constructor moved to the .cpp so that\n`varrow_flight_result_writer.h`, which 13 TUs include, does not include\n`runtime/query_context.h`; only the .cpp needs its definition. The new\nlines also take a few clang-tidy fixes (the destructor calls\n`ArrowFlightBatchReaderBase::Close()` explicitly instead of a virtual\ncall, `std::scoped_lock`), and the test starts its BRPC server inside\n`ScopedLeakCheckDisabler`, like the other BE tests that start one,\nbecause brpc registers process-lifetime built-in services on its first\nstart.\n\nNot in this PR: the FE route. A remote reader\u0027s abort, and an abort on\none result BE, still do not reach the query\u0027s other BEs; their fragments\nand buffers wait for the timeout as before. The follow-up adds\n`FrontendService.cancelFlightQuery` (same signature as branch-4.x), the\nBE broadcast to the FEs and the FE routing table, together with the FE\nunit test and the `test_flight_cancel_cleanup` regression test.\n\n| Situation | Before | After |\n|---|---|---|\n| Client cancels `DoGet` while the result buffer is empty (local reader)\n| Server thread waits until the query produces data or ends | Returns\nwithin 20 ms; the query is cancelled on this BE |\n| Client closes a local stream before EOF | Query keeps running,\nback-pressured until it times out if the result does not fit; unread\nblocks kept for minutes | Query cancelled on this BE; all of its Flight\nbuffers there freed at once |\n| Client cancels while a remote fetch is pending | Waits for the result\nBE, up to 300 s | BRPC cancelled within 20 ms |\n| FE cancels the query (`cancel_plan_fragment`, abort reason) | Unread\nblocks kept until the delayed cleanup, 300 s later | Flight buffers\nfreed at once; `LIMIT_REACH` / `FINISHED` still keep them |\n| `fetch_arrow_data` for a removed buffer | RPC never completes; caller\nwaits for its timeout | Completes at once with an error |\n| Stream read to EOF, then closed | Nothing cancelled | Nothing\ncancelled |\n| New `DoGet` with the ticket of a stream aborted on its result BE |\nReads the remaining blocks, or EOS | Fails: the buffer is gone |\n\n#### Results\n\n- `ArrowFlightCancelTest` (9 tests, new): early close and destruction\nfree a back-pressured buffer and cancel the query; sibling buffers are\nfreed after their `QueryContext` expired; EOF followed by close is not a\ncancellation; cancellation interrupts an empty-buffer wait; empty waits\nreturn for retry and do not end the stream; a conversion error cancels\nthe query; a real Flight server sees a client-side `Cancel()` through\n`ServerCallContext` and cleans up.\n- `ArrowFlightRemoteCancelTest` (3 tests, new, against an in-process\nBRPC `PInternalService`): cancellation interrupts a pending\n`fetch_arrow_data`; an aborting `cancel_plan_fragment` frees the buffer\nand a following fetch completes with an error inside its 1 s timeout;\n`LIMIT_REACH` / `FINISHED` keep the unread results.\n- On a local single-BE cluster, branch-4.x\u0027s regression\n`test_flight_cancel_cleanup` (it aborts one endpoint after the first\nbatch, by closing it and by an explicit cancel, and requires the query\u0027s\npipelines to disappear within 20 s) fails on master with `Cancelled\nFlight query retained pipelines: [...: 1 pipeline fragment contexts in\nquery ...]` and passes with this PR. The suite comes with the follow-up,\nbecause on a multi-BE cluster it also needs the FE route. The 20\n`arrow_flight_sql_p0` suites and the 14 `external_table_p0/remote_doris`\nsuites, which read results over Arrow Flight, pass before and after.\n- The 12 new tests plus `ResultBufferMgrTest` (6) and\n`ArrowResultBlockBufferTest` (4) pass 10 times in a row in one ASAN\nprocess. Ten single-point mutations of the change (dropping the buffer\ncleanup in `cancel_plan_fragment`, the `on_failure` call, the local\ncancel in `close()`, the cancellation check in the local read loop or in\nthe BRPC wait, the sibling loop, the EOF flag, the `QueryContext`\ncancel, the cancel in the destructor, the `eos \u003d false` on an empty\nwait) each turn their target tests red."
    },
    {
      "commit": "28cbcfc233e81218e5a3e38b83676637fa53772e",
      "tree": "0a36e16349a0ee6983457a5d5ca08a30b6b41600",
      "parents": [
        "b5996fbf10831d331c37c38867ad79718b5b5910"
      ],
      "author": {
        "name": "Gabriel",
        "email": "liwenqiang@selectdb.com",
        "time": "Wed Nov 04 10:12:58 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 10:12:58 2026 +0800"
      },
      "message": "[fix](build) Enable Parquet modular encryption in Arrow (#70087)\n\n### What problem does this PR solve?\n\nThe third-party Arrow build leaves `PARQUET_REQUIRE_ENCRYPTION` at its\ndefault `OFF`. Parquet still installs encryption APIs and exports some\nencryption symbols, but its archive contains\n`encryption_internal_nossl.cc.o`. Reading a standard PME-encrypted\nParquet file with the correct key then throws:\n\n```text\nCalling encryption method in Arrow/Parquet built without OpenSSL\n```\n\nLinking `libssl.a` and `libcrypto.a` into the consumer does not replace\nthose compiled stubs. This change supplies the build prerequisites for\nnative encrypted Iceberg reads.\n\nThe change:\n\n- Enables `PARQUET_REQUIRE_ENCRYPTION` and selects static OpenSSL.\n- Pins OpenSSL headers and both archives to the Doris third-party\nprefix, and fails before CMake if any required input is missing.\n- Installs `encryption_internal.h` and `internal_file_decryptor.h` from\nthe same pinned Arrow source. Arrow normally excludes these internal\nheaders, but module-level decryption is needed to retain the native\nParquet scan path.\n\nExisting third-party prebuilt packages must be rebuilt to pick up these\nchanges. SQL-level encrypted Iceberg read integration remains separate\nwork.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test:\n    - [x] Manual test (details below)\n- Behavior changed:\n- [x] Yes. Third-party Parquet is built with PME enabled; missing owned\nOpenSSL inputs fail before configure.\n- Does this need documentation?\n- [x] No. This is the third-party build prerequisite, with no new\nSQL-facing configuration.\n\n#### Validation\n\nOn Linux x86_64, extracted Arrow 24.0.0 from the archive pinned by\n`thirdparty/vars.sh`, applied the current Doris Arrow patches, and\nexecuted the changed `build_arrow()` function in an isolated prefix\nseeded with existing third-party dependencies. The complete Arrow build\nand install succeeded. CMake selected PME `ON` and OpenSSL 1.1.1s from\nthat prefix. The rebuilt archive contains `encryption_internal.cc.o` /\n`openssl_internal.cc.o`, with no `nossl` object.\n\nA standalone C++ reader linked directly to the checked/rebuilt static\narchives, including the same prefix\u0027s OpenSSL archives, used the public\nApache [Parquet test\nfixtures](https://github.com/apache/parquet-testing/tree/master/data):\n\n| Check | Before: checked PME-disabled package | After: rebuilt package\n|\n| --- | --- | --- |\n| `uniform_encryption.parquet.encrypted`, correct public test key |\nOpenSSL-disabled runtime error | All-column scan succeeds: 50 rows |\n| Same file, wrong key | OpenSSL-disabled runtime error | Authentication\nfailure: `Failed decryption finalization` |\n| Same file, no key | Missing decryption-properties error | Missing\ndecryption-properties error |\n| `alltypes_plain.parquet` | All-column scan succeeds: 8 rows |\nAll-column scan succeeds: 8 rows |\n\nAdditional checks:\n\n- Module-level AES-GCM round trip and rejection of a different data-page\nAAD pass.\n- Both installed internal headers match the source and compile/link with\n`InternalFileDecryptor` and the module decryptors.\n- Removing each required OpenSSL input rejects the build before CMake.\nThe baseline function reaches CMake for the same incomplete inputs.\n- Captured Linux and Darwin invocations contain the PME option and\nexplicit static OpenSSL paths; this is an argument check, not a macOS\nbuild.\n- `bash -n thirdparty/build-thirdparty.sh` and `git diff --check` pass.\n\nValidation limits: no full Doris BE build, Doris SQL test, AArch64 build\nor native macOS build was run. No runtime reader behavior is enabled by\nthis PR alone.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label"
    },
    {
      "commit": "b5996fbf10831d331c37c38867ad79718b5b5910",
      "tree": "0b58c8cf774441dcba8b92b17196951cf05cbb7e",
      "parents": [
        "11bfb7388944340181e7e4bef55c7629ea44e902"
      ],
      "author": {
        "name": "Socrates",
        "email": "suyiteng@selectdb.com",
        "time": "Wed Nov 04 10:12:43 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 10:12:43 2026 +0800"
      },
      "message": "[chore](paimon) Upgrade paimon-rust and reserve ANN Thrift fields (#70075)\n\n### What problem does this PR solve?\n\nRelated PR: #69493\n\nProblem Summary: Master still builds paimon-rust v0.4.0-rc1, while the\nPaimon vector-search work in #69493 needs v0.5.0-rc1 and additional wire\nfields. Upgrade the native dependency and reserve its Thrift layout on\nmaster so later changes cannot reuse the same identifiers for different\nmeanings.\n\nThe dependency change updates the source URL, archive name, source\ndirectory and MD5 checksum together. The schema change adds the\nfollowing optional fields and their Paimon payload/options types:\n\n| Structure | Field | ID |\n| --- | --- | --- |\n| `TPaimonFileDesc` | `vector_payload` | 24 |\n| `TExternalSearchRequest` | `paimon_options` | 6 |\n| `TFileScanRangeParams` | `external_search_request` | 42 |\n\n`TPaimonVectorPayloadType` keeps `BUCKET_SPLIT_BYTES\u003d0` and\n`MATERIALIZE_OPAQUE_BYTES\u003d1`. Both payload members and all Paimon\nsearch-option members are optional. Master does not emit these fields\nyet; this PR reserves the protocol without enabling SQL/TVF ANN search\nor upgrading the Paimon Java SDK.\n\nCompatibility notes:\n\n- `TPaimonFileDesc.20` remains `original_file_path`; comments reserve\nbranch-4.1\u0027s `blob_files\u003d21`, `anchor_first_row_id\u003d22` and\n`anchor_row_count\u003d23`. The vector payload uses 24 rather than the\nconflicting 21 from the original #69493 diff.\n- Comments reserve the existing branch-4.1/4.2 shared-search fields\n`search_query\u003d2`, `search_filter\u003d3`, `vector_search_options\u003d4` and\n`prefilter\u003d5`. Their query types can be introduced independently on\nmaster.\n- `TFileScanRangeParams.41` remains `fluss_properties`.\n- The affected definitions were reviewed against current master and\nbranches 3.1, 4.0, 4.1 and 4.2. Existing field IDs and enum values\nretain their meanings.\n\n### Release note\n\nUpgrade the native Paimon Rust reader dependency to v0.5.0-rc1.\n\n### Check List (For Author)\n\n- Test: No local build, Thrift generation or unit tests on master, as\nrequested; CI validation pending. During the preceding branch-4.2 work,\nthe archive checksum was verified and the v0.5.0-rc1 dependency was\nbuilt with Rust 1.94.0. This is not a completed master BE build or BE\nunit-test run.\n- Behavior changed: Yes; the bundled native Paimon dependency is\nupgraded. The new Thrift fields are reserved and remain unset by master\nproducers.\n- Does this need documentation: No"
    },
    {
      "commit": "11bfb7388944340181e7e4bef55c7629ea44e902",
      "tree": "d699e2b5f7ec47e914db39fad1370c7677305505",
      "parents": [
        "cbc30b98137f06353e8d5ce3953280d42849bef1"
      ],
      "author": {
        "name": "924060929",
        "email": "lanhuajian@selectdb.com",
        "time": "Wed Nov 04 10:08:42 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 10:08:42 2026 +0800"
      },
      "message": "[fix](metacache) Complete guarded external table lookup cleanup (#69806)\n\n### What problem does this PR solve?\n\nRelated PR: #69501, #69629, #69632, #69720\n\nProblem Summary: Complete guarded cleanup of stale external table lookup\nstate after confirmed remote absence, preserving refreshed or same-name\nrecreated tables and unrelated numeric lookup routes.\n\nExternal metadata reads can discover a dropped table after resolving a\ncached handle while FE names, table objects and numeric ID routes remain\nstale. A refreshed names list can also omit a resident old object. In\ncase-folding modes, targeted cleanup previously cleared unrelated\nnumeric routes.\n\nRoute confirmed provider absence through a typed SPI exception carrying\nthe exact remote database/table identity. Capture the existing lookup\nobservation before handle resolution and retain it through base/system\nschemas (including pinned system schemas), latest/pinned row counts,\nordinary write-schema binding and row-level DML validation,\npartition-values and partition-name metadata (SHOW PARTITIONS and\nPARTITIONS), eager/deferred scan partition views, MVCC\npartition/freshness reads and deferred loads, and scan\ncolumn/property/filter/estimation and synchronous/deferred split reads.\nSystem relations repair their shared source identity; query TVFs have no\nshared lookup identity. Missing files, transport/permission failures and\nunrelated resources do not authorize cleanup. Preserve live views, cache\nTTLs, HMS exception wrapping and comment display fallbacks.\n\nRuntime, names, object and metadata-refresh guards reject delayed\ncleanup after refresh or same-name recreation. Catalog-wide,\nmetadata-only and explicit lookup refreshes use the catalog-owned\ngeneration and active count across their full interval, protecting\nreplacement database/table objects admitted by normal eviction or\npublished during hot/cold invalidation. Database refresh and truncate\nreplay include their cold-cache branches; rename replay, overlapping\nrefreshes and errors use the same owner guard. Existing per-table\nmetadata generations remain part of the observation. Ordinary writes\nremain metadata-only and do not reload table names. Targeted wide\ninvalidation retires only the target\u0027s ROOT-folded ID aliases while\nretaining unrelated numeric routes. Provider IO remains outside the\ncatalog monitor.\n\nPinned system schema reads use the existing schema-session lifecycle\nhooks. Their live statement owns the borrowed scope until all aliases\nand scans finish; the schema read does not prematurely close it.\n\nThe guarded resolver preserves master\u0027s view classification before\ntable-endpoint access, including connectors that return non-404 errors\nfor views; live view deletion still permits guarded cleanup.\n\nHive file-size estimates preserve confirmed table absence until guarded\nFE cleanup, then retain the unknown estimate fallback. Sampled ANALYZE\nfile-size reads retain the same observation and their existing\nfailure/session-closure behavior. Ordinary listing failures still\ndegrade statistics to unknown.\n\nNative Hive table and table-scoped partition-name RPCs preserve\nexact-source absence under the existing HmsClientException boundary;\nboth ordinary/fresh and cancellation-aware listings use the same RPC\nnormalization. Individual partition absence, ordinary failures and\nsuccessful empty lists keep their existing behavior.\n\nMTMV partition warmup retains its pre-handle MetadataRead through\nsnapshot binding and partition enumeration. The combined interface\nretains current master\u0027s DEFAULT-column and partition-filter APIs.\n\nNative Iceberg write-shaping loads preserve exact-source table absence\nthrough the existing shared converter, while ordinary write exceptions,\nresource tracking and statement ownership keep their prior contracts.\n\nThe shared absence exception bumps the connector plugin API to 29.0. No\ncache owner framework, persisted ID format, or journal format changes.\n\n### Release note\n\nRepair stale external-table lookup state after confirmed table/view\nabsence while preserving refreshed or recreated tables and unrelated\nnumeric lookups."
    },
    {
      "commit": "cbc30b98137f06353e8d5ce3953280d42849bef1",
      "tree": "7da009ab89c630ec9077e6cb896fc7d982214b12",
      "parents": [
        "b65a564ea5bc6228977ef46a09085359be7bcc66"
      ],
      "author": {
        "name": "hui lai",
        "email": "laihui@selectdb.com",
        "time": "Wed Nov 04 10:06:53 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 10:06:53 2026 +0800"
      },
      "message": "[fix](fe) Align Thrift wire IDs with branch-4.1 (#70057)\n\n### What problem does this PR solve?\n\nProblem Summary:\n\nDuring a rolling upgrade from 4.1 to master, the same Thrift enum value\nor field ID can have different meanings on FE and BE:\n\n- `TSchemaTableType` diverges starting at value 62. For example, 4.1\nsends `SCH_FILE_CACHE_INFO \u003d 62`, which master interprets as\n`SCH_BLACKHOLE`.\n- `TRLTaskTxnCommitAttachment.firstErrorMsg` uses field 12 in 4.1 but\nfield 13 in master. Master uses field 12 for the Kinesis progress\nstruct, so mixed-version readers skip the error message because the wire\ntypes do not match.\n\nAlign master\u0027s shared enum values with branch-4.1 (also matching\nbranch-4.2), place master-only database properties and table stream\nvalues at 70–72, and preserve extensions/TSO at 73/74. Restore\n`firstErrorMsg` to field 12 and move the optional Kinesis progress field\nto 13.\n\nCompatibility review: checked both definitions on branch-3.1,\nbranch-4.0, branch-4.1, branch-4.2 and master. Branch-4.0 already\ndiffers from 4.1 for `SCH_BLACKHOLE`, `SCH_COLUMN_DATA_SIZES`,\n`SCH_LOAD_JOBS` and `SCH_BE_COMPACTION_TASKS`; this master-only fix does\nnot resolve that existing release-to-release mismatch. These corrections\nalso change the affected wire IDs relative to unpatched master.\n\n### Release note\n\nFix schema table enum values and routine load commit attachment field\nIDs on master to match Doris 4.1 and 4.2 during rolling upgrades."
    },
    {
      "commit": "b65a564ea5bc6228977ef46a09085359be7bcc66",
      "tree": "b53b63bd15b3c4573789ef9e11652dcc199ff30c",
      "parents": [
        "b78c656c0cbcc255b28509c5dc79a9d9d4b282f2"
      ],
      "author": {
        "name": "daidai",
        "email": "changyuwei@selectdb.com",
        "time": "Wed Nov 04 10:00:12 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 10:00:12 2026 +0800"
      },
      "message": "[fix](be) Route glibc allocation entry points through jemalloc (#69819)\n\n### What problem does this PR solve?\n\nProblem Summary:\nOn glibc \u003c 2.25, dynamic TLS can be allocated by libc and freed by\njemalloc, crashing during thread cleanup. Route all seven glibc\nallocation entry points through the existing jemalloc wrappers, preserve\nallocator API semantics, and add native interposition tests.\n\n### Release note\n\nFix a BE crash during dynamic TLS cleanup on older glibc systems and\nkeep glibc allocation entry points paired with jemalloc.\n\n### Check List (For Author)\n\n- Test\n    - [ ] Regression test\n    - [x] Unit Test\n    - [ ] Manual test (add detailed scripts or steps below)\n    - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n        - [ ] Previous test can cover this change.\n        - [ ] No code files have been changed.\n        - [ ] Other reason\n\n`./run-be-ut.sh --jemalloc-hook --run -j 2`: 12/12 passed on Linux\nx86_64/glibc 2.32. Full BE, older glibc, ARM64 and customer JNI E2E were\nnot run.\n\n- Behavior changed:\n    - [ ] No.\n- [x] Yes. Glibc allocation entry points now consistently use jemalloc,\nwith allocator boundary semantics preserved.\n\n- Does this need documentation?\n    - [x] No.\n    - [ ] Yes.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label"
    },
    {
      "commit": "b78c656c0cbcc255b28509c5dc79a9d9d4b282f2",
      "tree": "33757143fa82345e1f0fe102babe074280a5308a",
      "parents": [
        "6802f57005ee4f9e96caa609cc17ec264e368f6f"
      ],
      "author": {
        "name": "Chenjunwei",
        "email": "138805230+xylaaaaa@users.noreply.github.com",
        "time": "Wed Nov 04 09:59:11 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 09:59:11 2026 +0800"
      },
      "message": "[fix](iceberg) Commit every Iceberg file under the table\u0027s own location (#69392)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #69082\n\nProblem Summary:\n\n**In short.** DELETE, UPDATE and MERGE on an Iceberg table recorded the\ndelete files they write (v2 position delete files and v3 deletion vector\nfiles) under the internal `s3://` address BE writes them at, instead of\nthe table\u0027s own location (`oss://`, `s3a://`, `abfss://`, ...). Doris\ncould not always read such a table back, and other engines and\nmaintenance procedures resolved those files with the wrong file system.\nThis PR commits delete files under the table\u0027s location. It also makes\nevery Iceberg file Doris writes take both of its paths from one place in\nBE and pass one check in FE, so the next writer cannot repeat the\nmistake without failing.\n\n#### Background\n\nEvery Iceberg sink FE sends carries two forms of the table\u0027s data\nlocation:\n\n- a normalized one BE can address the storage with (`output_path`): FE\nrewrites `oss://`, `cos://`, `cosn://`, `obs://`, `bos://`, `gs://`,\n`s3a://`, `s3n://` and Azure `abfs[s]://` / `wasb[s]://` locations\n(except OneLake) to `s3://`;\n- the raw one Iceberg metadata must record: `original_output_path` on\n`TIcebergTableSink`, `table_location` on `TIcebergDeleteSink`, both on\n`TIcebergMergeSink`.\n\nBE writes each file under the first form and reports it in\n`TIcebergCommitData.file_path`, which FE writes into the manifests as it\nis. So the reported path must be in the second form.\n\n#### The problem, and what it cost\n\nReproduce on an Iceberg table located at `oss://bucket/...`:\n\n```sql\nCREATE TABLE t (id INT, v STRING) PROPERTIES (\u0027format-version\u0027 \u003d \u00272\u0027);\nINSERT INTO t VALUES (1, \u0027a\u0027), (2, \u0027b\u0027), (3, \u0027c\u0027);\nDELETE FROM t WHERE id \u003d 1;\nSELECT file_path FROM t$delete_files;  -- s3://bucket/.../data/delete_pos_xxx.parquet\nSELECT file_path FROM t$data_files;    -- oss://bucket/.../data/...\n```\n\n- Whether Doris can still read the table depends on the catalog and\nstorage binding. In a real Azure ADLS Hadoop catalog test with explicit\nSharedKey, Doris SELECT failed with `No storage properties found for\nschema: s3`, and Spark failed with `UnsupportedFileSystemException: No\nFileSystem for scheme \"s3\"`.\n- Other engines and maintenance operations such as `expire_snapshots` /\n`remove_orphan_files` may resolve the wrong file system from the\nrecorded scheme, depending on the engine and FileIO. Doris\u0027s own\n`remove_orphan_files` refuses to run on such a table (`Cannot determine\nwhether listed and reachable file locations are equivalent`).\n- Cause: data files were written under `write_path` and committed under\n`original_write_path` (`VIcebergPartitionWriter`), but\n`VIcebergDeleteSink` had no such split; it reported the\n`output_path`-based path it wrote.\n- Why it went unnoticed: the two writers mapped the paths in different\nways, `VIcebergDeleteFileWriter` reported its write path by default, and\nFE accepted any path. CI did run on such a location: the Iceberg REST\nfixture\u0027s warehouse is `s3a://`, which FE normalizes to `s3://`. But the\nDML suites only checked that delete file paths contain\n`/data/delete_pos_`, and a stray\n`test_iceberg_update_delete_basic.out.org` in the repository records the\nwrong `s3://warehouse/wh/...` paths.\n\n#### How this PR fixes it\n\n1. `[fix](be) Commit Iceberg delete files under the table\u0027s own\nlocation`, `[test](be) Verify Iceberg delete file commit paths` and\n`[test](be) Initialize the Arrow memory pool in the Iceberg position\ndelete write test`: `VIcebergDeleteSink` writes under `output_path` and\ncommits position delete files and deletion vectors under\n`table_location`. An FE unit test pins that both sinks carrying\n`table_location` name the same directory as `output_path`, and\n`DataSinks.thrift` documents the contract. The position delete write\ntest sets up the Arrow memory pool the Parquet writer allocates from, so\nit also passes when run alone (CI review bot finding on this PR).\n2. `[refactor](be) Take both paths of every Iceberg file from one output\nlocation`:\n- The new `IcebergOutputLocation` (header-only) holds one directory in\nboth forms and gives the write path and the commit path of a file name,\nand of a partition subdirectory.\n- The data writer (`VIcebergTableWriter` / `VIcebergPartitionWriter`),\nthe delete file writer and the Puffin path all take their paths from it.\n`VIcebergDeleteFileWriter` now reports the commit path itself instead of\nits write path.\n- A table or delete sink without both location fields is rejected in\n`open()` / `init_properties()` instead of committing a write path or a\nrelative path. The FE of every branch (3.1, 4.0, 4.1, 4.2, master) sets\nboth fields on every Iceberg sink.\n   - Directories join file names with exactly one `/`.\n3. `[improvement](fe) Reject Iceberg commit data reported outside the\nsink\u0027s location`:\n- `planWrite` registers on the transaction the raw location(s) each sink\nit builds carries, read back from the built sink.\n- `IcebergConnectorTransaction.addCommitData`, where every report from\nevery writer enters FE, rejects a file that is not under a registered\nlocation, or has no path. FE then answers the report as not accepted,\nand BE deletes the files under the paths it wrote them at and cancels\nthe query, so a wrong path fails the statement and leaves no orphan\nfiles.\n- Thrift comments state the contract, including that \"the same\ndirectory\" can differ in more than the scheme (Azure moves the account\nout of the authority). Comments only, no protocol change.\n4. `[test](regression) Check that every Iceberg write records its files\nunder the table location`:\n- The new `test_iceberg_write_file_locations` runs INSERT, INSERT\nOVERWRITE PARTITION, branch INSERT, DELETE, UPDATE, MERGE and\n`rewrite_data_files` on a v2 table, and INSERT, DELETE, UPDATE and MERGE\non a v3 table. After each operation every file in `$all_files` must\nstart with the table\u0027s LOCATION plus `/data/`. A new write operation\nbelongs in this suite too.\n- `test_iceberg_update_delete_basic` checks the prefix of its delete\nfiles as well, and the stray `.out.org` is removed.\n\n| Scenario | Before | After |\n| --- | --- | --- |\n| DELETE / UPDATE / MERGE on a v2 table at `oss://`, `s3a://`,\n`abfss://`, ... | position delete files recorded as `s3://...` |\nrecorded under the table\u0027s location |\n| The same on a v3 table | deletion vector (Puffin) files recorded as\n`s3://...` | recorded under the table\u0027s location |\n| A BE writer reports a file outside the location its sink carries |\ncommitted as reported | the statement fails; BE deletes the file |\n| A sink plan without `original_output_path` / `table_location` (no\nsupported FE sends one) | data files committed as `/p\u003d1/x.parquet`,\ndelete files as `s3://...` | rejected before any file is written |\n| `write.data.path` ending with `/` | data files at\n`.../data//p\u003d1/x.parquet` | `.../data/p\u003d1/x.parquet`, as Iceberg\u0027s own\nlocation provider places them |\n\nDelete files that an older version already committed with `s3://` keep\nthat address until they are rewritten or expired.\n\nUpgrade BE before FE, as usual. An upgraded FE rejects the `s3://`\ndelete file paths that a BE without this fix reports, so until that BE\nis upgraded, DELETE / UPDATE / MERGE on a table whose location FE\nnormalizes fails instead of committing the wrong path.\n\n#### Classes, and how they call each other\n\n- `IcebergWritePlanProvider.planWrite` (FE) builds the sink and\nregisters the raw location(s) it carries with the transaction\n(`commitLocationsOf`).\n- `IcebergConnectorTransaction.addCommitData` (FE) accepts a reported\nfile only under a registered location; `IcebergWriterHelper` then turns\nthe fragments into `DataFile` / `DeleteFile`.\n- `IcebergOutputLocation` (BE, new) is one directory in its write form\nand its commit form.\n- `VIcebergTableWriter` builds the data location in `open()` and hands\neach `VIcebergPartitionWriter` its partition directory\n(`WriteInfo.location`). `VIcebergDeleteSink` writes position delete\nfiles through `VIcebergDeleteFileWriter` and deletion vectors into one\nPuffin file, both from its location. `VIcebergMergeSink` builds one of\neach.\n\n```\nFE planWrite ── sink: output_path + raw location ──────────────► BE\n  │ addCommitLocation(raw location)                              IcebergOutputLocation(output_path, raw location)\n  ▼                                                                ├─ VIcebergTableWriter ─► VIcebergPartitionWriter   data files\nIcebergConnectorTransaction                                        ├─ VIcebergDeleteSink ──► VIcebergDeleteFileWriter  position deletes\n  ▲ addCommitData: file_path under a registered                    │                     └─► Puffin writer             deletion vectors\n  │ location, else reject the report                               └─ VIcebergMergeSink ──► both of the above\n  └──────────── TIcebergCommitData.file_path \u003d commit path ◄───────┘\n```\n\nNot changed: Thrift fields (comments only), the scan side, and how FE\nnormalizes locations."
    },
    {
      "commit": "6802f57005ee4f9e96caa609cc17ec264e368f6f",
      "tree": "594d42f78bb68f2a90fc8ae819f7745aac9d7fb8",
      "parents": [
        "01400351428fb39147d7af17bc4352087c8a06f3"
      ],
      "author": {
        "name": "Chenyang Sun",
        "email": "sunchenyang@selectdb.com",
        "time": "Wed Nov 04 09:41:00 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 09:41:00 2026 +0800"
      },
      "message": "[chore](build) Require new thrift/proto ids to land on master first (#70066)\n\n Require new thrift/proto ids to land on master first"
    },
    {
      "commit": "01400351428fb39147d7af17bc4352087c8a06f3",
      "tree": "b7bf84db080fa688874138bab54621ba56b15600",
      "parents": [
        "7866f5a36c02e15dc014aaef30e537a395f4059d"
      ],
      "author": {
        "name": "yujun",
        "email": "yujun@selectdb.com",
        "time": "Wed Nov 04 08:15:30 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Wed Nov 04 08:15:30 2026 +0800"
      },
      "message": "[fix](fe) Remove misleading missing-progress warnings for INSERT jobs (#70077)\n\nA successful `INSERT INTO dst SELECT ... FROM src` can repeatedly emit\n`ProgressManager` missing-meta warnings. Materialized view refreshes use\nthe same INSERT execution path and can produce large volumes of these\nwarnings.\n\nINSERT jobs have a load job ID but do not register scan progress.\nReports may also arrive after a bulk load removes its progress entry. A\nmissing entry therefore does not indicate an error. Remove the\ndiagnostic entirely and continue updating only registered entries,\nconsistent with `addTotalScanNums`. Backend statistics reporting and\nprogress registration/cleanup remain unchanged."
    },
    {
      "commit": "7866f5a36c02e15dc014aaef30e537a395f4059d",
      "tree": "3e7d8ccd322bfaa8ce23171bc5877f28cefc9d79",
      "parents": [
        "6b3481f28b93650e9457996052986e6d703bfd4a"
      ],
      "author": {
        "name": "Calvin Kirs",
        "email": "guoqiang@selectdb.com",
        "time": "Tue Nov 03 22:29:08 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Tue Nov 03 22:29:08 2026 +0800"
      },
      "message": "[fix](mtmv) Batch non-empty partition checks during MV compensation (#69966)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: None\n\nProblem Summary:\n\nCloud MV partition compensation checks each valid mapped MV partition\nthrough a singleton non-empty request. Cached-empty or unknown\npartitions can therefore cause many serial MS version requests, repeated\nacross related partition tables, even when the MV is not ultimately\nselected.\n\nCollect unique mapped valid MV partition IDs and call the existing bulk\ninterface in batches of at most 5,000. Reuse the returned ID set across\nrelated tables, preserving partition removal, base-table compensation,\ncache handling and exception propagation.\n\nThe 5,000-partition cap is a fixed code constant, not a runtime\nconfiguration. It bounds the three request ID lists to at most 165,000\nbytes at the worst protobuf integer width, plus the remaining request\nfields. This is a request-size guard, not a claimed performance-optimal\nbatch size. Tests verify that 1,020 candidates use one bulk call and\n10,001 use three; these are mocked interface call counts, not a live\nCloud benchmark."
    },
    {
      "commit": "6b3481f28b93650e9457996052986e6d703bfd4a",
      "tree": "2746eefb3fe9a817b028f6fb308b7c68a3272742",
      "parents": [
        "856194ece0e1e0ad45b46b96f26a3e2859412e06"
      ],
      "author": {
        "name": "Gavin Chou",
        "email": "gavin@selectdb.com",
        "time": "Tue Nov 03 22:16:58 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Tue Nov 03 22:16:58 2026 +0800"
      },
      "message": "[chore](fe) Upgrade BDB JE to 18.3.18 (#69970)\n\n### What problem does this PR solve?\n\nUpgrade the FE BDB JE dependency to the latest Doris snapshot requested\nfor all active target branches.\n\n### What is changed and how does it work?\n\n- Update `je.version` in `fe/pom.xml` to `18.3.18-doris-SNAPSHOT`.\n\n### Related PRs\n\n- Canonical Apache Doris master PR:\nhttps://github.com/apache/doris/pull/69970\n\n### Check List\n\n- [x] No local build or test was run, as requested for this\ndependency-only version update.\n\nCo-authored-by: gavinchou \u003cgavinchou@apache.org\u003e"
    },
    {
      "commit": "856194ece0e1e0ad45b46b96f26a3e2859412e06",
      "tree": "b8559232bb310ac1ee40fdce66fce08f41e4a376",
      "parents": [
        "1f71a665654f60ac5895440a3c86e02ea8a36825"
      ],
      "author": {
        "name": "Gabriel",
        "email": "liwenqiang@selectdb.com",
        "time": "Tue Nov 03 18:37:08 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Tue Nov 03 18:37:08 2026 +0800"
      },
      "message": "[test](regression) Check ORC lazy reads using row results (#70059)\n\n### What problem does this PR solve?\n\nIssue Number: N/A\n\nRelated PR: N/A\n\nThe ORC lazy materialization suite asserts exact counters from merged\nquery profiles. Reader counters can be absent when an empty scanner\nsupplies the profile merge template, making the suite fail independently\nof query correctness.\n\nReplace profile polling and metric assertions with full-row comparisons\nagainst an eager scan with ORC min/max filtering and lazy\nmaterialization disabled. Expected rows are filtered independently in\nGroovy. Exercise all four combinations of the two options with six\npredicates covering equality, partial and complete groups, empty\nresults, tail reads, and ranges spanning groups. Disable result and\ncondition caches during the checks and restore the modified session\nsettings in `finally`.\n\nThis suite now checks result equivalence rather than exact internal\nfiltering counts. No production code is changed.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test\n  - [ ] Regression test\n  - [ ] Unit Test\n  - [x] Manual test (details below)\n  - [ ] No need to test or manual test.\n\nValidation completed:\n\n- `git diff --check` passed.\n- Regression runner dry-run passed: `./run-regression-test.sh --run -d\nexternal_table_p0/hive -s test_orc_lazy_mat_profile -dryRun`.\n- An offline Groovy/H2 harness executed the suite against rows read from\nits existing ORC fixture: all 24 filtered queries passed. Fault\ninjections detected missing rows, duplicate rows, incorrect payloads,\nignored predicates, and an empty fixture. Session settings were restored\nafter both successful execution and assertion failures; disabling Hive\nskipped all SQL.\n\nThe dry-run and offline harness do not execute the Doris/Hive read path.\nFull Hive-backed integration validation has not been run locally.\n\n- Behavior changed:\n- [x] No production behavior change. Only regression assertions change\nas described above.\n  - [ ] Yes.\n\n- Does this need documentation?\n  - [x] No.\n  - [ ] Yes.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label"
    },
    {
      "commit": "1f71a665654f60ac5895440a3c86e02ea8a36825",
      "tree": "280b4b4c9cc3748a57c1e1caa041f5f638778564",
      "parents": [
        "e4c2dc261a7a8f1dca9286f2509a4e4fca062ce7"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Tue Nov 03 10:27:49 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Tue Nov 03 18:27:49 2026 +0800"
      },
      "message": "[fix](fe) Keep sibling capabilities on iceberg-on-HMS tables pinned at a snapshot (#69999)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #69299, #69488\n\nProblem Summary:\n\n#### In short\n\nAn Iceberg table stored in HMS and read through an HMS catalog\n(`\u0027type\u0027\u003d\u0027hms\u0027`) loses every capability it inherits from the embedded\nIceberg connector in any statement that pins the table at a snapshot.\nSince #69299 that is every statement that reads it: it gets no Top-N\nlazy materialization and no nested column pruning, and `DEFAULT(column)`\nin the conditions of `UPDATE` and `MERGE` silently matches nothing. This\nPR makes the snapshot overload of `HiveConnectorMetadata#getTableSchema`\npass the inherited capabilities on, as the latest-schema overload\nalready does. It is a master-only fix (branch-4.x has no connector SPI),\nleft over from #69488 (review comment 4355508100).\n\n#### Background\n\nfe-core resolves a table-scoped capability\n(`PluginDrivenExternalTable.hasCapability`) from the catalog\u0027s connector\nor from the per-table set in the table\u0027s schema. It never asks the\nembedded connector that built the schema of an iceberg-on-HMS table. So\n`HiveConnectorMetadata` copies the capabilities that fe-core resolves\nper table (`SIBLING_INHERITABLE_CAPABILITIES`: auto and sampled analyze,\nTop-N lazy materialization, nested column pruning, field-id access\npaths, storage predicate pruning, nested column DDL and write-only\ndefaults) from the owning connector into the per-table set of each\nschema it delegates. Iceberg declares seven of the eight; it does not\ndeclare sampled analyze.\n\nIt did this only in `getTableSchema(session, handle)`.\n`getTableSchema(session, handle, snapshot)` returned the sibling\u0027s\nschema unchanged. fe-core uses that overload for explicit time travel.\nSince #69299 it also uses it for the latest-snapshot pin of every\nstatement that reads an Iceberg table: the pin retains its schema\n(`retainSchema(true)`), `PluginDrivenMvccExternalTable` builds the\nschema at that pin, and `getSchemaCacheValue()`, where `hasCapability`\nreads the per-table set, returns it for the rest of the statement.\n\n#### The problem, and what it cost\n\nReproduced on a local cluster with a standalone Hive 3.1.2 metastore.\nTwo identical format-version 3 tables `(id INT, c INT, s STRUCT\u003ca:INT,\nb:STRING\u003e)` get two rows, then `ADD COLUMN d INT DEFAULT 42`, `RENAME\nCOLUMN s.a TO a2` and a third row with `d \u003d 7`. The same statements then\nrun against one table through the Iceberg catalog and against the other\nthrough the HMS catalog.\n\n| Statement | Iceberg catalog | HMS catalog, master | HMS catalog, this\nPR |\n|---|---|---|---|\n| `EXPLAIN SELECT * FROM t ORDER BY id LIMIT 1`, also with `FOR VERSION\nAS OF` | `VMaterializeNode` | no lazy materialization |\n`VMaterializeNode` |\n| `EXPLAIN SELECT struct_element(s, \u0027a2\u0027) FROM t` | `all access paths:\n[s.a2]` | no pruning | `all access paths: [s.a2]` |\n| `UPDATE t SET c \u003d 100 WHERE d \u003d DEFAULT(d)` | updates rows 0 and 1 |\nupdates nothing | updates rows 0 and 1 |\n| `MERGE INTO t USING (SELECT 0 AS id) s ON t.id \u003d s.id AND t.d \u003d\nDEFAULT(t.d) WHEN MATCHED THEN UPDATE SET c \u003d 300` | updates row 0 |\nupdates nothing | updates row 0 |\n| `DELETE FROM t WHERE d \u003d DEFAULT(d)` | fails: `cannot be used in a\nDELETE` | deletes nothing | fails the same way |\n| `SELECT DEFAULT(d) FROM t` | fails: `must reference a column of the\ntable that this statement writes` | returns NULL | fails the same way |\n\nThe `UPDATE`, `MERGE` and `DELETE` rows fail silently: the statement\nsucceeds and changes nothing. Reads of the renamed field return the\nright values in all three columns: on master the HMS catalog does not\nprune the struct at all, and with this PR it prunes it with field-id\naccess paths, as the Iceberg catalog does. `INSERT ... VALUES\n(DEFAULT(d))` and `INSERT ... SELECT DEFAULT(d)` already wrote 42\nthrough both catalogs.\n\n#### How this PR fixes it\n\nBoth overloads now go through one private helper,\n`delegatedTableSchema`. It resolves the owning sibling once, asks it for\nthe schema (at the pin, for the snapshot overload), and passes the\nresult through the existing `reflectSiblingCapabilities`. The snapshot\nreaches the sibling unchanged. A hudi-on-HMS table is unaffected: the\nHudi connector declares no capabilities, so its schema comes back as it\nwas. A plain Hive table never reaches this branch.\n\n#### Results\n\nThrough the HMS catalog, every statement in the table above now plans\nand answers as it does through the Iceberg catalog."
    },
    {
      "commit": "e4c2dc261a7a8f1dca9286f2509a4e4fca062ce7",
      "tree": "fc386d2661ef655271280990b6620c5bc8cee0d8",
      "parents": [
        "f109e130f77c672f763a5082c9be856380bd32ae"
      ],
      "author": {
        "name": "Mingyu Chen (Rayner)",
        "email": "morningman.cmy@gmail.com",
        "time": "Tue Nov 03 10:21:39 2026 +0000"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Tue Nov 03 18:21:39 2026 +0800"
      },
      "message": "[fix](be) Reject Arrow timestamps outside the DATETIMEV2 and TIMESTAMPTZ ranges (#69697)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #69581 #68516\n\nProblem Summary:\n\n**In short.** An Arrow timestamp whose civil year is outside 0000..9999\nis read into DATETIMEV2 or TIMESTAMPTZ as a different, valid-looking\nvalue: timestamp(us) 2005980768000000000, which is 65537-01-01, reads as\na non-null 0001-01-01. This PR checks the year before the value is\nnarrowed into Doris\u0027 16-bit year field, as the ORC path already does,\nand fails the read otherwise. The review of #69581 found it on the Lance\npath; the code is shared by every Arrow reader.\n\n**Background.** `DataTypeDateTimeV2SerDe::read_column_from_arrow` and\n`DataTypeTimeStampTzSerDe::read_column_from_arrow` decode Arrow\ntimestamps for ADBC, Arrow stream load, paimon-rust, remote Doris and,\nonce #69581 lands, Lance, nested leaves included. Both convert epoch\nseconds with `DateV2Value::from_unixtime()`, which passes cctz\u0027s 64-bit\nyear to `unchecked_set_time(uint16_t year, ...)` (\"there\u0027s no overflow\ncheck since it\u0027s hot path\"). #68516 added a check before this narrowing\nto the ORC path (`orc_timestamp_to_datetime()`).\n\n**The problem, and what it cost.**\n\n| Arrow timestamp(us) | DATETIMEV2 before | TIMESTAMPTZ before |\n|---|---|---|\n| 253402300800000000 (10000-01-01) | stored as year 10000, outside the\ntype\u0027s range | error |\n| 2005980768000000000 (65537-01-01) | 0001-01-01 00:00:00 | 0001-01-01\n00:00:00+00:00 |\n| a year before 0000 | wraps to 655xx or back into 0000..9999 | error,\nor a wrapped valid year |\n\nIn the wrapped cases the read succeeds with a wrong value, and nothing\nin the result tells the user.\n\n**How this PR fixes it.**\n\n- DATETIMEV2: convert to civil time in the value\u0027s time zone (UTC for a\ntimezone-naive Arrow timestamp, the session time zone for a zoned one)\nand reject a year outside 0000..9999 before packing, like\n`orc_timestamp_to_datetime()`. After packing, `is_valid_date()` is\nchecked too: cctz counts year 0 as a leap year and Doris does not\n(`is_leap(0)` is false), so the civil date 0000-02-29 passes the year\ncheck, and before this PR it was stored as an invalid date.\n- TIMESTAMPTZ is stored in UTC, so\n`append_timestamptz_from_utc_epoch_micros` checks the instant against\n0000-01-01 00:00:00 UTC .. 9999-12-31 23:59:59.999999 UTC before\nconverting. Its runtime `is_valid_date()` check after packing stays, for\nthe same leap day: timestamp(us) -62162121600000000 (0000-02-29 UTC) is\ninside the range but packs into an invalid value. The range message now\nsays 0000-9999, the range `is_valid_date()` accepts.\n- Both fail the read with DataQualityError, as the other range errors of\nthese readers do. NULL slots are still skipped before conversion.\n- Not changed: the Parquet INT64 / INT96 timestamp paths validate the\nUTC range before packing (`parquet_timestamp_micros()`,\n`parquet_int96_timestamp_micros()`), so they could not wrap. Arrow reads\ninto the deprecated DATE / DATETIME (v1) types go through\n`VecDateTimeValue::from_unixtime()` with the same unchecked narrowing;\nthey are left as they are. `orc_timestamp_to_datetime()` checks only the\nyear, so an ORC timestamp on 0000-02-29 is still stored as an invalid\ndate; it is not changed here.\n\n**Merge order: after #69581.** A NULL STRUCT, LIST or MAP row may hide\nany child value, an out-of-range timestamp included, because Arrow\nleaves the payload under a NULL parent unspecified. #69581 masks those\nchild slots in the shared nested SerDes\n(`mask_arrow_children_hidden_by_null_parents`, df95b41a331) before any\nleaf SerDe reads them. Without it, the checks here fail a batch whose\nvisible values are valid: a STRUCT with rows `[NULL, {ts: 0}]` whose\nNULL row carries 2005980768000000000 reads with DataQualityError on this\nPR alone, and reads both rows on a tree with both PRs.\n\n**Results.**\n\n| Arrow timestamp(us) | DATETIMEV2 after | TIMESTAMPTZ after |\n|---|---|---|\n| 253402300799999999 (9999-12-31 23:59:59.999999) | read | read |\n| -62167219200000000 (0000-01-01) | read | read |\n| -62162121600000000 (0000-02-29, not a Doris date) | DataQualityError |\nDataQualityError |\n| 10000-01-01, 65537-01-01, -62167219200000001 (year -1) |\nDataQualityError | DataQualityError |\n| zoned, at +08:00: 253402271999999999 / 253402272000000000 | 9999-12-31\n23:59:59.999999 / DataQualityError | |"
    },
    {
      "commit": "f109e130f77c672f763a5082c9be856380bd32ae",
      "tree": "868191e8c41f2ee6b81c7e51491860cf6df4a4c2",
      "parents": [
        "173a0e2064e529bfa6f017608c40ff003c2f1066"
      ],
      "author": {
        "name": "hutiefang76",
        "email": "137664623+hutiefang76@users.noreply.github.com",
        "time": "Tue Nov 03 02:21:21 2026 -0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Tue Nov 03 18:21:21 2026 +0800"
      },
      "message": "[fix](arrow-flight) Support table type metadata and filters (#67899)\n\n### What problem does this PR solve?\n\nRelated to #67578 (partial A3).\n\nFlight SQL clients can obtain FlightInfo for GetTableTypes, but reading\nits ticket fails with UNIMPLEMENTED. GetTables also forwards only the\nfirst table_types entry to a service that treats only VIEW as a filter:\n[VIEW, BASE TABLE] loses base tables, while [BASE TABLE] or an unknown\ntype can return excluded rows.\n\nImplement the type stream using the same Doris names as GetTables (BASE\nTABLE, SYSTEM VIEW and VIEW), sorted according to Flight SQL. Match the\ncomplete filter against the existing authorized listing before\ncollecting optional schemas. Request only name/type metadata to avoid\nunused table statistics, and retain the session guard and Arrow buffer\ncleanup on failed sends.\n\nA type is the TABLE_TYPE that information_schema.tables reports for the\nobject: table streams are VIEW, and external catalogs report their\ntables and views as BASE TABLE. The service lists every authorized\nobject and the filter runs on that listing. Requested names match\ncase-insensitively, and TABLE and SYSTEM TABLE, which Flight SQL calls\ncommonly supported, select BASE TABLE and SYSTEM VIEW: the ADBC Flight\nSQL driver asks for `table` (Doris\u0027s own ADBC catalog lists a Doris\nsource with `getObjects(..., {\"table\"})`), and JDBC tools ask for\n`TABLE`. Any other name matches only a type of that name, so UNKNOWN\nselects nothing.\n\nBecause the filter now runs on the full listing,\n`FrontendServiceImpl.listTableStatus` refreshes a cloud follower\u0027s\npartition versions from the meta service only when UPDATE_TIME is\nrequested, the one column that reads them. GetTables, which asks for\nnames and types, never waits on those RPCs or is cut short when they\nfail, and information_schema.tables scans that do not read UPDATE_TIME\nskip the refresh too.\n\nCo-authored-by: morningman \u003cyunyou@selectdb.com\u003e\nCo-authored-by: Gabriel \u003cliwenqiang@selectdb.com\u003e"
    },
    {
      "commit": "173a0e2064e529bfa6f017608c40ff003c2f1066",
      "tree": "859b152a8066917f305ead8ac04c703f15e9e6ac",
      "parents": [
        "74b477d3c1431e10954add7698b6496ca370a59e"
      ],
      "author": {
        "name": "shuke",
        "email": "shuke@selectdb.com",
        "time": "Tue Nov 03 17:45:20 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Tue Nov 03 17:45:20 2026 +0800"
      },
      "message": "[improvement](ci) Upgrade review runtime to Codex 0.166.0 (#69990)\n\n### What problem does this PR solve?\n\nRelated PR: https://github.com/shuke987/codex/pull/6\n\nUpgrade the Code Review Runner and its runtime canary from Codex 0.162.1\nto the validated [0.166.0 fork\nrelease](https://github.com/shuke987/codex/releases/tag/exec-goal-v0.166.0),\nretaining goal completion and same-session resume support.\n\nCodex 0.166.0 distinguishes `policy_capacity_unavailable` from ordinary\noverloads. Accept both canonical terminal capacity messages so the\nexisting bounded recovery policy continues to apply. Preserve the\noriginal diagnostic and exact submission verification; unrelated\nfailures remain non-retryable.\n\nUpdate the archive, version and CLI checksum pins together. The workflow\nchanges are limited to seven package pin replacements. Existing model\nselection and delivery checks are preserved."
    },
    {
      "commit": "74b477d3c1431e10954add7698b6496ca370a59e",
      "tree": "6184ff263215fb609e53ba6a20a3e60facc8635b",
      "parents": [
        "1aa505cb0be7870232288641bf9a5592687d328f"
      ],
      "author": {
        "name": "Gabriel",
        "email": "liwenqiang@selectdb.com",
        "time": "Tue Nov 03 17:39:12 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Tue Nov 03 17:39:12 2026 +0800"
      },
      "message": "[fix](fe) Avoid database object loading for schemata scans (#70010)\n\n### What problem does this PR solve?\n\nIssue Number: N/A\n\nProblem Summary:\n\nQueries such as:\n\n```sql\nSELECT COUNT(*) FROM example_catalog.information_schema.schemata;\n```\n\nonly need database and catalog names. However, `SchemaSchemataScanner`\nuses the default `getDbNames` RPC, whose FE handler calls\n`catalog.getAllDbs()` to also return database IDs. For external\ncatalogs, this resolves every database object even though SCHEMATA never\nreads those IDs. A large namespace list can therefore churn a much\nsmaller database-object cache and allocate substantial temporary\nmetadata while waiting inside a synchronous BE-to-FE RPC. A warm\ndatabase-name cache does not remove this work.\n\nThis PR adds the optional `include_db_ids` request field, defaulting to\n`true`, and lets SCHEMATA set it to `false`. The FE then calls\n`catalog.getDbNames()` and omits the unused database IDs. Both paths\nshare the existing privilege checks, pattern matching, and schema-name\nformatting. Callers that need IDs keep the object-enumeration path\nwithout additional intermediate lists.\n\nThe new field uses ID 8; field 7 remains reserved. Definitions were\nchecked on master and branch-3.1/4.0/4.1/4.2. Older FE nodes ignore the\noptional field and keep serving the existing response; older BE callers\nretain the default ID-inclusive behavior on a new FE.\n\n### Validation\n\n- Added a 100,000-namespace test that rejects any database-object\nenumeration and verifies the result-array lengths and omitted IDs.\n- Covered privilege/pattern filtering, qualified schema names,\nempty/unavailable catalogs, existing ID-returning callers, and binary\ndeserialization of requests without the new field.\n- Confirmed the new behavior tests fail on the original handler before\nimplementing the change. All 7 targeted FE tests pass, including the\nexisting `FrontendServiceImplTest#testGetDBNames`.\n\n```bash\nbash run-fe-ut.sh --run \u0027org.apache.doris.service.FrontendServiceGetDbNamesTest,org.apache.doris.service.FrontendServiceImplTest#testGetDBNames\u0027\n```\n\n- FE Checkstyle: passed with zero violations.\n- Changed BE scanner: clang-format 16 and C++ syntax-only compilation\nwith regenerated Thrift headers passed.\n\nAn in-process FE-handler benchmark uses the actual `ExternalCatalog` /\n`TestExternalCatalog` name and object-cache implementations, a synthetic\nmetadata provider, and stubbed catalog routing and privilege checks. It\ncompares the original handler source at master commit\n`bdb90856e6e468eb495e5bfd67010de90570bfbc` with the patched handler,\nusing the same dependencies and fixture. Each trial starts with a warm\nname cache and cold object cache, whose capacity is 1,000. There are\n100,000 synthetic namespaces plus the two system databases. Settings:\nJDK 17, 2 GiB heap, 4 active processors; discard one warm-up pair and\nreport medians from three measured trials.\n\n| Metric | Original master | Patched SCHEMATA path |\n| --- | ---: | ---: |\n| FE handler elapsed time | 2,984.2 ms | 1,331.1 ms |\n| Cumulative current-thread allocation | 2,117.2 MiB | 726.2 MiB |\n| Database objects constructed | 100,002 | 0 |\n| Name-provider invocations | 1 | 1 |\n| Returned rows | 100,002 | 100,002 |\n\nIn this fixture, handler time falls by 55.4% (2.24x faster) and\ncumulative allocation falls by 65.7%. Allocation is total bytes\nallocated during the call, not peak live heap. This benchmark excludes\nnetwork transport, Thrift serialization, and SQL execution; it does not\nmeasure a deployed PostgreSQL/Iceberg environment. Full BE linking and a\ndeployed end-to-end SQL test were not run locally.\n\n### Release note\n\nImprove `information_schema.schemata` enumeration for large catalogs by\navoiding unnecessary database-object loading.\n\n### Check List (For Author)\n\n- Test\n    - [ ] Regression test\n    - [x] Unit Test\n    - [x] Manual test (FE-handler benchmark described above)\n    - [ ] No need to test or manual test.\n- Behavior changed:\n- [x] No. SQL result semantics and existing ID-returning requests are\npreserved.\n    - [ ] Yes.\n- Does this need documentation?\n    - [x] No.\n    - [ ] Yes.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label"
    },
    {
      "commit": "1aa505cb0be7870232288641bf9a5592687d328f",
      "tree": "715a09001a6192caf6b4c9f6f0b33c043afb65b9",
      "parents": [
        "a35707da74f8df0d6502df95feaf734dd9d20d9d"
      ],
      "author": {
        "name": "Calvin Kirs",
        "email": "guoqiang@selectdb.com",
        "time": "Tue Nov 03 17:04:13 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Tue Nov 03 17:04:13 2026 +0800"
      },
      "message": "[fix](fe) Preserve non-UTF-8 prepared statement parameters (#69805)\n\n### What problem does this PR solve?\n\nRelated PR: #69314\n\nJDBC `setBytes()` and Connector/ODBC binary bindings can send\n`MYSQL_TYPE_VAR_STRING` / `MYSQL_TYPE_STRING` rather than BLOB. FE\ndecoded those bytes with Java\u0027s replacement UTF-8 decoder, silently\nchanging arbitrary binary data. For example, `0080FF4100C328` became\n`00EFBFBDEFBFBD4100EFBFBD28` in `SELECT HEX(?)`. The BLOB handling added\nby #69314 did not cover these wire types.\n\nDecode string parameters strictly and retain non-UTF-8 payloads as\nbinary literals. Bind those payloads through a BE-side cast to the\noriginal string category, preserving string function and comparison\ntyping without replacing bytes. Valid UTF-8 retains the existing text\nliteral types, and BLOB types retain their existing binary behavior. Use\nvalue-based binary literal child hashes so equivalent casts remain equal\nafter rebinding and forwarding.\n\n### Release note\n\nPreserve non-UTF-8 bytes in MySQL server-prepared string parameters,\nincluding JDBC `setBytes()` and Connector/ODBC binary bindings.\n\n### Check List (For Author)\n\n- Test:\n- [x] Regression test: generated and replayed\n`prepared_stmt_p0/binary_parameters`; replayed\n`prepared_stmt_p0/send_long_data`. Covers all byte values, malformed\nUTF-8, parameter reuse, stored HEX/MD5, string expression predicates,\nNULL/empty values and ordinary Unicode text.\n- [x] Unit Test: 17 tests passed across `MysqlParameterLiteralTest` and\n`MysqlConnectProcessorLongDataTest`, including string wire aliases,\ncached types, uploads and forwarding.\n- [x] Manual test: Connector/J 8.2.0 server-prepared `setBytes`,\n`setBinaryStream` and `setBlob` preserve `0080FF4100C328`.\n- [x] Manual test: Linux Connector/ODBC 5.3.14 Unicode, 17 cases passed,\nincluding 1-MiB binary/Unicode inputs. An additional 17-case\nproxy-rewritten VARCHAR wire-type probe also passed.\n    - [x] FE build and Checkstyle passed.\n- Behavior changed:\n- [x] Yes. Non-UTF-8 string-coded parameters preserve the input bytes\ninstead of substituting U+FFFD. Valid UTF-8 text behavior is unchanged.\n- Does this need documentation?\n    - [x] No.\n\nRuntime regression checks used the newly built master FE with an\nexisting 4.1.4 BE and condition cache disabled. The predicate regression\nuses `CONCAT(value, \u0027\u0027) \u003d ?`; direct constant predicate pushdown on a\ncurrent master BE was not validated. No BE, storage-format or\nprotocol-field changes are included."
    },
    {
      "commit": "a35707da74f8df0d6502df95feaf734dd9d20d9d",
      "tree": "ff7345ca00da142ac20b457a9fc139553aea6a61",
      "parents": [
        "c7d1df40c00914ed9dffcdba61aada94b5e5b3cd"
      ],
      "author": {
        "name": "lihangyu",
        "email": "lihangyu@selectdb.com",
        "time": "Tue Nov 03 16:37:26 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Tue Nov 03 16:37:26 2026 +0800"
      },
      "message": "[fix](variant) Write Variant V2 sparse keys in byte order (#70001)"
    },
    {
      "commit": "c7d1df40c00914ed9dffcdba61aada94b5e5b3cd",
      "tree": "2788a223330dbc5df18ce59f1515167b24979397",
      "parents": [
        "c8a75b273e999370e354ed39ae65db2c483c0ad0"
      ],
      "author": {
        "name": "Gabriel",
        "email": "liwenqiang@selectdb.com",
        "time": "Tue Nov 03 16:25:39 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Tue Nov 03 16:25:39 2026 +0800"
      },
      "message": "[fix](external) Preserve source values and alias semantics in ES scans (#69672)\n\n### What problem does this PR solve?\n\nElasticsearch scans can return values that fail SQL predicates when\nindexed values differ from the values Doris reads, or when a scan\nswitches all selected columns from docvalues to source. Parallel alias\nscans can also bypass alias filters/routing, and supported mapping\nshapes can fail during planning.\n\nFor example, projecting a normalized keyword forces a source read for\nits siblings. A numeric alias then reads NULL although its docvalue is\n5; a declared empty ARRAY remains nonnull although ES considers it\nmissing; a date-only string reads local midnight instead of the indexed\nUTC midnight. Boolean empty strings and coerced integer strings must\nalso remain readable: `\"\"` means false and `\"1.2e3\"` means 1200. Keeping\ntheir indexed predicates while pruning SQL residuals returns incorrect\nrows.\n\n#### Execution flow\n\n1. Select a populated mapping independently of alias response order.\nDistinguish typeless flags/root objects from legacy type wrappers,\nincluding a legacy type named `runtime`, and preserve array annotations.\nAccept empty/configuration-only alias backings during safety analysis\nwhile retaining standalone schema diagnostics.\n2. Resolve the selected keyword query path and docvalue read path\nseparately on every backing, including filter-only columns. Fetch index\nmapping defaults with the existing metadata request and honor per-field\noverrides for every alias backing; inherited `ignore_malformed` can\nremove indexed values while retaining source values. Ignore\ntransformations on unused sibling paths and unrelated source filters.\nMissing/read-changing paths disable the affected docvalue entry. Root\n`enabled:false` and disabled source (`enabled:false` or `mode:disabled`)\nrestrict every affected column.\n3. Determine the final scan-wide read mode before building DSL. In\nsource mode, STRING/ARRAY predicates and predicates on rounded numbers,\naliases, scripted fields and default dates retain local residuals\nwithout narrowing the ES query. Eligible pure docvalue projections\npreserve their indexed-value behavior; scalar `_id` metadata remains\npushable.\n4. Keep searchability separate from value equivalence: supported\ndoc-value-only fields remain queryable with `index\u003dfalse`. Require\nexistence for SQL inequalities. Normalize retained equality negation,\ndouble negation and Boolean negation inside the connector. Retained\nranges keep the original ES query complement with an existence guard: an\ninverse existential range changes matching on arrays such as [1,2]. This\npreserves NULL rejection when optimizer NOT simplification is disabled.\n5. Pass residual tracking and the existing early-LIMIT restrictions\nthrough the current scan contract. Native `esquery` remains remote-only:\nnative-only expressions and independent AND conjuncts work; an\nindivisible Boolean expression requiring local native evaluation fails\nexplicitly during planning.\n6. Use both readers\u0027 shared parser for supported source representations.\nBoolean empty strings become false in scalar and ARRAY paths. Scalar\nnumeric/Boolean/date singleton source arrays read their element; empty\nand null-only arrays read NULL, while STRING/JSONB and declared ARRAY\ncontainers remain intact. JSONB singleton, empty and null-element arrays\nretain their original JSON container. Declared ARRAYs preserve nullable\nmembers; numeric empty strings become SQL NULL in scalar and ARRAY\npaths, while Boolean empty strings remain false. Integer coercion\nfollows the ES family: byte/short/integer strings use double rounding\nbefore truncation, while long strings retain exact decimal/exponent\nhandling and 64-bit precision. Bounds are checked before narrowing.\nDeclared integer ARRAY elements share the family-specific conversion.\nNumeric floating tokens are converted directly with checked bounds,\npreserving numeric INT64_MIN and the Java upper-endpoint conversion.\nFloating strings and narrow integer strings accept Java hexadecimal\nexponents, type suffixes and outer ASCII whitespace; FLOAT strings round\ndirectly to float rather than through double. Epoch-millis source dates\naccept zero, short/signed decimal strings and fractional numeric values,\nwith checked bounds and second-precision floor semantics. Four-digit\ndefault-format years retain their calendar interpretation; pre-epoch\nformatted docvalues use the same floor.\n7. For a source read involving disabled-source backings, pass an\noptional `read_full_source` flag through the existing ES property map.\nBE requests `_source:true`: disabled hits have missing values, while\nenabled alias backings retain their source. Ordinary source scans keep\nfield projection and pure docvalue scans keep docvalue requests.\n8. Scan aliases once by their original name so ES applies alias filters\nand search routing without duplicate shared-shard scans. Physical\nindices retain shard-parallel scans.\n\n#### Effects and tradeoffs\n\nThis change preserves source values, supported decoding and SQL\nfiltering across both readers and read modes. Source fallbacks can fetch\nmore candidate rows because unsafe predicates remain local. Mixed\naliases containing disabled-source backings can fetch additional source\nfields to avoid ES rejecting projected-source requests. Alias scans use\none coordinating scroll to preserve alias semantics; physical-index\nparallelism remains. No throughput improvement is claimed. There are no\nnew public SQL/catalog/session properties or SPI/Thrift/Protobuf\ndefinitions; the optional internal flag uses the existing flat property\nmap.\n\n#### Known limitation\n\nMixed-generation FE/BE execution during smooth upgrades is outside this\nchange\u0027s scope. Source plans depending on the updated decoder require\nthe updated BE for correct results; this PR does not add a\nbackend-generation gate.\n\n### Release note\n\nFix Elasticsearch source/index value discrepancies, supported\nBoolean/floating/integer source decoding, default epoch-millis date\ndecoding, disabled-source reads, legacy/configuration-only mapping\nhandling, retained negation NULL semantics and alias filter/routing\nbypass. Preserve eligible doc-value-only queries and reject native-query\ncombinations requiring local evaluation during planning.\n\n### Check List (For Author)\n\n- Test:\n- [x] ES connector reactor package: 163 reported tests, 162 passed, one\noptional ES 9 compatibility class skipped; zero failures/errors. All\nnine reactor modules passed FE Checkstyle.\n- [x] Standard `build.sh --be` ASAN build and all 28\n`EsScrollParserTest` unit tests passed. The combined parser/reader run\nreported 47 tests: 46 passed and one optional environment-dependent live\nsmoke test skipped; the separate live regression run exercised both\nreaders. Clang-format 16, scoped clang-tidy and build hygiene passed for\nall affected C++ files. The local BE ELF requires no GLIBC version above\n2.10.\n- [x] Reproduced the reported alias/script, ARRAY, date-only,\ndisabled-source, boolean, root-disabled, integer and retained-NOT\nfailures on the previous head with live ES 8.11.3. FE/BE unit tests also\ndemonstrated failures before the relevant fixes. Legacy ES6 type\ncompatibility is covered by mapping-response unit tests.\n- [x] Seven live ES 8.11.3 suites passed 1,860 stored checks and 1,764\nindependent SQL comparisons, preserving 88 expected native planning\nerrors. All 1,780 prior stored checks remain unchanged.\n- [x] `test_es_source_semantics`: 612 stored / 516 independent checks;\n`test_es_numeric_source_semantics`: 264 / 264;\n`test_es_mapping_boundaries`: 272 / 272;\n`test_es_source_fallback_types`: 304 / 304;\n`test_es_scalar_materialization`: 160 / 160;\n`test_es_source_value_boundaries`: 168 / 168.\n- [x] New source-boundary coverage reproduces all six review findings:\nJSONB singleton/empty/null-element containers, numeric-double INT64_MIN\nand upper endpoint, nullable declared ARRAYs, numeric empty strings,\nJava numeric grammar, direct FLOAT rounding, and inherited\nmalformed/coercion settings with field overrides and mixed aliases. All\neight reader/docvalue/parallel combinations passed. New BE and FE\ncontrols failed before their corresponding fixes.\n- [x] `test_es_epoch_millis` adds 80 stored checks and 80 independent\ncomparisons across both readers, docvalue/parallel modes and UTC/+08:00.\nIt covers normalized-sibling source fallback, zero/short/fractional\nepoch strings, numeric doubles, negative instants, NULL/empty/singleton\narrays, local predicates and pure-docvalue controls. All outputs were\ngenerated by the repository runner and verified without generation; all\n80 sections also match an independent date/predicate computation. New\nunit coverage checks legacy/V2 date types, declared arrays,\ncalendar-year precedence and invalid numeric/civil-year bounds.\n- [x] New scalar-materialization coverage includes floating/Boolean/date\nsingleton, empty and null-only arrays; narrow integer rounding\nboundaries and NaN, 64-bit precision controls, declared integer ARRAYs\nand retained range NOT on multi-value numeric docvalues. New coverage\nspans both readers, docvalue modes and parallel modes, source fallback\nplus pure-docvalue controls, disabled source indices/mixed aliases,\npopulated disabled roots, non-UTC date-only values and retained NOT with\noptimizer simplification disabled. Session settings are restored. New\noutputs were generated by the repository runner and verified separately\nwithout generation; the final date control avoids dependence on the FE\nJVM timezone.\n- [x] Original FE package built with `build.sh --fe`; connector updates\npackaged through the documented module reactor recipe and deployed with\nthe rebuilt BE for live validation.\n- Behavior changed:\n- [x] Yes: correctness fallbacks, supported mapping/decoding/query\nhandling and native-only planning restrictions above.\n- Does this need documentation?\n    - [x] No new public SQL syntax or configuration."
    },
    {
      "commit": "c8a75b273e999370e354ed39ae65db2c483c0ad0",
      "tree": "5796b7a00221876a6d7f1f031edbb3badd816001",
      "parents": [
        "7acfd9bfc27f01f0b953e8ac0cdc3482a46e3c59"
      ],
      "author": {
        "name": "wudi",
        "email": "wudi@selectdb.com",
        "time": "Tue Nov 03 16:10:48 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Tue Nov 03 16:10:48 2026 +0800"
      },
      "message": "[fix](fe) Require ADMIN for remote CDC driver URLs (#69617)\n\n### What problem does this PR solve?\n\nIssue Number: N/A\n\nProblem Summary:\n\nUsers with permission to create CDC jobs can supply HTTP/HTTPS JDBC\ndriver URLs that load code into the FE JVM. Require global ADMIN\nprivilege when these URLs are supplied during CDC job creation,\nalteration, or `cdc_stream` construction, before source initialization\nor driver loading.\n\nShare the permission check across the FROM-TO and TVF paths. Preserve\nexisting local driver permissions and S3 streaming behavior."
    },
    {
      "commit": "7acfd9bfc27f01f0b953e8ac0cdc3482a46e3c59",
      "tree": "4c45d25f3738a9d23f0720638bb05165a979dcfa",
      "parents": [
        "cb0b56d408a76a59e5b7d248071907537796e47e"
      ],
      "author": {
        "name": "linrrarity",
        "email": "linzhenqi@selectdb.com",
        "time": "Tue Nov 03 15:08:06 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Tue Nov 03 15:08:06 2026 +0800"
      },
      "message": "[improvement](be) Reduce transitive dependencies in shared headers (#69717)\n\nRelated PR: https://github.com/apache/doris/issues/66901\n\n### What problem does this PR solve?\n\nSeveral shared BE headers transitively include protobuf, Thrift,\nthread-pool, and storage-reader implementations. Translation units that\nonly need declarations must still parse these definitions, increasing\ncompilation work and the rebuild scope when those headers change.\n\nRecent CI builds failed during CMake configuration because the\nreverse-reach counts for threadpool.h and rle_encoding.h exceeded the\nbuild-hygiene limits. This PR audits all 32 header-budget entries and\nreduces unnecessary dependencies without changing the budgets or\nexception lists.\n\nThe changes:\n\n- Extract `ColumnIteratorOptions` into a dedicated header so\n`vexpr_context.h` does not require the full `column_reader.h` dependency\nchain.\n- Use forward declarations for `thread-pool`, `protobuf`, `cache`, and\n`WorkloadGroup `types where complete definitions are unnecessary.\n- Move `UniqueId` conversions, `protobuf` methods in numeric `SerDe`,\nand `selected construction/destruction` operations into `.cpp` files.\n- Extract `SyncSizeCallback` into a dedicated header to reduce RPC\ndependencies propagated through the runtime-filter interface.\n- Add direct includes at actual use sites and verify independent\ncompilation.\n\nObject layouts, move semantics, explicit template instantiations, and\nper-value decoding implementations are preserved. Moving some\ndefinitions into .cpp files changes their visibility for inlining in\nordinary separate compilation. Query runtime performance has not been\nmeasured in this PR.\n\n### Dependency reductions\n\nThe following counts come from Ninja dependencies after completed\nbuilds, covering 1,531 production translation units and 52 generated\ntranslation units:\n\n| Header | TUs Before | TUs Now | New Limit |\n|---|---:|---:|---:|\n| util/rle_encoding.h | 846 | 231 | 250 |\n| util/threadpool.h | 1,056 | 585 | 600 |\n| gen_cpp/types.pb.h | 1,862 | 1,225 | 1,250 |\n| gen_cpp/segment_v2.pb.h | 1,231 | 1,156 | 1,200 |\n| gen_cpp/internal_service.pb.h | 777 | 557 | 600 |\n| gen_cpp/FrontendService_types.h | 593 | 293 | 300 |\n| gen_cpp/cloud.pb.h | 325 | 30 | 50 |\n| gen_cpp/BackendService_types.h | 195 | 115 | 150 |\n| gen_cpp/data.pb.h | 228 | 90 | 100 |\n| storage/options.h | 502 | 293 | 350 |\n| io/fs/s3_file_system.h | 65 | 31 | 50 |\n| gen_cpp/Types_types.h | 2,078 | 2,073 | 2,100 |\n| gen_cpp/PaloInternalService_types.h | 1,093 | 1,093 | 1,100 |\n| util/brpc_closure.h | 62 | 35 | 50 |\n| gen_cpp/Status_types.h | 37 | 37 | 41 |\n| runtime/workload_group/workload_group.h | 43 | 38 | 47 |\n\n\nAcross the 16 monitored heavy headers, measured header-to-TU dependency\npairs decrease from 8,108 to 6,617, an 18.39% reduction. These removals\naffect 742 distinct translation units. The number of production\ncompilation units remains unchanged.\n\nThe transitive project-header closure of dependency.h decreases from 325\nto 246 headers, a 24.31% reduction. This uses the hygiene script’s\nstatic include graph and is reported separately from the measured Ninja\ndependency counts.\n\n### Query runtime performance\n\n### TPC-H SF1000\n\n| Date / Version | Hot-min (s) | PR vs. Master |\n|---|---:|---:|\n| master 2026-10-08 | 59.615 | -9.93% |\n| master 2026-10-13 | 60.776 | -11.65% |\n| master 2026-10-15 | 57.021 | -5.83% |\n| master 2026-10-20 | 55.877 | -3.90% |\n| master 2026-10-27 | 65.987 | -18.62% |\n| PR #69717 | 53.697 | — |\n\n### SSB SF1000\n\n| Date / Version | Hot-min (s) | PR vs. Master |\n|---|---:|---:|\n| master 2026-10-08 | 11.418 | +3.71% |\n| master 2026-10-13 | 11.749 | +0.79% |\n| master 2026-10-15 | 10.653 | +11.16% |\n| master 2026-10-20 | 10.773 | +9.92% |\n| master 2026-10-27 | 11.471 | +3.23% |\n| PR #69717 | 11.842 | — |\n\n### ClickBench\n\nConfiguration: cluster_type\u003dsingle, ecs.c9i.4xlarge_16c32g, one 500 GB\ndisk, full analyze, PR #69717 release package.\n\n| Date / Version | Hot-min (s) | PR vs. Master |\n|---|---:|---:|\n| master 2026-10-08 | 24.717 | -1.83% |\n| master 2026-10-13 | 26.406 | -8.11% |\n| master 2026-10-15 | 25.079 | -3.25% |\n| master 2026-10-20 | 24.629 | -1.48% |\n| master 2026-10-27 | 23.643 | +2.63% |\n| PR #69717 single rerun | 24.265 | — |\n\n### TPC-DS SF1000 (1 TB)\n\nComparisons use the PR result from 2026-10-29. The 2026-10-28 rerun is\nlisted separately.\n\n| Date / Version | Hot-min (s) | PR vs. Master |\n|---|---:|---:|\n| master 2026-10-08 | 168.926 | -7.40% |\n| master 2026-10-13 | 170.676 | -8.35% |\n| master 2026-10-15 | 163.170 | -4.13% |\n| master 2026-10-20 | 163.926 | -4.58% |\n| master 2026-10-27 | 195.853 | -20.13% |\n| PR #69717 2026-10-29 main result | 156.423 | — |\n\n\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test: Three complete Release builds; standard ASAN BE/FE build and\nfinal incremental build; all BE UT sources compiled and linked; 29\nselected unit tests passed with -j50 --coverage; 1,024-row service read,\naggregate, shuffle join, sort and cast checks; header hygiene and\nchanged-line format checks passed. Clang-tidy covered all 71 files and\nretained two existing warnings in relocated protobuf methods (function\nsize and signedness).\n- Behavior changed: No\n- Does this need documentation: No"
    },
    {
      "commit": "cb0b56d408a76a59e5b7d248071907537796e47e",
      "tree": "bf1ea8490578d957635083a37c51aae9b3490493",
      "parents": [
        "61906ce9a1c89d76bb450f1d6b4a10acbb712efe"
      ],
      "author": {
        "name": "Jamie",
        "email": "lianyukang@selectdb.com",
        "time": "Tue Nov 03 14:41:49 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Tue Nov 03 14:41:49 2026 +0800"
      },
      "message": "[fix](cloud) Fix deadlock between tablet encryption check and meta lock writers (#69950)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #64574\n\nProblem Summary:\n\n`is_tablet_encrypted()`, which serves `/api/check_tablet_encryption`,\nholds the tablet `_meta_lock` read lock in `traverse_rowsets()` and\ntakes it again through `tablet_schema()` in the visitor. Since #64574\n`_meta_lock` is the writer-preferring `BthreadSharedMutex`, so a writer\nthat arrives between the two reads deadlocks with the check:\n\n- The check holds the first read lock and waits for the second, which is\nblocked by the pending writer.\n- The writer waits for the first read lock to be released.\n\nThe window is wide because each rowset\u0027s files are read from remote\nstorage under the lock, and compaction and rowset sync take the write\nlock often. Once this happens the tablet stays stuck until the BE\nrestarts:\n\n- The HTTP request never returns.\n- The compaction keeps renewing its lease but never finishes, so other\nBEs get `JOB_TABLET_BUSY` for this tablet.\n- Every later reader of this tablet\u0027s meta blocks behind the pending\nwriter, including the periodic tablet sync.\n\nWe hit this in a regression test: the compaction stopped right after its\nrowset sync, and the check timed out on all 10 retries over about 2h47m.\n\nThis PR reads the schema once before the check and checks a snapshot of\nthe rowsets (`get_snapshot_rowset()`) with no lock held, which also\nstops holding the meta lock across remote file reads. It adds a\ntest-only sync point so a unit test can observe the lock at that moment.\n\n### Release note\n\nFix a deadlock between the tablet encryption check API and tablet meta\nwriters such as compaction, which could block a tablet until the BE\nrestarts.\n\n### Check List (For Author)\n\n- Test \u003c!-- At least one of them must be included. --\u003e\n    - [ ] Regression test\n    - [x] Unit Test\n    - [x] Manual test (add detailed scripts or steps below)\n    - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n        - [ ] Previous test can cover this change.\n        - [ ] No code files have been changed.\n        - [ ] Other reason \u003c!-- Add your reason?  --\u003e\n\nUnit test\n`CheckEncryptionActionTest.RowsetsAreCheckedWithoutHoldingMetaLock`\nchecks that `_meta_lock` is free while a rowset is being checked. It\nfails on the code before this PR and passes with it.\n\nManual test: a temporary unit test (not committed) started a writer from\ninside the rowset check, then waited for the check to return. On the\ncode before this PR the check had not returned after 20s. With this PR\nit returned at once.\n\n- Behavior changed:\n    - [x] No.\n    - [ ] Yes. \u003c!-- Explain the behavior change --\u003e\n\n- Does this need documentation?\n    - [x] No.\n- [ ] Yes. \u003c!-- Add document PR link here. eg:\nhttps://github.com/apache/doris-website/pull/1214"
    },
    {
      "commit": "61906ce9a1c89d76bb450f1d6b4a10acbb712efe",
      "tree": "4764282c0c73d4fec4f19757415754faea1b96e1",
      "parents": [
        "deb079478d0114d79b04166b22861918fcdd7a51"
      ],
      "author": {
        "name": "daidai",
        "email": "changyuwei@selectdb.com",
        "time": "Tue Nov 03 14:39:46 2026 +0800"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "time": "Tue Nov 03 14:39:46 2026 +0800"
      },
      "message": "[fix](be) Preserve signed decimal bounds in Iceberg ORC writes (#69888)\n\n### What problem does this PR solve?\n\nIssue Number: N/A\n\nRelated PR: N/A\n\nProblem Summary:\n\nORC `Int128` statistics already contain two\u0027s-complement bits, but the\nIceberg ORC writer negates negative values again before encoding them. A\nfile containing `[-100, -1]` consequently reports bounds `[100, 1]`, so\na negative predicate can prune a file that contains matching rows.\n\nORC footer statistics also trim trailing fractional zeros. For example,\n`-1.00` is returned as `value\u003d-1, scale\u003d0`; a `DECIMAL(...,2)` manifest\nneeds the unscaled value `-100`. Preserving the sign alone therefore\nleaves incorrect bounds.\n\nRestore each bound to the declared column scale with the existing ORC\nscaling utility, then serialize its original signed bits using minimal\nbig-endian encoding. Preserve necessary sign prefixes such as `128 -\u003e 00\n80` and `-129 -\u003e FF 7F`.\n\nThe fix applies to newly collected bounds. Existing incorrect metrics\nrequire recomputation or data-file rewriting.\n\n### Release note\n\nCorrect decimal bounds emitted by Iceberg ORC writes to prevent\nincorrect predicate pruning for negative values and values with trailing\nfractional zeros.\n\n### Check List (For Author)\n\n- Test \u003c!-- At least one of them must be included. --\u003e\n    - [x] Regression test\n    - [x] Unit Test\n    - [x] Manual test (add detailed scripts or steps below)\n    - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n        - [ ] Previous test can cover this change.\n        - [ ] No code files have been changed.\n        - [ ] Other reason\n\nAdded BE tests cover 29 encoding cases, real ORC statistics at\nprecisions 1/9/18/38, scale restoration through scale 38, nullable\ncolumns, and all-null columns. The external regression checks manifest\npruning for negative and mixed decimal data across decimal32/64/128.\n\nLocal validation:\n- An isolated codec harness built from the production function passed\nall 29 encoding cases.\n- Five real ORC files passed independent Iceberg 1.12.0 manifest\nwrite/read and predicate-pruning probes.\n- clang-format 16, build hygiene, changed-line clang-tidy, and Groovy\n4.0.19 syntax checks passed. Static analysis used real ASAN UT\ncompilation commands with PCH expanded as headers.\n- The full local BE-UT build was stopped during the monolithic\ndependency build before tests executed.\n- Regenerated `test_iceberg_write_stats2.out` with the standard\nregression runner after restoring decimal scales. Only `sql_4/5/7/9`\nchanged, and the decimal bounds match both stored rows and the Parquet\nmetrics.\n- Re-ran `test_iceberg_write_stats2` and\n`test_iceberg_orc_decimal_bounds` locally using this PR\u0027s CI FE/BE\nartifacts: 2 suites passed, 0 failed, 0 skipped.\n\n- Behavior changed:\n    - [ ] No.\n- [x] Yes. Newly written ORC decimal bounds preserve the sign and the\ncolumn scale.\n\n- Does this need documentation?\n    - [x] No.\n    - [ ] Yes.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label"
    }
  ],
  "next": "deb079478d0114d79b04166b22861918fcdd7a51"
}