)]}' { "log": [ { "commit": "8d14eba8b41f9ddad018903861a5d0d3c4a8a668", "tree": "f7e9680a24731ef24830ad3cdf20d9cc819bac3c", "parents": [ "500a64aa99756dab57e31304c0ac02e8a5c527ef" ], "author": { "name": "Chenyang Sun", "email": "sunchenyang@selectdb.com", "time": "Thu Sep 10 15:41:00 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 15:41:00 2026 +0800" }, "message": "[fix](index) Reject \u0027default\u0027 for partition.inverted_index_storage_format (#67583)\n\npartition inverted index storage format only supports \"V2, V3 and SNII\"\n\nIssue Number: close #xxx\n\nRelated PR: #xxx\n\nProblem Summary:\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test \u003c!-- At least one of them must be included. --\u003e\n - [ ] Regression test\n - [ ] Unit Test\n - [ ] Manual test (add detailed scripts or steps below)\n - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n - [ ] Previous test can cover this change.\n - [ ] No code files have been changed.\n - [ ] Other reason \u003c!-- Add your reason? --\u003e\n\n- Behavior changed:\n - [ ] No.\n - [ ] Yes. \u003c!-- Explain the behavior change --\u003e\n\n- Does this need documentation?\n - [ ] No.\n- [ ] Yes. \u003c!-- Add document PR link here. eg:\nhttps://github.com/apache/doris-website/pull/1214 --\u003e\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label \u003c!-- Add branch pick label that this PR\nshould merge into --\u003e\n\nCo-authored-by: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "500a64aa99756dab57e31304c0ac02e8a5c527ef", "tree": "8170e6850cc87cead21b0b6f88f123045f1c2273", "parents": [ "0ded66a27bf9112a006999420380e6076af5c1f2" ], "author": { "name": "Mingyu Chen (Rayner)", "email": "morningman.cmy@gmail.com", "time": "Thu Sep 10 15:39:55 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 15:39:55 2026 +0800" }, "message": "[fix](arrow-flight) Report the real client address of an Arrow Flight SQL session (#67576)\n\n### What problem does this PR solve?\n\nProblem Summary:\n\n`FlightSqlChannel` had `getRemoteIp()` and `getRemoteHostPortString()`\nstubbed out as `0.0.0.0` and\n`0.0.0.0:0`, and `FlightSqlConnectContext` routed every client-address\naccessor through them. So an\nArrow Flight SQL session reported `0.0.0.0:0` everywhere an operator\nlooks for one:\n\n- the `Host` column of `SHOW PROCESSLIST` and of\n`information_schema.processlist`;\n- the audit log\u0027s `client_ip` (`AuditLogHelper` reads\n`ctx.getClientIP()`);\n- the `kill query from ...` and connection-timeout warnings in `fe.log`.\n\nWith every Flight session showing the same placeholder, there was no way\nto tell where one came\nfrom — which client to talk to, which one to `KILL`.\n\nThe address is already resolved and already on the session:\n`FlightRemoteIpServerStreamTracer`\ncaptures it from the gRPC transport when the bearer token is issued, and\n`FlightSessionsManager.buildConnectContext()` stores it via\n`setRemoteIP()`. This PR reports that,\nfalling back to the tracer\u0027s `0.0.0.0` sentinel when the address could\nnot be resolved. The two\nplaceholder methods on `FlightSqlChannel` have no callers left and are\nremoved.\n\nOnly the address is reported, not the `host:port` pair MySQL reports: a\nFlight session has no stable\npeer port, because each gRPC call of a session may arrive on its own\nconnection. A port that changes\nunder the operator is worse than no port." }, { "commit": "0ded66a27bf9112a006999420380e6076af5c1f2", "tree": "9d572bfec2f28b6ab0d453dc9c71bfbe39bcc927", "parents": [ "8dd6aafade836c862730d66449c5e3a70c6f8f69" ], "author": { "name": "yujun", "email": "yujun@selectdb.com", "time": "Thu Sep 10 14:52:20 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 14:52:20 2026 +0800" }, "message": "[feature](ivm) Support incremental refresh for array_agg and collect_list aggregates (#67575)\n\nMVs containing `ARRAY_AGG` or `COLLECT_LIST` can now be refreshed\nincrementally (IVM): the visible array is the full aggregate state, so\nno hidden state column is needed. The delta aggregate emits insert-side\nand delete-side arrays and apply merges them as a multiset\n`except_all(concat(coalesce(old, []), ins), del)`, which keeps duplicate\nelements and NULLs correct.\n\n`COLLECT_LIST` skips NULL rows, so its polarity columns use the\nconditional-argument idiom. `ARRAY_AGG` keeps NULL elements, so\nNULL-filtering cannot build its polarity arrays; instead every change\nrow is packed into a single `array_agg(struct(dml_factor, elem))`\naggregate output and the delta top project derives the insert/delete\npolarity columns above the aggregate with `array_filter`/`array_map`,\nkeeping the same two transient polarity slots for the multiset apply.\n\n`ARRAY_AGG` over JSONB/VARIANT elements (struct/map/array constructors\nall reject those types) is rejected during analysis with a precise\nreason, surfaced to the user at `CREATE MATERIALIZED VIEW` time.\nDISTINCT and the LIMIT variant of `collect_list` fall back to complete\nrefresh.\n\n## Key changes\n- add ARRAY_AGG and COLLECT_LIST aggregate kinds sharing one\n`IvmAggArrayProcessor` (two transient polarity delta slots, multiset\napply, no hidden state); ARRAY_AGG derives its polarity columns above\nthe delta aggregate through a new per-processor top-project hook\n- centralize polarity condition expressions and the typed empty array\nliteral in `IvmAggExpressionBuilder`\n- regression suite `test_ivm_agg_array_1` covering grouped, scalar and\nmixed (array_agg + count + sum) MVs over insert/update/delete windows\nwith NULL and non-NULL values\n\nTracked in https://github.com/apache/doris/issues/65418" }, { "commit": "8dd6aafade836c862730d66449c5e3a70c6f8f69", "tree": "1d91e41bb291902e4c0d195aec149bcc5f9f2c85", "parents": [ "8565db222d694a52e1acd2ce6d7d7829a0d08aff" ], "author": { "name": "linrrarity", "email": "linzhenqi@selectdb.com", "time": "Thu Sep 10 14:39:15 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 14:39:15 2026 +0800" }, "message": "[Fix](ai_func) Parse final text from Responses API output (#67671)\n\n### What problem does this PR solve?\n\nIssue Number: close #xxx\n\nRelated PR: #xxx\n\nProblem Summary:\n\nThe Responses API returns heterogeneous items in the `output` array.\nBesides the final `message`, it may contain `reasoning` and tool-related\nitems.\n\nThe previous OpenAI adapter treated every output item as response text.\nThis caused two problems:\n\n1. An OpenAI-type reasoning item with an empty `content` array was\nreported as an invalid response.\n2. An OpenAI-type `reasoning_text` item was incorrectly included in\nbatch results, causing errors such as `expected 1 items but got 2`.\n\nThis change updates the Responses API parser to:\n\n- Ignore non-`message` output items, including reasoning and tool items.\n- Only extract `output_text` parts from message content.\n- Preserve validation for malformed message and output-text structures.\n- Update the response format comments to match the current API\nstructure.\n\nfor example:\n```sql\nCREATE RESOURCE \u0027deepseek-responses\u0027\nPROPERTIES (\n \u0027type\u0027\u003d\u0027ai\u0027,\n \u0027ai.provider_type\u0027\u003d\u0027deepseek\u0027,\n \u0027ai.endpoint\u0027\u003d\u0027https://api.deepseek.com/responses\u0027,\n \u0027ai.model_name\u0027 \u003d \u0027deepseek-v4-flash\u0027,\n \u0027ai.api_key\u0027 \u003d \u0027sk-xxx\u0027\n);\n\n\nSELECT id, ai_TRANSLATE(\u0027deepseek-responses\u0027, str_val, tar_language) AS Result\nFROM ai_test WHERE str_val IS NOT NULL;\n```\n\nbefore:\n```text\nDoris\u003e SELECT id, ai_TRANSLATE(\u0027deepseek-responses\u0027, str_val, tar_language) AS Result\n -\u003e FROM ai_test WHERE str_val IS NOT NULL;\nERROR 1105 (HY000): Exception, msg: (127.0.0.1)[RUNTIME_ERROR]Failed to parse ai_translate batch result, expected 1 items but got 2\n```\n\n\nnow\n```text\n+------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| id | Result |\n+------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| 4 | Satz 4, Apache Doris ist ein MPP-basiertes Echtzeit-Data-Warehouse, das für seine hohe Abfragegeschwindigkeit bekannt ist. |\n| 3 | phrase 3, Apache Doris est un entrepôt de données en temps réel basé sur MPP, connu pour sa vitesse de requête élevée. |\n| 6 | предложение 6, Apache Doris — это хранилище данных реального времени на основе MPP, известное высокой скоростью выполнения запросов. |\n| 2 | 句子2：Apache Doris 是一个基于 MPP 的实时数据仓库，以高查询速度著称。 |\n| 5 | 文5、Apache DorisはMPPベースのリアルタイムデータウェアハウスであり、高速なクエリ処理で知られています。 |\n| 8 | Frase 8, Apache Doris é um data warehouse em tempo real baseado em MPP, conhecido por sua alta velocidade de consulta. |\n| 9 | 문장 9, Apache Doris는 높은 쿼리 속도로 알려진 MPP 기반 실시간 데이터 웨어하우스입니다. |\n| 1 | sentence 1, Apache Doris is an MPP-based real-time data warehouse known for its high query speed. |\n| 7 | oración 7, Apache Doris es un almacén de datos en tiempo real basado en MPP conocido por su alta velocidad de consulta. |\n| 10 | الجملة 10، Apache Doris هو مستودع بيانات في الوقت الفعلي يعتمد على MPP ويُعرف بسرعته العالية في معالجة الاستعلامات. |\n+------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n```\n\n### Release note\n\nFix parsing of OpenAI-compatible Responses API results containing\nreasoning output." }, { "commit": "8565db222d694a52e1acd2ce6d7d7829a0d08aff", "tree": "e2868c2ff0cc7953cc18ec96d8eb3e551c91d430", "parents": [ "ee4a91720f66ec3e2f6023a990e4ebcb9d66fd77" ], "author": { "name": "yujun", "email": "yujun@selectdb.com", "time": "Thu Sep 10 14:18:25 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 14:18:25 2026 +0800" }, "message": "[fix](ivm) Repair the removed rewrite-context constructor call in IvmNormalizeMTMVJoinTest (#67775)\n\n#67646 replaced `IvmRewriteContext`\u0027s public three-argument constructor\nwith factory methods. #67669, merged a few minutes later, added a test\nthat still called the removed constructor, so the fe-core test sources\nno longer compile on master:\n\n```\nfe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmNormalizeMTMVJoinTest.java:1053: error:\nconstructor IvmRewriteContext cannot be applied to given types;\n required: (Mode, MTMV, String, boolean, ExecutionKind, Optional\u003cIvmDryRunLimit\u003e, Map\u003c...\u003e, Optional\u003cStreamReadMode\u003e)\n found: (Mode, MTMV, boolean)\n```\n\nThe two changes do not conflict textually, so each PR was green on its\nown and the breakage only showed up once both had landed.\n\n### What is changed?\n\nConstruct the context in\n`testFullKeysSinkMaterializesSameNamedUnprojectedKey` through the\nfactory the CREATE MATERIALIZED VIEW analyze flow already uses,\n`IvmRewriteContext.normalize(mtmv)` (see `MTMVPlanUtil`), instead of the\nremoved constructor. Test-only change; no production code is touched.\n\n`normalize(mtmv)` is needed here rather than `create(mtmvName)`: the\ntest enables full keys through `mtmv.getIvmInfo().setUseFullKeys(true)`,\nwhich `IvmNormalizeMTMV.resolveUseFullKeys()` only reads on its fallback\npath — when the context carries a null `useFullKeys` but a non-null\nMTMV. `create(...)` leaves the MTMV null, which would silently resolve\nthat setting to `false`, so the test would compile but fail.\n`Mode.NORMALIZE` also avoids the `Mode.INCREMENTAL` plan-signature\nvalidation, and nothing in this test asserts on the mode.\n\n### Test\n\n- `IvmNormalizeMTMVJoinTest` — 44 tests, 0 failures, 0 errors\n- `mvn test-compile -pl fe-core -am` — BUILD SUCCESS, no errors; all\nfe-core test sources compile again" }, { "commit": "ee4a91720f66ec3e2f6023a990e4ebcb9d66fd77", "tree": "67a2e4ceb913ad65dffc22acffa7bdc2005f924d", "parents": [ "da298089d0755677f70b6fec12ccefbdc632fe14" ], "author": { "name": "linrrarity", "email": "linzhenqi@selectdb.com", "time": "Thu Sep 10 12:23:16 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 12:23:16 2026 +0800" }, "message": "[Enhance](resource) Restrict modification of root-created AI resources (#67659)\n\n\n- Persist whether an `AI resource` was created by root.\n- Only `root` can alter or drop root-created AI resources.\n- `ADMIN` users can still alter or drop AI resources created by `ADMIN`." }, { "commit": "da298089d0755677f70b6fec12ccefbdc632fe14", "tree": "5d335d7c5fcf49ca34a7c063654aa05e1ab4498b", "parents": [ "093f6b1b27af28bfa5e827060e374fbb73b12292" ], "author": { "name": "morrySnow", "email": "zhangwenxin@selectdb.com", "time": "Thu Sep 10 12:18:59 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 12:18:59 2026 +0800" }, "message": "[fix](aggregate) Ignore lambda-local slots in aggregate validation (#67742)\n\n## Problem\n\nValid HAVING predicates were rejected when a lambda function consumed an\naggregate result. Both map and array forms failed because the analyzer\ntreated\nthe lambda\u0027s local parameters as ungrouped table columns.\n\n## Root cause\n\nLambda parameters are represented by `ArrayItemSlot`, which extends\n`SlotReference`. `FillUpMissingSlots.Resolver` recursively visits HAVING\nexpressions and applied normal GROUP BY validation to every\n`SlotReference`,\nincluding these lambda-local slots. They are bound by their\n`ArrayItemReference` and are not inputs from the aggregate child; the\ngeneral\nexpression input-slot collector already excludes them for the same\nreason.\n\n## Reproduction\n\n```sql\nSELECT id, COUNT(*) AS n\nFROM (SELECT 1 id UNION ALL SELECT 1 id) t\nGROUP BY id\nHAVING map_exists((k, v) -\u003e v \u003e 1, map(1, COUNT(*)));\n\nSELECT id, COUNT(*) AS n\nFROM (SELECT 1 id UNION ALL SELECT 1 id) t\nGROUP BY id\nHAVING array_match_any(array_map(x -\u003e x \u003e 1, array(COUNT(*))));\n```\n\nThe map query reported an internal map-entry parameter as ungrouped, and\nthe\narray query reported `x` as ungrouped. Both should return `(1, 2)`.\n\n## Fix\n\nSkip `ArrayItemSlot` at the missing-slot resolver entry point. The\nlambda\nbinder owns these local slots, so no aggregate output or GROUP BY\nvalidation\nis needed for them. Ordinary `SlotReference` handling is unchanged, and\na\nreal ungrouped input column inside the surrounding expression is still\nrejected.\n\n## Tests\n\n- Added analyzer coverage for both map and array lambda parameters in\nHAVING.\n- Added a negative analyzer case proving an ordinary ungrouped input\nremains\n rejected.\n- Added execution-level regression coverage for both valid queries and\nthe\n invalid-column boundary.\n- Focused FE tests passed: 13 tests, 0 failures.\n- Regression suite passed: 1 suite, 0 failed suites.\n- Sandbox verification returned `(1, 2)` for both valid queries and\npreserved\n the expected GROUP BY error for `ungrouped_col`." }, { "commit": "093f6b1b27af28bfa5e827060e374fbb73b12292", "tree": "313e7c72ddbafc6e82af5460419c69f97c9ba79b", "parents": [ "5a84d88a9c7b9bb359019494fa58db836733e4c9" ], "author": { "name": "morrySnow", "email": "zhangwenxin@selectdb.com", "time": "Thu Sep 10 12:18:45 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 12:18:45 2026 +0800" }, "message": "[fix](aggregate) Preserve AVG accumulator width in distinct rewrite (#67740)\n\n## Problem\n\nWhen multiple DISTINCT aggregates trigger the AVG decomposition rewrite,\n`AVG(DISTINCT BIGINT)` can return an incorrect value. For example,\naveraging\n`9223372036854775807` and `9223372036854775806` produced `-1.5`; a\npredicate\nsuch as `avg_value \u003e 0` could therefore discard a row that should match.\n\n## Root cause\n\nThe rewrite decomposed `AVG(DISTINCT BIGINT)` into\n`SUM(DISTINCT BIGINT) / COUNT(DISTINCT BIGINT)`. Native AVG uses a\nLARGEINT\naccumulator for BIGINT input, while SUM keeps a BIGINT accumulator. The\nSUM\noverflowed before the division result was converted to AVG\u0027s return\ntype.\n\n## Reproduction\n\n```sql\nSELECT AVG(DISTINCT x)\nFROM (\n SELECT CAST(9223372036854775807 AS BIGINT) AS x\n UNION ALL\n SELECT CAST(9223372036854775806 AS BIGINT) AS x\n) t;\n```\n\nWith the multi-distinct rewrite enabled, the result was `-1.5` instead\nof\napproximately `9.223372036854776e18`.\n\n## Fix\n\nLosslessly widen a BIGINT AVG argument to LARGEINT before constructing\nthe\nreplacement SUM and COUNT. The same widened expression is reused by both\naggregates, preserving the shared DISTINCT argument required by the\nmulti-distinct rewrite while matching AVG\u0027s original accumulator width.\n\n## Tests\n\n- Added a focused rewrite unit test that verifies the SUM uses LARGEINT\nand\n the generated COUNT shares the same widened argument.\n- Added a regression case using the two BIGINT boundary values above\ntogether\n with another DISTINCT aggregate and an outer positive-value filter.\n- The focused FE unit test passed: 1 test, 0 failures.\n- The regression suite passed: 1 suite, 0 failed suites.\n- Sandbox verification changed the result from `-1.5`/0 matching rows to\n `9.223372036854776e18`/1 matching row." }, { "commit": "5a84d88a9c7b9bb359019494fa58db836733e4c9", "tree": "753a50ba4e1d3e7c8549530946a40456f21b8a16", "parents": [ "fd4cc9cc144f30574e80089fc549c3e2becb6b2c" ], "author": { "name": "morrySnow", "email": "zhangwenxin@selectdb.com", "time": "Thu Sep 10 12:14:07 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 12:14:07 2026 +0800" }, "message": "[fix](aggregate) Normalize projected count slots before null safety checks (#67732)\n\n## Problem\n\nCounting a projected alias over a nullable indexed column can return an\nincorrect non-zero result when the filter retains only null rows. Mixing\nthe alias count with `COUNT(*)` or another count exposes the problem:\n\n```sql\nSELECT COUNT(x), COUNT(*)\nFROM (SELECT k AS x FROM t WHERE k IS NULL) q;\n```\n\nFor two matching null rows, the correct result is `(0, 2)`, but the\nstorage-layer index-count path can return `(2, 2)`.\n\n## Root cause\n\nThe FE implementation rule validates `IS NULL` and OR predicates before\npushing count aggregation to the storage layer. In the Project variant,\nthis validation used the aggregate-side alias slot, while the filter\nbelow the Project refers to the source slot. Their expression IDs\ndiffer, so the null-safety guard did not recognize that the filter and\n`COUNT` referenced the same nullable value.\n\nThe rule normalized the aggregate argument to the source slot only\nlater, after the safety decision had already been made.\n\n## Reproduction\n\n```sql\nCREATE TABLE t (\n id INT NOT NULL,\n k INT NULL,\n INDEX idx_k (k) USING INVERTED\n)\nDUPLICATE KEY(id)\nDISTRIBUTED BY HASH(id) BUCKETS 1\nPROPERTIES (\"replication_num\" \u003d \"1\");\n\nINSERT INTO t VALUES (1, NULL), (2, NULL), (3, 1);\n\nSELECT COUNT(x), COUNT(*)\nFROM (SELECT k AS x FROM t WHERE k IS NULL) q;\n\nSELECT COUNT(x), COUNT(id)\nFROM (SELECT k AS x, id FROM t WHERE k IS NULL) q;\n```\n\nBefore this change, both queries return `(2, 2)` and the plan contains\n`pushAggOp\u003dCOUNT_ON_INDEX`. Both queries should return `(0, 2)`.\n\n## Fix\n\nNormalize aggregate arguments through the Project before collecting the\nslots used by the predicate safety checks. The count slots and filter\nslots are now compared in the same source expression-ID domain. If `IS\nNULL` targets a counted source slot, the FE rejects the index-count\npushdown and preserves the column\u0027s null values.\n\nThis change is limited to the FE planner.\n\n## Tests\n\n- Added a FE plan test for `COUNT(projected_alias) + COUNT(*)` above an\n`IS NULL` filter, verifying that the count-on-index implementation rule\nis rejected.\n- Ran `PhysicalStorageLayerAggregateTest`: 7 tests passed.\n- Deployed the FE to a local sandbox and reran both SQL reproductions.\nThey return `(0, 2)`, and the scan plan reports `pushAggOp\u003dNONE`." }, { "commit": "fd4cc9cc144f30574e80089fc549c3e2becb6b2c", "tree": "449f2185caf2ae86b2bff04ef5691e1e84fa4f35", "parents": [ "906b706c579ebbdfebbeee08badb4e70a8378ea5" ], "author": { "name": "morrySnow", "email": "zhangwenxin@selectdb.com", "time": "Thu Sep 10 12:13:34 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 12:13:34 2026 +0800" }, "message": "[fix](fe) Quote unsafe view comments in exported DDL (#67708)\n\n## Problem\n\n`SHOW CREATE VIEW` inserted view comments directly between single\nquotes. A comment containing an apostrophe produced invalid,\nnon-replayable DDL.\n\n## Root cause\n\nBoth view DDL construction paths appended the stored comment text\nwithout applying SQL string-literal escaping. For example, the comment\n`O\u0027Reilly` was emitted as `COMMENT \u0027O\u0027Reilly\u0027`, where the apostrophe\nprematurely ended the literal.\n\n## How to reproduce\n\n```sql\nCREATE VIEW v COMMENT \"O\u0027Reilly\" AS SELECT 1 AS c;\nSHOW CREATE VIEW v;\n```\n\nBefore this change, replaying the returned `CREATE VIEW` statement\nfailed because its comment literal was malformed.\n\n## Fix\n\nCentralize view-comment rendering in a shared helper used by both DDL\ngeneration paths. Comments containing apostrophes or backslashes are\npassed to `SqlUtils.quoteStringLiteral` with the active\n`NO_BACKSLASH_ESCAPES` mode, so the resulting literal is valid for the\ncurrent SQL mode. Comments that require no escaping keep the existing\nsingle-quoted output format.\n\nThe unit test exports a view whose comment contains an apostrophe,\nreplays the exported DDL under a new view name, and verifies that the\ncomment is preserved exactly.\n\n## Tests\n\n- `./run-fe-ut.sh --run CreateViewTest`: 10 tests passed, 0 failures\n- `./build.sh --fe`: all 73 FE modules built successfully\n- Manual sandbox validation confirmed that `SHOW CREATE VIEW` emits\n`COMMENT \"O\u0027Reilly\"` and the exported statement is replayable" }, { "commit": "906b706c579ebbdfebbeee08badb4e70a8378ea5", "tree": "cea966ad812e08b35f7007f4a912d3bc5f25781b", "parents": [ "a6a7e52a7336fcca8dd1564cbeb0f3d726304cc4" ], "author": { "name": "Mingyu Chen (Rayner)", "email": "morningman.cmy@gmail.com", "time": "Thu Sep 10 12:08:13 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 12:08:13 2026 +0800" }, "message": "[feat](authorization) introduce an authorization plugin SPI and move the Ranger sources out of fe-core (#66770)\n\n### User-visible changes at a glance\n\nEvery row below is expanded, with the reasoning behind it, in the\n**Release note** section. This\nsection is the short form: what an operator has to do at upgrade, and\nwhat behaves differently\nafterwards.\n\n#### 1. Deployment changes — action required\n\n| # | Change | What a deployment must do | If it does not |\n|---|---|---|---|\n| **D1** | The Ranger sources (`ranger-doris`, `ranger-hive`) are no\nlonger part of `fe-core.jar`. The release installs them under\n`plugins/authorization/`. | Carry `plugins/authorization/` over to the\nupgraded deployment. | `fe.conf` with `access_controller_type \u003d\nranger-doris`: **the FE does not start** — the instance-wide source is\nbuilt while the FE is starting; the startup error names the directory to\ncopy. A catalog naming a source in `access_controller.class`: the first\nstatement reaching that catalog fails. |\n| **D2** | Each Ranger plugin now initialises **its own** audit stack\nfrom its own plugin directory. Previously whichever Ranger source\ninitialised first decided the audit configuration for both. | A\ndeployment running `access_controller_type \u003d ranger-doris` alongside a\ncatalog bound to `ranger-hive` must now also supply\n`ranger-hive-audit.xml` in the `ranger-hive` plugin directory. | That\nsource audits to a no-op provider, and only a Ranger log line says so. |\n| **D3** | Hive Ranger data policies that were hand-edited to upper-case\naccess types (a workaround for **B1**). | Switch them back to the\nspelling the Ranger UI produces. | Those policies stop matching. |\n| **D4** | Hive Ranger **row filters** and `CUSTOM` masks are free SQL\ntext and are *not* translated: `\\|\\|` is string concatenation in Hive\nand `OR` in Doris — use `concat()`. | Review every existing Hive row\nfilter and `CUSTOM` mask **before** upgrading, because they start taking\neffect (**B1**). | A filter such as `tenant_id \\|\\| \u0027\u0027 \u003d \u0027x\u0027` restricts\nnothing it was meant to, and reports nothing. |\n| **D5** | `\"access_controller.class\" \u003d \"default\"` is a **new** accepted\nvalue (`default` is now reserved in both selectors). | Do not use it\nuntil every FE is upgraded. | A catalog created with it is unusable on a\nfollower that has not been upgraded yet, and after a rollback — the\nolder code has no branch for it and throws. A catalog naming no source\nat all is unaffected and remains the way to select the built-in model. |\n\n#### 2. Behavior changes\n\n| # | Area | Before | After |\n|---|---|---|---|\n| **B1** | Row filters and column masks written against a Ranger\n**`hive`** service | Never took effect. The lookup asked Ranger with the\naccess type spelled `SELECT` (what the Doris service definition\ndeclares) while the stock Hive definition declares `select`, so the\npolicy matched nothing and the query came back unfiltered and unmasked,\nwith no error anywhere. | Take effect. **For any deployment with Hive\ndata policies configured**, a column that used to come back in the clear\nnow comes back masked, and rows that used to be returned are now\nfiltered out. Policies on a Doris service are unaffected. |\n| **B2** | Column mask types on a Hive service | (unreachable — see B1)\n| `MASK`, `MASK_SHOW_LAST_4`, `MASK_SHOW_FIRST_4`, `MASK_HASH` and\n`MASK_DATE_SHOW_YEAR` are rendered with the same expressions\n`ranger-servicedef-doris.json` declares, instead of the Hive UDFs the\nstock definition supplies (`mask_show_last_n`, `mask_hash`, …), which\nDoris does not have. A type outside that set and outside `MASK_NULL` /\n`MASK_NONE` / `CUSTOM` fails the statement, naming the source, the\npolicy and the column, rather than failing on an unknown function. |\n| **B3** | A row filter whose text is **not a predicate** (e.g. `1`) |\nCoerced — `cast(1 as boolean)`, a restriction that admits every row and\nsays nothing. | Fails the statement. This applies to whoever produced\nthe filter — an authorization source and `CREATE ROW POLICY` alike — so\na stored row policy with such a predicate starts failing the queries it\ngoverns instead of silently not restricting them. Checked where the type\nis knowable before binding (a literal payload); a payload naming columns\nor functions is still resolved by binding, as before. |\n| **B4** | A Ranger source whose policy engine has **no answer** —\n`RangerBasePlugin` answers null until it has downloaded the service\u0027s\npolicies; an FE restarted while the Ranger admin is unreachable is the\nordinary way to meet it | Both data policy paths read it as \"this source\ndefines no policy\" and planned the table unfiltered and unmasked, with\nnothing anywhere saying so. (The privilege path already read the same\nnull as a refusal.) | Fails the statement. Queries against\nRanger-governed tables therefore **fail during such a window** where\nthey previously succeeded and returned more than they should have. |\n| **B5** | SQL mode a row policy predicate / column mask expression is\nparsed under | The `sql_mode` of the session running the query — and the\n**global** one for already stored policies. `sql_mode` needs no\nprivilege to set, so a restricted user could change what their own\nrestriction said. | A fixed default mode, for stored policies too. **A\ncluster that has run `SET GLOBAL sql_mode` to something that changes\nwhat SQL text means (`NO_BACKSLASH_ESCAPES`, `PIPES_AS_CONCAT`) and has\nrow policies whose predicates depend on it will see those policies\nfilter differently after the upgrade** — and differently on each FE\nduring a rolling upgrade. Each such policy is named in `fe.log` with\nboth readings as it is loaded; there is nothing to search for if the\nwarning does not appear. `CREATE ROW POLICY` under a session mode that\nreads the predicate differently still succeeds, and now reports both\nreadings to the client rather than only to `fe.log`. |\n| **B6** | `ranger-doris` configuration scope | One Ranger controller\nper FE: a second catalog binding to this source with different\n`access_controller.properties.*` had its configuration **discarded in\nsilence** — including `ranger.defer_to_global_scope_authority`, which\ndecides whether an instance administrator can read what Ranger governs.\n| Per binding. Each configuration gets a controller of its own over the\nsame shared Ranger plugin, so both bindings mean what they say, and\n`ranger.defer_to_global_scope_authority` is no longer inherited from\nwhichever binding was created first. It is now parsed strictly: anything\nother than `true` or `false` fails the statement that configures it\nrather than being read as `false`. |\n| **B7** | `ALTER CATALOG … SET PROPERTIES` on a Ranger-governed catalog\n| Detaching and re-attaching the access controller tore down and rebuilt\nthe Ranger plugin: the teardown interrupted the policy refresher and\njoined it without a timeout on the DDL thread, every check against that\ncatalog was refused until policies were downloaded again, both data\npolicy paths threw, and against an unreachable Ranger admin the `ALTER`\nitself blocked for the whole REST timeout. | Both Ranger sources keep\npolling across `ALTER CATALOG`. The plugin is shared per Ranger service\nand outlives any one binding (`ranger-hive` as well as `ranger-doris`);\nan idle one costs one policy download timer. Catalogs configured alike\nshare one controller, configured differently they get one each. |\n| **B8** | SQL result cache for a table under a Ranger row filter or\ncolumn mask | Never hit. The cache\u0027s \"did the policies move?\" check\ncompared freshly built objects by identity and decided \"changed\" every\ntime. | Can hit — the payloads gained value equality, so the check\ncompares them by content. Cached results are still keyed per user and\nstill re-validated against the source on every hit. |\n| **B9** | `information_schema.extensions` | No AUTHORIZATION rows\nexisted. | An AUTHORIZATION family lists every source this FE can\nselect: `default` for the built-in privilege model, the Ranger sources\ninstalled under `plugins/authorization/` with `SOURCE \u003d EXTERNAL`, and\nany source implementing the deprecated `AccessControllerFactory`. |\n| **B10** | `doris_fe_thread_pool` metrics | Included the `ranger-hive`\naudit flush timer. | That timer is gone from the metric — a plugin\ncannot register into the FE\u0027s thread-pool registry. |\n\n#### 3. Compatibility of existing configuration and third-party\ncontrollers\n\n| Surface | Status |\n|---|---|\n| `access_controller_type` | **Unchanged** — same accepted values. |\n| `access_controller.class` | Keeps accepting every class name it\naccepted before; the one name that moved out of the kernel is mapped by\nan alias table inside the kernel. It additionally accepts `default` —\nnew, see **D5**. |\n| A catalog naming no source at all | **Unaffected**, and remains the\nway to select the built-in model. |\n| A plugin publishing a source named `default` | Refused at load — the\nname is reserved. |\n| Third-party controller on the **deprecated** `CatalogAccessController`\n| Keeps working through an adapter; the interface is deprecated, not\nremoved. The removed `checkDbPriv(boolean hasGlobal, …)`-style default\nmethods are reproduced by the adapter, so it **refuses nothing it did\nnot refuse before** and does not lose the \"granted at global scope\"\nexemption. |\n| Third-party controller implementing **column masking or row\nfiltering** | **Must be recompiled** — the four payload classes it used\nwere replaced by neutral ones. Erasure hides this at class-load time, so\nit surfaces during execution: `NoClassDefFoundError` if the method body\nnames a deleted type, otherwise an `IllegalStateException` naming the\nsource, raised where the answer crosses back into the engine. |\n| Third-party controller implementing **neither** | Unaffected. |\n| Comparing `PrivPredicate` with `\u003d\u003d` | Still works for the questions\nthe engine asks by name — `SHOW_RESOURCES` and `SHOW_WORKLOAD_GROUP`\nincluded; they name the same actions and are told apart by the\nrequirement the engine derived. A check built for a single statement\nstill has to be recognised by reading `getPrivs()` and `getOp()`, as it\nalways did. |\n\nNo access decision of the built-in `GRANT` model changes: the golden\nmatrix in\n`AccessControlBehaviorBaselineTest` — every combination of default\nsource × catalog kind × user ×\nall 17 `PrivPredicate` constants × probe point — is byte-for-byte\nunchanged, with the one caveat\nrecorded under **Check List (For Author)**.\n\n---\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: None\n\nProblem Summary:\n\nDeciding what a user may access is wired into `fe-core`.\n`CatalogAccessController` is an `fe-core`\ninterface with one method per kind of object, the two Ranger\nintegrations implement it from inside\nthe kernel, and the row-filter / data-mask payloads handed to the\nplanner are kernel classes.\nAnything outside this repository that wants to decide access has to be\ncompiled against `fe-core`\ninternals, and the Ranger sources ship inside `fe-core.jar` whether a\ndeployment uses them or not.\n\nThis PR turns \"who decides access\" into a plugin contract, and makes\nboth the built-in `GRANT` model\nand the Ranger sources implementations of that contract.\n\n**New modules**\n\n- `fe-authorization-api` — a neutral vocabulary with no `fe-core` types:\nactions, requirements\n(\"any of these\" / \"all of these\"), the resources that can be asked\nabout, subjects, and the\n row-filter / data-mask payloads.\n- `fe-authorization-spi` — the contract itself: check a requirement,\ncheck one action, ask for row\nfilters, ask for column masks, plus a lifecycle and an\n`AuthorizationContext` carrying the\nquestions a decision needs (the roles of a subject; whether the\ninstance-wide source has already\n granted it).\n- `fe-authorization-plugins/{ranger-common,ranger-doris,ranger-hive}` —\nthe Ranger sources, now\n outside the kernel and shipped as installable plugins.\n\n**What changed in the engine**\n\n- Every access check now goes through one decision point.\n`AccessControllerManager` keeps only the\nrouting — a table from resource kind to the source governing it — and\nconverts a refusal back into\n the boolean its existing callers expect.\n- The manager no longer computes a global verdict of its own and no\nlonger ORs two sources\u0027 answers\ntogether. Each source now grants or refuses its own exemptions, which is\nwhat makes the policy in\nforce on an object readable from the configuration rather than from two\nplaces at once. Three\nexemptions predate that rule and stay the engine\u0027s own; they are\nenumerated in\n `fe/fe-authorization/README.md`.\n- A refusal is thrown and carries its reason, instead of being a boolean\nthat drops it.\n- The Ranger plugins are loaded by the same machinery the other plugin\nfamilies already use: an API\nversion gate, a child-first class loader, a per-plugin directory under\n`plugins/authorization/`,\nand registration in `information_schema.extensions`. A checkpoint `Env`,\nwhich replays metadata\nand authorizes nothing, builds no source at all — one that did would\nstart Ranger threads nothing\n ever stops, twice per checkpoint round.\n\n**Compatibility**\n\n- `access_controller.class` keeps accepting the class names it accepted\nbefore. The one name that\nmoved out of the kernel is mapped by an alias table inside the kernel,\nand a third-party\ncontroller still implementing the old interface keeps working through an\nadapter (the old\n interface is deprecated, not removed).\n- `access_controller_type` is unchanged; its accepted values are the\nsame.\n- The deprecated interface\u0027s scoped methods came in pairs —\n`checkDbPriv(boolean hasGlobal, ...)` in\nfront of `checkDbPriv(...)` — and the engine computed `hasGlobal` from\nwhoever governed instance\nscope, so a caller holding the privilege globally was granted without\nthe controller being asked.\nThose default methods are gone; the adapter reproduces the exemption, so\na third-party controller\n refuses nothing it did not refuse before.\n- A third-party controller that implemented **column masking or row\nfiltering** must be recompiled:\nthe four payload classes it used were replaced by neutral ones. Erasure\nhides this at class-load\ntime, so it surfaces during execution — as `NoClassDefFoundError` if the\nmethod body names a\ndeleted type, otherwise as an `IllegalStateException` naming the source,\nraised where the answer\n crosses back into the engine.\n- A third-party controller is handed the `PrivPredicate` the caller\nnamed, `SHOW_RESOURCES` and\n`SHOW_WORKLOAD_GROUP` included - they name the same actions, and are\ntold apart by the requirement\nthe engine derived from each. Comparing with `\u003d\u003d` therefore keeps\nworking for the questions the\nengine asks by name; a check built for a single statement still has to\nbe recognised by reading\n `getPrivs()` and `getOp()`, as it always did." }, { "commit": "a6a7e52a7336fcca8dd1564cbeb0f3d726304cc4", "tree": "39def7da268df0ce037ca8c18c92f0e396ce2b3e", "parents": [ "a565aca4478ee22bb5f9332fd0ba69668dcb001e" ], "author": { "name": "morrySnow", "email": "zhangwenxin@selectdb.com", "time": "Thu Sep 10 12:07:59 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 12:07:59 2026 +0800" }, "message": "[improvement](parser) Add case-insensitive stream fast paths (#67452)\n\n### What problem does this PR solve?\n\nProblem Summary:\n\nEvery SQL lexer currently copies its input through\n`CharStreams.fromString()` and calls `Character.toUpperCase()` for every\ncase-insensitive lookahead. This is visible in lexer allocation and\nidentifier-heavy parsing time.\n\nAdd two strictly compatible fast paths:\n\n- Fold ASCII `a-z` with arithmetic and retain `Character.toUpperCase()`\nfor every other code point.\n- Read strings without UTF-16 surrogates directly, avoiding ANTLR\u0027s\ncopied code-point buffer. Inputs containing any surrogate retain the\noriginal ANTLR stream so code-point indices, `getText`, navigation, and\nerrors remain unchanged.\n\nThe public arbitrary-`CharStream` constructor is preserved. All\nproduction String entry points now use the factory. ANTLR\u0027s native\n`caseInsensitive` option was evaluated but rejected because it changed\nexisting Unicode behavior such as the handling of `ſelect`.\n\n### Benchmark\n\nEnvironment and method:\n\n- Baseline: `cf33a08bcd5`, artifact SHA-256\n`51700bef953361218912abc6aec348cecdec3fe61031eba10cc1ebc534bc9183`\n- Candidate: `7aa7b5e6487`, artifact SHA-256\n`6e8edf78befc0dc349be398aae454c172b5cb968ee98d88a0e056d4c21a9c5d0`\n- macOS 15.0.1 arm64, OpenJDK 17.0.20.1, 1 GiB heap, JMH 1.37\n- 2 forks, 4 x 300 ms warmup, 7 x 400 ms measurement, `-prof gc`\n- Interleaved baseline/candidate runs; one candidate run affected by\nunrelated host load was discarded and repeated. The table averages two\nvalid runs per artifact. Individual JMH scores use 99.9% confidence\nintervals.\n\n```shell\njava -Xms1g -Xmx1g -jar \u003cbenchmark.jar\u003e \\\n \u0027CaseInsensitiveStreamBenchmark.(createLexer|foldPrebuiltCharacters|parseStatement|tokenize)\u0027 \\\n -p workload\u003dshortQuery,lowercaseIdentifiers,stringAndComment,unicode \\\n -f 2 -wi 4 -i 7 -w 300ms -r 400ms -prof gc -rf json\n```\n\n| Benchmark | Baseline us/op (B1 / B2) | Candidate us/op (C1 / C3) | Mean change | Baseline → candidate B/op |\n| -------------------------------------- | --------------------------: | --------------------------: | ----------: | ------------------------: |\n| Tokenize 64 lowercase identifiers | 10.826±0.078 / 11.408±0.125 | 8.708±0.071 / 8.704±0.096 | -21.7% | 14,799 → 10,464 (-29.3%) |\n| Parse 64 lowercase identifiers | 54.775±7.360 / 56.365±4.224 | 49.860±0.191 / 52.340±1.160 | -8.0% | 99,395 → 95,017 (-4.4%) |\n| Tokenize `select 1` | 0.451±0.004 / 0.485±0.011 | 0.348±0.069 / 0.324±0.010 | -28.2% | 1,008 → 816 (-19.0%) |\n| Parse strings/comments with Unicode | 27.047±5.371 / 28.435±2.932 | 24.744±0.401 / 24.743±0.425 | -10.8% | 24,167 → 23,784 (-1.6%) |\n| Tokenize supplementary Unicode control | 32.034±1.398 / 33.910±0.752 | 32.303±1.108 / 32.020±0.738 | -2.5% | 24,630 → 24,594 (-0.1%) |\n\n\nThe isolated `createLexer` measurement for the surrogate-containing\nstring/comment workload regresses by 5.9% because the compatibility\nguard scans for surrogates before falling back. The corresponding\ncomplete tokenize and parse paths improve by 3.6% and 10.8%; no\nend-to-end control workload regressed. Prebuilt lowercase character\nfolding improves by 10.6%.\n\nThe performance gains come from eliminating the copied input buffer for\nBMP-only SQL, avoiding its allocation, and replacing the common\nlowercase ASCII `Character.toUpperCase()` call with an arithmetic\nbranch.\n\nCorrectness corpus:\n\n- 4,610 tracked SQL files; baseline and candidate parse signatures match\nin legacy and ANSI modes. SHA-256:\n`69c811d0d80c52b40d8cd854e925ff4930aa6601c7d1e66541f2eb29f35db50a`.\n- 9,220 lexer cases (4,610 SQL files x both `noBackslashEscapes` modes);\ncomplete token tuple and lexer-error signatures match. SHA-256:\n`66780d4e1d9224ea27c8f715ab33edd247faed1571030ba9e90a8ef3f6212180`." }, { "commit": "a565aca4478ee22bb5f9332fd0ba69668dcb001e", "tree": "af33a52ec4cf0fdca12ed134cd7413777d3ecc77", "parents": [ "7129a3e8c6bf087cb5a3a818e53343f5fee2c297" ], "author": { "name": "yujun", "email": "yujun@selectdb.com", "time": "Thu Sep 10 11:32:59 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 11:32:59 2026 +0800" }, "message": "[fix](regression) Stop MTMV task waits from latching onto the previous task (#67710)\n\nThe shared MTMV task waits (`waitingMTMVTaskFinishedByMvName`,\n`waitingMTMVTaskFinishedByMvNameAllowCancel`, `waitingMTMVTaskFinished`,\n`waitingMTMVTaskFinishedWithoutAnalyze`,\n`waitingMTMVTaskFinishedNotNeedSuccess`) read the newest row of\n`tasks(\u0027type\u0027\u003d\u0027mv\u0027)` with `order by CreateTime DESC limit 1` and trusted\nit as the task they had just submitted. A task that just finished is\nbriefly missing from that list: the job removes it from its running list\nbefore the MV history gets it, so a poll that lands in that window reads\nthe previous task of the same MV, whose status is already terminal. The\ncommitted-not-visible regression hit this on CI (expected SUCCESS but\nread the previous FAILED task).\n\nKey changes:\n- Add `pollMTMVTaskTerminal()`: a terminal row is trusted only when the\nsame task was seen running by this wait, or is seen terminal twice in a\nrow and no earlier wait in the suite returned it (so a wait cannot latch\nonto an already-reported task).\n- The five `waitingMTMVTaskFinished*` helpers now share that wait\ninstead of five copies of the polling loop; their SQL, logging,\nassertions and analyze step are unchanged.\n- Add `SuiteMTMVTaskWaitTest` covering a task seen running, a terminal\nrow that needs confirmation, the previous task showing through the gap,\nand a repeated wait for a task that already finished (bounded fallback)." }, { "commit": "7129a3e8c6bf087cb5a3a818e53343f5fee2c297", "tree": "469b2ba286e93b629c34b09b6fa46f64b73fe60b", "parents": [ "c1e522e24918e91039db6010381819b004921b11" ], "author": { "name": "yujun", "email": "yujun@selectdb.com", "time": "Thu Sep 10 11:22:11 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 11:22:11 2026 +0800" }, "message": "[fix](ivm) Propagate and compensate failures of the IVM excluded-trigger-tables ALTER (#67665)\n\nFollow-up of the #62606 review round.\n\nALTER MATERIALIZED VIEW ... SET (\u0027excluded_trigger_tables\u0027 \u003d ...) on an\nIVM MTMV transitions the base-table streams alongside the property. That\ntransition used to run inside `processAlterMTMV`, whose catch swallowed\nany `UserException`: a mid-transition failure (e.g. a stream create\nthrowing after an earlier create was already journaled) reported success\nto the client while the property stayed unchanged and stray streams\nremained.\n\nKey changes:\n- Live ALTER PROPERTY statements now run through the new\n`Alter.processAlterMTMVProperty`, which propagates failures to the\nclient; the journal replay path keeps the tolerant `processAlterMTMV`\nbehavior.\n- The stream transition is reordered: create the streams of newly\nun-excluded bases first (compensated by dropping exactly the streams\ncreated in this call on failure), then apply the property, then\nbest-effort drop the streams of newly excluded bases - a failed drop\nonly leaks a stream of an already excluded table and never fails the\nALTER after the property took effect.\n- Two test debug points (count-based, independent of the base-table\niteration order) inject stream create/drop failures.\n\nTests: `AlterMTMVTest` two new cases covering create-failure\ncompensation and drop-failure best effort with multi-table excluded-set\nchanges; the full class (24 tests) passes.\n\nTrace issue: https://github.com/apache/doris/issues/65418" }, { "commit": "c1e522e24918e91039db6010381819b004921b11", "tree": "680c89e5904a0e9969e8e4f5f8182da04889c2cb", "parents": [ "ecae3c4a4a6c7c6503b43ff3b4679cb22a5b4641" ], "author": { "name": "Gabriel", "email": "liwenqiang@selectdb.com", "time": "Thu Sep 10 11:21:24 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 11:21:24 2026 +0800" }, "message": "[fix](be) Decouple Parquet metadata size limit from Thrift (#67631)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #25194\n\nProblem Summary: FileScannerV2 reused `thrift_max_message_size` as the\nParquet footer allocation limit, so valid files with metadata larger\nthan the 100 MiB RPC ceiling were rejected. This PR adds an independent\nmutable `parquet_metadata_size_limit` with a 256 MiB default. It\npreserves the structural `footer_size \u003c\u003d file_size - 8` validation and\nkeeps the metadata cap check before allocation and any second read.\n\n### Release note\n\nFileScannerV2 now limits Parquet metadata with the independent\n`parquet_metadata_size_limit` configuration, which defaults to 256 MiB.\n\n### Check List (For Author)\n\n- Test: Unit Test\n - `NewParquetReaderTest.NativeFooter*` (7 tests passed under ASAN)\n- Behavior changed: Yes (valid Parquet metadata between the Thrift RPC\nceiling and the new metadata limit is accepted.)\n- Does this need documentation: No" }, { "commit": "ecae3c4a4a6c7c6503b43ff3b4679cb22a5b4641", "tree": "e678f9a85b8e3d067948280c31324885fc9adc6e", "parents": [ "8da964acac241c898c5f9ced550e41fc42ce1134" ], "author": { "name": "Gabriel", "email": "liwenqiang@selectdb.com", "time": "Thu Sep 10 11:20:22 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 11:20:22 2026 +0800" }, "message": "[fix](iceberg) Support fractional timestamps in time travel (#67705)\n\n### What problem does this PR solve?\n\nIceberg `FOR TIME AS OF` rejected timestamps containing fractional\nseconds, even though the snapshots metadata exposes `committed_at` with\nfractional precision. Truncating the value to whole seconds can also\nselect the wrong snapshot when multiple snapshots are committed within\nthe same second.\n\n### What is changed and how does it work?\n\n- Accept an optional 1-9 digit fractional-second component in Iceberg\ntime-travel timestamp literals while preserving whole-second\ncompatibility.\n- Keep session time-zone interpretation unchanged.\n- Add unit coverage for millisecond and microsecond-formatted literals.\n- Update the Iceberg time-travel regression matrix to feed the\nfractional `committed_at` value back into `FOR TIME AS OF` directly.\n\n### Check List\n\n- [x] Unit test: `IcebergTimeUtilsTest` (6 tests)\n- [x] FE Checkstyle\n- [x] `git diff --check`" }, { "commit": "8da964acac241c898c5f9ced550e41fc42ce1134", "tree": "33f2bc87e1cbbd9d540296fc99ad4ad2a0528484", "parents": [ "4f3abce2c25e7b1e3d24eeaa97cffcfd76abe531" ], "author": { "name": "TengJianPing", "email": "tengjianping@selectdb.com", "time": "Thu Sep 10 11:15:10 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 11:15:10 2026 +0800" }, "message": "[fix](p2) fix p2 regression failures (#67731)\n\nrelated PR: #65609\n\n### What problem does this PR solve?\n\nIssue Number: close #xxx\n\nRelated PR: #xxx\n\nProblem Summary:\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test \u003c!-- At least one of them must be included. --\u003e\n - [ ] Regression test\n - [ ] Unit Test\n - [ ] Manual test (add detailed scripts or steps below)\n - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n - [ ] Previous test can cover this change.\n - [ ] No code files have been changed.\n - [ ] Other reason \u003c!-- Add your reason? --\u003e\n\n- Behavior changed:\n - [ ] No.\n - [ ] Yes. \u003c!-- Explain the behavior change --\u003e\n\n- Does this need documentation?\n - [ ] No.\n- [ ] Yes. \u003c!-- Add document PR link here. eg:\nhttps://github.com/apache/doris-website/pull/1214 --\u003e\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label \u003c!-- Add branch pick label that this PR\nshould merge into --\u003e" }, { "commit": "4f3abce2c25e7b1e3d24eeaa97cffcfd76abe531", "tree": "b39e2203f03499304b9439b5a992c32f7e68efe2", "parents": [ "c1bff0d27c2b1327de70ac583d322db7561878de" ], "author": { "name": "yujun", "email": "yujun@selectdb.com", "time": "Thu Sep 10 11:13:13 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 11:13:13 2026 +0800" }, "message": "[fix](ivm) Resolve IVM identity keys by slot identity and materialize unprojected keys (#67669)\n\nTwo follow-ups from the #62606 review round, both around same-named\nidentity keys of aggregate/full-keys MVs:\n\n1. **Resolve IVM aggregate group keys by slot identity instead of name**\n— an aggregate delta over a join of two tables that both expose the same\ncolumn name (`GROUP BY l.id, r.id`) resolved its group keys through\nname-keyed lookups. A name index collapses same-named slots onto the\nlast match, so both keys were bound to one delta slot and a `(10, 20)`\ngroup was written as `(20, 20)`, diverging from COMPLETE. The delta-side\ngroup keys are now resolved and carried by slot identity (ordered with\nthe aggregate metadata), the apply project emits each key output from\nits own delta slot, and the `ivm_use_full_keys` identity conjuncts\nresolve their delta side by identity first. The full-keys regression\nwith a non-full-keys counterpart covers plain INSERT, full-keys, and MOW\nUPDATE refresh, each cross-checked against COMPLETE.\n\n2. **Materialize unprojected full-keys identity keys at the project\nlayer** — with `ivm_use_full_keys` every identity key must be\nmaterialized in the stored layout, either as a visible output or under a\nhidden key column. The result sink judged \"already projected\" by column\nname, so a same-named key from another table (selecting only `l.id`\nwhile grouping by `l.id, r.id`) was silently dropped and the key set\nlost one dimension. The sink now checks projection by slot identity, and\nmaterialization moves down to the first project that drops a key, so\nCREATE (result sink) and refresh (olap-table sink) plans produce the\nsame hidden-key layout. A new regression asserts the hidden key column\nexists (DESC with hidden columns) and that INCREMENTAL matches COMPLETE.\n\nTests: `IvmAggDeltaHandlerTest` (33) and `IvmNormalizeMTMVJoinTest` new\ncases pass; the new regression cases pass and the full ivm regression\nset (85 suites) is green on the materialization change.\n\nTrace issue: https://github.com/apache/doris/issues/65418" }, { "commit": "c1bff0d27c2b1327de70ac583d322db7561878de", "tree": "312ed06a7c1507eaf09f98b8db045f4f37885be2", "parents": [ "958aaf46c8f6b7bd6fb661a18e5b7cbbf5c40999" ], "author": { "name": "yujun", "email": "yujun@selectdb.com", "time": "Thu Sep 10 11:08:46 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 11:08:46 2026 +0800" }, "message": "[fix](ivm) Fail aggregate IVM refresh when the previous refresh txn is not visible yet (#67646)\n\nThis PR contains two follow-ups from the #62606 review round:\n\n1. **Derive IVM hidden column names from `IVM_HIDDEN_COLUMN_PREFIX`** —\nIVM hidden column names are now derived from the shared prefix constant\ninstead of duplicated literals.\n\n2. **Fail aggregate IVM refresh when the previous refresh txn is not\nvisible yet** — an aggregate delta joins the MV\u0027s old rows with the new\ndelta; when the previous refresh txn committed but its data is not\nvisible yet (an MV partition\u0027s committed version is ahead of its visible\nversion), that join misses the old rows and permanently loses the delta.\nThe delta rewriter now fails a non-empty aggregate delta with\n`MV_COMMIT_NOT_VISIBLE` and the refresh falls back to a rebuild that\nrecomputes from the base tables without reading old MV state.\n\nAdds a nonConcurrent regression that holds a refresh txn in COMMITTED\nvia the new `DatabaseTransactionMgr.finishTransaction.block_visible`\ndebug point, then verifies the next strict incremental refresh fails\nwhile a COMPLETE refresh can still report SUCCESS with its txn not yet\nvisible, and that everything converges once the stuck txns publish.\n\nTrace issue: https://github.com/apache/doris/issues/65418" }, { "commit": "958aaf46c8f6b7bd6fb661a18e5b7cbbf5c40999", "tree": "0756ada1279a35cf6662c1a6e1ef2b5cd3a5bb54", "parents": [ "0d36c66f8f1bf2415690ecee59b0edef8444af7b" ], "author": { "name": "Calvin Kirs", "email": "guoqiang@selectdb.com", "time": "Thu Sep 10 10:44:30 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 10:44:30 2026 +0800" }, "message": "[fix](protocol) Prevent Connector/J cursor fetch from hanging on empty results (#67520)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #61050, #61062\n\nAfter `CLIENT_DEPRECATE_EOF` is negotiated, affected Connector/J clients\nusing `useCursorFetch\u003dtrue` and a positive fetch size can hang when a\nserver-prepared statement returns an empty result. The driver consumes\nthe first terminator after column definitions while checking whether a\nserver cursor was created. With no rows, this consumes the only\nresult-end marker, so the driver waits for a packet that Doris will\nnever send.\n\nThe reported Connector/J 8.2.0 case is covered by real-driver tests.\nThis PR preserves the final result marker for affected clients and\navoids inserting the compatibility marker for identified Connector/J\n9.5+ and MariaDB clients. It also preserves this behavior through FE\nforwarding and fixes related packet/capability regressions.\n\n### What is changed?\n\n1. **Cursor result boundaries.** Read `CURSOR_TYPE_READ_ONLY` from each\n`COM_STMT_EXECUTE`, retain it in the request context, and classify\ndriver behavior using connection attributes. For affected cursor clients\nwith deprecated EOF enabled, insert a compatibility ResultSet OK after\nmetadata so the real final marker remains available. Identified modern\nclients and ordinary non-cursor requests keep the standard sequence.\n2. **FE forwarding.** Forward cursor intent and negotiated capabilities\nthrough optional Thrift fields. The master restores them before\nserializing results. A new follower adapts buffered results from an\nolder master that has not confirmed protocol adaptation:\ninsert/remove/convert metadata boundaries as appropriate and convert the\nfinal legacy EOF when needed. Row payloads are retained; confirmed\nresults are not adapted twice.\n3. **Complete OK information.** Preserve the original forwarded DML/DDL\nOK and ERR packets rather than rebuilding a partial successful OK.\nRetain affected rows, warnings, info, load labels and transaction IDs.\nPreserve warnings and server status when producing/converting result-set\nterminators. SQL is not re-executed by this adaptation.\n4. **Legacy protocol compatibility.** Configure the serializer using the\nintersection of client/server capabilities, rather than server defaults.\nLimit the empty OK-info compatibility byte to the applicable negotiated\nformat, preserving legacy OK behavior.\n5. **Arrow follower compatibility.** Access MySQL-channel capabilities\nonly for MySQL connections, so the shared forwarding path does not call\nthe unsupported MySQL-channel getter on an Arrow Flight SQL context.\n6. **Clients without connection attributes.** Remove the new fail-fast\nrejection of anonymous cursor clients. When client behavior is unknown,\nprioritize the older Connector/J compatibility sequence. This retains\nthe older-client target but cannot simultaneously guarantee\ncompatibility with anonymous clients requiring the standard sequence.\n7. **File-load compatibility.** In both LOAD handlers, require\n`CLIENT_LOCAL_FILES` only for client-side `LOCAL INFILE` uploads.\nFE-side file reads do not require a client-upload capability and remain\nusable when that bit is absent.\n\nFull PR diff against the merged master, at `32d01e5df19`:\n\n| Category | Files | Added lines | Removed lines | Net added lines |\n|---|---:|---:|---:|---:|\n| Production code, including Thrift IDL | 12 | 218 | 40 | 178 |\n| Tests and expected output | 12 | 787 | 13 | 774 |\n\nThe test total includes 11 automatically generated expected-output\nlines. Environment configuration and review documents are excluded.\n\n### Compatibility boundaries\n\n- This preserves streamed results; it does not implement MySQL\nserver-side cursors or `COM_STMT_FETCH` batching.\n- **New follower → old master** is covered by the mixed-version client\nmatrix. **Old follower → new master** may have already lost cursor\nintent; the master cannot reconstruct it. The historical cursor hang on\nthat route remains until the client-facing follower is upgraded.\nExisting ordinary prepared-statement behavior is retained.\n- Identified Connector/J 9.5+ and MariaDB clients use the standard path.\nAnonymous clients requiring that sequence remain outside the\nolder-client fallback guarantee.\n- The OIDC provider and authentication routing implementation are not\nchanged by this PR.\n\n### Release note\n\nFix affected Connector/J cursor queries hanging on empty prepared\nresults. Preserve negotiated MySQL packet formats, forwarded DML/DDL OK\ninformation, Arrow Flight SQL forwarding and FE-side file loading.\nUpgrade the client-facing follower to preserve cursor intent during an\nFE rolling upgrade.\n\n### Check List (For Author)\n\n- Test:\n- [x] Unit Test: **113 targeted FE tests passed, 0 failures/errors**,\nbefore the final master/JUnit 5 merge. Coverage includes driver\nclassification, cursor boundaries, forwarding, capabilities, packet\nserialization, authentication routing and both LOAD handlers.\n- [x] Regression test: `prepared_stmt_p0/cursor_fetch_empty_result`,\n`prepared_stmt_p0/prepared_show`, and `arrow_flight_sql_p0/test_ddl`\nthrough a real follower passed. Cursor expected output was generated by\nthe regression runner.\n- [x] Manual test: **120 successful real-JDBC driver/endpoint/transport\nruns** across baseline/candidate direct endpoints, a same-version\nfollower and a new follower targeting an old master, with plaintext/TLS\nmodes. Each driver JAR runs in its own JVM to avoid classpath version\ncollisions.\n- MySQL Connector/J: **5.1.49, 8.0.28, 8.0.33, 8.2.0, 8.4.0, 9.0.0,\n9.4.0, 9.5.0, 9.6.0**. MariaDB Connector/J: **3.5.6**.\n- Each run exercises 12 requested combinations of cursor/server-prepare\nswitches and fetch sizes **0/1/10000**, repeated\nStatement/PreparedStatement empty→nonempty→empty results on reused\nconnections, and wrong-password rejection. Drivers may couple cursor and\nserver-prepare settings internally. Connection/read timeouts bound\nfailures; a timeout is not counted as success.\n- Additional checks passed: explicit TLS 1.2/1.3 (16 groups), MySQL\nShell 8/9 ordinary authentication (16), legacy EOF via PyMySQL (8),\nSQL-error connection reuse (8), and suppressed connection attributes\nwith older drivers (12).\n- Actual baseline/candidate FE-file and client-file imports passed;\nforwarded INSERT retained label/status/txnId. Real LDAP/local-password\nrouting, rejected credentials and connection reuse passed 20\nbaseline/candidate checks across plaintext/TLS.\n- Independent FE-only raw-protocol tests passed 12 configurations\ncovering multi-result status, cursor transitions, RESET/CLOSE and reuse\nafter errors. These are packet tests, not additional real JDBC runs or\nproof of mixed-version multi-statement forwarding.\n- FE build passed before the final master merge. After the merge,\nstandalone `mvn checkstyle:check -pl fe-core -Dcheckstyle.skip\u003dfalse`\npassed with **0 violations**. Compilation and unit tests were not rerun\nafter the JUnit 5 migration.\n- The multi-version JDBC matrix was executed locally with retained\nscripts/logs; it is not an automatic all-version CI matrix.\n- Review and validation limits:\n- The latest independent review of the complete final diff found **0\nBlocker / 0 Major production findings**. It noted two deterministic\nSQL-result assertions that still use `assertEquals` rather than\nregression `qt` output, and missing dedicated tests for errors occurring\nafter partial metadata or rows have been buffered. Those error paths\nwere inspected in source, not fault-injected; the OK/ERR preservation\nunit test does not cover buffered intermediate errors.\n- Connector/J 6.0.6 failed handshakes on both baseline and candidate in\nsix runs; it is not counted as passing or newly supported.\n- Four broader prepared suites failed identically on baseline/candidate\nwith the available older BE because it cannot execute the plan.\nAuthentication-integration metadata regression also remains incomplete\nwith that BE. Four independent SELECT configurations without a BE were\nblocked and not counted as passing.\n- Full product OIDC-provider login/new TLS-extension E2E and the\ncustomer\u0027s exact SmartBI environment were not validated. Ordinary\npassword/TLS and authentication unit tests do not replace these checks.\n- Behavior changed:\n- [x] Yes. Correct cursor packet boundaries, preserve complete forwarded\nOK information and use negotiated capabilities, with the compatibility\nboundaries above.\n- Does this need documentation?\n - [x] No.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label" }, { "commit": "0d36c66f8f1bf2415690ecee59b0edef8444af7b", "tree": "3db48150b81dc29720ced3570504c64456ed426e", "parents": [ "29583a18fa76416a8fe50d20cbf82d8663bcdea6" ], "author": { "name": "yiguolei", "email": "guolei@selectdb.com", "time": "Thu Sep 10 10:32:02 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 10:32:02 2026 +0800" }, "message": "Revert \"[chore](docker)update docker image to almalinux:8\" (#67752)\n\nReverts apache/doris#65516\n\nwe should never change centos version, because of glibc compatable" }, { "commit": "29583a18fa76416a8fe50d20cbf82d8663bcdea6", "tree": "a5c85b73c47390d87c9b41372f4d1d4ff5b6de51", "parents": [ "4e90d4b43cb28fc088f54bc272f4e1b7328b0732" ], "author": { "name": "HappenLee", "email": "happenlee@selectdb.com", "time": "Thu Sep 10 10:29:12 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 10:29:12 2026 +0800" }, "message": "[improvement](build) Upgrade Snappy to 1.2.1 and enable SIMD paths (#67727)\n\n### What problem does this PR solve?\n\nDoris builds Snappy 1.1.10 without enabling its available SIMD paths.\nUpgrade to 1.2.1 and match the BE instruction-set target:\n\n- On x86_64, enable SSE4.2, including Snappy\u0027s byte-shuffle and CRC32\nhashing paths. Add `-mavx2` by default and when `USE_AVX2` is enabled;\n`USE_AVX2\u003d0` or `OFF` selects the SSE4.2 baseline. This follows the\nexisting third-party `USE_AVX2` convention used by CRoaring.\n- On ARM, enable CRC32 through `ARM_MARCH` (default `armv8-a+crc`).\n\nSnappy is a prebuilt static library, so its third-party build must use\nthe same target as the BE. Changing only the BE\u0027s `USE_AVX2` setting\ndoes not rebuild or retarget an existing Snappy archive. We do not\nenable `SNAPPY_REQUIRE_AVX2`, because upstream 1.2.1 adds BMI2 through\nthat option on GCC/Clang, while BMI2 is not guaranteed by AVX2.\n\nPreserve RTTI for `SnappySlicesSource` and remove the sign-compare patch\nalready included upstream. Existing codec interfaces and default\ncompression selection remain unchanged; bRPC\u0027s embedded `butil::snappy`\nis outside this change.\n\nAdd coverage for binary data, 64 KiB boundaries, empty and uneven\nslices, truncated and invalid streams, insufficient output capacity, and\nfrozen 1.1.10 compressed data. Use byte-exact comparisons in the\nexisting round-trip tests.\n\nFresh seven-run median CPU-time microbenchmarks on a Xeon Platinum\n8457C, using Clang 16.0.6 for all libraries, gave the following\nthroughput changes relative to 1.1.10. Both new targets were built\nthrough the updated third-party script. These results replace the\nmeasurements for the earlier SSSE3-only recipe.\n\n| Input | SSE4.2 compression | SSE4.2 decompression | AVX2 compression |\nAVX2 decompression |\n| --- | ---: | ---: | ---: | ---: |\n| 1 MiB repeated binary | -7.4% | +140.5% | -0.4% | +140.0% |\n| Upstream HTML corpus | -27.4% | +10.3% | -28.8% | +21.6% |\n| Upstream URL corpus | -10.3% | +4.5% | -10.3% | +0.9% |\n| Upstream geo.protodata | -35.2% | -3.5% | -33.7% | +5.0% |\n\nCompression regresses on some inputs; this is not a universal speedup.\nMeasurements are single-threaded, pinned to one logical CPU on a shared\nhost, with hot input, preallocated output buffers, and at least 0.1 CPU\nseconds per sample. Twelve inputs were measured. These are library\nmicrobenchmarks, not SQL or ARM performance claims. The larger upstream\nhash table adds up to 32 KiB of temporary compression memory.\n\n### Release note\n\nUpgrade the BE Snappy dependency to 1.2.1 and enable SIMD paths for the\nselected CPU target. Third-party Snappy builds enable AVX2 by default;\nuse `USE_AVX2\u003d0` or `OFF` for a non-AVX2 BE. Snappy data remains\nformat-compatible; performance and compressed bytes can vary by input.\n\n### Check List (For Author)\n\n- Test:\n- [x] Current recipe: build SSE4.2 and AVX2 libraries through\n`USE_AVX2\u003dOFF/ON thirdparty/build-thirdparty.sh -j 48 snappy` with Clang\n16.0.6.\n- [x] Current recipe: check 30 architecture/flag combinations, including\nunset, numeric and textual boolean values and both ARM architecture\nnames.\n- [x] Inspect both archives: CRC32 instructions present in both, no AVX\ninstructions in the SSE4.2 archive, and 256-bit vector instructions in\nthe AVX2 archive. BMI2 detection remains disabled for both.\n- [x] Verify 24 inputs across all 9 encoder/decoder combinations of\n1.1.10, 1.2.1 SSE4.2 and 1.2.1 AVX2; run seven-trial\ncompression/decompression microbenchmarks on 12 inputs.\n- [x] Bash syntax, Snappy recipe shfmt and `git diff --check` pass.\nShellCheck introduces no new warnings; 7 existing warnings remain\nelsewhere in the script.\n- Previous revision: `run-be-ut.sh -j 48 --run\n--filter\u003d\u0027BlockCompressionTest.*\u0027` passed all 5 tests under ASAN, with\nbuild hygiene, clang-format 16 and changed-line clang-tidy passing.\nThese BE tests ran before the Apache-master cherry-pick and before the\nSSE4.2/AVX2 flag update; they were not rerun for the final targets. CI\nmust validate final BE integration.\n- ARM CRC32 intrinsic code generation was checked in the previous\nrevision; no ARM hardware test performed.\n- Behavior changed:\n- [x] Yes. CPU build flags, compression output and performance can\nchange; the format and codec interfaces remain compatible.\n- Does this need documentation?\n - [x] No. Third-party build behavior is recorded in the changelog." }, { "commit": "4e90d4b43cb28fc088f54bc272f4e1b7328b0732", "tree": "2ac8c0d078342dfbe98a943689badeee2db680b9", "parents": [ "bd993f3f3569f903b7100a681e09a527fd208002" ], "author": { "name": "wudi", "email": "wudi@selectdb.com", "time": "Thu Sep 10 10:15:29 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 10:15:29 2026 +0800" }, "message": "[improvement](fe) Improve audit logs for S3 streaming insert jobs (#67489)\n\n### What problem does this PR solve?\n\nProblem Summary:\n\nS3 streaming insert tasks execute internal INSERT statements without\nemitting query audit events. This makes it difficult to identify the\nexact files imported after wildcard paths are resolved.\n\nThis change emits internal audit events only for S3 streaming insert\ntasks. The audited statement contains the rewritten S3 URI and masks\nsensitive TVF properties. Both successful and failed executions are\nrecorded. CDC streaming insert tasks keep the existing behavior and do\nnot emit these audit events." }, { "commit": "bd993f3f3569f903b7100a681e09a527fd208002", "tree": "e04281aa07ecfc2b1bb503a957db52a3b5197cf8", "parents": [ "f47b64d80c15f76c13355ce492976012b2447d5f" ], "author": { "name": "Calvin Kirs", "email": "guoqiang@selectdb.com", "time": "Thu Sep 10 10:13:11 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 10:13:11 2026 +0800" }, "message": "[improvement](hive) Batch Hive metastore partition access (#67186)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: None\n\nProblem Summary:\n\nHive tables with very large partition counts could either issue one HMS\npartition-object RPC per partition on legacy caller paths or send every\npartition name in one unbounded `getPartitionsByNames` request. The\nfirst form creates excessive serial RPC latency; the second risks\nThrift/HMS message limits and large temporary allocations.\n\nThis PR narrows the change to the shared HMS partition-object boundary.\nCallers continue to submit one logical partition-name list through\n`HmsClient#getPartitions`; the existing cache aggregates misses, one HMS\nbatch executor owns bounded chunking, adaptive fallback and strict\nresponse validation, and a leaf transport performs one\n`getPartitionsByNames` invocation per physical attempt. Query,\nstatistics, write and Hive-backed MTMV callers therefore receive the\nsame batching behavior without implementing their own chunk/retry loops.\n\n#### Common HMS batch execution\n\n- `hive.hms_partitions_batch_size_per_rpc` bounds each physical\npartition-object request; the default is 5,000.\n- Explicit message/frame/request-size and partition-limit failures halve\nthe effective batch size until success or the minimum batch size of one.\n- The reduced successful size is reused for the remaining partitions in\nthe logical request.\n- The reduction ladder is naturally bounded by the configured maximum\nand minimum batch sizes. Individual blocking calls continue to use the\nexisting HMS connection/socket timeout; this PR does not advertise a\nseparate fallback wall-clock deadline that cannot interrupt an active\nsynchronous RPC.\n- Ordinary connection outages, authentication/setup failures, malformed\nresults and local failures are not replayed through the halving ladder.\n- Hive\u0027s standard `hive.metastore.limit.partition.request` / “partitions\nscanned ... exceeds limit” failure is recognized.\n- With `hive.metastore.client.pool.size\u003d0`, successful chunks in one\nlogical request reuse one temporary HMS client. A failed physical call\ntaints and destroys that client before a fallback attempt creates\nanother.\n- Hive and Hudi bind and validate the same batch-size setting through\n`HmsClientConfig`.\n- Batch request and transport types remain package-private HMS\nimplementation details.\n\n#### Strict result integrity\n\n- Requested names are parsed once per layer into canonical ordered\npartition-value identities.\n- Duplicate request identities and inconsistent partition-key layouts\nfail before HMS access.\n- Every physical response is checked for missing, duplicate, unexpected,\nnull and invalid-arity partition objects.\n- HMS response order is not trusted; a valid response is reconstructed\nin exact request order.\n- Any integrity mismatch fails the whole logical request with bounded\ndiagnostics. Partial results are neither returned nor published to\ncache.\n- Mixed cache hit/miss requests fetch all misses in one logical delegate\ncall, rebuild caller order, and retain the existing\ninvalidation-generation fence.\n\n#### Narrow MTMV bulk adapter\n\n- `MTMVRelatedTableIf#getPartitionSnapshots` has a compatibility default\nthat retains the existing scalar loop for non-bulk table\nimplementations.\n- The plugin-driven external-table adapter overrides it and calls the\nconnector bulk freshness API once for the requested table/partition\nunion.\n- Hive implements that bulk API with one logical\n`HmsClient#getPartitions` call; the common executor then splits it into\nbounded physical requests.\n- `MTMVRefreshContext` keeps only a request-scoped table → partition →\nsnapshot cache. It unions mapped base partitions before the existing\nloops in sync, need-refresh, display, persistence and rewrite paths.\n- Persisted partition-name mismatches are rejected locally before remote\nfreshness loading.\n- `MTMVTask` preloads the complete need-refresh union before splitting\nexecution groups, so the default one-partition group size cannot regress\nfirst/manual/COMPLETE refreshes to singleton HMS requests.\n- Existing MTMV mapping semantics, base-version, lock and\npersisted-snapshot lifecycles remain unchanged. Task-captured MVCC pins\nare threaded through mapping/alignment and the bulk loader, and the new\nbulk freshness load runs outside the task\u0027s table locks.\n\nWith the default batch size, a cold 120,000-partition logical object\nrequest becomes 24 bounded requests instead of one 120,000-name request.\nA 160,000-partition Hive-backed MTMV union becomes one logical bulk load\nand 32 bounded physical requests, rather than one object request per\nmapped partition.\n\n#### Query Profile observability\n\n- Hive table scans publish one aggregated `Connector Metadata Access`\nprofile through the existing `ConnectorScanProfile` hook.\n- The profile reports logical requests and requested items, physical\nbatch attempts and items, smallest/largest batch sizes, fallback\nreductions, total logical elapsed time, total batch-call elapsed time,\nand maximum batch-call latency.\n- Partition-batch scan mode aggregates all asynchronous 1,024-partition\nscan batches before publishing the profile.\n- Synchronous planning failures and asynchronous dispatch that stops\nbefore every logical batch is submitted still drain completed metadata\ndiagnostics exactly once; profile-finalization failures do not mask the\nprimary planning failure.\n- Cache hits remain visible as logical requested items with zero\nphysical batch attempts.\n- The implementation returns immutable result-plus-stats data from the\ncommon executor; it does not put observers, callbacks, or mutable\nexecution state into `HmsPartitionRequest`.\n\nScope boundaries:\n\n- This PR targets the master Thrift-HMS path used by Hive/Hudi. Iceberg,\nPaimon and non-HMS metadata protocols keep their own implementations.\n- 4.0/4.1 backports require separate path-specific changes and\nvalidation.\n- Query cancellation/deadline propagation through name listing,\nauthentication, pool/client creation, retry and active wire calls is out\nof scope.\n- A separate fallback wall-clock deadline is also out of scope;\nimplementing one correctly requires the same client-taint and late-call\ncleanup lifecycle as active-call cancellation.\n- Connector-wide process metrics, source tagging, and non-scan metadata\nspans remain out of scope; this PR adds only lightweight Hive scan\nProfile output through the existing scan-profile SPI.\n- Cache single-flight/admission/progressive publication, statistics\nsampling-policy changes and Cloud MTMV preload policy are out of scope.\n- Split-assignment first-split timeout and `SplitSource` lifecycle\nbehavior are unchanged.\n- Real HMS performance results and their end-to-end scope boundary are\ndocumented below.\n\n### Release note\n\nHive Metastore partition-object access now uses configurable bounded RPC\nbatches, strict response validation, and adaptive fallback for explicit\noversized-request failures. Hive-backed MTMV partition freshness is\naggregated into bulk logical requests before HMS batching. Hive Query\nProfile also shows the resulting partition-batch request shape and\nelapsed time.\n\n#### Deterministic request-shape evidence\n\n| Scenario | Previous / unsafe shape | This PR |\n| --- | --- | --- |\n| Master 120,000-partition object load | 1 unbounded request containing\n120,000 names | 24 requests, each at most 5,000 names |\n| Legacy scalar caller shape, 120,000 objects | Approximately 120,000\nobject requests | 24 bounded object requests |\n| Hive-backed MTMV, 160,000 mapped objects | Approximately 160,000\nsingleton object requests | 1 logical bulk load, split into 32 physical\nrequests |\n| Injected server limit above 625 names | Large request fails | `5000 →\n2500 → 1250 → 625`, then all objects complete |\n| Pool disabled, successful multi-chunk request | A new client per chunk\nin the initial implementation | One temporary client reused for all\nsuccessful chunks |\n\nThese rows describe deterministic orchestration and request shape. Real\nHMS measurements follow.\n\n#### Real HMS performance evidence\n\nBoth measurements used a Hive 2.3.2 Metastore backed by PostgreSQL 9.5.3\nover loopback Thrift TCP. The partition cache was excluded from the\ntimed A/B reads, returned counts and checksums matched, and the default\nphysical batch size was 5,000.\n\nThe large-scale transport benchmark reproduced the exact before/after\nHMS API shapes against 120,000 real partitions. The before side issued\none `getPartitionsByNames(singleton)` call per partition; the after side\nissued bounded `getPartitionsByNames` calls:\n\n| Partitions | Before | This PR shape | Speedup | Latency reduction |\n| ---: | ---: | ---: | ---: | ---: |\n| 1,000 | 4.180 s / 1,000 RPCs | 0.056 s / 1 RPC | 74.24x | 98.653% |\n| 10,000 | 41.670 s / 10,000 RPCs | 0.535 s / 2 RPCs | 77.92x | 98.717%\n|\n| 120,000 | 509.368 s / 120,000 RPCs | 5.622 s / 24 RPCs | 90.60x |\n98.896% |\n\nA separate small-scale run used freshly compiled current-head\n(`cd5db406e18cb5a7808e6c8b393d82fb82684d74`) Doris production classes.\nClass-load tracing confirmed the path `CachingHmsClient -\u003e\nThriftHmsClient -\u003e HmsPartitionBatchExecutor -\u003e real HMS/PostgreSQL`.\nBoth sides preconstructed the same names and warmed the connection\nbefore timing; the before side reproduced the singleton Doris call shape\nand the after side made one logical bulk call:\n\n| Partitions | Before | Current-head batch | Speedup | Latency reduction\n|\n| ---: | ---: | ---: | ---: | ---: |\n| 10 | 65.900 ms / 10 RPCs | 8.119 ms / 1 RPC | 8.12x | 87.680% |\n| 100 | 552.685 ms / 100 RPCs | 16.462 ms / 1 RPC | 33.57x | 97.021% |\n| 1,000 | 4,657.364 ms / 1,000 RPCs | 64.102 ms / 1 RPC | 72.66x |\n98.624% |\n| 5,000 | 22,420.043 ms / 5,000 RPCs | 281.926 ms / 1 RPC | 79.52x |\n98.743% |\n\nThe 120,000-partition run isolates the real HMS transport/database\nbottleneck and does not include Doris conversion. The current-head run\nincludes Doris conversion, cache lookup, identity parsing, strict\nreordering and batch execution, but directly invokes the production\nclasses rather than running a full FE/BE SQL or MTMV refresh. End-to-end\nimprovement therefore still depends on the share of refresh/query\nlatency originally spent in HMS metadata access.\n\n\n#### Validation\n\n- Latest review increment: 13 HMS batch-executor tests and 20\nPluginDriven scan batch/profile tests passed; Hive/Hudi catalog-property\ntests also passed.\n\n- 111 focused FE-core tests passed: MTMV refresh context, partition\nutilities, rewrite, task, and plugin-driven MVCC table paths.\n- 72 focused connector tests passed: HMS batching/cache/Thrift\nintegration, Hive freshness, and connector SPI surface.\n- The final no-cache 60-module Maven `validate` reactor passed with zero\nCheckstyle violations.\n- `git diff --check` passed.\n- Effective PR diff against its master base: 43 files, 3,086 additions\nand 207 deletions, excluding the uncommitted design/review documents.\n- Three independent final review scopes converged with no new P1/P2\nfindings after fixing task preloading, pool-disabled client reuse, and\nHive\u0027s standard partition-limit classifier.\n- The standard targeted FE test runner compiled the current-head\n60-module reactor successfully after the worktree\u0027s standard prebuilt\nthird-party package was restored. `HmsPartitionBatchExecutorTest` ran 13\ntests with no failures or errors.\n#### Full Doris MTMV refresh version A/B\n\nA real binary-version A/B ran `REFRESH MATERIALIZED VIEW ... COMPLETE`\nagainst a Hive table with 100 partition metadata rows and empty S3\nprefixes in local MinIO. The before side used the released Doris\n4.1.3-rc02 FE (`7126cf65d96`); the after side used this PR’s FE\n(`cd5db406e18cb5a7808e6c8b393d82fb82684d74`). Both sides used the same\nrunning Doris 4.1.3-rc02 BE, Hive Metastore, PostgreSQL, MinIO data,\ncatalog properties, and MV definition. To preserve the exact MV and\ncluster state, current-head FE started from a copy of the measured 4.1.3\nFE metadata and upgraded it in place.\n\nBoth Doris partition caches were disabled. Each FE version had one\nexcluded warm-up followed by five serialized measured COMPLETE\nrefreshes.\n\n| FE version | Five measured refreshes | Mean | Median | Physical\npartition-object RPCs / refresh |\n| --- | --- | ---: | ---: | ---: |\n| Doris 4.1.3-rc02 (`7126cf65d96`) | 2641, 2317, 2410, 2299, 2143 ms |\n2362.0 ms | 2317 ms | 300 `get_partition` |\n| PR head (`cd5db406e18`) | 661, 648, 515, 598, 525 ms | 589.4 ms | 598\nms | 3 `getPartitionsByNames` |\n\nThis is a measured **4.01x full-refresh speedup**, **75.05% mean latency\nreduction**, and **100x physical RPC reduction (99%)**. The three\nremaining batched calls are the refresh path’s three independent\npartition-metadata stages; each stage loads all 100 objects in one\nphysical request, whereas 4.1.3 issues 100 scalar requests per stage.\nDurations are FE task-start to `MTMVService.refreshComplete` timestamps;\nRPC counts and method names come from the Hive Metastore log.\n\nEnvironment: same running Doris 4.1.3-rc02 BE (`7126cf65d96`) for both\nsides; Hive Metastore 2.3.2 + PostgreSQL 9.5.3 + MinIO on the same host.\nEmpty partition data deliberately isolates metadata/MV orchestration\nfrom file-scan cost, so this is a real end-to-end MTMV refresh version\nA/B but not a representative data-scan benchmark.\n\nScaling note: with `N` partitions, Doris 4.1.3 performs approximately\n`3N` remote partition-object calls in this refresh path. This PR\nperforms approximately `3 × ceil(N / batchSize)` calls (the default\nbatch size is 5,000). Both versions still read, deserialize, and process\n`N` partition objects, so total work retains an O(N) component; the\nimprovement removes the per-partition network round trips rather than\nmaking refresh time constant. The speedup therefore generally grows with\npartition count until HMS serialization, Doris object processing, or MV\npartition work becomes dominant." }, { "commit": "f47b64d80c15f76c13355ce492976012b2447d5f", "tree": "76a120c0014d16961bb3b0d618849e5eb96bdc58", "parents": [ "8966bdfc699cd67ffe9b5eabda861edaeaabc1ee" ], "author": { "name": "Chenyang Sun", "email": "sunchenyang@selectdb.com", "time": "Thu Sep 10 10:08:20 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 10:08:20 2026 +0800" }, "message": "[refactor](storage) Merge SegmentWriter into VerticalSegmentWriter (#67468)\n\n1. This PR keeps one class, `VerticalSegmentWriter`, built from\n`SegmentWriter`\u0027s column group protocol plus the load path\u0027s whole-block\nfeed:\n\n```cpp\nfor (column_group : column_groups) {\n writer.init(column_group, has_key);\n writer.append_block(block, ...); // any number of times\n writer.finalize_columns(\u0026index_size); // data pages, index sections, key indexes\n}\nwriter.finalize_footer(\u0026file_size);\n```\n\nA load flush hands over one whole-schema group at once.\n\n```cpp\nwriter.set_derived_column(derived_column); // only if the table has a row store column\nwriter.write_block(block, 0, num_rows); // opens the group itself\nwriter.finalize_columns(\u0026index_size);\nwriter.finalize_footer(\u0026file_size);\n```\n\n2. **`config::enable_vertical_segment_writer` is gone**, with its fuzzy\nentry\n\nIssue Number: close #xxx\n\nRelated PR: #xxx" }, { "commit": "8966bdfc699cd67ffe9b5eabda861edaeaabc1ee", "tree": "b29a729e925a9e268727a128c152cd239c807222", "parents": [ "204eca6591b0d62c5349295394f33f4be2dc4dfb" ], "author": { "name": "Mingyu Chen (Rayner)", "email": "morningman.cmy@gmail.com", "time": "Thu Sep 10 09:59:22 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 09:59:22 2026 +0800" }, "message": "[fix](regression) Deflake test_sql_cache_over_arrow_flight (#67734)\n\n### What problem does this PR solve?\n\nIssue Number: close #xxx\n\nRelated PR: #67381\n\nProblem Summary:\n\n`arrow_flight_sql_p0/test_sql_cache_over_arrow_flight` is flaky. It\nprimes four sql\ncache entries through the MySQL control session, runs the same\nstatements over Arrow\nFlight, and then asserts the entries are still there, to prove that a\nflight query\nnever consumes the cache. That closing block fails intermittently on\nboth master and\nbranch-4.1 — six times since the suite was added on 2026-09-02, most\nrecently in p0\nbuild 124525, where the `select 1 as c, \u0027x\u0027 as s` entry was gone\n**299ms** after it had\nbeen primed:\n\n```\nException in arrow_flight_sql_p0/test_sql_cache_over_arrow_flight.groovy(line 163):\n assertTrue(hasSqlCache(constantSql))\norg.opentest4j.AssertionFailedError: expected: \u003ctrue\u003e but was: \u003cfalse\u003e\n```\n\n**Root cause.** The FE sql cache is a single Caffeine map shared by\nevery session\n(`NereidsSqlCacheManager.sqlCaches`), bounded by\n`Config.sql_cache_manage_num`, which\ndefaults to **100**. Caffeine admits a newcomer through a window sized\nat **1% of that\nbound**, so at the default the admission window holds a single entry: a\njust cached\nstatement has frequency ~1 and loses the admission contest to an\nestablished victim as\nsoon as any other session caches anything.\n`SessionVariable.enableSqlCache` defaults to\n`true`, so the rest of the p0 suite running concurrently against the\nsame FE is already\nenough to evict it.\n\nThe audit log rules out ordinary LRU pressure: only **15 distinct\nselects** ran cluster\nwide during that 299ms window, far fewer than the 100 an LRU would have\nneeded. A local\nrun against caffeine 3.2.4 reproduces the admission behaviour directly:\n\n```\nmaximumSize\u003d100 1 other insert after mine -\u003e survived 7/20\nmaximumSize\u003d100 15 other inserts after mine -\u003e survived 8/20\nmaximumSize\u003d10000 1 other insert after mine -\u003e survived 20/20\nmaximumSize\u003d10000 15 other inserts after mine -\u003e survived 20/20\n```\n\nThis is not an FE bug — the sql cache is best effort and gives no\nretention guarantee.\nIt is the suite asserting a property the cache does not provide. The\nother five sql\ncache suites in the repo (`mv_with_sql_cache`, `mtmv_with_sql_cache`,\n`parse_sql_from_sql_cache`, `union_all_compensate`,\n`union_rewrite_grace_big`) already\nraise the bound at their start for exactly this reason; this one was\nmissing it. In p0\nbuild 124525 the failing suites ran at 21:54 and 22:01, before any of\nthose suites\nraised the bound at 22:12.\n\n**Fix.** Raise `sql_cache_manage_num` to 10000 while the suite runs, and\nrestore the\nprevious value afterwards. The restore is deliberate: this suite runs\nabout half an\nhour earlier in the p0 run than the five existing ones, and the ones\nthat raise the\nbound without restoring it left **97k live `SqlCacheContext` instances**\nin the FE heap\nin the same build (post-GC live heap peaked at 4G of 8G in that window).\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test\n - [x] Regression test\n - [ ] Unit Test\n - [ ] Manual test (add detailed scripts or steps below)\n - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n - [ ] Previous test can cover this change.\n - [ ] No code files have been changed.\n - [ ] Other reason\n\n- Behavior changed:\n - [x] No.\n - [ ] Yes.\n\n- Does this need documentation?\n - [x] No.\n - [ ] Yes.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label\n\n🤖 Generated with [Claude Code](https://claude.com/claude-code)\n\nhttps://claude.ai/code/session_01Njd8iDxdqc19QbLdNtZ7Pt\n\nCo-authored-by: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "204eca6591b0d62c5349295394f33f4be2dc4dfb", "tree": "483697b83ba3f288cb851c211dc2f3335d3867a0", "parents": [ "0557668f40516aa11458ce663006f63474412917" ], "author": { "name": "yiguolei", "email": "guolei@selectdb.com", "time": "Thu Sep 10 07:31:50 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 07:31:50 2026 +0800" }, "message": "[bugfix](glibc) fix glibc compatible bugs (#67701)\n\n### What problem does this PR solve?\n\nRust std weak-links this glibc 2.18 entry point and has an internal\nfallback when it is absent. Since the LDB sysroot exposes it, the final\nlinker would otherwise record GLIBC_2.18.\n\nSee https://github.com/rust-lang/rust/issues/57497 for more details.\n\nI have checked the output.\n\nobjdump -T output/be/lib/doris_be | grep -o \u0027GLIBC_[0-9.]*\u0027 | sort -Vu |\ntail\nGLIBC_2.3\nGLIBC_2.3.2\nGLIBC_2.3.3\nGLIBC_2.3.4\nGLIBC_2.4\nGLIBC_2.5\nGLIBC_2.7\nGLIBC_2.8\nGLIBC_2.9\nGLIBC_2.10\n\nRelated PR: #xxx\n\nProblem Summary:\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test \u003c!-- At least one of them must be included. --\u003e\n - [ ] Regression test\n - [ ] Unit Test\n - [ ] Manual test (add detailed scripts or steps below)\n - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n - [ ] Previous test can cover this change.\n - [ ] No code files have been changed.\n - [ ] Other reason \u003c!-- Add your reason? --\u003e\n\n- Behavior changed:\n - [ ] No.\n - [ ] Yes. \u003c!-- Explain the behavior change --\u003e\n\n- Does this need documentation?\n - [ ] No.\n- [ ] Yes. \u003c!-- Add document PR link here. eg:\nhttps://github.com/apache/doris-website/pull/1214 --\u003e\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label \u003c!-- Add branch pick label that this PR\nshould merge into --\u003e" }, { "commit": "0557668f40516aa11458ce663006f63474412917", "tree": "06757e2be30f2a6094cd028a3650678183641288", "parents": [ "bfe46ec2ca50cdd1d2c92976676b0eaae7c33c54" ], "author": { "name": "Luwei", "email": "814383175@qq.com", "time": "Wed Sep 09 20:42:12 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 20:42:12 2026 +0800" }, "message": "[fix](cloud) Exclude covered rowsets from compaction minimum timestamps (#67617)\n\nProblem Summary:\n\nVersioned rowset reads add load metadata versions to the reader minimum\nbefore compact rowsets remove covered loads from the returned set. With\nloads L2[2,2]@80 and L3[3,3]@90 covered by A[2,3]@150, plus L4[4,4]@170,\nreading [2,4] returns A and L4 but records a minimum of 80. Later\ncompaction logs can consequently retain successive compacted rowsets for\na snapshot that only needs L2/L3.\n\nKeep each candidate\u0027s metadata Versionstamp alongside its rowset, then\nmerge only the final returned candidates into the existing minimum after\nboth scans succeed. The returned set, scan/coverage rules, snapshot\nbounds and error propagation are unchanged. Previous stats and other\nreal dependencies remain tracked, and persisted historical logs are\nunchanged.\n\nThe regression exercises two real MetaService compactions and the\nRecycler reference checker using MemTxnKv: snapshot@100 still protects\nthe first log containing L2/L3, while the later log has\nmin_timestamp\u003d150 and is no longer protected by that snapshot.\n\n### Release note\n\nFix unnecessary retention of later compacted rowsets when snapshots\nprotect older load rowsets already covered by compaction." }, { "commit": "bfe46ec2ca50cdd1d2c92976676b0eaae7c33c54", "tree": "0136f8e93735892c72c7f43d50b2b8c80b9a5586", "parents": [ "00b08ad55455ccaca77d719a978772a302357569" ], "author": { "name": "Luwei", "email": "814383175@qq.com", "time": "Wed Sep 09 20:40:44 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 20:40:44 2026 +0800" }, "message": "[fix](row binlog) make time-based incremental reads use a safe fence (#67594)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #65850\n\nProblem Summary:\n\nTime-based `@incr` reads previously allowed an `endTimestamp` later than\nthe FE\u0027s current TSO. Such a future boundary cannot be closed by the\ntransaction waiting logic: a transaction may start after the request\nbegins but still fall before that future boundary, so the scan cannot\nguarantee a complete result.\n\nIn addition, waiting only for currently running transactions is\ninsufficient unless the TSO boundary, transaction watermark, transaction\nvisibility, and table visible versions are coordinated under one\nauthoritative fence. Otherwise, especially on a follower FE, a\ntransaction can already be `VISIBLE` while the local catalog still\nexposes an older visible version, causing the incremental scan to miss\nrows.\n\nThis PR establishes a safe read fence for both classic and cloud modes:\n\n1. Acquire the current TSO from the master FE.\n2. Capture the transaction ID watermark after the TSO snapshot.\n3. Wait for transactions at or below the watermark that involve the\nscanned OLAP tables to reach a final visible or aborted state.\n4. In classic mode, synchronize table publishers and wait for follower\njournal replay so catalog visible versions cross the same fence.\n5. In cloud mode, refresh the latest visible versions from MetaService\nafter the transaction wait.\n6. Reject an `endTimestamp` later than the captured\n`CURRENT_TSO_PHYSICAL_TIME`.\n7. Convert a physical timestamp `P` to TSO boundary `(P, 0)`, preserving\nhalf-open range semantics: `[startTimestamp, endTimestamp)`.\n\n#### Future endTimestamp error example\n\nRequest:\n\n```sql\nSELECT id, value, __DORIS_BINLOG_OP__\nFROM example_table@incr(\n \"startTimestamp\" \u003d \"2026-09-01 00:00:00\",\n \"endTimestamp\" \u003d \"2999-01-01 00:00:00\",\n \"incrementType\" \u003d \"DETAIL\"\n);\n```\n\nResponse:\n\n```text\nERROR 1105 (HY000): errCode \u003d 2, detailMessage \u003d\nendTimestamp exceeds the maximum supported time for an INCR read:\nrequestedEndTimestampMs\u003d\u003crequested epoch milliseconds\u003e,\nCURRENT_TSO_PHYSICAL_TIME\u003d\u003cmaximum supported epoch milliseconds\u003e\n```\n\nThe maximum currently supported boundary can be obtained from:\n\n```sql\nSELECT CURRENT_TSO_PHYSICAL_TIME\nFROM information_schema.tso_status;\n```\n\n### Release note\n\nTime-based `@incr` reads now reject `endTimestamp` values after\n`CURRENT_TSO_PHYSICAL_TIME`. Timestamp ranges use half-open semantics\n`[startTimestamp, endTimestamp)`, with each physical timestamp mapped to\nlogical counter zero.\n\n### Check List (For Author)\n\n- Test: FE unit tests, FE and BE builds, and\n`row_binlog_p0/test_binlog_changes_syntax` regression test\n- Behavior changed: Yes\n- Future `endTimestamp` values are rejected with the maximum supported\nTSO physical time in the error message.\n- Physical timestamp boundaries are interpreted as `(physicalTime, 0)`.\n- Does this need documentation: No" }, { "commit": "00b08ad55455ccaca77d719a978772a302357569", "tree": "491ac989c6db42fdfca8e47399f368fdc86d1d00", "parents": [ "2f6358db28ddec0d304fc84dff5934aaed2ca91c" ], "author": { "name": "morrySnow", "email": "zhangwenxin@selectdb.com", "time": "Wed Sep 09 18:04:42 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 18:04:42 2026 +0800" }, "message": "[fix](window) Respect frames when simplifying window functions (#67706)\n\n## Problem\n\nThe optimizer simplifies window functions when the partition keys are\nunique because each partition contains at most one row. It replaces\nfunctions such as `SUM(value)` with `value` and a non-null `COUNT` with `1`.\n\nThat replacement produces incorrect results when the window frame\nexcludes the current row. For a one-row partition, a preceding-only or\nfollowing-only frame is empty, so `SUM` must return `NULL` and `COUNT` must return `0`.\n\n## Root cause\n\nThe simplification rule used partition cardinality alone and did not\ncheck whether the normalized frame actually contains the partition\u0027s only row.\n\n## How to reproduce\n\nCreate a merge-on-write unique-key table with one row and run `SUM` and\n`COUNT(*)` over a window partitioned by the unique key with `ROWS\nBETWEEN 1 PRECEDING AND 1 PRECEDING` (or the equivalent following-only\nframe). Before this change, the optimizer removed the window and\nreturned the current value and `1`; the correct result is `NULL` and `0`.\n\n## Fix\n\nCheck the normalized frame boundaries before simplifying frame-dependent\nfunctions. `COUNT`, `SUM`, `MIN`, `MAX`, `AVG`, `FIRST_VALUE`, and\n`LAST_VALUE` are simplified only when the frame contains the current\nrow. Ranking functions retain their existing simplification because\ntheir result does not depend on frame membership.\n\n## Tests\n\n- Added a regression suite covering preceding-only and following-only\nframes that must retain `PhysicalWindow` and return `NULL`/`0`.\n- Added current-row and centered-frame cases that continue to eliminate\n`PhysicalWindow` and return the simplified values.\n- Ran the new regression suite in verification mode: 1 suite passed, 0\nfailed.\n- Built the FE successfully with all reactor modules passing and no\ncheckstyle violations." }, { "commit": "2f6358db28ddec0d304fc84dff5934aaed2ca91c", "tree": "bc40a3a5dfdd2b79a0376cfe3d6ce685f17a0c6f", "parents": [ "1335022902a85d83fcfd2f82094b211b5e1b8edc" ], "author": { "name": "morrySnow", "email": "zhangwenxin@selectdb.com", "time": "Wed Sep 09 18:04:08 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 18:04:08 2026 +0800" }, "message": "[fix](aggregate) Guard aggregate not-null inference by null semantics (#67668)\n\n### What problem does this PR solve?\n\nProblem Summary: `InferAggNotNull` reused predicate-style not-null\ninference for aggregate arguments. This is unsafe across two separate\nNULL-semantics boundaries:\n\n1. Replacing a slot with NULL may fold an expression to FALSE or another\nnon-NULL value. FALSE rejects a filter predicate, but remains a valid\naggregate argument. For example, `COUNT(r.v IS NOT NULL)` must count\nnull-extended outer-join rows.\n2. Not every aggregate ignores NULL input rows. For example, `ARRAY_AGG`\nretains NULL as an array element, so adding `r.v IS NOT NULL` changes\nits result.\n\nThis change separates aggregate argument inference from predicate\ninference. Aggregate inference accepts only a folded SQL NULL result,\nnot FALSE. It also adds an explicit null-input contract: only `COUNT`,\n`AVG`, `SUM`, `MAX`, and `MIN` opt in because they ignore rows with NULL\narguments. Every other current or future aggregate is excluded by\ndefault unless it explicitly declares the same contract. If a global\naggregate contains any non-opted-in function, no common not-null filter\nis generated.\n\nAfter the fix:\n\n- `COUNT(r.v IS NOT NULL)` and `COUNT(COALESCE(r.v, 0))` keep the left\nouter join and return 3.\n- `COUNT(r.v)` still infers not-null, changes the join to inner, and\nreturns 1.\n- `COUNT(r.v), ARRAY_SIZE(ARRAY_AGG(r.v))` keeps the left outer join and\nreturns `1, 3`.\n\n### Release note\n\nFix aggregate not-null inference so it does not discard FALSE values or\nNULL inputs required by NULL-sensitive aggregates." }, { "commit": "1335022902a85d83fcfd2f82094b211b5e1b8edc", "tree": "d043688fd7b60bfa0efc4b38afef59c9edce5417", "parents": [ "2e89f3a81cbf0effe06491a383109ab36ec72923" ], "author": { "name": "morrySnow", "email": "zhangwenxin@selectdb.com", "time": "Wed Sep 09 17:06:57 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 17:06:57 2026 +0800" }, "message": "[fix](variable) Apply sql_select_limit to materialized CTE results (#67656)\n\n### What problem does this PR solve?\n\nProblem Summary:\n\nWhen a CTE was materialized, `AddDefaultLimit` rebuilt the\n`LogicalCTEAnchor` without visiting its result child. As a result, the\nquery result could bypass `sql_select_limit`, for example when two\n`UNION ALL` branches referenced the same CTE.\n\nThis change visits the CTE result branch while preserving the producer.\nIt also adds a unit test for the rewritten plan shape and a regression\ntest covering the materialized CTE query result.\n\n### Release note\n\nFix `sql_select_limit` not taking effect for queries using materialized\nCTEs." }, { "commit": "2e89f3a81cbf0effe06491a383109ab36ec72923", "tree": "fe3bcfefaf0f2ca829c2af634df9ff2ff6329d5a", "parents": [ "ec886f33e1ebab98f5751c6279bec32ed7f75287" ], "author": { "name": "morrySnow", "email": "zhangwenxin@selectdb.com", "time": "Wed Sep 09 16:19:12 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 16:19:12 2026 +0800" }, "message": "[fix](mv) Prevent snapshot reads from using current MVs (#67700)\n\n## Problem\n\nAn external-table query that explicitly reads an older snapshot can\nstill enter materialized view rewrite. The candidate materialized view\nrepresents the table state at its refresh snapshot, so using a current\nmaterialized view for a historical query can silently return data from\nthe wrong point in time.\n\n## Root cause\n\n`LogicalFileScan` stores standard `FOR TIME AS OF` and `FOR VERSION AS\nOF` clauses in `tableSnapshot`. The materialized-view eligibility\nchecker rejected scans with table samples or scan parameters, but did\nnot inspect `tableSnapshot`. As a result, the historical scan was\ntreated like an ordinary latest-snapshot scan.\n\n## How to reproduce\n\n1. Create an Iceberg table and insert an initial row, then record that\nsnapshot ID.\n2. Insert newer data and refresh a materialized view over the table at\nthe current snapshot.\n3. Enable materialized-view rewrite and query the Iceberg table with\n`FOR VERSION AS OF \u003cold_snapshot_id\u003e` (the same issue applies to `FOR\nTIME AS OF`).\n4. Before this change, the historical query can be considered eligible\nfor rewrite by the current-snapshot materialized view, producing current\nrather than historical results.\n\nThe same condition can be reproduced directly in the optimizer by\nbuilding a `LogicalFileScan` with a non-empty `tableSnapshot`: the\ntable-query-operator checker previously returned false.\n\n## Fix\n\nTreat a non-empty `LogicalFileScan.tableSnapshot` as a table-level query\noperator, alongside table samples and scan parameters. This\nconservatively prevents materialized-view rewrite until the optimizer\ncan prove that the query snapshot and materialized-view refresh snapshot\nare semantically equivalent.\n\nAdd a unit test that constructs a file scan with a version snapshot and\nverifies that the checker rejects it from ordinary rewrite eligibility." }, { "commit": "ec886f33e1ebab98f5751c6279bec32ed7f75287", "tree": "74fbe9c0445c08ddf9cbc5bc8a06474aeb5d9359", "parents": [ "62327d3a0054f1ffadc93ee2900a5fe0ddf5405c" ], "author": { "name": "minghong", "email": "zhouminghong@selectdb.com", "time": "Wed Sep 09 15:54:22 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 15:54:22 2026 +0800" }, "message": "[fix](nereids) Disambiguate NULL/OFFSET metadata from same-named nested fields (#65805)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #65591, #66380\n\nProblem Summary:\n\nNested-column pruning (feature series #59263 / #61888 / #64535) makes\nDoris read only the referenced parts of a struct/map/array column. Two\nkinds of special *meta* access paths serve that purpose: a terminal\n`NULL` component means the query only needs the null flag (`IS [NOT]\nNULL` on a column or nested field), and a terminal `OFFSET` component\nmeans only offsets are needed (`length`/`cardinality`-style functions).\nThe BE satisfies them in `NULL_MAP_ONLY` / `OFFSET_ONLY` meta-read modes\nthat skip the payload data entirely. While the optimization itself is\nsound, the way meta paths were expressed on the wire and handled on both\nsides had several correctness holes:\n\n* **Wire ambiguity.** Every access path was sent as a `DATA`-typed list\nof string components, and `NULL`/`OFFSET` lived in the same list as real\nchild names. There was no way to tell a metadata selector apart from a\nstruct field literally named `OFFSET` or `NULL`, so such a field could\nbe swallowed as metadata and its data silently pruned away (wrong query\nresults), and there was no FE/BE version contract to evolve the encoding\nduring rolling upgrades.\n* **FE null-map correctness.** A slot can be nullable for two different\nreasons: its physical column has a null bitmap, or an outer join merely\nmade the output slot nullable. The old logic keyed on expression/slot\nnullability, so a physical `NOT NULL` dimension column of a `LEFT JOIN`\nreceived a `[col, NULL]` path and the BE aborted trying to read a null\nmap from a `NOT NULL` column. Meta paths were also generated for\nnon-nullable struct fields / non-null structs and for variant\nsub-columns, which do not support meta-only reads.\n* **FE pruning bookkeeping.** `NULL`/`OFFSET` were detected by string\nsuffix, so real fields with those names collided inside the pruning\nlogic too; and predicate meta paths were stripped or broadened\n(map-star) whenever regular data paths existed, losing metadata-only\nreads or over-reading payload data.\n* **BE routing/read-mode logic.** A terminal `NULL`/`OFFSET` was treated\nas current-level metadata at every container, meta-only modes could be\nentered even when a sibling `DATA` path still required payload reads,\nsibling `META` paths were dropped when unrelated data projections\nexisted, and Map descendant routing depended on physical key/value child\ncolumn names while silently ignoring unknown selectors.\n\nWhat this PR does:\n\n* **Versioned, type-selected access paths.** Adds an optional `version`\nto `TColumnAccessPath` (thrift + protobuf + FE/BE descriptor conversion;\n`TCOLUMN_ACCESS_PATH_VERSION_LEGACY \u003d 0`,\n`TCOLUMN_ACCESS_PATH_VERSION_TYPED \u003d 1`). In the typed format the path\ntype is authoritative: `DATA` selects `data_access_path`, `META` selects\n`meta_access_path`, and `NULL`/`OFFSET` are only ever emitted as typed\n`META` paths. New BEs still decode the legacy all-`DATA` encoding from\nold FEs, so the supported rolling upgrade is: upgrade all BEs first,\nthen let FEs send typed paths.\n* **FE** (`AccessPathExpressionCollector`, `NestedColumnPruning`,\n`AccessPathPlanCollector`, `DescriptorToThriftConverter`): marks\n`NULL`/`OFFSET` collector contexts as `META`; checks the physical\ncolumn\u0027s nullability (`getOriginalColumn().isAllowNull`) instead of slot\nnullability before emitting null-map paths, so outer-join-nullable `NOT\nNULL` columns and non-nullable fields/structs no longer get `NULL`\npaths; emits the parent-struct `NULL` path for `element_at(...) IS NULL`\nonly when actually needed; falls back to plain data reads for variant\nsub-columns; keeps the exact meta path when a sibling data path is also\nread; preserves fields referenced by unchanged predicates; and keys\npruning on the path type instead of string suffixes, rejecting\nunsupported `META` paths.\n* **BE** (`ColumnIterator` / `column_reader.cpp`): validates path\nversions and type-selected payloads; partitions current data / current\nmetadata / descendant routing with explicit per-container ownership\n(Struct owns the null map; Map/Array and string-like scalar columns own\nboth null map and offsets); derives `NULL_MAP_ONLY` / `OFFSET_ONLY` only\nwhen no current or predicate data path requires payload, and marks data\nchildren `SKIP` consistently; keeps sibling `META` paths explicit so an\nunrelated `DATA` projection cannot disable a metadata-only predicate\nread; routes Map children by the logical `KEYS`/`VALUES` selectors\n(wildcard expansion, arbitrary nesting) independent of physical child\ncolumn names; rejects unrecognized Map selectors instead of silently\npruning everything; and documents the full routing flow.\n* **Tests \u0026 docs.** BE `ColumnReaderTest` coverage for typed-vs-legacy\nencoding, metadata-vs-same-named-data disambiguation, meta-read-mode\nlegality, sibling-data-path handling and Map selector validation;\n`SlotDescriptor` protobuf round-trip of versions and payloads; FE\n`PruneNestedColumnTest` / `DescriptorToThriftConverterTest` cases; the\n`nereids_rules_p0/column_pruning` regression suites updated to assert\nthe typed `META` paths, plus the new `left_join_not_null_column` suite\nguarding the outer-join null-map crash.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test:\n- FE unit test: `./run-fe-ut.sh --run\n\u0027org.apache.doris.nereids.rules.rewrite.PruneNestedColumnTest,org.apache.doris.analysis.DescriptorToThriftConverterTest\u0027`\n(74 passed on the final rebased FE tree)\n- BE unit test: `./run-be-ut.sh --run\n--filter\u003d\u0027ColumnReaderTest.*:SlotDescriptorTest.AccessPathsPreservedThroughProtobuf\u0027`\n(50 passed before the final rebase; the final rebuild was blocked before\nGTest by the missing master-added\n`thirdparty/installed/lib64/liblance_c.a` in this environment)\n- Regression test: `nereids_rules_p0/column_pruning` —\n`string_length_column_pruning`, `null_column_pruning`,\n`nested_container_offset_pruning`, `lambda_null_pruning`,\n`left_join_not_null_column` (passed before the final rebase; the final\nrerun was blocked by the same missing dependency)\n - Format checks: `build-support/check-format.sh`, `git diff --check`\n- Behavior changed: Yes. Typed `META` access paths with an explicit\nversion replace the legacy all-`DATA` encoding between new FEs and BEs\n(upgrade BEs before FEs); struct fields literally named `NULL`/`OFFSET`\nare now read correctly instead of being pruned as metadata; `IS NULL` on\nouter-join-nullable `NOT NULL` columns no longer crashes the BE; invalid\nMap descendant selectors now return an internal error instead of being\nsilently ignored.\n- Does this need documentation: No\n\n---------\n\nCo-authored-by: Hu Shenggang \u003chushenggang@selectdb.com\u003e" }, { "commit": "62327d3a0054f1ffadc93ee2900a5fe0ddf5405c", "tree": "36f57975fe0d469dd0d16d8cca55971e5e6c20be", "parents": [ "acd7e31685e9160395224f35c823373cb10436d8" ], "author": { "name": "morrySnow", "email": "zhangwenxin@selectdb.com", "time": "Wed Sep 09 15:51:58 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 15:51:58 2026 +0800" }, "message": "[opt](ci) update mtmv code owner (#67526)\n\nupdate mtmv code owner" }, { "commit": "acd7e31685e9160395224f35c823373cb10436d8", "tree": "5a1cdd5ea12a61bf3d0aacc02fc8def2cd636355", "parents": [ "3763638e6c8e3ebbc1b7079266f0d97213f0278d" ], "author": { "name": "morrySnow", "email": "zhangwenxin@selectdb.com", "time": "Wed Sep 09 15:45:58 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 15:45:58 2026 +0800" }, "message": "[fix](rbo) Preserve nullability in distinct window rewrites (#67666)\n\n### What problem does this PR solve?\n\nProblem Summary:\n\nDISTINCT window rewrites for SUM and GROUP_CONCAT manually reconstructed\ntheir multi-distinct functions with the default non-nullable flag. For\nempty window frames over NOT NULL inputs, the backend consequently\nmaterialized empty aggregate states (0 or an empty string) instead of\nNULL.\n\nThis change reuses each aggregate function\u0027s canonical multi-distinct\nconversion, preserving the always-nullable contract established during\nwindow-function analysis. COUNT(DISTINCT) behavior remains unchanged.\n\n### Release note\n\nFix SUM(DISTINCT ...) and GROUP_CONCAT(DISTINCT ...) window functions to\nreturn NULL for empty frames." }, { "commit": "3763638e6c8e3ebbc1b7079266f0d97213f0278d", "tree": "76c6aaed6f7f0168b2431d82b15b937e45f2bf4f", "parents": [ "18967f28a3976e73c236b4db02fa53ffa9ed8e92" ], "author": { "name": "morrySnow", "email": "zhangwenxin@selectdb.com", "time": "Wed Sep 09 15:02:30 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 15:02:30 2026 +0800" }, "message": "[improvement](parser) Avoid empty identifier suffix contexts (#67685)\n\n### What problem does this PR solve?\n\nProblem Summary:\n\nThe `errorCapturingIdentifier` rule always entered a nullable helper for\nvalid identifiers. In the tracked SQL corpus this created 25,406 empty\nCST contexts. Move the optionality to the caller and make\n`errorCapturingIdentifierExtra` non-empty, while preserving the existing\nunquoted-identifier error action. This avoids allocating an empty\n`RealIdentContext` on the common path and keeps the grammar structure\nstraightforward.\n\nThis PR also removes stale `PostProcessor` references from two JMH\nharnesses. Identifier post-processing already moved into grammar actions\nand the class no longer exists, so those references prevented the\nbenchmark profile from compiling.\n\nAdd a multipart-identifier JMH workload and a direct CST regression\ntest. Across two JMH runs, normalized allocation for that workload\ndecreased from 174,368.9 to 159,004.1 B/op (-8.81%) end-to-end and from\n146,097.7 to 130,741.3 B/op (-10.51%) with pre-tokenized input." }, { "commit": "18967f28a3976e73c236b4db02fa53ffa9ed8e92", "tree": "04781bbf8e6c1659204d9e262f468f15b2e49f42", "parents": [ "300d532864e453fd237b8dca0ddd5a2624231b6f" ], "author": { "name": "hui lai", "email": "laihui@selectdb.com", "time": "Wed Sep 09 12:58:20 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 12:58:20 2026 +0800" }, "message": "[fix](test) Deflake insert overwrite auto detect failure (#67571)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #64580\n\nProblem Summary: INSERT OVERWRITE auto-detect must fail when source rows\ndo not match an existing target partition and automatic partition\ncreation is disabled. With multiple parallel sink instances, several\nequivalent failure messages can race to reach the client. The regression\ntest accepted only one message, making repeated executions flaky. Accept\nthe known equivalent errors while still requiring every statement to\nfail.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test: Regression test updated; not run locally as requested, CI will\nexecute it\n- Behavior changed: No\n- Does this need documentation: No" }, { "commit": "300d532864e453fd237b8dca0ddd5a2624231b6f", "tree": "ab85558b3d8ceba678d97e29fcffbc798d5e7bf2", "parents": [ "6fe0b1904bb408d0fcacfac7ee0a9e67d75f0f40" ], "author": { "name": "TsukiokaKogane", "email": "cby141994@gmail.com", "time": "Wed Sep 09 12:35:21 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 12:35:21 2026 +0800" }, "message": "[fix](table stream) Preserve table stream offsets during cleanup (#67533)\n\n### What problem does this PR solve?\n\nIssue Number: close #67094" }, { "commit": "6fe0b1904bb408d0fcacfac7ee0a9e67d75f0f40", "tree": "d3375ee64b790a99992491edd4ed611c791780dc", "parents": [ "e4093c3cc2fdf8f59c61297f958ec90dbaf01a74" ], "author": { "name": "foxtail463", "email": "foxtail463@gmail.com", "time": "Wed Sep 09 11:40:54 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 11:40:54 2026 +0800" }, "message": "[improvement](pk lookup) Avoid copying segment key bounds (#67618)\n\nPrimary-key lookup copied each rowset’s segment key bounds into a\ntemporary vector before pruning candidate segments. This caused repeated\nallocation and protobuf object copying on the lookup hot path, with\noverhead increasing alongside the number of segments.\n\nCo-authored-by: yangtao555 \u003cyangtao555@jd.com\u003e" }, { "commit": "e4093c3cc2fdf8f59c61297f958ec90dbaf01a74", "tree": "7f2a908819af78a67bc73c06fab16f7125010328", "parents": [ "85e9da508c71483366bebe6a50e017580b1d4894" ], "author": { "name": "Jamie", "email": "lianyukang@selectdb.com", "time": "Wed Sep 09 10:43:59 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 10:43:59 2026 +0800" }, "message": "[fix](regression) Stabilize compaction regression cases (#67613)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #66337\n\nProblem Summary: Schema change compaction cases created only five\npost-alter versions. After the NOTREADY tablet policy started protecting\nits latest ten versions, no rowsets remained eligible for cumulative\ncompaction, so the cases expected a merge that could no longer occur.\nThe inverted-index cumulative compaction case also assumed that the\nfirst trigger always found input and that an initial run_status\u003dfalse\nmeant completion, which races the BE visible-version refresh.\n\nCreate enough double-write versions to exercise both the eligible and\nprotected ranges, assert the resulting active rowset topology in the\nno-restart, FE-restart, and BE-restart variants, and use completion\nchecks that cannot finish before compaction starts. For the\ninverted-index case, retry only the transient E-2000 response while\nforcing tablet reports and wait for observable rowset reduction." }, { "commit": "85e9da508c71483366bebe6a50e017580b1d4894", "tree": "e794dd4a4ecfc72a14506944c08b8c70966bcd66", "parents": [ "3c83b696c84b00fbc9bcc1ed4d4fadbd3a8a6560" ], "author": { "name": "HappenLee", "email": "happenlee@selectdb.com", "time": "Wed Sep 09 10:34:24 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 10:34:24 2026 +0800" }, "message": "[fix](be) Fix array_range overflow and batch integer output (#67584)\n\n### What problem does this PR solve?\n\nIssue Number: N/A\n\nRelated PR: N/A\n\nProblem Summary:\n\n`sequence(2147483646, 2147483647, 2)` and its `array_range` equivalent\nshould return `[2147483646]`. The current Int32 generator overflows when\nincrementing past the last element and can keep appending instead of\nterminating. The wider preflight size calculation does not protect the\nInt32 generation loop.\n\nUse Int64 distance/cursor arithmetic, check the exact element count\nbefore allocation, resize the nested output once per multi-element row,\nand fill it with a counted loop. Initialize the element null map once\nper block. Empty and singleton fast paths avoid division and\ngeneral-fill overhead for short ranges. The Int32 return type, exclusive\nupper bound, invalid-input NULLs and array-size limit remain unchanged.\n\nFour BE unit tests cover both function names, all constant/vector\nargument combinations, overflow boundaries, default arguments,\nNULL/invalid inputs, mixed growing/empty/NULL rows, and the exact size\nlimit and limit+1.\n\n### Performance\n\nFunction-level Google Benchmark calls through the real function factory,\nincluding output allocation. Both binaries use the same benchmark and\nmaster baseline `eea19b3f3cfef9e1bbbd559f9ea42954d8891e0f`; only the\ngenerator implementation differs.\n\nIntel Xeon Platinum 8457C, Clang 20.1.8, Release `-O3`, AVX2, pinned to\nCPU 24. Each case uses non-constant input columns, `start \u003d row % 17`,\nand the length/step below. Run order: before, after, after, before; 7\nrepetitions per run, minimum 0.3 s measurement and 0.2 s warmup. Values\nare the median CPU time across 14 samples per case.\n\n| Input rows | Elements per array | Step | Before (µs/batch) | After\n(µs/batch) | Speedup |\n|---:|---:|---:|---:|---:|---:|\n| 4096 | 0 | 1 | 9.34 | 9.27 | 1.01× |\n| 4096 | 1 | 1 | 15.64 | 11.37 | 1.38× |\n| 4096 | 16 | 1 | 119.95 | 32.01 | 3.75× |\n| 4096 | 256 | 1 | 2737.57 | 931.27 | 2.94× |\n| 4096 | 1024 | 1 | 20430.45 | 3763.61 | 5.43× |\n| 4096 | 256 | 7 | 2615.14 | 942.04 | 2.78× |\n| 1 | 1000000 | 1 | 1973.84 | 214.92 | 9.18× |\n\nThis is a shared host with CPU scaling enabled, so the empty-array\nresult should be treated as unchanged. These are BE function timings,\nnot end-to-end SQL speedups. Raw samples show higher variation for the\nlarge multi-row allocation cases; the reported values are medians.\n\nReproduce with the registered `BM_ArrayRange` benchmark:\n\n```bash\nBUILD_TYPE\u003dRelease ./build.sh --benchmark -j 48\ntaskset -c 24 be/build_Release/bin/benchmark_test \\\n --benchmark_filter\u003dBM_ArrayRange \\\n --benchmark_min_time\u003d0.3s --benchmark_min_warmup_time\u003d0.2 \\\n --benchmark_repetitions\u003d7 --benchmark_out\u003dresults.json \\\n --benchmark_out_format\u003djson\n```\n\n### Release note\n\nFix integer overflow in `sequence`/`array_range` near INT32_MAX and\naccelerate integer range generation with batched output filling.\n\n### Check List (For Author)\n\n- Test\n - [ ] Regression test\n- [x] Unit Test: `./run-be-ut.sh --run\n--filter\u003d\u0027FunctionArrayRangeTest.*\u0027 -j 48` — all 4 ASAN tests passed and\nthe process exited normally.\n - [x] Manual test: actual-function Release benchmarks above.\n- Behavior changed:\n - [ ] No.\n- [x] Yes: valid ranges whose final increment exceeds INT32_MAX now\nterminate with the expected array.\n- Does this need documentation?\n - [x] No.\n - [ ] Yes.\n\nValidation notes:\n- clang-format 16 and build-hygiene checks passed.\n- The Release benchmark compiled, linked, installed and ran\nsuccessfully. The build script\u0027s subsequent generic packaging step fails\nbecause benchmark-only builds do not populate `be/output/bin/*`.\n- The repository clang-tidy script was attempted but did not pass\nbecause of the existing unmatched `NOLINTEND` in `be/src/core/types.h`\nand signed comparisons in unchanged array-range code. No unrelated fixes\nare included.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label" }, { "commit": "3c83b696c84b00fbc9bcc1ed4d4fadbd3a8a6560", "tree": "d91a797321d45bd5e91d336440d7b5b164bb21be", "parents": [ "b3766d781b529f89ec2dadcf53ecf712aec0ae7b" ], "author": { "name": "deardeng", "email": "dengxin@selectdb.com", "time": "Wed Sep 09 10:16:16 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 10:16:16 2026 +0800" }, "message": "[fix](build) Update Doris compose to Debian 12 with Liberica JDK 17 (#67664)\n\nProblem Summary: Building the Doris compose image fails during apt-get\nupdate because openjdk:17.0.1-jdk-slim is based on Debian 11. Bullseye\nLTS ended on August 31, 2026, and its security repository metadata\nsubsequently expired. The same expired metadata is served by both the\nAliyun mirror and Debian upstream, so switching mirror hosts does not\nresolve the failure.\n\nUse bellsoft/liberica-openjdk-debian:17, currently based on Debian 12,\nwhile preserving the JDK_IMAGE build argument and the base image\u0027s Java\nenvironment. Update mirror rewriting to support both legacy .list and\ndeb822 .sources files, and install python-is-python3 because Debian 12\ndoes not provide the old python package. Document the new default image.\n\n### Release note\n\nThe Doris compose image now uses Debian 12 with BellSoft Liberica JDK\n17. The python command now runs Python 3.\n\n### Check List (For Author)\n\n- Test: Manual test\n- Build the complete Docker image using existing local output artifacts.\n - Run docker build --check: no warnings.\n- Verify mirror rewriting for both legacy .list and deb822 .sources\nfiles.\n- Verify Debian 12, Java/javac 17, JAVA_HOME/libjvm, Python, MySQL\nclient, and JaCoCo agent loading in the built image.\n- Run FE start_fe.sh --version and BE doris_be --version successfully.\n - Run git diff --check.\n- No kernel rebuild or full cluster regression: this changes image\npackaging.\n- Behavior changed: Yes; update the base OS/JDK distribution and use\nPython 3.\n- Does this need documentation: Yes; update the Doris compose README in\nthis commit.\n\n### What problem does this PR solve?\n\nIssue Number: close #xxx\n\nRelated PR: #xxx\n\nProblem Summary:\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test \u003c!-- At least one of them must be included. --\u003e\n - [ ] Regression test\n - [ ] Unit Test\n - [ ] Manual test (add detailed scripts or steps below)\n - [x] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n - [ ] Previous test can cover this change.\n - [ ] No code files have been changed.\n - [ ] Other reason \u003c!-- Add your reason? --\u003e\n\n- Behavior changed:\n - [x] No.\n - [ ] Yes. \u003c!-- Explain the behavior change --\u003e\n\n- Does this need documentation?\n - [x] No.\n- [ ] Yes. \u003c!-- Add document PR link here. eg:\nhttps://github.com/apache/doris-website/pull/1214 --\u003e\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label \u003c!-- Add branch pick label that this PR\nshould merge into --\u003e" }, { "commit": "b3766d781b529f89ec2dadcf53ecf712aec0ae7b", "tree": "4f3204f1a654e02954e17426abdb2ba14f20c620", "parents": [ "77cdd9cc9d5abcaa89dcddc0b3eba6a37b385637" ], "author": { "name": "shuke", "email": "shuke@selectdb.com", "time": "Wed Sep 09 10:09:39 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 10:09:39 2026 +0800" }, "message": "[fix](bin) Read final config lines without trailing newlines (#67663)\n\nProblem Summary:\n\nFE/BE startup and shutdown scripts silently skip the final environment\nassignment in `fe.conf` or `be.conf` when the file has no trailing\nnewline. For example, a final `PID_DIR\u003d/custom/pid` is ignored, so the\nscript uses the default or an earlier PID directory.\n\nBash `read` sets `line` but returns failure when EOF terminates a\npartial line. Keep each of the four configuration loops running when\nthat final line is nonempty so the existing parser exports it.\n\n### Release note\n\nFix FE/BE startup and shutdown scripts ignoring the last configuration\nassignment when the file has no trailing newline." }, { "commit": "77cdd9cc9d5abcaa89dcddc0b3eba6a37b385637", "tree": "61a1999e8b423a0d89e550bc446c368e69b930a3", "parents": [ "0796543f6a025a16948d507ecc161761da574d58" ], "author": { "name": "Xin Liao", "email": "liaoxin@selectdb.com", "time": "Wed Sep 09 09:57:15 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 09:57:15 2026 +0800" }, "message": "[fix](cloud) Set file cache TTL on cloud schema change output (#67454)\n\nProblem Summary:\n\n`CloudSchemaChangeJob::_convert_historical_rowsets()` builds its\n`RowsetWriterContext` with `write_file_cache` but never sets\n`file_cache_ttl_sec`. For a table with `file_cache_ttl_seconds`, the\nschema change output is therefore written into the NORMAL/INDEX file\ncache queues instead of the TTL queue. The other two cloud write paths\nboth set it:\n\n- load: `CloudRowsetBuilder::init()` — `context.file_cache_ttl_sec \u003d\n_tablet-\u003ettl_seconds();`\n- compaction: `CloudCompactionMixin::construct_output_rowset_writer()` —\n`ctx.file_cache_ttl_sec \u003d _tablet-\u003ettl_seconds();`" }, { "commit": "0796543f6a025a16948d507ecc161761da574d58", "tree": "edce5f74375793e2045767f91839eb403b267c77", "parents": [ "1d629e40636308d768042a75bd3f111352212083" ], "author": { "name": "Mingyu Chen (Rayner)", "email": "morningman.cmy@gmail.com", "time": "Wed Sep 09 09:52:43 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 09:52:43 2026 +0800" }, "message": "[fix](arrow-flight) Forward a statement to the master FE without touching the MySQL channel (#67569)\n\n### What problem does this PR solve?\n\nRelated PR: #61050\n\nProblem Summary:\n\n**1. A Flight session could not forward a statement at all.**\n\nAn Arrow Flight SQL session has no `MysqlChannel`:\n`FlightSqlConnectContext` overrides\n`getMysqlChannel()` to throw `getMysqlChannel not in mysql connection`.\nSince #61050,\n`FEOpExecutor.buildStmtForwardParams()` reads the client\u0027s\n`CLIENT_DEPRECATE_EOF` capability\nstraight off that channel, so on a multi-FE deployment every statement a\nFlight connection has to\nforward to the master fails with that message before the request is even\nsent:\n\n- any DDL issued over Arrow Flight SQL to a follower or observer FE;\n- any statement at all when `force_forward_all_queries` is on.\n\n`CLIENT_DEPRECATE_EOF` is a MySQL protocol capability, so only read it\nfor a MySQL connection. The\nthrift field is `optional` with no IDL default, and leaving it unset for\nother protocols puts the\nmaster back on the packet layout it used before #61050 —\n`ConnectProcessor` applies it only when\n`isSetClientDeprecatedEOF() \u0026\u0026 isClientDeprecatedEOF()`, so unset is\nequivalent to set-false for\nevery reader in every rolling-upgrade direction.\n\n**2. Once the forward succeeds, nothing carries the master\u0027s answer back\nto the Flight client.**\n\n`ConnectProcessor.finalizeCommand()` is the only place a forwarded\nstatement\u0027s status and result set\nare replayed, and it is MySQL-only: it opens with\n`Preconditions.checkState(connectType.equals(ConnectType.MYSQL))`.\nNothing else reads\n`getProxyStatusCode()` / `getShowResultSet()` / `getOutputPacket()`\nexcept the audit log. So after\nthe RPC, `ctx.getState()` stays at the `OK` that `executeQuery()` set\nwith `reset()`, the\n`FlightSqlChannel` stays empty, and `DorisFlightSqlProducer` answers\nwith `addOKResult()`\u0027s\nsynthesized one-row `StatusResult \u003d 0`. With a client connected over\nArrow Flight SQL to a follower\nor observer:\n\n- a DDL that **failed** on the master was reported to the client as\nsuccess — only the follower\u0027s\n audit event recorded the real error;\n- a forwarded statement that returns rows (`SHOW FRONTENDS`, `SHOW\nLOAD`, … — about 33\n`ShowCommand` subclasses both forward and return rows) returned that\nsingle `StatusResult` row\ninstead of the master\u0027s rows. The same statement run against the master\nreturns real rows, because\n`StmtExecutor.sendResultSet()` already has a working Arrow Flight\nbranch.\n\nThis gap is older than #61050 (`finalizeCommand()` was already\nMySQL-only before it), so fixing only\npart 1 would have turned a loud, diagnosable error into a silent wrong\nanswer, which the\nrepository\u0027s \"Error Means Failure\" invariant does not allow. Part 2 is\ntherefore fixed here too:\n\n- `ConnectProcessor.carryForwardedOutcomeToFlightSession()` is the Arrow\nFlight counterpart of\n`finalizeCommand()`\u0027s forwarded-statement branch. It copies a non-zero\nmaster status into\n`ctx.getState()` (and logs it at WARN — the follower used to say\nnothing), and otherwise replays\nthe master\u0027s result set through the existing `sendResultSet()` Arrow\nFlight branch.\n- It is called from the branch of `executeQuery()` that is **already**\nscoped to\n`ARROW_FLIGHT_SQL`, so it structurally cannot fire on a MySQL connection\nand\n `finalizeCommand()`\u0027s `getStateType() !\u003d ERR` condition is untouched.\n- `StmtExecutor` now refuses to forward a *query* on an Arrow Flight SQL\nconnection, before the RPC.\nA query result comes back as MySQL wire packets in\n`TMasterOpResult.queryResultBufList`, which\nonly `finalizeCommand()` can replay and which cannot be converted to\nArrow batches; answering it\nwith a synthesized empty success would be the same silent wrong answer.\nThis is reachable only\nunder `force_forward_all_queries`, which `Config` documents as \"For\ntesting purposes\".\n\nThe MySQL path is byte-for-byte unchanged." }, { "commit": "1d629e40636308d768042a75bd3f111352212083", "tree": "1672c05f3b7afaaecd0b41fe8aceb7b76b15ac95", "parents": [ "d86ad920d37c62c0adc494f9bd23c394bbbfb919" ], "author": { "name": "wudi", "email": "wudi@selectdb.com", "time": "Wed Sep 09 09:49:44 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 09:49:44 2026 +0800" }, "message": "[improvement](streaming-job) Adjust CDC request timeout and split batch defaults (#67559)\n\n### What problem does this PR solve?\n\nSlow CDC control requests, including snapshot split discovery for\nunevenly distributed keys, can exceed the default 60-second BE HTTP\ntimeout. Retrying split discovery can repeat expensive work before the\nFE receives progress.\n\nThis PR changes only two defaults:\n\n- Increase `request_cdc_client_timeout_ms` from `60000` to `120000` (120\nseconds).\n- Reduce `streaming_cdc_fetch_splits_batch_size` from `100` to `16` to\nreturn smaller batches on batched split-discovery paths." }, { "commit": "d86ad920d37c62c0adc494f9bd23c394bbbfb919", "tree": "08c49c8f31bca704f3c401975e71113554b73021", "parents": [ "303d121e85a2845b90b0e3deb90148bc93e63cfc" ], "author": { "name": "Gabriel", "email": "liwenqiang@selectdb.com", "time": "Wed Sep 09 09:48:47 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 09:48:47 2026 +0800" }, "message": "[fix](http) Clamp range reads to file size (#67634)\n\n### What problem does this PR solve?\n\nIssue Number: DORIS-28543\n\nRelated PR: None\n\nProblem Summary:\n\n`HttpFileReader` expands small reads to a 1 MiB read-ahead request. Near\nEOF, the generated byte range can extend beyond the file size. Some\nRange-capable object stores respond to that overlong range with HTTP 200\nand the complete object, so Doris incorrectly reports that Range support\nchanged after it was successfully detected during open.\n\nThis change clamps speculative read-ahead to the known file boundary. It\nalso adds a local HTTP server unit test that reproduces the 206-at-open\nfollowed by 200-on-overlong-range behavior.\n\n### Release note\n\nFix HTTP range reads near EOF for servers that return the complete\nobject for ranges extending past EOF.\n\n### Check List (For Author)\n\n- Test\n - [ ] Regression test\n - [x] Unit Test\n - [ ] Manual test (add detailed scripts or steps below)\n - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n - [ ] Previous test can cover this change.\n - [ ] No code files have been changed.\n - [ ] Other reason\n\n- Behavior changed:\n - [ ] No.\n - [x] Yes. HTTP read-ahead now respects the known EOF boundary.\n\n- Does this need documentation?\n - [x] No.\n - [ ] Yes.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label" }, { "commit": "303d121e85a2845b90b0e3deb90148bc93e63cfc", "tree": "f795317086eb5d2311d142c3b937ddab13ed1fdc", "parents": [ "e08bcd1e94c6f69342450a0c48b06b1873314111" ], "author": { "name": "Yixuan Wang", "email": "wangyixuan@selectdb.com", "time": "Wed Sep 09 09:27:04 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 09:27:04 2026 +0800" }, "message": "[fix](fe) Preserve active export jobs and label mappings during cleanup (#67539)\n\n### What problem does this PR solve?\n\n`ExportMgr.removeOldExportJobs()` removes export jobs when the number of\nstored jobs exceeds `Config.max_export_history_job_num`. The old\nimplementation removed the oldest jobs without checking their states, so\nPENDING, EXPORTING, or IN_QUEUE jobs could also be deleted.\n\nAn active export job can still write a later `OP_EXPORT_UPDATE_STATE`\nedit log. After FE restart or master transfer, journal replay then looks\nup the job by ID in `exportIdToJob`. Because the cleanup task may have\nalready removed the job, the lookup returns null and replay fails with:\n\n```\njava.lang.NullPointerException: Cannot invoke \"org.apache.doris.load.ExportJob.replayExportJobState(...)\" because \"job\" is null\n```\n\nThe cleanup also removed label mappings by label only. If an old\ncancelled job and a newer active job used the same label, deleting the\nold job could remove the newer job\u0027s label mapping and allow an invalid\nduplicate export job to be created.\n\n### How does this PR solve the problem?\n\n- Only remove export jobs in `CANCELLED` or `FINISHED` states when\nenforcing the maximum history-job limit.\n- Keep active jobs even when their total number is greater than\n`Config.max_export_history_job_num`, because active jobs may still\nproduce state-transition edit logs.\n- Iterate through the sorted job list safely and stop when there are no\nmore removable candidates, avoiding an endless cleanup loop when all\njobs are active.\n- Remove a label mapping only when both the label and the mapped job ID\nmatch the job being deleted. This prevents an old job from deleting the\nmapping owned by a newer job with the same label.\n\n### Test\n\n- Add coverage to verify that PENDING and EXPORTING jobs are retained\nwhile terminal jobs are removed.\n- Add coverage to verify that all active jobs are retained when the\nhistory limit is smaller than the number of active jobs.\n- Add coverage to verify that a running job keeps its label reservation\nafter an older job with the same label is cleaned up." }, { "commit": "e08bcd1e94c6f69342450a0c48b06b1873314111", "tree": "f7f8193e51930e206e84cc45ebfb59403cad3b06", "parents": [ "1fc400bec62fb698c213702a2233a0904848d310" ], "author": { "name": "Gabriel", "email": "liwenqiang@selectdb.com", "time": "Wed Sep 09 09:25:20 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 09:25:20 2026 +0800" }, "message": "[fix](iceberg) Fix V2 reads across nested schema evolution (#67574)\n\n### What problem does this PR solve?\n\nIssue Number: N/A\n\nRelated PR: N/A\n\nProblem Summary:\n\nIceberg V2 scans could retain a stale nested file projection after a\npredicate on a missing child was demoted, shifting projected sibling\nvalues by physical ordinal. Separately, ID-less migrated files fell back\nto current-name mapping after the explicit default name mapping was\nremoved, exposing stale physical values instead of NULL.\n\nThis PR reapplies the finalized scan projection to column mappings and\nrequires authoritative name mapping for V2 ID-less file reads.\n\n### Release note\n\nFix Iceberg V2 reads when a rejected nested predicate widens projection\nand when migrated ID-less files lose their default name mapping.\n\n### Check List (For Author)\n\n- Test: Unit Test and Regression test\n- All 82 `IcebergV2ReaderTest` tests and all 15 `ColumnMapperTest` tests\npassed.\n- The four equality-delete tests affected by the stricter ID-less\nmapping invariant passed after explicitly mapping their projected result\ncarrier.\n- Both new Iceberg external-table regressions passed: nested projection\nalignment and migrated ID-less nested NULL semantics.\n- The existing `test_gen_iceberg_by_api` suite passed while validating\nthe required-field error for an ID-less fixture without authoritative\nname mapping.\n - ASAN BE build and FE build passed.\n - Build hygiene and clang-format 16 checks passed.\n- Behavior changed: Yes. Iceberg V2 nested projections preserve field-ID\nalignment, and migrated ID-less fields without default name mapping\nmaterialize as NULL.\n- Does this need documentation: No" }, { "commit": "1fc400bec62fb698c213702a2233a0904848d310", "tree": "12eadefcf345ccd90495567f2bb80e054035630d", "parents": [ "88b470c757dd382e1864ec3c4f582d8774d9aa02" ], "author": { "name": "hui lai", "email": "laihui@selectdb.com", "time": "Wed Sep 09 09:21:21 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 09:21:21 2026 +0800" }, "message": "[fix](be) Initialize bthread context key before first use (#67580)\n\n### What problem does this PR solve?\n\nA background sender could finish all of its work and then crash while\ndestroying\n`AddThreadMemTrackerConsumer`:\n\n```text\nAddThreadMemTrackerConsumer::~AddThreadMemTrackerConsumer()\nVTabletWriter::_send_batch_process()\nperiodic_send_batch(void*)\nbthread::TaskGroup::task_runner(...)\n```\n\nThe core file shows that `thread_context()` returned `nullptr`; the\n`ThreadMemTrackerMgr` and its consumer stack were not corrupted. The\noriginal\n`ThreadContext` was still present in the bthread key table, but it could\nno longer be\nretrieved:\n\n```text\nglobal btls_key: { index \u003d 0, version \u003d 1 }\nbthread key-table slot 0: { version \u003d 0, ptr \u003d valid ThreadContext }\n```\n\n`btls_key` was zero-initialized as `{0, 0}` and was not created until\n`PInternalService` was constructed. The BE starts its Thrift backend\nservice before\nconstructing the BRPC internal service, so work can arrive during that\nstartup window.\nA bthread could therefore store its context with `{0, 0}`. BRPC accepts\nthat value\nbefore the real key is created because key slot 0 also starts at version\n0.\n\nLater, `bthread_key_create()` changed the global key to `{0, 1}`. A\nlookup using the\nnew version did not match the existing version-0 entry and returned\n`nullptr`, causing\nthe crash during scoped tracker cleanup.\n\nThis is a key-initialization race. It is not caused by a bthread\nmigrating between\nworker pthreads or by `pthread_context_ptr_init` changing during\nmigration.\n\n### What is changed?\n\n- Move the bthread ThreadContext key and its deleter into the\n`thread_context` module.\n- Initialize the key with `std::call_once` before any\n`ThreadLocalHandle` creates or\naccesses a context. The BE\u0027s early `SCOPED_INIT_THREAD_CONTEXT()`\ninitializes it\nbefore services start, while the lazy guard also protects other entry\npoints.\n- Keep the key valid for the complete process lifetime instead of\ndeleting it with\n`PInternalService`, so existing bthread contexts cannot be invalidated\nby service\n lifetime.\n- Revert the earlier context-pointer caching and lookup-order changes.\nThose changes\n only masked the null lookup and did not fix the key-version mismatch.\n\nThe change does not alter memory-accounting semantics. It guarantees\nthat every\nbthread context is stored and retrieved with the same valid key version." }, { "commit": "88b470c757dd382e1864ec3c4f582d8774d9aa02", "tree": "996d9b9e00fd6205d73e1b3dc94b07b5bf88084c", "parents": [ "9957987d5bf43fa6a33aa59918435fed65e2912a" ], "author": { "name": "bobhan1", "email": "baohan@selectdb.com", "time": "Tue Sep 08 18:30:37 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 18:30:37 2026 +0800" }, "message": "[fix](fe) Adapt warm-up scheduler tests to JUnit 5 (#67655)\n\n### What problem does this PR solve?\n\nRelated PR: #67527, #67396\n\nProblem Summary: #67527 added seven JUnit 4 `Assert` calls to\n`ConfigTest` after #67396 migrated the class to JUnit 5, causing\n`fe-common:testCompile` to fail with `cannot find symbol: Assert`. Use\nthe existing JUnit 5 `Assertions` import. Also migrate the scheduler\ntest introduced by the same PR to JUnit 5 assertions and lifecycle\nannotations, consistent with the FE test migration.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test: `./run-fe-ut.sh --run\norg.apache.doris.common.ConfigTest,org.apache.doris.cloud.CacheHotspotManagerSchedulerTest`\ncompleted with BUILD SUCCESS: ConfigTest 10/10 and\nCacheHotspotManagerSchedulerTest 4/4 passed, with no failures, errors,\nor skipped tests. Checkstyle passed for fe-common and fe-core with zero\nviolations; `git diff --check` passed.\n- Behavior changed: No\n- Does this need documentation: No" }, { "commit": "9957987d5bf43fa6a33aa59918435fed65e2912a", "tree": "ba29c163daf110333fbb54f51fd323b6789c47f8", "parents": [ "396ad795b059a44fef51942f8950588f83e7be0a" ], "author": { "name": "Yixuan Wang", "email": "wangyixuan@selectdb.com", "time": "Tue Sep 08 17:07:30 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 17:07:30 2026 +0800" }, "message": "[fix](cloud) Return existing rowset for concurrent commits (#67637)\n\nProblem Summary:\n\nDifferent BEs may generate different rowsets for the same transaction\nand tablet.\n\nAfter rowset A is committed, the temporary rowset key points to A while\nthe recycle rowset key for B still exists. The recycle key check ran\nbefore comparing rowset IDs, so commit_rowset incorrectly treated tmp(A)\nand recycle(B) as an invalid conflict.\n\nCompare rowset IDs first and only check the tmp and recycle key conflict\nwhen they belong to the same rowset. For different rowsets, return\nALREADY_EXISTED with rowset A." }, { "commit": "396ad795b059a44fef51942f8950588f83e7be0a", "tree": "c1db2ebe7a8b8bfb2df29bacfb15c6f9094819cd", "parents": [ "c88e889515ecc33e90a503c274bf58e76a5758a5" ], "author": { "name": "bobhan1", "email": "baohan@selectdb.com", "time": "Tue Sep 08 16:43:35 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 16:43:35 2026 +0800" }, "message": "[fix](test) Isolate TPC-H MOR-as-DUP tables from compaction (#67566)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #61762\n\nProblem Summary:\n\nThe TPC-H SF100 MOR suite loads the same data twice into MOR and DUP\ntables, then compares their results with `read_mor_as_dup_tables`\nenabled. Automatic compaction can merge the MOR versions before the\ncomparison, leaving fewer rows than the DUP tables.\n\nAdd eight dedicated `*_mor_as_dup` tables with automatic compaction\ndisabled. Prepare them in the existing two-round `load.groovy` flow by\nreusing the existing S3 import SQL with a distinct table name and load\nlabel. Route the MOR-as-DUP queries to these tables, preserving the\nshared MOR table definitions used by the normal TPC-H and\npredicate-pushdown cases.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test\n - [ ] Regression test\n - [ ] Unit Test\n - [ ] Manual test\n - [x] No need to test or manual test. Explain why:\n- [x] Other reason: Test fixture change. Both modified Groovy scripts\npassed compilation with the existing regression framework\u0027s Groovy\ncompiler; all eight dedicated DDL schemas and import substitution\nanchors were checked; shared MOR DDLs are unchanged; `git diff --check`\npassed. The full SF100 suite was not run locally.\n- Behavior changed:\n - [x] No. Test setup only.\n - [ ] Yes.\n- Does this need documentation?\n - [x] No.\n - [ ] Yes.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label" }, { "commit": "c88e889515ecc33e90a503c274bf58e76a5758a5", "tree": "88db7c4280f7c025a9c0c4478e3f48aa24c20358", "parents": [ "a42928c5caf1e064dbc4f6b7450430d49a1549e7" ], "author": { "name": "YanzhiJin5", "email": "297311832+YanzhiJin5@users.noreply.github.com", "time": "Tue Sep 08 16:15:48 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 16:15:48 2026 +0800" }, "message": "[feature](function) Add ST_IsClosed function (#67350)\n\n### What problem does this PR solve?\n\nIssue Number: ref #48203\n\nRelated PR: apache/doris-website#4102\n\nProblem Summary:\n\nAdd the `ST_IsClosed` spatial scalar function.\n\nFor a valid LineString, the function returns:\n\n* `true` when the first and last points are exactly equal\n* `false` when the LineString is open\n* `NULL` for a `NULL` input\n* `NULL` for a non-LineString or invalid encoded geometry\n\nThe implementation uses exact S2 point equality without introducing a\ntolerance.\n\nThis PR also adds Nereids registration and visitor support, FE and BE\nunit tests, and a self-asserting SQL regression suite.\n\n### Release note\n\nAdd the `ST_IsClosed` spatial function." }, { "commit": "a42928c5caf1e064dbc4f6b7450430d49a1549e7", "tree": "d81d5533422dac65375db4a67bdea8093991ea0d", "parents": [ "bd01e3322679b374da54892ae8f932542a8a7cf6" ], "author": { "name": "deardeng", "email": "dengxin@selectdb.com", "time": "Tue Sep 08 16:02:14 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 16:02:14 2026 +0800" }, "message": "[fix](be) Avoid repeated peer connect failures (#67464)\n\nProblem:\nPeer reads create an uncached BRPC client for every request. When a peer\nis unreachable, readers across tablets repeatedly connect to the same\naddress with the default 2-second connect timeout and 10 retries.\n\nImpact:\nAn unavailable cache peer can repeatedly delay reads before\nremote-storage fallback and generate unnecessary connection attempts and\nwarning logs.\n\nFix:\n- Add a BE-wide, address-level circuit breaker shared by peer readers.\n- Open the circuit after a configurable number of consecutive connection\nor RPC failures (3 by default) for a configurable cooldown (30 seconds\nby default).\n- Reject reads while the circuit is open, allow one probe after\ncooldown, and clear the failure state after a successful RPC.\n- Make uncached BRPC connect timeout and retry count configurable at the\ncall site, and use a 200 ms timeout with no retries for peer reads.\n\nTest:\nStop the peer test server and verify that four reads issue only three\nRPC attempts, with the final read rejected by the circuit breaker." }, { "commit": "bd01e3322679b374da54892ae8f932542a8a7cf6", "tree": "b2b839a5bfc39a27679d414edf32d3f8dda70c8d", "parents": [ "94ad62a4aeaefbb3f41336f50fa0ecf8b4cdf203" ], "author": { "name": "Mingyu Chen (Rayner)", "email": "morningman.cmy@gmail.com", "time": "Tue Sep 08 15:29:54 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 15:29:54 2026 +0800" }, "message": "[refactor](qe) Merge the duplicate ConnectType enum and drop the unreachable forward branch (#67572)\n\n### What problem does this PR solve?\n\nProblem Summary:\n\nTwo pieces of dead weight in the connect layer, found while reading the\nArrow Flight SQL and MySQL\npaths side by side. No behavior change.\n\n**1. `ConnectType` was declared twice.** Once in `ConnectContext` and\nonce in `ConnectProcessor`, so\na processor\u0027s `connectType` and its session\u0027s `connectType` were\nunrelated types that only happened\nto carry the same two constants — nothing stopped them from disagreeing,\nand neither could be passed\nwhere the other was expected. Keep the `ConnectContext` one, which is\nthe session\u0027s own property,\nand let the processors import it. Every subclass (including the four\ntest-only ones) was resolving\nthe enum through inheritance, so they now import it explicitly.\n\n**2. `FrontendServiceImpl.createForwardProcessor()` dispatched on an\noutcome that cannot happen.** It\nbranched on the connect type of the forwarded context, but the context\ncomes from\n`createForwardContext()` one line above, which builds it with `new\nConnectContext(null, true,\nsessionId)` — and that constructor always sets `ConnectType.MYSQL`. So\nthe Arrow Flight branch and\nthe `unknown ConnectType` throw were unreachable: the master replays a\nforwarded statement over a\n`ProxyMysqlChannel` no matter which protocol the client speaks on the\norigin FE. The MySQL processor\nis now constructed directly at the one call site, with a comment\nrecording why that is the only\npossibility." }, { "commit": "94ad62a4aeaefbb3f41336f50fa0ecf8b4cdf203", "tree": "a1c5688f720be2004410de090c58744c4cab5ff1", "parents": [ "c275e05579b23e00907f129be052609acd46c8e9" ], "author": { "name": "bobhan1", "email": "baohan@selectdb.com", "time": "Tue Sep 08 15:20:10 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 15:20:10 2026 +0800" }, "message": "[fix](cloud) prevent warm-up job scheduling starvation (#67527)\n\nProblem Summary:\n\nWhen cloud warm-up jobs outnumber the configured active slots, the\nscheduler can over-submit tasks because a job is recorded as active only\nafter its worker starts. The direct-handoff thread pool previously\ndiscarded rejected tasks, and unordered map traversal could repeatedly\nfavor the same jobs, leaving other runnable jobs pending indefinitely.\n\nThis PR keeps the direct-handoff pool and makes scheduling bounded and\nretryable:\n\n- Scan runnable jobs once and keep only the jobs needed by the current\nslots in a bounded priority heap.\n- Reserve the active slot before submission and roll it back when\nsubmission is rejected.\n- Schedule the least-recently-run jobs first, with never-scheduled\none-time jobs taking the first turn.\n- Surface pool rejection so the scheduler can retry the job in the next\ncycle.\n- Apply mutable scheduler concurrency and interval configuration without\nrequiring an FE restart.\n\n### Release note\n\nFix cloud warm-up jobs remaining pending when the scheduler thread pool\nis saturated." }, { "commit": "c275e05579b23e00907f129be052609acd46c8e9", "tree": "64fdf324d5add357d351019e43813b7c27b905fa", "parents": [ "798eb24b176c6862bdc66c94bfa875b8e861341e" ], "author": { "name": "Gabriel", "email": "liwenqiang@selectdb.com", "time": "Tue Sep 08 15:11:14 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 15:11:14 2026 +0800" }, "message": "[fix](catalog) Restore DLF support for Iceberg and Paimon (#67545)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: None\n\nProblem Summary:\n\nThe connector metastore SPI refactor removed DLF provider registration\nand the plugin-local DLF client path, so existing Iceberg and Paimon\ncatalogs using `dlf` can no longer be created or replayed on master.\n\nThis PR restores DLF support for both connectors:\n\n- Register engine-specific DLF metastore providers and restore legacy\nproperty aliases, endpoint derivation, catalog-id fallback, and\nsensitive credential handling.\n- Restore the Iceberg DLF catalog and client pool, with catalog-scoped\nclient caching and OSS-backed `S3FileIO`.\n- Route Paimon DLF through its Hive catalog using\n`ProxyMetaStoreClient`, while preserving legacy alias-only OSS and\nOSS-HDFS storage configurations.\n- Bundle the matching Hive 3 and Hive 2 DLF clients into the existing\nIceberg/HMS and Paimon private shades, including their Thrift and Tea\nnamespace isolation.\n\n### Release note\n\nRestore Aliyun DLF metastore support for Iceberg and Paimon catalogs.\n\n### Check List (For Author)\n\n- Test\n - [ ] Regression test\n - [x] Unit Test\n - [x] Manual test (add detailed scripts or steps below)\n- Targeted Iceberg, Paimon, metastore provider, property-binding, and\nDLF catalog tests passed.\n- The 19-module connector package build passed with tests skipped; FE\nCheckstyle reported zero violations.\n- Verified both generated private shade jars contain the DLF proxy\nclient and relocated Thrift and Tea classes.\n- The full Iceberg connector suite was also attempted; 1 of 1,376 tests\nfailed in the unrelated\n`IcebergWritePlanProviderTest.planMergePreservesExplicitlyEmptyReadAcrossConcurrentFirstAppend`\ncase. The focused rerun reproduces the same failure, and this PR does\nnot modify that code path.\n - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n - [ ] Previous test can cover this change.\n - [ ] No code files have been changed.\n - [ ] Other reason\n\n- Behavior changed:\n - [ ] No.\n- [x] Yes. Iceberg and Paimon catalogs can use the DLF metastore backend\nagain.\n\n- Does this need documentation?\n - [x] No. Existing DLF catalog properties and behavior are restored.\n - [ ] Yes.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label" }, { "commit": "798eb24b176c6862bdc66c94bfa875b8e861341e", "tree": "82b0111e27df2af5a55143de766e0c29a3838328", "parents": [ "e1df149d617d16066fec9c97d770c14b3ac045b2" ], "author": { "name": "Luwei", "email": "814383175@qq.com", "time": "Tue Sep 08 15:00:54 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 15:00:54 2026 +0800" }, "message": "[fix](binlog) Fix row binlog recovery for multi-tablet transactions (#67508)\n\n### What problem does this PR solve?\n\nIssue Number: close #67091\n\nRelated PR: None\n\nProblem Summary: A BE restart indexed row-binlog rowsets only by\ntransaction ID, so multiple tablet pairs in one transaction could attach\nthe wrong companion rowset. Persist each base tablet\u0027s companion ID and\nrecover using both transaction and tablet IDs.\n\n### Release note\n\nFix incorrect row-binlog companion recovery after a BE restart for\nmulti-tablet transactions.\n\n### Check List (For Author)\n\n- Test: Unit Test\n- Added and ran\nGroupRowsetBuilderTest.recoverMultipleRowBinlogPairsInOneTxn and\nGroupRowsetBuilderTest.*\n- Behavior changed: Yes. BE restart recovery now attaches each base\nrowset to its persisted companion tablet.\n- Does this need documentation: No" }, { "commit": "e1df149d617d16066fec9c97d770c14b3ac045b2", "tree": "28f9ce5606db1e20c98d6cc7c5299387881315d0", "parents": [ "ac7244ff1978b368a985a2874a48a3637b0df815" ], "author": { "name": "Mryange", "email": "yanxuecheng@selectdb.com", "time": "Tue Sep 08 14:55:15 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 14:55:15 2026 +0800" }, "message": "[fix](local shuffle) Prevent local exchange under serial parent pipeline (#67490)\n\nWhen FE local shuffle planning inserted a local exchange into a pipeline\nwhose parent pipeline was serial, it could increase only the downstream\npipeline task count while the paired sink and source operators remained\none-to-one. This produced inconsistent pipeline concurrency and could\nfail during execution. Root cause: FE did not propagate the serial state\nof a parent pipeline across pipeline boundaries. This change tracks that\nstate and skips local exchange insertion when the parent pipeline is\nserial, matching the existing BE planning constraint.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test \u003c!-- At least one of them must be included. --\u003e\n - [ ] Regression test\n - [ ] Unit Test\n - [ ] Manual test (add detailed scripts or steps below)\n - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n - [ ] Previous test can cover this change.\n - [ ] No code files have been changed.\n - [ ] Other reason \u003c!-- Add your reason? --\u003e\n\n- Behavior changed:\n - [ ] No.\n - [ ] Yes. \u003c!-- Explain the behavior change --\u003e\n\n- Does this need documentation?\n - [ ] No.\n- [ ] Yes. \u003c!-- Add document PR link here. eg:\nhttps://github.com/apache/doris-website/pull/1214 --\u003e\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label \u003c!-- Add branch pick label that this PR\nshould merge into --\u003e" }, { "commit": "ac7244ff1978b368a985a2874a48a3637b0df815", "tree": "d923a0934dad4abed3a1ca803bf1aa7072a6fcfa", "parents": [ "685e91f962a068ce098280d1ca84ddb6bc782c1d" ], "author": { "name": "shuke", "email": "shuke@selectdb.com", "time": "Tue Sep 08 14:46:06 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 14:46:06 2026 +0800" }, "message": "[fix](ci) Skip Codex auth with reused refresh tokens (#67635)\n\nProblem Summary: A review that fails with `refresh_token_reused` leaves\nits credentials eligible because the runner only records usage limits.\nLater reviews can repeatedly select the same unusable credentials.\n\nRecord explicit reuse failures in a separate OSS object keyed by the\nselected refresh token\u0027s SHA-256, then check that marker before\nselecting a downloaded auth snapshot. A replacement refresh token\nrestores eligibility; metadata-only changes do not. Separate objects\nkeep late failures and usage-context writes from overwriting another\ntoken\u0027s quarantine state. Marker read/write failures are surfaced\nexplicitly.\n\nThe marker applies to the selected token version. Runs that already\nselected credentials before the marker was written can still fail. This\ndoes not add credential leasing or change auth writeback." }, { "commit": "685e91f962a068ce098280d1ca84ddb6bc782c1d", "tree": "376529a31146113a53f459a8c5c61087585c3f39", "parents": [ "53264e01c6ab7fcb10c53ad6c00c38c537a2b2ba" ], "author": { "name": "TengJianPing", "email": "tengjianping@selectdb.com", "time": "Tue Sep 08 11:24:27 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 11:24:27 2026 +0800" }, "message": "[feature](timestamp_ns) Add end-to-end TIMESTAMP_NS support (#66761)\n\nDocs: https://github.com/apache/doris-website/pull/4091\n\nProblem Summary:\nDoris DATETIME only support at most six fractional digits, it\u0027s not\nsufficient for event times, traces, etc which require higher precision\nof sub-seconds.\n\nIntroduce `TIMESTAMP_NS` as a separate fixed-scale SQL type backed by\nsigned Int64 Unix-epoch nanoseconds. It always has scale 9 and supports\nvalues from `1677-09-21 00:12:43.145224192` through `2262-04-11\n23:47:16.854775807`. Wire the type through Thrift and protobuf metadata,\nthe FE catalog and Nereids, BE columns and SerDe, storage and indexes,\nresult protocols, JNI and Java UDF transport, complex and VARIANT types,\nand schema metadata.\n\nAdd exact casts, comparisons, type inference, and constant folding\nacross date-like types while preserving nanoseconds and rejecting\npartial or precision-losing implicit conversions. Add timestamp-aware\nscalar and calendar functions, aggregation, joins and ASOF joins, scan\npredicates, runtime filters, partition pruning, query-cache\nnormalization, automatic partitions, and MTMV roll-up. Keep FE folding\nand BE runtime behavior consistent at signed endpoints, leap years, DST\ngaps, calendar-clamped boundaries, and fractional formatting and\nrounding boundaries.\n\nKeep existing DATETIME and DATETIMEV2 scale semantics unchanged.\nFROM_UNIXTIME retains a DECIMAL64 microsecond path for scale 0 through 6\nand uses the DECIMAL128 nanosecond path only for scale 7 through 9. The\n%f formatter rounds half-up to microseconds with second carry, while %n\npreserves the nanosecond fraction.\n\n\n- Test: Unit Test and Regression Test\n- Added targeted BE unit coverage for TIMESTAMP_NS values, casts,\nfunctions, SerDe, predicates, aggregates, joins, and compatibility paths\n- Added targeted FE unit coverage for literals, casts, constant folding,\ntype inference, partitions, query-cache and MTMV boundaries, and\nstatement-time semantics\n- Added regression coverage under datatype_p0/timestamp_ns and focused\nMTMV and Java UDF suites\n - Built FE and BE with the standard build scripts\n- Behavior changed: Yes. Adds TIMESTAMP_NS and nanosecond function\nsemantics without changing the scale limits of DATETIME or DATETIMEV2.\n- Does this need documentation: Yes. Document TIMESTAMP_NS syntax,\nrange, casts and coercion rules, function behavior, and integration\nsupport.\n\n### Release note\n\nAdd `TIMESTAMP_NS`, a fixed-scale nanosecond timestamp type stored as\nsigned Unix-epoch nanoseconds. It supports SQL literals, casts,\ndate/time functions, storage and indexes, partitions, joins, aggregates,\ncomplex types, Java UDF/JNI transport, and FE constant folding and type\ninference. Existing DATETIME and DATETIMEV2 types remain limited to\nscale 0 through 6.\n\n### Check List (For Author)\n\n- Test \u003c!-- At least one of them must be included. --\u003e\n - [x] Regression test\n - [x] Unit Test\n - [ ] Manual test (add detailed scripts or steps below)\n - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n - [ ] Previous test can cover this change.\n - [ ] No code files have been changed.\n - [ ] Other reason \u003c!-- Add your reason? --\u003e\n\n- Behavior changed:\n - [ ] No.\n - [x] Yes. \u003c!-- Explain the behavior change --\u003e\n\n- Does this need documentation?\n - [ ] No.\n- [x] Yes. \u003c!-- Add document PR link here. eg:\nhttps://github.com/apache/doris-website/pull/1214 --\u003e\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label \u003c!-- Add branch pick label that this PR\nshould merge into --\u003e" }, { "commit": "53264e01c6ab7fcb10c53ad6c00c38c537a2b2ba", "tree": "fc2e3422cde04e1e6150ddc47f6e5507e9c12310", "parents": [ "62b8b4b3b8816fda32faa335f433ebabbff2f779" ], "author": { "name": "Gabriel", "email": "liwenqiang@selectdb.com", "time": "Tue Sep 08 11:04:38 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 11:04:38 2026 +0800" }, "message": "[fix](be) Reduce Iceberg equality delete index memory (#67570)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: None\n\nProblem Summary:\n\nFileScannerV2 keeps a node-based hash-to-row map for every row in a\nmulti-column Iceberg equality delete file. It also rebuilds that map for\nevery data split even though the parsed delete file is shared. This\namplifies memory usage and repeats index construction.\n\nThis change stores hash candidates in a sorted contiguous index and\nshares the immutable index through the split cache. Full-key comparison\nremains in place to resolve hash collisions. The optimization is\nintentionally scoped to FileScannerV2; the legacy format reader is\nunchanged.\n\n### Optimization approach\n\nBefore this change, FileScannerV2 cached the parsed equality delete\nblock, but every split-local `EqualityDeletePredicate` still recomputed\nall delete-row hashes and inserted them into a `std::multimap\u003chash,\nrow_index\u003e`. This caused repeated `O(N log N)` construction and one\ntree-node allocation per delete row for every predicate.\n\nThe new flow is:\n\n1. When an equality delete file is loaded for the first time, compute\none hash for each delete row.\n2. Store `{hash, row_index}` entries in a contiguous vector and sort it\nby hash.\n3. Cache this immutable index together with the parsed delete block.\n4. Pass a shared reference to every split-local predicate instead of\nrebuilding the index.\n5. Use `lower_bound` and `upper_bound` to find candidates with the same\nhash, then compare the complete equality key to preserve correctness\nunder hash collisions.\n\nThis removes the per-row node and allocator overhead of `std::multimap`,\nimproves cache locality, and changes index construction from once per\nsplit to once per cached delete file. Runtime profile counters expose\nindex cache hits, misses, and index memory usage for validation.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test\n - [ ] Regression test\n - [x] Unit Test\n - [x] Manual test (add detailed scripts or steps below)\n - Full BE build\n - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n - [ ] Previous test can cover this change.\n - [ ] No code files have been changed.\n - [ ] Other reason\n\n- Behavior changed:\n - [x] No.\n - [ ] Yes.\n\n- Does this need documentation?\n - [x] No.\n - [ ] Yes.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label" }, { "commit": "62b8b4b3b8816fda32faa335f433ebabbff2f779", "tree": "ee08fb6719d69af8e87b5aa7dd8206bb663cef93", "parents": [ "53e1f62096e85fda10d24dd7ef65547ecdc694d0" ], "author": { "name": "Gabriel", "email": "liwenqiang@selectdb.com", "time": "Tue Sep 08 11:04:20 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 11:04:20 2026 +0800" }, "message": "[fix](be) Preserve rows for untrusted Parquet null counts (#67573)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: None\n\nLegacy parquet-mr writers and Arrow versions through 5 can under-report\nParquet null counts. Format v2 treated a reported zero as proof that a\nrow group or page had no NULL values, so `IS NULL` pruning could drop\nmatching rows.\n\n### What is changed?\n\n- Derive null-count trust from Parquet `created_by` in format v2.\n- Treat the legacy `parquet-cpp` identity and `parquet-cpp-arrow`\nversions before 6.0.0 as untrusted for null-count pruning.\n- Conservatively retain NULL candidates at both row-group and page-index\npruning layers while preserving min/max pruning.\n- Add an affected PyArrow 3.0.0 dictionary-encoded fixture and Arrow\n3/4/5/6 boundary coverage.\n\n### Release note\n\nAvoid incorrect `IS NULL` pruning for Parquet files produced by affected\nlegacy writers.\n\n### Check List (For Author)\n\n- Test: Unit Test\n- `NativeParquetStatisticsTest.*:ParquetScanTest.*Null*` (24 tests\npassed with ASAN)\n- Behavior changed: Yes. Format v2 no longer trusts null counts from\naffected legacy Parquet and Arrow writers for metadata pruning.\n- Does this need documentation: No" }, { "commit": "53e1f62096e85fda10d24dd7ef65547ecdc694d0", "tree": "1a4c6026294597808a7d1caca7655cc79e6db6a8", "parents": [ "be7c013c119e4311f84794067d6a8c40e80cad17" ], "author": { "name": "shee", "email": "13843187+qzsee@users.noreply.github.com", "time": "Tue Sep 08 10:34:48 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 10:34:48 2026 +0800" }, "message": "[Improve](cloud) Only register watershed txn id once when BE enters decommissioning state (#67322)\n\nRelated PR: #30243\n\nProblem Summary:\n**This caused issues:**\n\nWrong guard flag. The block checked isDecommissioned() (the final\ndecommissioned state) instead of isDecommissioning() (the in-progress\nstate). As long as the BE had not yet reached the terminal\nDECOMMISSIONED state, every scheduling round of\n[CloudClusterChecker](copilot://navigate?keyword\u003dCloudClusterChecker)\nwould re-enter this branch and call registerWaterShedTxnId(be.getId())\nagain. The watershed txn id is only meaningful to be registered once per\ndecommissioning session; repeatedly registering it on every check\ninterval produces redundant RPCs/log entries and pollutes the upgrade\nmanager\u0027s internal state.\n\n\n**Fix**\nChange the guard from `!be.isDecommissioned()` to\n`!be.isDecommissioning()`, so the branch is only entered on the first\ntransition into the decommissioning state.\nMove `be.setDecommissioning(true)` inside the try block, right after\nregisterWaterShedTxnId succeeds. This guarantees the in-memory\ndecommissioning flag is only flipped when the watershed txn id has been\nregistered successfully; if the RPC fails, the BE stays in its previous\nstate and will be retried on the next check round.\n\n**Impact**\nEliminates repeated registerWaterShedTxnId calls for a BE that is stuck\nin DECOMMISSIONING state.\nMakes the decommissioning state transition atomic with respect to the\nwatershed txn id registration: either both succeed, or neither takes\neffect (and the operation is retried next round).\nNo behavior change for BEs that have already reached\nNODE_STATUS_DECOMMISSIONED." }, { "commit": "be7c013c119e4311f84794067d6a8c40e80cad17", "tree": "c993d207ca6191019aa807e7888593bba4bf881a", "parents": [ "2be8fba29d7180f8bb153135d397ed2bda4254fa" ], "author": { "name": "Calvin Kirs", "email": "guoqiang@selectdb.com", "time": "Tue Sep 08 09:49:38 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 09:49:38 2026 +0800" }, "message": "[fix](fe) Upgrade gRPC and remove unused LZ4 dependencies (#67585)\n\n### What problem does this PR solve?\n\nRelated PR: #67065\n\nProblem Summary:\n\nUpgrade the managed gRPC BOM from 1.65.1 to 1.75.0, matching #67065 and\nreplacing the `grpc-netty-shaded` version affected by CVE-2025-55163.\n\nExclude `org.lz4:lz4-pure-java` centrally from the managed\n`odps-sdk-core` and `odps-sdk-table-api` dependencies. Neither\nMaxCompute connector needs a replacement Java LZ4 dependency: the FE\nuses the ODPS metadata/session APIs, and the BE uses Table API Arrow\nreaders/writers with ZSTD. The legacy SDK Tunnel LZ4 implementations are\noutside the active Doris data path. The BE connector retains its local\nJackson exclusions.\n\nLimit `flight-sql-jdbc-driver` to test scope. Only the FE test client\nloads the JDBC driver; production code uses the retained `FlightServer`\nand `FlightSqlClient` protocol libraries. The regression framework keeps\nits independent JDBC driver dependency.\n\nAdd a ZSTD round-trip test through the ODPS Arrow writer/reader\nfactories, with the Java 17 direct-buffer access options required by the\ntest JVM. This verifies 1,024 rows including a null after removing LZ4,\nalongside the existing isolated ODPS class-loading test.\n\n### Release note\n\nUpgrade gRPC to 1.75.0. Stop bundling the unused Java LZ4 dependency in\nMaxCompute connectors and the test-only Flight SQL JDBC driver in the FE\nruntime distribution.\n\n### Check List (For Author)\n\n- Test\n - [ ] Regression test\n - [x] Unit Test\n - [x] Manual test (details below)\n\n Validation of the final implementation in `d9613e32c86`:\n- `EXTRA_FE_MODULES\u003dmaxcompute\u003dbe-java-extensions/max-compute-connector\n./run-fe-ut.sh --run\n\u0027org.apache.doris.connector.maxcompute.*Test,org.apache.doris.maxcompute.*Test,!org.apache.doris.connector.maxcompute.OdpsLiveConnectivityTest\u0027`:\n146 FE and 14 BE connector tests passed, including ODPS class-loader\nisolation and the new ZSTD round trip. The successful test runtime\nclasspaths were checked and contain no `net.jpountz.lz4` classes.\n- `build.sh --fe` passed with `DISABLE_BE_JAVA_EXTENSIONS\u003dOFF`,\n`FE_MAVEN_THREADS\u003d4`, `MVN_OPT\u003d-Dmaven.build.cache.enabled\u003dfalse`, and\n`--be-extension-ignore\niceberg-metadata-scanner,hadoop-hudi-scanner,java-common,java-udf,jdbc-scanner,paimon-scanner,trino-connector-scanner,preload-extensions,hadoop-deps,java-writer`.\nAll 64 reactor modules passed, including Checkstyle, FE compilation and\nFE/BE MaxCompute packaging. Maven wall time: 2m 31s.\n- Both connector dependency trees contain no Java LZ4 artifacts. The\nrebuilt FE plugin zip and BE connector jar contain no Java LZ4\njars/classes. The incremental output directory retained the previous\nbuild\u0027s LZ4 jar; that stale artifact was removed and the runtime plugin\ndirectory was rechecked.\n- `output/fe/lib` contains `grpc-netty-shaded-1.75.0.jar`; no Flight SQL\nJDBC driver jar is present under `output/fe`.\n- `git diff --check` passed. Live MaxCompute connectivity, cluster\nstartup and regression tests were not run." }, { "commit": "2be8fba29d7180f8bb153135d397ed2bda4254fa", "tree": "2e59d85cdb3b393ed70c8b84644537af4414fb72", "parents": [ "28c63a4809dabc4adc2b8b4de1d3099b53c5a2de" ], "author": { "name": "Luwei", "email": "814383175@qq.com", "time": "Mon Sep 07 22:25:24 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 22:25:24 2026 +0800" }, "message": "[improvement](snapshot) Add snapshot retained analysis interface (#67616)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: None\n\nProblem Summary: Add public Recycler HTTP routing and SnapshotManager\nextension methods for read-only snapshot retained diagnostics. The OSS\nSnapshotManager returns unsupported so enterprise implementations can\noverride the interface without public code depending on enterprise\ntypes.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test: Unit Test\n - ASAN Cloud build\n - MetaServiceHttpTest.ResolveHttpHandlerByVersion\n- Behavior changed: Yes. Adds two Recycler HTTP routes whose OSS\nimplementation reports unsupported.\n- Does this need documentation: No" }, { "commit": "28c63a4809dabc4adc2b8b4de1d3099b53c5a2de", "tree": "14f36ab32ff60718f332431b6c68f93197327c7d", "parents": [ "612896bff0d1f65d33b5e216157f8d8e7082948f" ], "author": { "name": "xy720", "email": "22125576+xy720@users.noreply.github.com", "time": "Mon Sep 07 22:12:27 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 22:12:27 2026 +0800" }, "message": "[fix](profile) Nereids Optimize Time shows N/A when MV pre-rewrite is skipped (#67532)\n\n`getPrettyNereidsOptimizeTime` computes the elapsed time as\n`optimizeFinish - preRewriteByMvFinish`,\n\nBut `preRewriteByMvFinish` is only set at the tail of\n`preMaterializedViewRewrite`, which early-exits whenever pre-rewrite is\nnot needed.\n\nThe start marker stays at -1, `getPrettyTime` sees -1 and returns \"N/A\"\n— even though CBO did run and `optimizeFinish` is set.\n\nThis misleads users on any query that skips MV pre-rewrite: \n1.MV refresh,\n2.INSERT, large joins using DpHyper,\n3.Sessions with `enable_materialized_view_rewrite\u003dfalse`, \n4.Queries that don\u0027t touch any MV at all\n\nExample:\nan MV refresh with\n`Rewrite Time: 4ms`,\n`Translate Time:239ms`\nstill prints `Optimize Time: N/A`, misleading anyone reading the profile\ninto thinking CBO was skipped.\n\nFall back the start marker through earlier phase finish times\n(`preRewriteByMv -\u003e collectTablePartition -\u003e rewrite`) so the elapsed\ntime is shown whenever the upstream phase actually finished." }, { "commit": "612896bff0d1f65d33b5e216157f8d8e7082948f", "tree": "364f328919110104eae8d628627f41ea09d253c4", "parents": [ "c7a44c738e6a8cbe6b03fd50129f9ca77eba4ef9" ], "author": { "name": "shuke", "email": "shuke@selectdb.com", "time": "Mon Sep 07 21:05:38 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 21:05:38 2026 +0800" }, "message": "[improvement](ci) Verify Litefuse subagent trace completeness (#67567)\n\nThe Litefuse exporter currently posts both the main review trace and\nselected subagent session traces, but `--verify` only reads back the\nmain trace. An OTLP HTTP success does not prove that all subagent\nobservations have become queryable, so a complete main trace can hide\npartially ingested or missing subagent traces." }, { "commit": "c7a44c738e6a8cbe6b03fd50129f9ca77eba4ef9", "tree": "d25e802eab4c5ee05fe78ccbfe1ca43ca7768bca", "parents": [ "54f5c41220568ddcd1cf11c7ce2ba360cf1ebe73" ], "author": { "name": "Jerry Hu", "email": "hushenggang@selectdb.com", "time": "Mon Sep 07 20:46:49 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 20:46:49 2026 +0800" }, "message": "[fix](auth) Add privilege checks to SHOW/EXPLAIN/REFRESH DICTIONARY (#67343)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #66218\n\nProblem Summary:\n\n`SHOW DICTIONARIES` and `EXPLAIN DICTIONARY` did not check any\nprivilege. Any\nuser who can `USE` a database (which only needs a privilege on some\ntable of\nthat database) could list every dictionary of the database together with\nits\nsource table name, status and BE data distribution, and describe its\ncolumns.\n`REFRESH DICTIONARY` only failed inside the internal `INSERT INTO`,\nafter the\ndictionary had been looked up and switched to `LOADING`.\n\nThis is inconsistent with `SHOW TABLES`, which hides tables the user\ncannot\nshow, and with `CREATE/DROP DICTIONARY`, which already require\nprivileges on the\ndictionary name (#66218).\n\nDictionaries are authorized like tables of the internal catalog, so:\n\n- `SHOW DICTIONARIES` now skips dictionaries the user has no `SHOW`\nprivilege\n on, the same way `SHOW TABLES` filters tables.\n- `EXPLAIN DICTIONARY` requires `SHOW` on the dictionary, like\n`DESCRIBE` on a\n table.\n- `REFRESH DICTIONARY` checks `LOAD` on the dictionary up front. This is\nthe\nprivilege the internal `INSERT INTO` already required, so nobody loses\nthe\nability to refresh; the check now happens before the dictionary is\nresolved\n and before its status is flipped to `LOADING`.\n\nThe checks run before the dictionary is looked up, so a denied user\ncannot\nprobe whether a dictionary exists either.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test\n- [x] Regression test: `auth_call/test_ddl_dictionary_auth` now covers a\nuser with a privilege on another table of the database (must not see,\ndescribe or refresh the dictionary), `SHOW_VIEW` on the database (sees\n the dictionary and its source table, may describe it, still cannot\n refresh), and `LOAD` on the database (may refresh).\n - [ ] Unit Test\n - [ ] Manual test (add detailed scripts or steps below)\n - [ ] No need to test or manual test. Explain why:\n\n- Behavior changed:\n - [ ] No.\n - [x] Yes. Users without `SHOW` on a dictionary no longer see it in\n `SHOW DICTIONARIES` and cannot `EXPLAIN DICTIONARY` it. `REFRESH\nDICTIONARY` still needs `LOAD` on the dictionary, but is now rejected\n before the dictionary is touched.\n\n- Does this need documentation?\n - [ ] No.\n- [x] Yes. The privilege requirements of the three statements should be\n documented.\n\nhttps://claude.ai/code/session_01X9KukfTLYxHmP6iYyEnQtW" }, { "commit": "54f5c41220568ddcd1cf11c7ce2ba360cf1ebe73", "tree": "0fca5a19d51e36c96649e1fed3eb0bcf02064ea8", "parents": [ "eea19b3f3cfef9e1bbbd559f9ea42954d8891e0f" ], "author": { "name": "morrySnow", "email": "zhangwenxin@selectdb.com", "time": "Mon Sep 07 18:16:18 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 18:16:18 2026 +0800" }, "message": "[improvement](cascades) Fuse project pushdown into join reorder (#67541)\n\n### What problem does this PR solve?\n\nProblem Summary:\n\n`AddProjectForJoin` inserts a `LogicalProject` above every Join before\nclassic Cascades exploration. Most of these Projects contain only Slots,\nbut the standalone `PushDownProjectThroughSemiJoin` and\n`PushDownProjectThroughInnerOuterJoin` exploration rules still expand\nchild groups and assemble temporary Plans before their predicates reject\nthe match. This creates substantial CPU and allocation overhead as the\nMemo grows.\n\nThis change:\n\n- extracts the existing Project normalization logic into a shared\nhelper;\n- invokes that helper only when a Project-aware Join reorder rule is\nready to produce an alternative;\n- covers inner associate/asscom/exchange, outer associate/asscom, and\nsemi-join transpose paths;\n- removes the two standalone PushDown rule factories from classic\n`OTHER_REORDER_RULES`;\n- keeps the standalone rules and their `AFTER_DPHYP_REORDER_RULES`\nregistration unchanged for DPHyp.\n\nLocal FE-only validation:\n\n- 24 targeted FE unit tests passed, including helper edge cases, all\nProject-aware reorder families, the existing standalone rule tests, and\na complete `AddProjectForJoin -\u003e classic optimizer` path.\n- TPC-H 22, TPC-DS 99, and one DPHyp smoke query produced identical\noptimizer mode, physical Plan fingerprint, root cost, and limit state\nbetween the registered baseline and candidate.\n- The candidate removed 162 redundant Memo expressions across six\nworkload queries without changing the selected Plan.\n- Across 12 STAR/DENSE × SLOT/COMPLEX × 3/8/16-table JMH pairs,\ncandidate mean planning time and allocation/op were lower in every\nconfiguration. The geometric baseline/candidate ratios were\n1.066x/1.114x for Slot Projects and 1.698x/1.617x for Complex Projects.\n- In a 9-table JFR workload, candidate main-thread allocation fell\n16.2%, matcher allocation 34.4%, Plan-assembly allocation 39.4%, and\n`withGroupExprLogicalPropChildren` allocation 40.7%.\n\n### Release note\n\nReduce Nereids classic Cascades planning overhead for queries with\nmultiple joins." }, { "commit": "eea19b3f3cfef9e1bbbd559f9ea42954d8891e0f", "tree": "9af42c413f0f76609e4c84c028235287e7c91904", "parents": [ "94f0d69ac8d31d06682e6b943c6a1b4495dd9ab2" ], "author": { "name": "foxtail463", "email": "foxtail463@gmail.com", "time": "Mon Sep 07 14:20:16 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 14:20:16 2026 +0800" }, "message": "[improvement](predicate) Extend monotonic predicate inference to storage scans (#67182)\n\n## Problem Summary:\nMonotonic-function predicates were only converted into bare-column\nranges during partition pruning. Consequently, OLAP zone maps and\nParquet/ORC min/max indexes could not use predicates such as\ndate_trunc(dt, ...) \u003d ..., date_format(dt, ...) \u003d ..., or substring(col,\n...) \u003d ... to skip irrelevant storage ranges.\n\n## Solution:\n- Reuse the monotonic predicate inference after CBO for OLAP scans, file\nscans, file TVFs, and storage-layer aggregate wrappers.\n- Preserve the original predicate and append only safety-checked\nnecessary conditions, avoiding changes to query semantics, cardinality\nestimation, join planning,\n and MV matching.\n\n- Derive tighter two-sided ranges for supported prefix, date-format, and\ndate rounding equalities.\n- Model adjacent bucket boundaries in date floor/ceil functions while\nconservatively excluding unsafe origins, timestamp-with-time-zone\nvalues, invalid\n boundaries, and non-ASCII prefix successors.\n\n- Prevent integer overflow when converting quarter periods into months.\n\nrelated pr: #66965\n\n## Performance Evaluation\n\n### Setup\n\n- **Planner benchmark:** two independent runs measured both end-to-end\nSQL planning latency and time spent in\n`AddMonotonicFunctionPruningPredicates`.\n- **Execution benchmark:** A-B-A order with the same FE binary, BE\nprocess, metadata, data, and session settings. A temporary local switch\nchanged only whether this post-processor ran.\n- **Data:** single-node, single-bucket OLAP table with 1,000,000 rows\nand 20 segments. Query, SQL, and condition caches were disabled,\ntogether with expression Zone Map pruning and count-on-index pushdown.\n- Positive cases used 5 warm-up and 20 measured runs; control cases used\n3 warm-up and 10 measured runs.\n\nThe BE binary was built with ASAN, so wall-clock speedups should be\ntreated as directional results for this targeted workload rather than\nproduction guarantees. Profile pruning ratios are deterministic.\n\n### Planner Overhead\n\nThe table shows P50 ranges from the two runs. Percentages are relative\nto the baseline total planning P50.\n\n| Plan shape | Rule time and share of planning time | Additional total planning time |\n| --------------------------------------- | -----------------------------------: | -------------------------------------------: |\n| Scan without a filter | 0.81–1.06 μs (~0.02%) | Approximately zero (-1.29% to +0.20%, noise) |\n| Simple unsupported filter | 15.93–16.02 μs (0.31%–0.37%) | 13.6–34.7 μs (+0.32%–0.68%) |\n| One supported inference | 59.25–324.15 μs (1.85%–8.01%) | 119.2–793.7 μs (+5.07%–19.61%) |\n| Complex OR/UNION or multiple inferences | 160.09–263.87 μs (1.65%–5.82%) | 328.5–1058.6 μs (+5.86%–9.39%) |\n\nThe supported cases covered `substring`, `year`, `date_format`,\n`date_trunc`, `date_floor`, `month_floor`, `month_ceil`, `to_date`, and\n`to_monday`. Ordinary single-scan queries incurred less than one\nmillisecond of additional planning time.\n\n### Execution Results\n\n| Scenario | Baseline P50 | PR P50 | Result |\n| ------------------------------------------------ | ---------------------------: | ---------------------------: | ----------------------------------------------------- |\n| Selective `date_trunc` | 1388.8 ms | 217.5 ms | 84.3% lower, 6.38× faster |\n| Selective `substring` | 2409.5 ms | 274.6 ms | 88.6% lower, 8.77× faster |\n| Mixed segment ranges with selective `date_trunc` | 1146–1310 ms | 707 ms | 38%–46% lower; midpoint: 42.4%, 1.74× faster |\n| Non-selective inferred year range | 1954.0 ms | 1934.7 ms | Approximately 1% difference; within measurement noise |\n| Unsupported predicate | Plans and profiles unchanged | Plans and profiles unchanged | No deterministic execution impact |\n\nFor the two selective, well-clustered cases, Zone Map pruning eliminated\n19 of 20 segments and reduced Profile `ScanRows` from 1,000,000 to\n50,000. `ScanBytes` decreased from 7.85 MB to 588.93 KB for\n`date_trunc`, and from 16.73 MB to 1.02 MB for `substring`.\n\nIn the mixed-layout case, every segment\u0027s min/max range covered the\nqueried day, so no segment was pruned and `ScanBytes` remained\nunchanged. The inferred raw `dt` range still vector-filtered 950,000\nrows before evaluating `date_trunc`, reducing expensive expression work\nwithout reducing storage I/O.\n\n### Conclusion\n\nThe optimization provides the largest benefit when the inferred range is\nselective and partition, segment, or row-group min/max ranges permit\npruning. Even without storage pruning, the raw-column predicate may\nreduce expensive expression evaluation. Non-selective and unsupported\npredicates show no deterministic execution benefit, while planner\noverhead remains small and sub-millisecond for ordinary plans.\n\n---------\n\nCo-authored-by: yangtao555 \u003cyangtao555@jd.com\u003e" }, { "commit": "94f0d69ac8d31d06682e6b943c6a1b4495dd9ab2", "tree": "3ed9d3cb950d02df37c7b50b31d7e0b0841de3f2", "parents": [ "efedf10c7e35877bc0aa36573cd1f2af621367b9" ], "author": { "name": "shuke", "email": "shuke@selectdb.com", "time": "Mon Sep 07 14:04:31 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 14:04:31 2026 +0800" }, "message": "[improvement](ci) Accept Astra and Fable 5.1 review receipts (#67561)\n\nProblem Summary:\nA converged local review using GPT-6 Astra or Claude Fable 5.1 is\nrejected by the PASS receipt validator with \"model is not allowed\", even\nwhen the review meets the existing effort, commit, reviewer, and\nfindings requirements." }, { "commit": "efedf10c7e35877bc0aa36573cd1f2af621367b9", "tree": "a3c85ebdd0ead91a1b99acc8fca25e4ceb9553c7", "parents": [ "f75e66f8810b1fa369c2f4d4003593be67fdd887" ], "author": { "name": "shuke", "email": "shuke@selectdb.com", "time": "Mon Sep 07 11:21:19 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 11:21:19 2026 +0800" }, "message": "[improvement](ci) Harden Litefuse OTLP reliability (#67416)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: #67413 (merged)\n\nProblem Summary:\n\nLarge code reviews repeatedly serialize growing batches during Litefuse\npre-chunking, transient HTTP failures stop export immediately, and the\nmain review trace can pass read-back verification after only a small\nsubset of its exported observations becomes visible.\n\nThis PR makes three focused changes:\n\n1. Makes legacy pre-chunk size accounting linear in the encoded event\ndata while preserving event order, exact encoded sizes, and the existing\nOTLP-aware truncation and adaptive HTTP 413 handling.\n2. Retries HTTP 429, 502, 503, and 504 within the configured per-export\nretry budget and delay, with an HTTP retry counter alongside existing\ntransport and payload-size counters.\n3. Strengthens read-back verification of the **main review trace**: it\nmust expose at least its exported observation count in unique, non-empty\nIDs, with no duplicate or missing IDs, in addition to the existing I/O\nand step checks.\n\nThe existing legacy-observations-first read order, v2 and trace-detail\nfallbacks, and pagination remain unchanged. Main and subagent exports\nboth use the chunking and transport improvements; read-back verification\nremains scoped to the main trace. Verifying separately exported subagent\ntraces is outside this PR\u0027s scope." }, { "commit": "f75e66f8810b1fa369c2f4d4003593be67fdd887", "tree": "3b30ace8fce181fb9f739efcfc9e79b2a4e09339", "parents": [ "cace2bb01f15ee6f8160eff17004052bb9aa18cc" ], "author": { "name": "Gabriel", "email": "liwenqiang@selectdb.com", "time": "Mon Sep 07 10:59:19 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 10:59:19 2026 +0800" }, "message": "[improvement](parquet) Prune array_contains with row group statistics (#67441)\n\n## Summary\n\n- Enable `array_contains(array_column, constant)` to participate in\nexpression-level ZoneMap evaluation for supported primitive array\nelements.\n- Resolve the logical ARRAY slot to its physical Parquet LIST element\nleaf in File Scanner V2 and use row-group footer min/max statistics to\nreject impossible matches.\n- Keep missing, incompatible, or unsafe statistics conservative, and\ndeliberately exclude repeated leaves from page-index pruning because\npages do not preserve parent-row boundaries.\n- Generalize the nested metadata probe path so Parquet Bloom and ZoneMap\npruning share the same validated physical leaf resolution.\n- Preserve scalar Page Index pruning when the predicate is wrapped by\n`RuntimeFilterExpr`.\n- Keep Page Index scan test doubles structurally equivalent to\nproduction scalar comparisons so the physical probe classifier sees\ntheir slot and literal children.\n\n## Fix boundary\n\n- The metadata-safe `array_contains` opt-in is limited to signatures\naccepted by the current BE execution dispatch. Complex ARRAY, MAP, and\nSTRUCT element signatures remain unsupported and are kept outside the\nsafe prefix so metadata pruning cannot suppress their residual runtime\nerror.\n- `RuntimeFilterExpr` is unwrapped only for Page Index capability and\npath classification. The original wrapper remains in the scan request\nand is used for actual evaluation.\n- This change does not add Page Index pruning for repeated ARRAY leaves;\n`array_contains` continues to use row-group footer statistics only.\n- The TeamCity follow-up changes only test fixtures. The production Page\nIndex admission gate remains conservative.\n\n## Testing\n\n- Added RED/GREEN coverage for complex-element `array_contains`\nsafe-prefix rejection.\n- Added RED/GREEN coverage for runtime-filter-only Page Index admission\nand range selection.\n- Reproduced all three BE UT failures from TeamCity build 1037684\nlocally before the test-fixture fix; all three pass afterward.\n- `TableReaderTest.*`, `NativeParquetStatisticsTest.*`,\n`ExprZonemapFilterTest.*`, and `ParquetBloomFilterPruningTest.*` (194\nrelated tests passed).\n- Page Index coverage from `NewParquetReaderTest` and `ParquetScanTest`\n(130 related tests passed).\n- clang-format 16 dry run on all changed C/C++ files.\n- `build-support/check-build-hygiene.sh`.\n- `git diff --check`.\n\n## Links\n\nNone" }, { "commit": "cace2bb01f15ee6f8160eff17004052bb9aa18cc", "tree": "fe1c98350004ec945e9b4fd44c236ca8af870da1", "parents": [ "4d9468cdba36a227f071ad9d7a56fcc8c03b8f16" ], "author": { "name": "shuke", "email": "shuke@selectdb.com", "time": "Mon Sep 07 10:58:38 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 10:58:38 2026 +0800" }, "message": "[improvement](workflow) Accept qualified local review receipts (#66959)\n\nProblem Summary:\nThe hosted code-review pipeline can run out of review tokens even when a\ncommitter has already completed the equivalent `doris-repo-review`\nworkflow locally. Accept a strictly formatted `doris-repo-review/v1`\nPASS comment as an alternative source for the existing `code-review`\ncommit status.\n\nThe trusted default-branch workflow now:\n\n- accepts only comments authored by users with effective `write` or\n`admin` repository permission;\n- requires the receipt commit to equal the live PR head;\n- requires the reviewed base to equal the live base or be its ancestor\nby no more than 48 hours of commit history;\n- enforces the exact Opus 5, Fable 5, and GPT-5.6 Sol model allowlist at\n`xhigh`, `max`, or `ultra` effort;\n- requires `PASS`, convergence, one to three rounds, and zero\nBlocker/Major findings;\n- writes `code-review: success` only after every check passes.\n\nThe 48-hour base rule is evaluated when the comment is created or\nedited. Runtime identity remains an auditable local declaration rather\nthan cryptographic proof, so the existing write-permission boundary and\nlater sampling remain part of the trust model." }, { "commit": "4d9468cdba36a227f071ad9d7a56fcc8c03b8f16", "tree": "7f8c0b60d9df7521eba93d711bd86cc1ac0d93a3", "parents": [ "cf1a553731a84a06b62391518ef42199bbb6718f" ], "author": { "name": "Calvin Kirs", "email": "guoqiang@selectdb.com", "time": "Mon Sep 07 10:47:28 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 10:47:28 2026 +0800" }, "message": "[fix](subquery) Resolve correlated qualified columns before dereference (#67438)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: None\n\nProblem Summary:\n\nIn a correlated subquery, a multipart reference can represent either a\nrelation-qualified column (`table_alias.column`) or a nested-field\ndereference (`column.field`). Nereids previously searched the inner\nscope completely before checking the outer scope. If an inner table had\na physical column with the same name as an outer table alias, the outer\nreference could therefore be interpreted as a nested field of that inner\ncolumn.\n\nFor scalar inner columns this raised a `No such field` analysis error.\nFor complex inner columns it could bind successfully to the wrong\nexpression and produce incorrect results.\n\n#### Example\n\n```sql\nCREATE TABLE outer_events (\n id INT,\n `@event_name` VARCHAR(32)\n)\nDISTRIBUTED BY HASH(id) BUCKETS 1\nPROPERTIES (\"replication_num\" \u003d \"1\");\n\nCREATE TABLE inner_events (\n id INT,\n t1 INT\n)\nDISTRIBUTED BY HASH(id) BUCKETS 1\nPROPERTIES (\"replication_num\" \u003d \"1\");\n\nINSERT INTO outer_events VALUES (1, \u0027blocked\u0027), (2, \u0027kept\u0027);\nINSERT INTO inner_events VALUES (1, 0);\n\nSELECT t1.id, t1.`@event_name`\nFROM outer_events t1\nWHERE NOT EXISTS (\n SELECT 1\n FROM inner_events inner_alias\n WHERE t1.`@event_name` \u003d \u0027blocked\u0027\n)\nORDER BY t1.id;\n```\n\nBefore this fix, the outer alias `t1` conflicted with the physical inner\ncolumn `inner_events.t1`. Nereids treated ``t1.`@event_name``` as a\nnested-field access on the inner scalar column and failed during\nanalysis:\n\n```text\nNo such field \u0027@event_name\u0027 in \u0027t1\u0027\n```\n\nAfter this fix, ``t1.`@event_name``` is correctly bound to the outer\nrelation alias and the query returns:\n\n```text\n+------+-------------+\n| id | @event_name |\n+------+-------------+\n| 2 | kept |\n+------+-------------+\n```\n\nThis change resolves relation-qualified columns in the current and outer\nscopes before falling back to first-part-as-column dereference. The\nrelation-only phase follows each analyzer\u0027s complete local scope order\nbefore searching the outer scope, so custom HAVING and QUALIFY scopes\npreserve normal nearest-relation shadowing.\n\nLambda lexical scope is preserved: in `array_map(x -\u003e x.value,\nx.items)`, the first `x` inside the lambda resolves to the lambda\nargument while the second `x` resolves to the enclosing table alias.\nWhen an outer qualified reference contains nested fields, such as\n`outer_alias.payload.k`, the underlying `payload` slot is also recorded\nas a correlated slot.\n\n### Release note\n\nFix incorrect Nereids column binding when an outer table alias conflicts\nwith an inner column name in a correlated subquery.\n\n### Check List (For Author)\n\n- Test: Regression test / Unit Test\n- `./run-fe-ut.sh --run\norg.apache.doris.nereids.rules.analysis.TestDereference`\n - Result: 10 tests passed, 0 failures\n- Added `query_p0/test_dereference` regression coverage for scalar and\ncomplex inner columns, lambda lexical scope, outer nested-field\ncorrelation, reused inner/outer aliases in HAVING and QUALIFY, and local\nqualifier shadowing\n - `./run-regression-test.sh --run -d query_p0 -s test_dereference`\n - Result: suite passed against the generated snapshot\n- Behavior changed: Yes (relation-qualified correlated references now\ntake priority over inner column-field dereference while preserving\ncomplete local-scope precedence)\n- Does this need documentation: No" }, { "commit": "cf1a553731a84a06b62391518ef42199bbb6718f", "tree": "04ed27f1042d69598c0e566f94121f30efe58f77", "parents": [ "66f237a9fbbc8048cb7c69a7a11829f722857aa0" ], "author": { "name": "TengJianPing", "email": "tengjianping@selectdb.com", "time": "Mon Sep 07 10:10:47 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 10:10:47 2026 +0800" }, "message": "[fix](be) Eliminate undefined float-to-wide-integer conversion (#67463)\n\n### What problem does this PR solve?\n\nIssue Number: None\n\nProblem Summary: Casting scaled Float or Double values to Decimal256\ndecomposed large floating-point values in UINT64_MAX-sized chunks.\nRounding could produce an out-of-range remainder and trigger undefined\nfloating-point-to-integer conversion. Decode IEEE-754 binary64 directly\ninto the wide integer, add safe Float32 and long-double paths, and\nremove the obsolete platform-specific Boost implementation TU.\n\n### Release note\n\nFix unstable Float and Double to Decimal256 cast results.\n\n### Check List (For Author)\n\n- Test: Unit Test\n- Behavior changed: Yes. Large finite floating-point values are\nconverted deterministically instead of relying on undefined behavior.\n- Does this need documentation: No\n\n### What problem does this PR solve?\n\nIssue Number: close #xxx\n\nRelated PR: #xxx\n\nProblem Summary:\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test \u003c!-- At least one of them must be included. --\u003e\n - [ ] Regression test\n - [ ] Unit Test\n - [ ] Manual test (add detailed scripts or steps below)\n - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n - [ ] Previous test can cover this change.\n - [ ] No code files have been changed.\n - [ ] Other reason \u003c!-- Add your reason? --\u003e\n\n- Behavior changed:\n - [ ] No.\n - [ ] Yes. \u003c!-- Explain the behavior change --\u003e\n\n- Does this need documentation?\n - [ ] No.\n- [ ] Yes. \u003c!-- Add document PR link here. eg:\nhttps://github.com/apache/doris-website/pull/1214 --\u003e\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label \u003c!-- Add branch pick label that this PR\nshould merge into --\u003e" }, { "commit": "66f237a9fbbc8048cb7c69a7a11829f722857aa0", "tree": "d4c50c96d39117485bb4bc728c2269162ccda4d8", "parents": [ "23b5ee5700fdda55857fdf805e6c2a2ec50fde36" ], "author": { "name": "starocean999", "email": "lichi@selectdb.com", "time": "Mon Sep 07 10:02:59 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 10:02:59 2026 +0800" }, "message": "[fix](config) Mask sensitive FE configs in SHOW FRONTEND CONFIG via @ConfField(sensitive) (#67338)" }, { "commit": "23b5ee5700fdda55857fdf805e6c2a2ec50fde36", "tree": "31224c76df9aacdaca7ab3f05ee4ad3379a42078", "parents": [ "287a334b63ce4d4fa83ecde7cef96d73f4426490" ], "author": { "name": "Raghvendra Singh", "email": "raghav@cashify.in", "time": "Mon Sep 07 07:25:02 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 09:55:02 2026 +0800" }, "message": "[fix](arrow-flight) Release a finished Flight query\u0027s coordinator instead of holding it until wait_timeout (#67504)\n\n### What problem does this PR solve?\n\nIssue Number: close #67503\n\nRelated PR: #64799 (kept the coordinator alive across GetFlightInfo →\nDoGet so the BE can fetch external-table splits; this PR narrows that to\nthe queries that actually need it and bounds how long an abandoned\nsession can hold the rest)\n\nProblem Summary:\n\nSince #64799 every Arrow Flight SQL query whose results are produced on\nthe BE kept its coordinator alive after `GetFlightInfo`, until the\nsession ran its next query or was closed. The coordinator holds the\nquery\u0027s workload-group queue slot and its `active_queries` registration.\nMost Flight clients open a session per query and never send\n`CloseSession` (the ADBC/JDBC drivers don\u0027t do it on connection close,\nas the comment in `DorisFlightSqlProducer.closeSession` notes), so each\nabandoned session pinned one slot for up to `wait_timeout` (8h by\ndefault).\n\nObserved on a 2-FE cluster with a `max_concurrency\u003d8` workload group:\neight finished Flight queries sat in `information_schema.active_queries`\nas RUNNING for over an hour, and every later query in the group failed\nwith `query queue timeout, timeout: 60000 ms`. Full write-up in #67503.\n\nThis PR fixes it in two steps:\n\n1. **Only defer the coordinator when the BE still needs it.** The\ndeferral added by #64799 is required only for an external-table scan in\nbatch mode, where the BE lazily fetches splits during `DoGet` from the\nsplit source the coordinator holds. Every other query (internal tables,\nexternal tables in non-batch mode) now closes its coordinator at the end\nof `GetFlightInfo` again, releasing the queue slot and the\n`active_queries` entry right away. That is safe: the BE buffers its\nresults independently of the coordinator, and finalizing the FE side\ndoes not cancel BE execution (`QeProcessorImpl.reportExecStatus` accepts\nreports for an unregistered query, and `Coordinator.close()` only\nreleases the queue token and the split sources). New predicates\n`ScanNode.hasBatchSplitSource()` / `Coordinator.hasBatchSplitSource()`\ndrive the gate in `StmtExecutor.executeAndSendResult`.\n\n2. **Bound the remaining deferred queries without killing the session.**\nNew mutable FE config `arrow_flight_deferred_query_idle_timeout_second`\n(default 3600). Once a Flight session has been sleeping for longer than\nthis since its last query started, the connection timeout checker\nfinalizes the session\u0027s deferred executors (releasing the slot and the\nregistration) and leaves the session alive; `wait_timeout` still governs\nthe session itself. The bound is floored at the execution timeout the\ndeferred query actually ran with, captured when the executor is deferred\n(`SET_VAR` hint values are reverted at the end of `execute()`, so the\nsession value cannot be read later). `0` disables the bound.\n\nWhy not kill the session (the first revision of this PR): a killed\nFlight session\u0027s bearer token stays in the token cache marked as already\nused, so the client\u0027s next call on it fails with `UserSession expire\nafter access` and has to re-handshake. With a 1h bound, every pooled or\nBI-tool Flight connection that idles for an hour would fail once on its\nnext use. Reaping only the deferred query releases the leaked resources\nwith no client-visible change.\n\n### Release note\n\nArrow Flight SQL: a query no longer holds its workload-group queue slot\nand `active_queries` entry after `GetFlightInfo` unless it is an\nexternal-table scan in batch mode, the only case where the BE still\nfetches splits from the FE during `DoGet`. For that case a new FE config\n`arrow_flight_deferred_query_idle_timeout_second` (default 1h) releases\nthe coordinator of an idle, never-closed session without killing the\nsession.\n\n### Check List (For Author)\n\n- Test \u003c!-- At least one of them must be included. --\u003e\n- [x] Regression test\n(`arrow_flight_sql_p0/test_arrow_flight_query_release`: a finished\nFlight query on an internal table no longer occupies a\n`max_concurrency\u003d1` group and is gone from `active_queries`;\n`external_table_p0/iceberg/test_iceberg_arrow_flight_split_source`: the\nbatch-mode scan stays registered after `DoGet` and is released by the\nidle reaper while the session survives)\n- [x] Unit Test (`FlightSqlDeferredQueryIdleTimeoutTest`: the reaper\nthrough `checkTimeout`, the exec-timeout floor, `0` disables, nothing\ndeferred, MySQL untouched; `ArrowFlightDeferralGateTest`: the\nbatch-split-source predicates;\n`StmtExecutorTest.testDeferForArrowFlightFreezesExecTimeoutInEffect`)\n- [x] Manual test (the first revision was verified on a test cluster\nwith an 8s bound, see the PR history; the current revision is covered by\nthe regression tests above)\n - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n - [ ] Previous test can cover this change.\n - [ ] No code files have been changed.\n - [ ] Other reason \u003c!-- Add your reason? --\u003e\n\n- Behavior changed:\n - [ ] No.\n- [x] Yes. (1) A Flight query that is not an external-table batch-mode\nscan releases its coordinator, queue slot and `active_queries` entry at\nthe end of `GetFlightInfo`, as it did before #64799. (2) The deferred\ncoordinator of a batch-mode scan on an idle session is released after 1h\nby default instead of at `wait_timeout`; set\n`arrow_flight_deferred_query_idle_timeout_second\u003d0` for the previous\nbehavior. Sessions are never killed by this change.\n\n- Does this need documentation?\n - [ ] No.\n- [x] Yes. New FE config\n`arrow_flight_deferred_query_idle_timeout_second` — doris-website PR to\nfollow once this is reviewed.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label \u003c!-- #64799 is in 4.0.8 / 4.1.4 --\u003e\n\n---------\n\nSigned-off-by: Raghvendra Singh \u003craghav@cashify.in\u003e\nCo-authored-by: Raghav \u003craghav@Raghavs-RMac.local\u003e\nCo-authored-by: Claude Fable 5 \u003cnoreply@anthropic.com\u003e\nCo-authored-by: morningman \u003cyunyou@selectdb.com\u003e" }, { "commit": "287a334b63ce4d4fa83ecde7cef96d73f4426490", "tree": "0b9603de70aa3e6cc678e2616ef14a9d049eec29", "parents": [ "74ccaf8938debee865e97d8f7fbd301017f5580f" ], "author": { "name": "Pxl", "email": "pxl290@qq.com", "time": "Mon Sep 07 09:53:09 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 09:53:09 2026 +0800" }, "message": "[feature](function) Support map arguments for inner_product (#67311)\n\nRelated PR: apache/doris-website#4113\n\nProblem Summary:\n\n`inner_product` currently only accepts dense `ARRAY\u003cFLOAT\u003e` vectors.\nThis PR extends it to sparse vectors represented as `MAP\u003cK, FLOAT\u003e`,\nwhere matching map keys identify dimensions.\n\nMap keys are intentionally limited to integral and string types. The BE\ndispatches to concrete native key types and hashes keys directly:\nnumeric keys use their native column data, while string keys use\nzero-copy `StringRef` access. The implementation does not serialize keys\nor use runtime type erasure. For each row, it builds a flat hash map\nfrom the smaller input map and probes it with the larger map, using O(m\n+ n) time and O(min(m, n)) temporary space. Existing dense array\nbehavior remains unchanged.\n\nThe implementation also validates unsupported key types in FE and BE,\npreserves NULL-key matching, and rejects NULL map values and NULL outer\nmaps.\n\n### Release note\n\nSupport `inner_product(MAP\u003cK, FLOAT\u003e, MAP\u003cK, FLOAT\u003e)` for integral and\nstring key types.\n\n### Check List (For Author)\n\n- Test\n - [x] Regression test\n - `test_map_inner_product`\n - [x] Unit Test\n - `FunctionMapInnerProductTest.*` (11 tests under ASAN)\n - [ ] Manual test (add detailed scripts or steps below)\n - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n - [ ] Previous test can cover this change.\n - [ ] No code files have been changed.\n - [ ] Other reason\n\nAdditional validation:\n\n- `DISABLE_BE_CDC_CLIENT\u003dON ./build.sh --be`\n- `DISABLE_BUILD_UI\u003dON ./build.sh --fe`\n- `build-support/check-build-hygiene.sh`\n- `build-support/check-format.sh`\n\n- Behavior changed:\n - [ ] No.\n- [x] Yes. `inner_product` now accepts compatible MAP arguments in\naddition to ARRAY arguments.\n\n- Does this need documentation?\n - [ ] No.\n - [x] Yes. apache/doris-website#4113\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label" }, { "commit": "74ccaf8938debee865e97d8f7fbd301017f5580f", "tree": "4c014934c4d97b04679cce4186e269318b9de079", "parents": [ "b58b2c53ff56354c692c2dc7634d724d8780838f" ], "author": { "name": "zhangstar333", "email": "zhangsida@selectdb.com", "time": "Sun Sep 06 16:43:42 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Sep 06 16:43:42 2026 +0800" }, "message": "[chore](lance) update lance version to tag 0.1.9 (#67496)\n\n### What problem does this PR solve?\nProblem Summary:\nupdate lance version to tag 0.1.9 \nand pick proto files have changed in branch-41\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test \u003c!-- At least one of them must be included. --\u003e\n - [ ] Regression test\n - [ ] Unit Test\n - [ ] Manual test (add detailed scripts or steps below)\n - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n - [ ] Previous test can cover this change.\n - [ ] No code files have been changed.\n - [ ] Other reason \u003c!-- Add your reason? --\u003e\n\n- Behavior changed:\n - [ ] No.\n - [ ] Yes. \u003c!-- Explain the behavior change --\u003e\n\n- Does this need documentation?\n - [ ] No.\n- [ ] Yes. \u003c!-- Add document PR link here. eg:\nhttps://github.com/apache/doris-website/pull/1214 --\u003e\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label \u003c!-- Add branch pick label that this PR\nshould merge into --\u003e" }, { "commit": "b58b2c53ff56354c692c2dc7634d724d8780838f", "tree": "b91f4f270329ddbfbdafe7e2ad5d8fc155047091", "parents": [ "caa7bbf468419db69c1f282d30143c84a40653e7" ], "author": { "name": "yiguolei", "email": "guolei@selectdb.com", "time": "Sat Sep 05 15:31:47 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Sep 05 15:31:47 2026 +0800" }, "message": "[enhancement](thirdparty) fix arrow build bug and clear dangerous env… (#67535)\n\n… (#67523)\n\nIssue Number: close #xxx\n\nRelated PR: #xxx\n\nProblem Summary:\n\nNone\n\n- Test \u003c!-- At least one of them must be included. --\u003e\n - [ ] Regression test\n - [ ] Unit Test\n - [ ] Manual test (add detailed scripts or steps below)\n - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n- [ ] Previous test can cover this change. - [ ] No code files have been\nchanged. - [ ] Other reason \u003c!-- Add your reason? --\u003e\n\n- Behavior changed:\n - [ ] No.\n - [ ] Yes. \u003c!-- Explain the behavior change --\u003e\n\n- Does this need documentation?\n - [ ] No.\n- [ ] Yes. \u003c!-- Add document PR link here. eg:\nhttps://github.com/apache/doris-website/pull/1214 --\u003e\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label \u003c!-- Add branch pick label that this PR\nshould merge into --\u003e\n\n### What problem does this PR solve?\n\nIssue Number: close #xxx\n\nRelated PR: #xxx\n\nProblem Summary:\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test \u003c!-- At least one of them must be included. --\u003e\n - [ ] Regression test\n - [ ] Unit Test\n - [ ] Manual test (add detailed scripts or steps below)\n - [ ] No need to test or manual test. Explain why:\n- [ ] This is a refactor/code format and no logic has been changed.\n - [ ] Previous test can cover this change.\n - [ ] No code files have been changed.\n - [ ] Other reason \u003c!-- Add your reason? --\u003e\n\n- Behavior changed:\n - [ ] No.\n - [ ] Yes. \u003c!-- Explain the behavior change --\u003e\n\n- Does this need documentation?\n - [ ] No.\n- [ ] Yes. \u003c!-- Add document PR link here. eg:\nhttps://github.com/apache/doris-website/pull/1214 --\u003e\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label \u003c!-- Add branch pick label that this PR\nshould merge into --\u003e" }, { "commit": "caa7bbf468419db69c1f282d30143c84a40653e7", "tree": "7bebefe4500e118dc6e9135e29bb86659569a42a", "parents": [ "050442dfc4757b5f042c31e64efd12ed9e9839d2" ], "author": { "name": "Mingyu Chen (Rayner)", "email": "morningman.cmy@gmail.com", "time": "Sat Sep 05 15:29:21 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Sep 05 15:29:21 2026 +0800" }, "message": "[fix](test) move StreamingInsertJobStatusTransitionTest to JUnit 5 (#67543)\n\n### What problem does this PR solve?\n\nIssue Number: close #xxx\n\nRelated PR: #67396, #67529, #66729\n\nProblem Summary:\n\n**master is red on `fe-core` checkstyle.** #67396 removed JUnit 4 from\nthe fe\nreactor and added a `banJUnit4` checkstyle rule so it could not come\nback.\n#67529 added `StreamingInsertJobStatusTransitionTest`, written with\nJUnit 4.\nThe two crossed in flight: the migration could not cover a file that did\nnot\nexist when it was written, and the new test was branched before the rule\nexisted.\n\n`mvn checkstyle:check -pl fe-core` on master reports exactly two errors,\nboth\nin this file, on its two JUnit 4 imports:\n\n```\nsrc/test/java/org/apache/doris/job/extensions/insert/streaming/\n StreamingInsertJobStatusTransitionTest.java:23 error\n StreamingInsertJobStatusTransitionTest.java:24 error\n```\n\nThe fix is mechanical: two imports and six `Assert.` call sites. All six\nare\none- or two-argument forms carrying no assertion message, so none is\naffected\nby the JUnit 4 -\u003e 5 message reordering (JUnit 4 puts an assertion\nmessage\n**first**, JUnit 5 puts it **last**). Every argument stays exactly where\nit is.\n\n**Why this is worth its own PR rather than waiting.**\n`junit-vintage-engine` is\nwhat runs a JUnit 4 test in this reactor, and it is on its way out -\n#66729\nremoves it once the `be-java-extensions` modules are migrated, which is\nthe last\nthing keeping it alive. Without that engine, the jupiter engine does not\nfail on\na JUnit 4 test, it **ignores** it: these three cases would stop running\nand\nnothing would say so. That silent-skip is exactly the failure #67396\nmade the\ngate a checkstyle rule for, and exactly why it held the engine back\nuntil the\ntree was clean. Landing this first keeps that ordering safe." }, { "commit": "050442dfc4757b5f042c31e64efd12ed9e9839d2", "tree": "2fb2c502f6c1b34d904b6ace05b067ad0f5e97ec", "parents": [ "ad4e9d730803f32196ec04acb9a36f196b6c33d1" ], "author": { "name": "Mingyu Chen (Rayner)", "email": "morningman.cmy@gmail.com", "time": "Sat Sep 05 14:02:26 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Sep 05 14:02:26 2026 +0800" }, "message": "[fix](test) remove JUnit 4 from the fe reactor and unblock the dependency licence review (#67396)\n\n### What problem does this PR solve?\n\nIssue Number: close #xxx\n\nRelated PR: #xxx\n\nProblem Summary:\n\n`Dependency License Review` fails any pull request that adds a JUnit\ndependency to a pom — in practice, any pull request that adds a Java\nmodule with a test. It happened twice on 2026-09-01 alone, on two\nunrelated branches:\n\n```\nThe following dependencies have incompatible licenses:\nfe/be-java-extensions/hive-apache-shade/pom.xml » junit:junit@ – License: EPL-1.0\nfe/be-java-extensions/jni-spi/pom.xml » org.junit.jupiter:junit-jupiter@ – License: LicenseRef-bad-non-standard\n...\n```\n\nThis PR fixes the immediate failure and then removes the thing that made\na licence exception necessary in the first place.\n\n---\n\n## 1. The workflow exception (`third_party_review.yml`)\n\nNeither reported licence is one the project has not approved:\n\n* **`org.junit.jupiter:junit-jupiter` is EPL-2.0**, which\n`allow-licenses` already carries. GitHub\u0027s dependency graph reports it\nas `LicenseRef-bad-non-standard`, so the check rejects a licence the\nproject already accepted — a data-quality gap on GitHub\u0027s side, not a\ndependency problem.\n* **`junit:junit` is EPL-1.0**, an ASF Category B licence, test scope,\nand no release artifact ships it.\n\nBoth are excluded by purl — the same package-specific shape the existing\n`caniuse-lite` exception uses — rather than by widening `allow-licenses`\nor dropping `development` from `fail-on-scopes`.\n`allow-dependencies-licenses` excludes a package from the **licence\ncheck only**, so vulnerability reporting for test-scope dependencies is\nunchanged.\n\nOne detail not in the action\u0027s README: it matches a purl on **type and\nname and ignores the version** (`purlsMatch` in its `src/purl.ts`). The\nversion-less entries added here therefore cover every version, and the\nversion pin on the existing `caniuse-lite` entry has no effect either.\n\n## 2. Removing JUnit 4 from the fe reactor\n\n`junit:junit` is the one EPL-1.0 artifact in this build. Nothing\ndeclares it — `junit-vintage-engine` drags it in, and that engine exists\nonly to run the JUnit 4 tests still in the tree. 450 files, 441 of them\nin `fe-core`, were what kept it alive. They are JUnit 5 now.\n\n| what moved | count |\n|---|---|\n| `Assert` → `Assertions` | ~8600 call sites |\n| **message argument moved from first to last** | 407 |\n| `@Test(expected \u003d X.class)` → `assertThrows` | 94 methods |\n| `@Rule ExpectedException` → `assertThrows` + message substring\nassertion | 18 sites, 8 files |\n| `@Rule TemporaryFolder` → `@TempDir Path` | 4 files |\n| `Assume.assumeTrue(msg, cond)` → `assumeTrue(cond, msg)` | 12 sites |\n| `@Before`/`@After`/`@BeforeClass`/`@AfterClass`/`@Ignore` | all |\n| `@RunWith(MockitoJUnitRunner)` → `MockitoAnnotations.openMocks` | 1 |\n| `@FixMethodOrder` → `@TestMethodOrder` | 1 |\n| `junit.framework.AssertionFailedError` → `java.lang.AssertionError` |\n2 files |\n\nTwo traps the compiler cannot catch, handled explicitly:\n\n* **JUnit 4 puts an assertion message FIRST, JUnit 5 puts it LAST.**\n`javac` catches the swap for\n`assertTrue`/`assertFalse`/`assertNull`/`assertArrayEquals`, but **not**\nfor `assertEquals`/`assertSame`/`assertNotEquals` when the last argument\nis itself a `String`. The 8 three-argument `assertEquals` calls that are\na float delta rather than a message were identified and left alone.\n* **`Assert.assertEquals(Object[], Object[])` compares arrays; the JUnit\n5 `assertEquals` it would become compares references.** One such call\nwas converted to `assertArrayEquals`.\n\n### Three things only running the tests found\n\n* `JdbcSourceOffsetProviderAsyncSplitTest` spelled its teardown\n**`@org.junit.After`, fully qualified** — no import, so no import scan\ncould see it. The jupiter engine does not fail on a JUnit 4 annotation,\nit **ignores** it: the teardown stopped running, its `MockedStatic`\nnever closed, and 27 tests died on `static mocking is already\nregistered`. This is why the checkstyle pattern below matches anywhere\non a line and not just an import.\n* `StatsCalculatorTest.testFilterOutofRange` was annotated\n`@org.junit.Test` in a class the jupiter engine already ran — **it has\nnever executed**. Spelled `@Test` it runs, and passes.\n* `CloudAuthTest extends TestWithFeService`, whose setup is driven by\nJUnit 5 annotations the JUnit 4 engine never saw: no cluster was started\nand the inherited `connectContext` was always `null`, which is what\nevery command in the class was handed. Moving the class to JUnit 5 would\nhave activated that setup for the first time, against a class that mocks\n`Env` and `ConnectContext` statically. The vestigial inheritance is\ndropped instead, with a comment.\n\n## 3. The gate\n\nA checkstyle rule, because checkstyle runs at the `validate` phase with\n`includeTestSourceDirectory`: a JUnit 4 import fails a plain `mvn test`\n**locally**, rather than waiting for CI. The pattern also covers\n`junit.framework.*` (how a JUnit 3 import had survived here) and matches\nfully qualified references, for the reason above.\n\n`fe/be-java-extensions` is suppressed for now and the suppression says\nwhy: #66729 is rewriting those modules, so migrating them here would\nonly conflict. `junit-vintage-engine` comes out of `fe/pom.xml` when\nthat lands — **not before**, because without it a JUnit 4 test is\nsilently not run rather than failed.\n\nDeliberately out of scope: `extension/kettle` and `samples/` are\nstandalone maven projects, outside this reactor and built by no workflow\nhere, so their JUnit 4 cannot be verified from this build. The\n`pkg:maven/junit/junit` exception therefore stays." }, { "commit": "ad4e9d730803f32196ec04acb9a36f196b6c33d1", "tree": "01c0d00e709e075647efa90546af0406c382b9c8", "parents": [ "dc848b3fc32899ffe0a213da489ba6ef032cd1ae" ], "author": { "name": "Oliveira", "email": "43768685+OIiveirra@users.noreply.github.com", "time": "Sat Sep 05 10:44:26 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Sep 05 10:44:26 2026 +0800" }, "message": "[feature](be) Support file metadata columns in V2 scans (#67207)\n\n### What problem does this PR solve?\n\nIssue Number: close #66816\n\nRelated PR: None\n\nProblem Summary: FileScannerV2 native Parquet/ORC scans did not expose\nstable file-location metadata for Iceberg and Paimon. This change adds\nIceberg `_file`/`_pos` and Paimon\n`__paimon_file_path`/`__paimon_row_index`, preserves original source\npaths, and rejects unsupported JNI or V1 reader paths.\n\n### Release note\n\nExpose file path and physical row position metadata columns for\nsupported native Iceberg and Paimon scans.\n\n### Check List (For Author)\n\n- Test: Unit Test / Manual test / Regression test pending\n - Targeted Iceberg and Paimon connector unit tests passed.\n - Private Iceberg cluster manual validation passed.\n- External Docker regression will be run through `run buildall` and a\nretained CI environment.\n- Behavior changed: Yes (supported native external scans expose\nfile-location metadata columns)\n- Does this need documentation: No" }, { "commit": "dc848b3fc32899ffe0a213da489ba6ef032cd1ae", "tree": "78f82e91f448f97881d435bb980505c685bd3951", "parents": [ "43db29b8dbb5489587cda890fbbc7cc5268f2278" ], "author": { "name": "Mingyu Chen (Rayner)", "email": "morningman.cmy@gmail.com", "time": "Fri Sep 04 23:28:52 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 04 23:28:52 2026 +0800" }, "message": "[fix](temp-table) Fix CTAS/DROP for temporary tables and unmute 5 P0 cases (#67529)\n\n### What problem does this PR solve?\n\nFive P0 regression cases are muted on TeamCity. Three of them fail on\n**every** master build — they only look green on `branch-4.1` because\nthe suite is skipped there (`if (true) { return }` at the top of\n`test_temp_table.groovy`), so the mute is hiding persistent master\nfailures rather than flakiness. The other two are genuinely flaky.\n\n| muted case | failures / last 300 P0 runs |\n|---|---|\n| `nereids_rules_p0/pkfk/eliminate_inner` | 251 |\n| `compaction/test_vertical_compaction_agg_state` | 254 |\n| `temp_table_p0/test_temp_table` | 250 |\n| `load_p0/routine_load/test_routine_load` | 32 |\n| `query_p0/cache/sql_cache_object_type` | 10 / 103 |\n\nTwo of them turned out to be real FE bugs.\n\n#### 1. `CREATE TEMPORARY TABLE ... AS SELECT` wrongly rejected\n\nA temporary table is created under `\u003csessionId\u003e#TEMP#\u003cname\u003e`, but the\nCTAS existence probe added in #66112\n(`CreateTableCommand.targetTableExists`) looked up the **bare** name:\n\n```java\nreturn database !\u003d null \u0026\u0026 database.isTableExist(qualifiedName.get(2));\n```\n\nSo any normal table sharing that name makes the statement fail with\n`Table \u0027x\u0027 already exists`, even though the table it would create does\nnot exist. This is user-visible, not just a test problem:\n\n```sql\nCREATE TABLE t (id INT) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES(\u0027replication_num\u0027\u003d\u00271\u0027);\nCREATE TEMPORARY TABLE t PROPERTIES(\u0027replication_num\u0027\u003d\u00271\u0027) AS SELECT * FROM src;\n-- ERROR 1105: errCode \u003d 2, detailMessage \u003d Table \u0027t\u0027 already exists\n```\n\nThe probe now mangles the name exactly the way\n`InternalCatalog.createTable` does. Plain `CREATE TEMPORARY TABLE t\n(...)` was never affected, because that path goes straight to\n`Env.createTable`, which mangles internally. This is the failure\n`temp_table_p0/test_temp_table` hits at line 456.\n\n#### 2. `DROP TEMPORARY TABLE IF EXISTS` ignored `IF EXISTS`\n\n`Database.getTableNullable` resolves the temporary table first and falls\nback to the normal table, so when a session owns no temporary table of\nthat name, `table !\u003d null` and the earlier `ifExists` branch is skipped.\nThe `mustTemporary` guard then raised `Unknown table` unconditionally:\n\n```sql\nDROP TEMPORARY TABLE IF EXISTS never_existed; -- OK, no-op\nDROP TEMPORARY TABLE IF EXISTS t; -- ERROR 1105: Unknown table \u0027t\u0027 (t is a normal table)\n```\n\n`IF EXISTS` is now honored there. Without `IF EXISTS` the statement\nstill reports `Unknown table`, and a `DROP TEMPORARY TABLE` still never\ndrops the normal table.\n\n### Case and baseline fixes\n\n- **`eliminate_inner`** — baseline went stale on 2026-07-16 when #65264\nadded `shapeInfo()` overrides to `Cast`, `IsNull` and `Not`, which now\npreserve the table qualifier (`cast(f as ...)` → `cast(fkt_not_null.f as\n...)`). `branch-4.1` has no such override, which is why it stayed green\nthere. Regenerated the 8 affected shape lines.\n- **`test_vertical_compaction_agg_state`** — the first assertion\ncompared a literal `collect_set_merge` ordering, but `collect_set` is\nbacked by `flat_hash_set`, whose iteration order is unspecified. Wrapped\nit in `array_sort`, matching the two sibling assertions already in the\nsame suite. The set contents were never wrong, only their order.\n- **`test_routine_load`** — the `load_to_single_tablet` section waited\nonly for the job to leave `NEED_SCHEDULE` (i.e. to be *scheduled*), not\nfor a batch to be committed, and its baseline recorded the empty table\nthat race produced. It now uses the same data-visibility wait as the\nother nine sections, and the baseline holds the rows that actually load.\n9 of the 14 most recent failures of this suite were exactly this tag.\n- **`sql_cache_object_type`** — asserted that a cache entry survived.\nThe FE map holds soft values under a bounded size\n(`Config.sql_cache_manage_num`) and the rows live in the BE result\ncache, so neither is guaranteed to persist. It re-primes the cache\ninstead; the assertion that each `return_object_data_as_binary` setting\nis served its own result is unchanged." }, { "commit": "43db29b8dbb5489587cda890fbbc7cc5268f2278", "tree": "e5dfa80cf1e8debc560ae03760bbbc35828b6ef5", "parents": [ "4ab2cd710954a279c931581a294113736a6eda23" ], "author": { "name": "Chenyang Sun", "email": "sunchenyang@selectdb.com", "time": "Fri Sep 04 22:36:15 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 04 22:36:15 2026 +0800" }, "message": "[fix](zonemap) Treat reversed zone map bounds as invalid (#67431)\n\nNaN and infinity are recorded in the has_nan/has_positive_inf/\nhas_negative_inf flags and never move min/max, so a page holding nothing\nelse leaves both at the values add_values() starts from: min \u003d DBL_MAX\nand max \u003d -DBL_MAX. Those are written out as if they were data, and the\nflags stand in for at most one side, so reading them back reports a\nDBL_MAX that is nowhere in the page -- SELECT MIN(v) answers with it\nwhile SELECT COUNT(*) WHERE v \u003d that value answers 0. On data written\nbefore 4.0 there is no flag at all and both bounds are used, which\nprunes the page away: 25 rows of NaN return 0 for WHERE v \u003e 5.\n\nThe bounds round-trip exactly, so neither the flags nor the parse\nfailure #67341 keys on can tell such a zone map from a sound one. Mark\nit pass_all when the bounds come back reversed, which only FLOAT and\nDOUBLE can do -- any value of any other type moves both. Do the same in\nflush() and finish() so newly written pages and segments record that\nthey have no bounds instead of leaving it to the reader." }, { "commit": "4ab2cd710954a279c931581a294113736a6eda23", "tree": "685ac9fb502d857b690ff127850e0c40c7ecc8e4", "parents": [ "146d35955664cc86abb87b99130425444b2d18d7" ], "author": { "name": "Mryange", "email": "yanxuecheng@selectdb.com", "time": "Fri Sep 04 14:42:20 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 04 14:42:20 2026 +0800" }, "message": "[feature](function) Add array_except_all scalar function (#67132)\n\nProblem Summary: Add array_except_all for ARRAY\u003cscalar\u003e arguments. The\nfunction applies multiset difference semantics, preserves unmatched\nleft-side duplicates and order, handles nullable elements and constant\ncolumns through ColumnArrayView, and rejects complex element types. Add\nBE unit coverage and regression coverage for scalar types, nulls,\nconstants, duplicates, and unsupported nested types.\n\n\n\ndoc https://github.com/apache/doris-website/pull/4089" }, { "commit": "146d35955664cc86abb87b99130425444b2d18d7", "tree": "00b49c87ddbb92ee3b210c8533029bb2075ba6c5", "parents": [ "de9ed8ee90397881b7d77f6272bd41f633d4c7d9" ], "author": { "name": "morrySnow", "email": "zhangwenxin@selectdb.com", "time": "Fri Sep 04 14:35:53 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 04 14:35:53 2026 +0800" }, "message": "[improvement](parser) Assign query organization to a single owner (#67429)\n\n### What problem does this PR solve?\n\nProblem Summary: The grammar allowed both `querySpecification` and its\nouter `query` to consume `ORDER BY` and `LIMIT`, so ANTLR repeatedly\nentered `queryOrganization` and reported full-context ambiguities. This\nchange gives every clause group one owner according to ANSI mode and\nquery level, makes the rule structurally non-empty, and preserves the\nexisting accepted/rejected SQL matrix. It intentionally uses ANTLR\u0027s\ndefault error reporting; three incomplete `ORDER BY`/`LIMIT` forms now\nreport the end-of-input position instead of re-anchoring the error to\nthe clause keyword.\n\n### Benchmark\n\nThe original P2 benchmark is reused without rerunning it. After\nmeasurement, the follow-up cleanup only removed error-position\ncompatibility state and grammar actions; it added no parsing decisions\nand removed work from the valid-SQL path. Lower latency is better. The\ntarget input is `SELECT a, b, c FROM t WHERE a \u003e 1 ORDER BY a, b DESC\nLIMIT 20 OFFSET 10`; the control input has the same SELECT without\nquery-organization clauses.\n\n- Host: MacBookPro17,1, Apple M1 (8 cores, 16 GB), macOS 15.0.1\n- Runtime: OpenJDK 17.0.20.1, ANTLR 4.13.1, JMH 1.37, 1 thread, 1 GB\nheap\n- JMH: 3 forks, 4 x 300 ms warmup, 7 x 400 ms measurement; C1-B1-C2\ninterleaving\n- Measurement baseline: `5e0eadb13e9`; parser jar SHA-256\n`530f0ed45c4bac3a096373a1e932aae0025c603a6f905c3d975a2ad53bd0e7ca`;\nbenchmark jar SHA-256\n`7543595b2a87f55b9b53538336cfedf55c13484302ffb487efcfb4fdd66b1c0b`\n- Measurement candidate: `773c4ee1027`; parser jar SHA-256\n`65730a22a5262370a832bda7a04ae4b8e0de5c107be65c5ae80f95c711842f59`;\nbenchmark jar SHA-256\n`47284ffd6779e9d534a19a682de5a4d468f83a62a98e048982807497f1f665b6`\n- Harness:\n`fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/QueryOrganizationBenchmark.java`\n\n#### Ordered SELECT target\n\nThe candidate change is calculated from the mean of C1 and C2 against\nB1. Values are us/op with JMH\u0027s 99.9% error estimate.\n\n| Path | Mode | Baseline B1 | Candidate C1 | Candidate C2 | Latency Improvement |\n|:------------------------|:-------|------------------:|------------------:|------------------:|--------------------:|\n| End-to-end | Legacy | 9.926 ± 0.347 | 9.303 ± 0.333 | 9.122 ± 0.267 | **7.2% faster** |\n| End-to-end | ANSI | 9.170 ± 0.437 | 8.709 ± 0.646 | 8.492 ± 0.241 | **6.2% faster** |\n| Pre-tokenized parser | Legacy | 7.653 ± 0.266 | 7.246 ± 0.244 | 7.069 ± 0.090 | **6.5% faster** |\n| Pre-tokenized parser | ANSI | 7.344 ± 0.105 | 7.181 ± 0.099 | 7.099 ± 0.076 | **2.8% faster** |\n\n#### Plain SELECT control\n\n| Path | Mode | Baseline (us/op) | Candidate (us/op) | Latency Improvement |\n|:---------------------|:-------|-----------------:|------------------:|--------------------:|\n| End-to-end | Legacy | 6.580 ± 0.710 | 6.657 ± 0.809 | 1.2% slower |\n| End-to-end | ANSI | 6.944 ± 0.648 | 6.448 ± 0.440 | **7.1% faster** |\n| Pre-tokenized parser | Legacy | 5.684 ± 1.205 | 5.084 ± 0.054 | **10.6% faster** |\n| Pre-tokenized parser | ANSI | 5.313 ± 0.125 | 5.193 ± 0.089 | **2.3% faster** |\n\nNo control-path latency regression exceeds the 3% threshold.\n\n#### Allocation\n\nThe same artifacts were measured separately with `-prof gc`. Timing\nunder the profiler was noisy, so this table uses only normalized\nallocation.\n\n| Path | Mode | Baseline (B/op) | Candidate (B/op) | Allocation Change |\n|:---------------------|:-------|----------------:|------------------:|------------------:|\n| End-to-end | Legacy | 15,473.6 | 15,719.6 | +246.0 (+1.59%) |\n| End-to-end | ANSI | 15,426.8 | 15,536.1 | +109.3 (+0.71%) |\n| Pre-tokenized parser | Legacy | 12,368.1 | 12,381.4 | +13.3 (+0.11%) |\n| Pre-tokenized parser | ANSI | 12,317.4 | 12,421.4 | +104.0 (+0.84%) |\n\nAll allocation changes are below the 3% threshold.\n\n#### Profile attribution\n\n- Before this PR, tracked SQL entered `queryOrganization` 13,196 times\nand examined 49,902 lookahead tokens; SSB and Trino profiling reported\n40 and 20 related ambiguities.\n- After this PR, the 4,275 parseable tracked SQL files enter the target\nrule 2,794 times and examine 8,645 lookahead tokens, a reduction of\nabout 79% and 83% respectively.\n- LL_EXACT reports zero fallback and zero ambiguity for both the\n`querySpecification` and `queryOrganization` target decisions.\n- The gain comes from eliminating duplicate ownership and adaptive\nlookahead, not from lexer or token changes.\n\n### Semantic differential\n\n- Original corpus: all 4,610 tracked `*.sql` files; manifest SHA-256\n`567e209d57e5eaf6546ff03bf887437b8d647ed5f7ecb85bc657b987dd04be10`\n- Original result: 4,275 parsed and 335 rejected in both artifacts and\nin both ANSI modes\n- The follow-up cleanup does not change any grammar decision or error\noccurrence, so accepted/rejected behavior remains unchanged; the full\ncorpus was not rerun\n- Three deliberately changed first-error positions are covered by unit\ntests:\n - ANSI `SELECT 1 ORDER BY`: pos 9 -\u003e 17\n - ANSI `SELECT 1 LIMIT`: pos 9 -\u003e 14\n - Legacy `SELECT 1 LIMIT 1 ORDER BY`: pos 17 -\u003e 25\n- Lexer and token behavior are unchanged by construction\n\n### Release note\n\nMalformed `ORDER BY` and `LIMIT` clauses now use ANTLR\u0027s default error\npositions." }, { "commit": "de9ed8ee90397881b7d77f6272bd41f633d4c7d9", "tree": "dbc5696c82258ad9314a93eb61e40ef545f39e7e", "parents": [ "60854dbfb52cf1b165bd9111d93b91fd3644f909" ], "author": { "name": "linrrarity", "email": "linzhenqi@selectdb.com", "time": "Fri Sep 04 12:18:04 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 04 12:18:04 2026 +0800" }, "message": "[fix](udf) Reject variadic user-defined functions (#67373)\n\nUser-defined functions exposed variadic DDL metadata without reliable\nend-to-end support. Reject variadic declarations during `CREATE\nFUNCTION` analysis for scalar, aggregate, table, and alias functions\nwhile retaining variadic signature parsing for `DROP`, `SHOW`, and\nhistorical metadata compatibility.\n\nbefore:\n```sql\nCREATE FUNCTION py_add(INT, INT, ...)\nRETURNS INT\nPROPERTIES (\n \"type\" \u003d \"PYTHON_UDF\",\n \"symbol\" \u003d \"evaluate\",\n \"runtime_version\" \u003d \"3.12.11\",\n \"volatility\" \u003d \"immutable\"\n)\nAS $$\ndef evaluate(a, b, c):\n return a + b + c\n$$;\n\nSELECT py_add(1, 2, 3);\n-- ERROR 1105 (HY000): errCode \u003d 2, detailMessage \u003d Index 2 out of bounds for length 2\n```\n\nIn `PythonUdfBuilder.java:82`:\n\n```java\npublic Pair\u003cPythonUdf, PythonUdf\u003e build(String name, List\u003c?\u003e arguments) {\n // exprs \u003d (1, 2, 3), size \u003d 3\n // argTypes \u003d [INT, INT], size \u003d 2\n List\u003cExpression\u003e exprs \u003d arguments.stream().map(Expression.class::cast).collect(Collectors.toList());\n List\u003cDataType\u003e argTypes \u003d udf.getSignatures().get(0).argumentsTypes;\n\n List\u003cExpression\u003e processedExprs \u003d Lists.newArrayList();\n for (int i \u003d 0; i \u003c exprs.size(); ++i) {\n // when i \u003d 2, argTypes.get(2), err occur!\n processedExprs.add(TypeCoercionUtils.castIfNotSameType(exprs.get(i), argTypes.get(i)));\n }\n return Pair.ofSame(udf.withFreshVolatileIdentity().withChildren(processedExprs));\n}\n```\n\nnow:\n```sql\nCREATE FUNCTION py_add(INT, INT, ...)\nRETURNS INT\nPROPERTIES (\n \"type\" \u003d \"PYTHON_UDF\",\n \"symbol\" \u003d \"evaluate\",\n \"runtime_version\" \u003d \"3.12.11\",\n \"volatility\" \u003d \"immutable\"\n)\nAS $$\ndef evaluate(a, b, c):\n return a + b + c\n$$;\n-- ERROR 1105 (HY000): errCode \u003d 2, detailMessage \u003d mismatched input \u0027,\u0027 expecting \u0027)\u0027(line 1, pos 31)\n```\n\n### Release note\n\nReject variadic declarations for user-defined functions.\n\n### Check List (For Author)\n\n- Test: Unit Test\n - ./run-fe-ut.sh --run org.apache.doris.catalog.CreateFunctionTest\n- Behavior changed: Yes. Variadic user-defined function declarations are\nrejected during analysis.\n- Does this need documentation:\nhttps://github.com/apache/doris-website/pull/4103" }, { "commit": "60854dbfb52cf1b165bd9111d93b91fd3644f909", "tree": "bc17e5314a43994ce599d4a108f9c6b3c0494b6b", "parents": [ "4eb013c29f78add70992ce7d84084c422bb91771" ], "author": { "name": "Mingyu Chen (Rayner)", "email": "morningman.cmy@gmail.com", "time": "Fri Sep 04 11:41:24 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 04 11:41:24 2026 +0800" }, "message": "[feat](stream) Support ALTER STREAM ... SET/MODIFY COMMENT (#67471)\n\n### What problem does this PR solve?\n\nIssue Number: close #65388, related #65418\n\nRelated PR: #65810\n\nProblem Summary:\n\n**1. `ALTER STREAM ... SET COMMENT` was not supported (#65388)**\n\nA table stream can be created with a comment and the comment is fully\nwired up everywhere except for changing it:\n\n| step | before this PR |\n| --- | --- |\n| `CREATE STREAM s ON TABLE t COMMENT \u0027x\u0027` | supported\n(`InternalCatalog#createTableStream`) |\n| persisted in the image | supported (`Table#comment`) |\n| `SHOW CREATE STREAM s` | prints the comment |\n| `information_schema.table_streams.STREAM_COMMENT` | exposes the\ncomment |\n| changing the comment | **not possible** |\n\nThere was no `ALTER STREAM` rule in `DorisParser.g4` at all — `STREAM`\nonly appeared in `CREATE STREAM`, `DROP STREAM`, `SHOW STREAMS` and\n`SHOW CREATE STREAM` — so the statement failed at parser stage:\n\n```\nerrCode \u003d 2, detailMessage \u003d no viable alternative at input \u0027ALTER STREAM\u0027(line 1, pos 6)\n```\n\n`ALTER TABLE` is not an alternative either: `Alter#processAlterTable`\nrejects the `STREAM` table type with `Do not support alter STREAM\ntable[...]`.\n\nThis PR adds:\n\n```sql\nALTER STREAM \u003cname\u003e SET COMMENT \u0027new comment\u0027;\nALTER STREAM \u003cname\u003e MODIFY COMMENT \u0027new comment\u0027; -- same thing\n```\n\n`MODIFY` is accepted alongside `SET` so the syntax stays consistent with\n`ALTER TABLE ... MODIFY COMMENT`, which is the existing Doris spelling\nfor the same operation on a table.\n\nImplementation notes:\n\n- The comment of a stream lives in the `Table` metadata only, so\n`Alter#processAlterStreamComment` reuses\n`ModifyCommentOperationLog.forTable(...)` and the existing replay path\n`Alter#replayModifyComment`, which already resolves a generic `Table`.\n**No new edit log operation and no meta version bump.**\n- Cloud Meta Service only stores stream offsets and ids\n(`CloudInternalCatalog#beforeCreateTableStream` /\n`#afterCreateTableStream`), so no extra RPC is needed and the behaviour\nis the same in cloud mode.\n- `AlterStreamCommand` extends `AlterCommand`, which already provides\n`ForwardWithSync` and `StmtType.ALTER`. It carries an `AlterType` enum\nso that other `ALTER STREAM` clauses can be added later without\nreshaping the command.\n- Privilege required is `ALTER` on the stream, matching `ALTER TABLE`.\nAltering a non-stream table through `ALTER STREAM` reports\n`ERR_WRONG_OBJECT`, the same way `SHOW CREATE STREAM` does.\n- `Config.enable_table_stream` gates the operation, consistent with\n`CREATE STREAM` and `DROP STREAM`.\n- The comment literal is decoded with\n`SqlLiteralUtils.parseStringLiteral`, so a doubled quote\ncollapses to one quote and backslash escapes follow the session sql\nmode, matching the lexer\n(`NereidsParser` drives the lexer with\n`SqlModeHelper.hasNoBackSlashEscapes()`).\n`CREATE STREAM ... COMMENT` was decoding the same literal differently --\nit unescaped\nbackslashes but never collapsed doubled quotes and ignored\n`NO_BACKSLASH_ESCAPES` -- so it was\nmoved onto the same decoder, otherwise the comment stored by CREATE and\nby ALTER would differ\nfor the same text. Not fixed here: `Env#addTableComment` quotes the\nvalue with single quotes\nwhile escaping only double quotes, so a comment holding a `\u0027` makes\n`SHOW CREATE` emit\nnon-parsable DDL. That is pre-existing, shared by all 19 call sites of\nevery table type, and\n will be filed separately.\n\n**2. Regression coverage for immutable binlog properties (#65383)**\n\n`ALTER TABLE ... SET (\"binlog.format\" \u003d ...)` on a ROW binlog table used\nto fail with a misleading light-schema-change error, because\n`AlterOperations#checkBinlogConfigChange` did not list `binlog.format` /\n`binlog.need_historical_value` and the statement was dispatched to the\ngeneric schema change path. That was fixed as a side effect of #65810\n(`f745ddf9e22`), but no test locked the behaviour in. This PR adds\n`test_binlog_property_alter_exception.groovy` covering:\n\n| statement (on a `binlog.format \u003d ROW` MOW table) | expected |\n| --- | --- |\n| `SET (\"binlog.format\" \u003d \"STATEMENT_AND_SNAPSHOT\")` | `not support\nchange binlog format from ROW to STATEMENT_AND_SNAPSHOT` |\n| `SET (\"binlog.need_historical_value\" \u003d \"false\")` | `not support change\nbinlog.need_historical_value from true to false` |\n| `SET (\"binlog.enable\" \u003d \"false\")` | `can\u0027t disable binlog when format\nis [Row]` |\n| `SET (\"binlog.format\" \u003d \"ROW\")` (same value) | accepted, no-op |\n| `SET (\"binlog.ttl_seconds\" \u003d \"7200\")` | accepted |\n| `SET (\"binlog.format\" \u003d \"ROW\")` on a table without binlog | `not\nsupport change binlog format from STATEMENT_AND_SNAPSHOT to ROW` |" }, { "commit": "4eb013c29f78add70992ce7d84084c422bb91771", "tree": "9788f3bf8fd7d3223626f7fbfdee82c2f27ee84b", "parents": [ "d326680d9ae286bf4f3a62ccfdf4f681a4ad7765" ], "author": { "name": "Socrates", "email": "suyiteng@selectdb.com", "time": "Fri Sep 04 11:39:51 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 04 11:39:51 2026 +0800" }, "message": "[fix](paimon) tolerate removed paimon-cpp session variable (#67472)\n\n### What problem does this PR solve?\n\nIssue Number: N/A\n\nRelated PR: #67385\n\nAfter removing `enable_paimon_cpp_reader`, older clients and replayed\nstate may still reference it. Add the name to\n`REMOVED_SESSION_VAR_NAMES` so existing compatibility handling silently\nignores SET and tolerates reads instead of reporting an unknown system\nvariable.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test:\n- [x] No local test needed; this is a one-entry compatibility allowlist\nupdate.\n - [x] `git diff --check`\n- Behavior changed:\n - [x] Yes. References to the removed variable are tolerated.\n- Does this need documentation:\n - [x] No" }, { "commit": "d326680d9ae286bf4f3a62ccfdf4f681a4ad7765", "tree": "a1b188054fedf15d883606fbf58dc76661a4d04c", "parents": [ "b502984a1cf34109221e343d2ba1766487a13bcb" ], "author": { "name": "morrySnow", "email": "zhangwenxin@selectdb.com", "time": "Fri Sep 04 11:19:50 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 04 11:19:50 2026 +0800" }, "message": "[refactor](statistics) Organize statistics classes by responsibility (#67483)\n\n### What problem does this PR solve?\n\nProblem Summary:\n\nThe `org.apache.doris.statistics` package mixed optimizer-facing\nstatistics models with collection workflows, cache internals, and\nstatistics-table persistence, making class ownership difficult to\nidentify.\n\nThis PR reorganizes those classes by responsibility:\n- keeps only cross-cutting statistics definitions in the `statistics`\nroot package;\n- moves statistics values, histograms, builders, ranges, and metric\ntypes to `statistics.model`;\n- moves collection jobs, tasks, scheduling, metadata, descriptors, and\nupdate events to `statistics.analysis`;\n- moves cache loaders, keys, invalidation, refresh, and synchronization\ntargets to `statistics.cache`;\n- moves internal statistics-table access, persisted row models, and\ncleanup to `statistics.repository`;\n- mirrors the same package layout in unit tests and documents the\nboundaries in `README.md` and `package-info.java`.\n\nNo runtime logic or persisted field format changes." }, { "commit": "b502984a1cf34109221e343d2ba1766487a13bcb", "tree": "ce65acdac5e54506c3678e42c1f2b3a78ba8bb04", "parents": [ "a560c746fa9961c2cf2a41635ef04c33844cc0a7" ], "author": { "name": "Dongyang Li", "email": "lidongyang@selectdb.com", "time": "Fri Sep 04 11:07:00 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 04 11:07:00 2026 +0800" }, "message": "[chore](build) Assign owners for workflows and thirdparty (#67519)\n\n### What problem does this PR solve?\n\nIssue Number: N/A\n\nRelated PR: N/A\n\nProblem Summary: GitHub Actions workflow files and the thirdparty\ndirectory did not have explicit ownership routing. Add the requested\nowners so changes to workflows request review from hello-stephen,\nmorningman, and yiguolei, while thirdparty changes request review from\nyiguolei and morningman.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test: No need to test (CODEOWNERS-only metadata change)\n - Verified with `git diff --check` and a focused rule-content check\n- Behavior changed: Yes (GitHub review-request routing for the covered\npaths)\n- Does this need documentation: No" }, { "commit": "a560c746fa9961c2cf2a41635ef04c33844cc0a7", "tree": "412f950ff70207e5e3db3ad4970521d0fb3c34e4", "parents": [ "099d5bd71dffb4f515c417e75227c15c4fcea922" ], "author": { "name": "yujun", "email": "yujun@selectdb.com", "time": "Fri Sep 04 10:06:40 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 04 10:06:40 2026 +0800" }, "message": "[feature](ivm) Support incremental view maintenance (IVM) for materialized tables (MTMV) (#62606)\n\n### What problem does this PR solve?\n\nImplement Incremental View Maintenance (IVM) for materialized tables\n(MTMV) in Apache Doris. When a base table of an MTMV changes, only the\naffected rows are derived from the base-table deltas (row binlog / Table\nStream) and transactionally applied to the materialized table, instead\nof recomputing the whole result with a full refresh. This enables\nmaintaining near-real-time aggregation and join results at a fraction of\nthe cost of full refresh.\n\nSupported operators: projection, filter, aggregate\n(COUNT/SUM/AVG/MIN/MAX, bitmap aggregates, expression arguments, bare\nGROUP BY, grouping sets), joins (INNER/CROSS/OUTER JOIN, nested join and\nself join), UNION ALL, subquery aliases, and OneRowRelation.\n\nSupported refresh: `REFRESH MATERIALIZED VIEW ...\nINCREMENTAL/PARTITIONS`, refresh explain, dry-run, and automatic\nfull-refresh fallback when incremental maintenance is not valid (e.g.\nbinlog broken).\n\nLifecycle: stream creation/lifecycle integrated with MTMV, chained IVM\nMTMV support.\n\nThis PR is the MTMV incremental-maintenance layer of the end-to-end\nincremental computation stack tracked by #65418.\n\nIssue Number: close #66719\n\nRelated issue: #65418 (tracking issue), #57921 (superseded umbrella\nissue)\n\n---------\n\nCo-authored-by: seawinde \u003cwusi@selectdb.com\u003e" }, { "commit": "099d5bd71dffb4f515c417e75227c15c4fcea922", "tree": "225c5fe9b6214091122a9ec28e44a12a4cfaab4d", "parents": [ "219c6193f24ed6c028c5f5469f18681be7c0292c" ], "author": { "name": "Mingyu Chen (Rayner)", "email": "morningman.cmy@gmail.com", "time": "Fri Sep 04 09:48:13 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 04 09:48:13 2026 +0800" }, "message": "[fix](arrow-flight) Do not take the point-query short circuit on an Arrow Flight connection (#67487)\n\n### What problem does this PR solve?\n\nIssue Number: close #67368\n\nRelated PR: #67381 (sibling fix in the same tracking series), #62259\n\nProblem Summary:\n\nA `UNIQUE KEY` point query that qualifies for the short-circuit path\nreturned no Flight endpoint over Arrow Flight SQL. The client failed\nwith\n\n```\nfetch arrow flight schema failed, no FlightSqlEndpointsLocations\n```\n\nand the row was silently dropped. The identical query with\n`SET_VAR(enable_short_circuit_query\u003dfalse)` returned it on the same\nconnection.\n\n**Root cause** — the short circuit produces no Arrow result at either\nend, and nothing prevented an Arrow Flight connection from planning one:\n\n* It executes on `PointQueryExecutor`, not a `Coordinator`, and\n`Coordinator`/`NereidsCoordinator` are the only places that register a\n`FlightSqlEndpointsLocation`. `StmtExecutor.executeAndSendResult` then\nreturns early through its Arrow Flight branch with nothing registered\nand without ever calling `getNext()`, so `GetFlightInfo` found an empty\nendpoint list.\n* The BE cannot be pointed at either. `tablet_fetch_data` serializes\nwith `VMysqlResultWriter` into `PTabletKeyLookupResponse.row_batch` and\nruns no fragment, so the `ArrowFlightResultBlockBuffer` that\n`fetch_arrow_flight_schema` looks up by finst id never exists.\n\n`LogicalResultSinkToShortCircuitPointQuery` did not look at the connect\ntype, and `enable_short_circuit_query` defaults to `true`, so every ADBC\n/ Arrow Flight JDBC point query on a MoW + light-schema-change +\n`store_row_column` table hit this. Prepared statements go through the\nsame `executeQueryStatement` and failed identically.\n\n**Fix** — keep Arrow Flight SQL on the normal execution path.\n\nThis has to be decided at plan time rather than when picking the\nexecutor: `OlapScanNode.computeTabletInfo` and several rewrite/property\nrules (`ChildOutputPropertyDeriver`, `ShuffleKeyPruner`,\n`NestedColumnPruning`, `PruneOlapScanPartition`) read\n`StatementContext.isShortCircuitQuery()` while the plan is being built,\nso flipping the flag later would run a coordinator over a plan shaped\nfor a different execution mode. MySQL connections keep the short circuit\nunchanged.\n\nReturning the point-query result from the FE instead was considered and\nrejected for now: `FlightSqlChannel.addResult` builds varchar vectors\nonly, so every column would come back as `Utf8`, inconsistent with the\nnormal Flight path. Full support (Arrow serialization in the BE lookup\nRPC plus a result buffer to hand out an endpoint) is a larger change and\nout of scope here.\n\nAlso refreshes a now-stale comment in `StmtExecutor` that said point\nqueries reach the Arrow Flight deferral gate." }, { "commit": "219c6193f24ed6c028c5f5469f18681be7c0292c", "tree": "14910a843238d0ac956e32fec9e7d3e19849b85f", "parents": [ "182022b593acfc959178d80cd9b7d110b93c2fe6" ], "author": { "name": "Gabriel", "email": "liwenqiang@selectdb.com", "time": "Thu Sep 03 20:02:55 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 03 20:02:55 2026 +0800" }, "message": "[fix](iceberg) Preserve nested field case in created schemas (#67166)\n\n### What problem does this PR solve?\n\nIssue Number: N/A\n\nProblem Summary:\n\nDoris used the lowercase runtime name of nested STRUCT fields when\nbuilding connector schemas. Creating an Iceberg table through Doris\ntherefore changed persisted mixed-case field names such as\n`CaseSensitive` to `casesensitive`, causing schema compatibility checks\nfrom engines such as Trino and Spark to fail.\n\nThe same identity mismatch also affected schema evolution, query\nexecution, and pruning:\n\n- complex `MODIFY COLUMN` could interpret a case-only spelling\ndifference as a nested rename;\n- flat Iceberg `MODIFY COLUMN` and its `AFTER` reference used caller\nspelling even though Iceberg update paths are case-sensitive;\n- nested `MODIFY COLUMN` error upgrading resolved only the leaf name and\ncould inspect an unrelated same-named top-level column;\n- runtime struct lookup depended on the JVM default locale, so locales\nsuch as Turkish could map distinct field names to the same key;\n- a Unicode field spelling displayed by `DESCRIBE`, such as `Σ` or `ẞ`,\ncould fail at execution because FE passed that external spelling to BE\nwhile the thrift struct descriptor contained the normalized runtime name\n(`σ` or `ß`);\n- cast-aware nested pruning used `equalsIgnoreCase`, which could map\ndistinct ROOT-normalized siblings such as `i` and `ı` to the wrong\nsource field.\n\nThis change keeps two explicit field identities: a locale-independent\nlowercase key for Doris runtime lookup and the original spelling for\nconnector/external metadata. Connector create-table paths, including\nFILE TVF CTAS, use the original spelling. Iceberg schema evolution\nresolves existing fields case-insensitively and then stages type,\ncomment, nullability, and position changes with the persisted canonical\npath. Only an explicit rename operation changes field spelling. Nested\nmodify error upgrading resolves the complete `ConnectorColumnPath` and\nremains best-effort so it cannot replace the original build error for a\nmissing target.\n\nAfter Nereids successfully resolves a STRUCT selector, it replaces the\nexternal spelling with the resolved field\u0027s normalized runtime name for\nthrift/BE execution. This canonicalization is applied to directly\nanalyzed `ElementAt` nodes, `ElementAt` produced while binding SQL\nfunction syntax from `UnboundFunction`, and `ElementAt` created for\ndotted access on a computed base such as `CAST(... AS\nSTRUCT\u003c...\u003e).field`. The latter two paths construct and return a new\nnode without revisiting `visitElementAt`. Consequently,\n`element_at(struct, \u0027field\u0027)` and dotted dereference both work while\n`DESCRIBE` and connector metadata continue to expose the original\nexternal spelling. Cast-aware pruning compares exact ROOT-normalized\nkeys so that distinct siblings remain distinct.\n\n#### Metadata and rolling-upgrade compatibility\n\n`StructField.name`, `StructType.fields`, and the current `fieldMap`\nlookup index can be present in FE image metadata. Before this change, an\nFE running with a locale such as `tr-TR` could therefore persist `I` as\nthe runtime key `ı`. A new FE using `Locale.ROOT` produces the lookup\nkey `i`, so exact ROOT lookup alone cannot read that pre-ROOT image.\n\nFor fields replayed from metadata that predates `originalName`, this PR\nrecords a runtime-only legacy marker through Catalog-to-Nereids\nconversion. Current fields use the exact ROOT lookup key. Legacy fields\nfirst accept an exact persisted runtime spelling; broader case matching\nis used only when it identifies a single legacy field. If multiple\nlegacy runtime names match, lookup rejects the ambiguous selector\ninstead of silently returning the wrong sibling. Newly created metadata\nalways has `originalName`, so valid ROOT-distinct names such as `i` and\n`ı` are not merged by the compatibility path.\n\nThis covers the supported rolling-upgrade direction where upgraded\nFollowers/Observers replay metadata written by an older Master, followed\nby upgrading the Master. It does not make an old FE understand metadata\nfirst written by a new FE. It also cannot reconstruct original spelling\nthat an old FE already discarded; it only preserves unambiguous lookup\ncompatibility for the persisted runtime name. Rebuilding `fieldMap`\nduring deserialization would not recover the old FE locale or discarded\nspelling because Doris replays these objects through Gson and that\ninformation was never persisted.\n\n#### Fix boundary\n\nThe fix is limited to nested field identity preservation, Iceberg schema\nevolution, STRUCT selector canonicalization in FE, unambiguous legacy\npre-ROOT struct lookup, and consistent cast-pruning identity. Doris\nruntime lookup remains case-insensitive for current metadata. BE\ncontinues to receive and compare normalized thrift names; this PR does\nnot add Unicode case folding to BE. Current metadata continues to\npersist the existing `fieldMap`; rebuilding or removing that lookup\nindex is a separate metadata-format change and is intentionally outside\nthis PR. Removed code paths such as `StructElement` and\n`IcebergScanNode` are not reintroduced; their current replacements\nalready route through the fixed lookup or preserve partition-column\ncase.\n\n### Release note\n\nPreserve mixed-case nested field names when creating and evolving\nIceberg schemas, allow displayed Unicode nested field names to be\nqueried, keep nested pruning correct for locale-sensitive Unicode names,\nand retain safe lookup compatibility with pre-ROOT FE metadata.\n\n### Check List (For Author)\n\n- Test\n - [x] Regression test\n - [x] Unit Test\n - [ ] Manual test\n - [ ] No need to test or manual test.\n\nAdded coverage for mixed-case nested fields in create-table and FILE TVF\nCTAS paths, STRUCT/ARRAY/MAP evolution, locale-independent runtime\nlookup and pruning, case-insensitive flat Iceberg `MODIFY\nCOLUMN`/`AFTER` resolution, full-path nested modify error handling,\npre-ROOT Turkish metadata replay across Catalog and Nereids, ambiguous\nlegacy-field rejection, cast pruning with ROOT-distinct sibling names,\nand execution of exact displayed Unicode names (`Σ` and `ẞ`). The\n`ExpressionAnalyzer` coverage constructs an `UnboundFunction` to\nexercise the same SQL function-binding path as `element_at(...)`, and\ncovers direct `ElementAt`, ordinary dotted dereference, and\ncomputed-base dotted dereference.\n\nLatest local validation passed 23 Iceberg connector column-evolution\ntests, all 8 `ExpressionAnalyzer` tests, and all 5\n`ColumnGsonSerializationTest` tests. Full FE Checkstyle passed all 74\nmodules. The external Spark/Iceberg regression requires the CI test\nenvironment and was added for CI execution.\n\n- Behavior changed:\n - [ ] No.\n- [x] Yes. External Iceberg schemas retain their original nested-field\nspelling, displayed Unicode nested field names remain executable,\nunambiguous legacy struct metadata remains queryable after upgrade,\nambiguous legacy selectors are rejected instead of reading the wrong\nfield, and pruning keeps ROOT-distinct field identities separate.\n\n- Does this need documentation?\n - [x] No.\n - [ ] Yes.\n\n### Check List (For Reviewer who merge this PR)\n\n- [ ] Confirm the release note\n- [ ] Confirm test cases\n- [ ] Confirm document\n- [ ] Add branch pick label" }, { "commit": "182022b593acfc959178d80cd9b7d110b93c2fe6", "tree": "2646ed391e14af7c90b9fd8b4c744adcfe120a5c", "parents": [ "b5148780cf7619f7b17eff5bf3b24c10cd071d1d" ], "author": { "name": "deardeng", "email": "dengxin@selectdb.com", "time": "Thu Sep 03 19:57:22 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 03 19:57:22 2026 +0800" }, "message": "[fix](file cache) keep the cache monitor off the LRU recorder lock (#67315)\n\nrun_background_monitor() ended its loop with\nupdate_shadow_queue_element_count_metrics(), which takes _mutex_lru_log.\nThe LRU log replay thread holds that lock for as long as it takes to\ndrain the log queue, so a slow consumer froze the monitor with it:\ncheck_disk_resource_limit(), check_need_evict_cache_in_advance() and\nevery gauge stopped running, and the disk resource limit mode stayed at\nwhatever value it happened to hold. Gauges were observed frozen for up\nto 40 minutes in production.\n\nThe call was redundant from the start. #64798 added the shadow queue\nelement count gauge and published it in two places: inside\nreplay_queue_event(), under the same lock that mutates the shadow queue,\nand again from the monitor every\nfile_cache_background_monitor_interval_ms as a periodic refresh. Nothing\noutside replay_queue_event() mutates a shadow queue, so that refresh\ncould only rewrite a value that had just been published and could not\nhave changed since. What it did add was a dependency from the disk\nprotection loop onto a lock owned by a background consumer.\n\nDrop the call, and update_shadow_queue_element_count_metrics() with it:\nit existed only for that refresh, and leaving a public method that takes\n_mutex_lru_log invites the next background loop to reintroduce the\ncoupling. The gauge is still published by replay, now on the replay\ninterval instead of the monitor interval. Its test is replaced by one\nasserting that replay publishes the gauge on its own.\n\nHow long replay holds the lock is a separate problem, addressed\nseparately." } ], "next": "b5148780cf7619f7b17eff5bf3b24c10cd071d1d" }