)]}' { "log": [ { "commit": "664a7be2ab2541d099694ba3d564881b8cdfe77e", "tree": "25957ad6af2d6280248ffc39294cc04cd62baadf", "parents": [ "9a219ee72d7ca676fd9fa02cba41abe626052b1e" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Wed Nov 11 20:08:49 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Nov 11 20:08:49 2026 -0700" }, "message": "ci: add cargo fmt and clippy workflow (#53)\n\nAdds a GitHub Actions workflow running `cargo fmt --all -- --check` and\n`cargo clippy --all-targets --all-features --workspace -- -D warnings`\non pushes to main and pull requests.\n\nRust is installed with `rustup` because the ASF action allowlist only\npermits `dtolnay/rust-toolchain` at a pinned SHA.\n\nNote: the workspace `Cargo.toml` is not on `main` yet (#45), so the\nclippy job needs that to merge first. I haven\u0027t run clippy locally\nagainst the current tree, so existing warnings may need fixing before\nthis goes green." }, { "commit": "9a219ee72d7ca676fd9fa02cba41abe626052b1e", "tree": "5e26e657490def91cbb50071fc7a69427321c130", "parents": [ "b947917a46baf348632340add9898b385c41b568" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Wed Nov 11 20:08:07 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Nov 11 20:08:07 2026 -0700" }, "message": "test: run sqllogictest files in CI (#51)\n\nAdds a `datafusion-spark-sqllogictest` crate whose `sqllogictests` test\ntarget runs every `.slt` file under `crates/sqllogictest/test_files`\nagainst a DataFusion session with the Spark functions registered and the\nSpark SQL dialect enabled. It reuses the DataFusion engine from\n`datafusion-sqllogictest` (branch-56). Also adds a GitHub Actions job\nrunning `cargo test -p datafusion-spark-sqllogictest`.\n\nRust is installed with `rustup` because the ASF action allowlist only\npermits `dtolnay/rust-toolchain` at a pinned SHA.\n\nRun locally: `cargo test -p datafusion-spark-sqllogictest -- [FILTER]`.\nAll 245 test files pass.\n\n**Depends on #45** (workspace `Cargo.toml` and the branch-56 pin); this\nbranch is stacked on it, so its commits appear here until #45 merges." }, { "commit": "b947917a46baf348632340add9898b385c41b568", "tree": "7c00a2bea2b3ad9f69110773d55d099b0714beaf", "parents": [ "4061d37755110e55bb25312b1bbc7eee8874d4d6" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Wed Nov 11 19:55:16 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Nov 11 19:55:16 2026 -0700" }, "message": "ci: add Apache RAT license header check (#50)\n\nAdds a GitHub Actions workflow that runs Apache RAT (via\n`dev/release/run-rat.sh`) over tracked files to verify license headers.\nLICENSE/NOTICE files are excluded via\n`dev/release/rat_exclude_files.txt`.\n\nNote: not yet verified in CI; RAT CLI flags and SPDX header recognition\nin RAT 0.17 should be confirmed by the first workflow run." }, { "commit": "4061d37755110e55bb25312b1bbc7eee8874d4d6", "tree": "7a5d148953668e69f17ff21a0c56bdb4a57dc22f", "parents": [ "420c28057a420a7dc379960ea155e5587b24f4b1" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Wed Nov 11 19:16:51 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Nov 11 19:16:51 2026 -0700" }, "message": "docs: add audit-datafusion-spark-expression agent skill and AGENTS.md (#52)\n\nMoves the `audit-datafusion-spark-expression` skill here from\napache/datafusion (removal PR:\nhttps://github.com/apache/datafusion/pull/27344) and adds a minimal\n`AGENTS.md`.\n\nPaths to the spark crate, SLT files and the test command were updated\nfor this repo\u0027s layout (`crates/spark`, `crates/sqllogictest`, `cargo\ntest -p datafusion-spark-sqllogictest`). References to other DataFusion\ncrates (e.g. `datafusion/functions`) still describe the main repo and\nwere left as is." }, { "commit": "420c28057a420a7dc379960ea155e5587b24f4b1", "tree": "cba0160a5d679a78279ca8cc990bec8c085465f6", "parents": [ "a71e37adcb6470e8197dbf3562abde4c8e0d2af9" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Wed Nov 11 19:15:11 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Nov 11 19:15:11 2026 -0700" }, "message": "docs: add README, LICENSE.txt and NOTICE.txt (#49)\n\nAdds a concise top-level README describing the crate, repository layout,\nand how to build and test.\n\nAlso adds the root `LICENSE.txt` and `NOTICE.txt`. The crate\u0027s\n`LICENSE.txt` and `NOTICE.txt` are symlinks to `../../LICENSE.txt` and\n`../../NOTICE.txt`, which have been dangling since the import.\n`LICENSE.txt` is the one from apache/datafusion without the Apache\nAurora section, which covers release scripts this repo does not have." }, { "commit": "a71e37adcb6470e8197dbf3562abde4c8e0d2af9", "tree": "006f54f7f7075594392a4169fb4ac44a8414726d", "parents": [ "ea1e78439bf8c9753281b8afc18a52ea236bcfa4" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Wed Nov 11 15:08:07 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Nov 11 15:08:07 2026 -0700" }, "message": "chore: add top-level workspace Cargo.toml (#45)\n\n## Which issue does this PR close?\n\nN/A\n\n## Rationale for this change\n\nThe code was moved over from the datafusion repo but there was no\ntop-level `Cargo.toml`, so nothing built.\n\n## What changes are included in this PR?\n\n- Add a workspace `Cargo.toml` (package metadata, shared dependencies,\nlints) with `crates/spark` as the member\n- Add `.gitignore` for `target/` and commit `Cargo.lock`\n- DataFusion crates are git dependencies on `apache/datafusion`\n`branch-56`, because the 56 crates are not released yet.\n- Revert #26703 (`hex.rs` only). This repo starts from the point\n`branch-56` was cut, and #26703 landed on `main` afterwards; it needs\n`datafusion_common::utils::map_dictionary_values`, which `branch-56`\nlacks.\n\n## Are these changes tested?\n\n`cargo build --all-targets --all-features` succeeds and `cargo test\n--lib` passes (324 tests).\n\n## Are there any user-facing changes?\n\nNo.\n\n## Versioning\n\nThe crate version is set to `56.0.0`. We will bump the DataFusion\ndependencies from the git `main` branch to the released `56` crates (and\nupdate the version if needed) once the 56 crates are published." }, { "commit": "ea1e78439bf8c9753281b8afc18a52ea236bcfa4", "tree": "9c80d7d054e23a63089b4943c46b6aef8a2ef32f", "parents": [ "ef4c88a20dea374365cbb8f876be4e05b41d56d1" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Wed Nov 11 13:22:32 2026 -0700" }, "committer": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Wed Nov 11 13:22:32 2026 -0700" }, "message": "chore: add `.asf.yaml` with repository settings\n" }, { "commit": "ef4c88a20dea374365cbb8f876be4e05b41d56d1", "tree": "bad10061629f9c1f8a1751be6bea8336d332c9c5", "parents": [ "d5000e42590d8ab4cca14fa094fddb251834b836" ], "author": { "name": "Jay Zhan", "email": "jayzhan211@gmail.com", "time": "Sat Nov 07 07:41:43 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Nov 07 07:41:43 2026 +0000" }, "message": "perf: stop dictionary work growing with the group count (#26703)\n\n## Which issue does this PR close?\n\n- Closes #26696.\n\n## Rationale for this change\n\nQueries that group on a high-cardinality dictionary column slow down as\nthe group count grows. On ClickBench `hits_partitioned` with\n`enable_rle_to_dictionary \u003d true`, this PR takes the suite from 78.8 s\nto 42.8 s, with Q33 and Q34 about 3.5x faster and peak memory slightly\nlower ([bot\nrun](https://github.com/apache/datafusion/pull/26703#issuecomment-6703599262)).\n\nA hash aggregate builds its output once and emits it in `batch_size`\nslices, and slicing a `DictionaryArray` keeps every value. Every slice\ntherefore carries every group\u0027s value, so hashing, functions such as\n`lower`, the repartition coalescer and the final aggregate do work per\nslice that grows with the group count.\n\n## What changes are included in this PR?\n\nStart reading at `next_batch` in\n`aggregates/aggregate_hash_table/common.rs`.\n\n- **Aggregate output, the main gain.** A slice with more values than\nrows now holds only the values its rows reference. No threshold is\nneeded: each group is emitted once, so the cost is fixed per group.\n- **A bound for other batches.** Filters and scans also keep whole\ndictionaries, so `hash_dictionary` and the new `map_dictionary_values`\nhandle at most `MAX_DICTIONARY_VALUES_PER_ROW` (64) values per row and\ncompact past that. Too high only gives up an improvement, while too low\nmakes cheap work slower: `lower` and `upper` break even at about 40\nvalues per row with keys in random order. The filter and scan queries I\nmeasured showed no visible difference, so this is a guard, not a\nmeasured speedup.\n- **`compact_dictionary`** does both, taking each referenced value once.\n`take(values, keys)` copies per row, and multi-column group keys repeat\ndictionary keys, so it can overflow `i32` offsets, need more rows than\nthe key type holds, and copy a long value thousands of times.\n- **`map_dictionary_values`** replaces\n`dictionary.with_values(f(values))` in the string functions, Spark\n`hex`, `array_agg`, `regexp_replace` and `substr`, and `clippy.toml`\ndisallows calling `with_values` directly. If `f` fails, it retries on\nonly the referenced values, because `f` can see values no row\nreferences. This replaces `substr`\u0027s own fallback.\n\nEach aggregate slice now has its own values `Arc`, so the hash cache\nproposed in #26816 would miss on these slices. The two haven\u0027t been\nmeasured together.\n\n## Benchmarks\n\n| ClickBench partitioned | main | this PR |\n|---|---|---|\n| Total | 78.8 s | 42.8 s |\n| Q33 | 14.9 s | 4.4 s |\n| Q34 | 14.8 s | 4.1 s |\n| Q18 | 7.6 s | 2.3 s |\n| Peak memory | 13.1 GiB | 12.7 GiB |\n\nQ35, Q32 and Q40 read 1.05x to 1.10x slower by minimum. None groups on a\ndictionary column, so this PR only adds a type check per output batch\nfor them, and the bot\u0027s noise is about 5-7% per query.\n\n\u003cdetails\u003e\u003csummary\u003eLocal runs on a busy 12-core laptop: `lower`, memory\nlimits, filter queries\u003c/summary\u003e\n\nResults were identical on every query. Treat small differences as noise.\n\n`lower` over a dictionary GROUP BY, which ClickBench doesn\u0027t cover:\n\n```sql\nSELECT lower(s), c FROM (\n SELECT \"SearchPhrase\" s, COUNT(*) c FROM hits WHERE \"SearchPhrase\" \u003c\u003e \u0027\u0027 GROUP BY s\n) ORDER BY c DESC, 1 LIMIT 10;\n```\n\n| partitions | 12 | 24 | 32 | 48 | 64 |\n|---|---|---|---|---|---|\n| ratio to main | 0.08x | 0.14x | 0.16x | 0.24x | 0.22x |\n\nQ33 under `--memory-limit`, one run each:\n\n| limit | main | this PR |\n|---|---|---|\n| 12g | fails: TopK resources exhausted | 6.3 s |\n| 8g | 23.6 s | 16.4 s |\n| 6g | 31.2 s | 19.4 s |\n| 4g | 20.2 s | 14.3 s |\n\nFilter and scan queries using `reverse`, `initcap`, `LIKE` and hashing\nafter a filter, with the bound at 40: 0.94x to 1.05x of main.\n\n\u003c/details\u003e\n\n## What is the testing strategy for this PR?\n\n- `dictionary.slt` compares each function on aggregate output and on\nfilter output with the same function on the decoded value. The\ncomparison is projected, because a filter on the group key is pushed\nbelow the aggregate.\n- Unit tests cover `next_batch`, `map_dictionary_values` and hashing:\nrepeated keys, NULL groups, more rows than the key type holds, and a\nvalue too long to copy per row.\n- The `dictionary_sliced` case in\n`datafusion/common/benches/with_hashes.rs` covers 8 and 128 values per\nrow.\n\n## Are there any user-facing changes?\n\nNo behavior changes. `datafusion_common::utils` gains\n`compact_dictionary`, `map_dictionary_values` and\n`MAX_DICTIONARY_VALUES_PER_ROW`.\n\nOriginal-commit: apache/datafusion@bc1be2c5a07c508c91160cb2b54ace0cdbbcaaf8\n" }, { "commit": "d5000e42590d8ab4cca14fa094fddb251834b836", "tree": "82e0f63d98f85491c8459b6e5a7b5574b1902fb9", "parents": [ "db6f43de3cac7e4eb4951aa499951c5b46f9f479" ], "author": { "name": "Yongting You", "email": "2010youy01@gmail.com", "time": "Tue Nov 03 19:45:06 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Nov 03 19:45:06 2026 +0000" }, "message": "chore: Complete the short Apache header migration (#27049)\n\n\u003e [!IMPORTANT]\n\u003e\n\u003e **Fork CI setup:** For pull requests from a fork to `main`, enable\nworkflows in your fork\u0027s **Actions** tab before you push.\n\u003e See the [fork CI setup\nguide](https://datafusion.apache.org/contributor-guide/#pull-requests-from-forks).\n\n## Which issue does this PR close?\n\n\u003c!--\nWe generally require a GitHub issue to be filed for all bug fixes and\nenhancements and this helps us generate change logs for our releases.\nYou can link an issue to this PR using the GitHub syntax. For example\n`Closes #123` indicates that this PR will close issue #123.\n--\u003e\n\nFollow-up to https://github.com/apache/datafusion/pull/26761.\n\n\n## Rationale for this change\n\n\u003c!--\nWhy are you proposing this change? If this is already explained clearly\nin the issue then this section is not needed.\nExplaining clearly why changes are proposed helps reviewers understand\nyour changes and offer better suggestions for fixes.\n\nPlease explain the problem you are trying to solve in terms of the\nuser-visible\nbehavior, rather than the implementation.\n\nFor example, \"The code in `foo.rs` doesn\u0027t handle nulls\" is a symptom of\nthe\nimplementation. \"COUNT(DISTINCT) returns wrong results when the column\ncontains\nnulls\" is the user-visible problem.\n--\u003e\nFrom #26761:\n\n\u003e Use the shorter Apache License 2.0 header instead. It saves tokens and\nscreen space.\n\nThe original configuration covered only `.rs`, `.py`, `.ts`, and `.slt`.\nOther file types still used the long header; for example, `.slt.part`\nfiles were not included.\n\n## What changes are included in this PR?\n\n\u003c!--\nThere is no need to duplicate the description in the issue here, but it\nis sometimes worth providing a summary of the individual changes in this\nPR.\n--\u003e\nExisting dev tool use hawkeye to check/generate license header:\nhttps://github.com/fast/hawkeye\n\n- commit 1: preparations, set up hawkeye config files\n- commit 2: ran the following command, all source license headers are\nauto-converted\n```\nhawkeye remove --config licenserc-legacy.toml \u0026\u0026\nhawkeye format --config licenserc.toml \u0026\u0026\nhawkeye check --config licenserc.toml\n```\n- commit 3: clean up\n\n## What is the testing strategy for this PR?\nUpdated `licenserc.toml`, to ensure those previously missed file types\ncan continue getting verified by CI (and auto-fixed by existing dev\nscript)\n\nALso let Codex to double checked no unintended changes.\n\u003c!--\nWe typically require tests for all PRs in order to:\n1. Prevent the code from being accidentally broken by subsequent changes\n2. Serve as another way to document the expected behavior of the code\n\nBriefly describe how this PR is tested, and point to the specific tests\nyou added. For example: \u0027This new feature is covered by the\n`sqllogictest` cases added in `foo.slt`\u0027.\n\nIf this PR does not add tests, explain why. For example, if the change\nis already covered by existing tests, please mention it.\n\nYou should also check the `codecov` bot reply on this PR to confirm the\nchanged code is exercised.\n--\u003e\n\n## Are there any user-facing changes?\n\n\u003c!--\nIf there are user-facing changes then we may require documentation to be\nupdated before approving the PR.\n\nIf there are any breaking changes to public APIs, please add the `api\nchange` label.\n--\u003e\n\nOriginal-commit: apache/datafusion@f673e62f3ea1a665b0c1ff1336866effacf7289c\n" }, { "commit": "db6f43de3cac7e4eb4951aa499951c5b46f9f479", "tree": "194375fa2ea0daa3ecc06768113d4fc411f14f9b", "parents": [ "99eee7c692be4e69a329a5577aa9fec66f4705e2" ], "author": { "name": "lazzypanda7", "email": "158449814+harsh-ande@users.noreply.github.com", "time": "Fri Oct 30 07:42:23 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 30 07:42:23 2026 +0000" }, "message": "fix(spark): floor instead of truncate in unix_millis/unix_seconds for pre-epoch timestamp (#26129)\n\n## Which issue does this PR close?\n\n- Closes #26128.\n\n## Rationale for this change\n\nThe Spark-compatible `unix_millis` and `unix_seconds` functions return\ndifferent results from Spark for timestamps before 1970-01-01 that have\na sub-unit component. Spark floors the result, but DataFusion truncates\ntoward zero. For example, `unix_millis` of a timestamp 1 µs before the\nepoch is `-1` in Spark and `0` in DataFusion. Results for non-negative\ntimestamps and exact multiples already match.\n\nThe cause is that `SparkUnixTimestamp::simplify` rewrites the call as a\ncast to a coarser `Timestamp` unit, and Arrow\u0027s cast truncates toward\nzero. Spark computes these values with `Math.floorDiv`\n(`TimestampToLongBase`).\n\n## What changes are included in this PR?\n\nIn `SparkUnixTimestamp::simplify`\n(`datafusion/spark/src/function/datetime/unix.rs`):\n\n- When the input timestamp unit is finer than the target unit, the raw\n`Int64` ticks are divided by the unit ratio (1e3, 1e6 or 1e9). Then `1`\nis subtracted when the remainder is negative: `ticks / d - CAST(ticks %\nd \u003c 0 AS BIGINT)`. This is an exact floor division over the full `i64`\nrange, with no intermediate unit conversion, so it adds no new overflow\ncases.\n- When the input unit is the same as or coarser than the target, the\nexisting direct cast is kept, so results for those inputs don\u0027t change.\n- NULL handling doesn\u0027t change; all operators involved propagate NULL.\n\n## What is the testing strategy for this PR?\n\nNew sqllogictest cases in\n`datafusion/sqllogictest/test_files/spark/datetime/unix.slt`:\n\n- **Runtime path:** a table column (not just literals, so constant\nfolding isn\u0027t the only path tested) holding pre-epoch values with and\nwithout a remainder, a positive value and NULL. Each is checked for both\n`unix_millis` and `unix_seconds`.\n- **Constant-folded path:** the same cases written as literals.\n- **Other input units:** `Timestamp(Nanosecond)` to millis and micros,\n`Timestamp(Millisecond)` to seconds, and an unchanged conversion to a\nfiner unit (`Millisecond` to micros).\n- **Extremes:** `i64::MIN` microseconds to seconds, and a large\nmillisecond value to seconds, to guard against overflow.\n\nExpected values for microsecond inputs came from Spark 4.0.0 (pyspark\nwith `spark.sql.session.timeZone\u003dUTC`), not from this implementation.\nSpark has no nanosecond timestamps, so the nanosecond cases use the same\nfloor rule.\n\nLocal runs: all `spark/*` sqllogictests, `cargo test -p\ndatafusion-spark`, `cargo fmt --all -- --check` and `cargo clippy -p\ndatafusion-spark --all-targets -- -D warnings` pass\n\n## Are there any user-facing changes?\n\nYes, as a bug fix. `unix_millis` and `unix_seconds` now return the\nSpark-compatible (floored) value for pre-epoch timestamps with a\nsub-unit component. No public API changes, so no `api change` label is\nneeded.\n\n---\n\nAI assistance: I used an AI coding assistant to help find this\ndivergence (by differential testing against Spark) and to draft the\nchange. A separate AI review pass caught overflow edge cases, which the\ntests now cover. I reproduced the behavior against Spark 4.0.0 myself\nand reviewed the change line by line.\n\nOriginal-commit: apache/datafusion@4ec76ade4e3800174bf72bc859a8f02eee45ab70\n" }, { "commit": "99eee7c692be4e69a329a5577aa9fec66f4705e2", "tree": "d674715df1d4c538d23472d89f72124f775ddc54", "parents": [ "efd0fc1eaeb17fea95f1b725e3b2a0e89c584491" ], "author": { "name": "lazzypanda7", "email": "158449814+harsh-ande@users.noreply.github.com", "time": "Thu Oct 29 21:03:50 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 29 21:03:50 2026 +0000" }, "message": "fix(spark): accept STRING and TIMESTAMP start dates in next_day (#26127)\n\n## Which issue does this PR close?\n- Closes #23889.\n\n## Rationale for this change\nSpark\u0027s `NextDay` casts its first argument to DATE, so\n`next_day(\u00272015-07-23\u0027, \u0027Mon\u0027)` and `next_day(TIMESTAMP \u00272015-07-23\n12:12:12\u0027, \u0027Mon\u0027)` both work. `datafusion-spark` declared `Exact(Date32,\nUtf8)` and failed at planning for STRING and TIMESTAMP inputs.\n\n## What changes are included in this PR?\n- `next_day` now uses a coercible signature: the first argument is\nimplicitly cast to DATE from STRING or TIMESTAMP, and the day-of-week is\nany string type (numeric is still coerced to string).\n - The kernel is unchanged.\n- In `next_day.slt`, three existing `query error` expectations changed\nto the new planner messages (arity and non-date first argument), and the\nunparseable-string case now returns a cast error. They still fail at\nplanning or execution as before.\n\n## What is the testing strategy for this PR?\nNew cases in `spark/datetime/next_day.slt`: string, string with a time\npart, timestamp literal, string column with NULL, timestamp column, and\n`LargeUtf8`/`Utf8View` day-of-week. I compared the expected results with\n`pyspark\u003d\u003d4.0.0`, ANSI on and off.\n\n## Are there any user-facing changes?\n`next_day` accepts more argument types. No API changes.\n\n## Known limitations / unknowns\n- An unparseable date string always errors. Spark returns NULL with ANSI\noff and errors with ANSI on. This looks tied to the ANSI work in #23890\nand #23929.\n- I did not verify timestamps with time zones near midnight against\nSpark\u0027s session-time-zone cast.\n- #23892 touches the same files, so I expect a small rebase if it merges\nfirst.\n- AI assistance was used to implement this. I reviewed and understand\nthe change, and checked the results against Spark.\n\n---------\n\nSigned-off-by: Harsh Ande \u003charsh.vardhan.ande@gmail.com\u003e\n\nOriginal-commit: apache/datafusion@d2bd93cc09ed3de872cdb27552dd8dc3314fd0bc\n" }, { "commit": "efd0fc1eaeb17fea95f1b725e3b2a0e89c584491", "tree": "b3761cb7a77c0225e1bd531d6c4291c152503948", "parents": [ "829bd8ad5464eb885e1a4e00afa03c90cd654e0e" ], "author": { "name": "Yongting You", "email": "2010youy01@gmail.com", "time": "Wed Oct 28 01:18:58 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 28 01:18:58 2026 +0000" }, "message": "chore: Use short Apache license header (#26761)\n\n## Which issue does this PR close?\n\n\u003c!--\nWe generally require a GitHub issue to be filed for all bug fixes and\nenhancements and this helps us generate change logs for our releases.\nYou can link an issue to this PR using the GitHub syntax. For example\n`Closes #123` indicates that this PR will close issue #123.\n--\u003e\n\n- Closes #.\n\n## Rationale for this change\n\n\u003c!--\nWhy are you proposing this change? If this is already explained clearly\nin the issue then this section is not needed.\nExplaining clearly why changes are proposed helps reviewers understand\nyour changes and offer better suggestions for fixes.\n\nPlease explain the problem you are trying to solve in terms of the\nuser-visible\nbehavior, rather than the implementation.\n\nFor example, \"The code in `foo.rs` doesn\u0027t handle nulls\" is a symptom of\nthe\nimplementation. \"COUNT(DISTINCT) returns wrong results when the column\ncontains\nnulls\" is the user-visible problem.\n--\u003e\nUse the shorter Apache License 2.0 header instead. It saves tokens and\nscreen space.\n\nThe shorter 3-lines form is explicitly allowed from ASF guide:\nhttps://www.apache.org/legal/src-headers.html?utm_source\u003dchatgpt.com#is-an-spdx-form-of-the-source-header-acceptable\n\nBefore:\nhttps://github.com/apache/datafusion/blob/350ca76505c9513d16e7dd9910bcd9bde2c911ed/datafusion/common/src/alias.rs#L1-L16\nAfter:\nhttps://www.apache.org/legal/src-headers.html?utm_source\u003dchatgpt.com#is-an-spdx-form-of-the-source-header-acceptable\n\n## What changes are included in this PR?\n\n\u003c!--\nThere is no need to duplicate the description in the issue here, but it\nis sometimes worth providing a summary of the individual changes in this\nPR.\n--\u003e\nExisting dev tool use `hawkeye` to check/generate license header:\nhttps://github.com/fast/hawkeye\n\ncommit 1: preparations, set up `hawkeye` config files\ncommit 2: ran the following command, all source license headers are\nauto-converted\n```\nhawkeye remove --config licenserc-legacy.toml \u0026\u0026\nhawkeye format --config licenserc.toml \u0026\u0026\nhawkeye check --config licenserc.toml\n```\ncommit 3: clean up\n\n## What is the testing strategy for this PR?\n\n\u003c!--\nWe typically require tests for all PRs in order to:\n1. Prevent the code from being accidentally broken by subsequent changes\n2. Serve as another way to document the expected behavior of the code\n\nBriefly describe how this PR is tested, and point to the specific tests\nyou added. For example: \u0027This new feature is covered by the\n`sqllogictest` cases added in `foo.slt`\u0027.\n\nIf this PR does not add tests, explain why. For example, if the change\nis already covered by existing tests, please mention it.\n\nYou should also check the `codecov` bot reply on this PR to confirm the\nchanged code is exercised.\n--\u003e\nI\u0027ve let codex double checked no missing files, and no unintended\nchanges.\n\nAlso CI validates new header:\n\nhttps://github.com/apache/datafusion/blob/350ca76505c9513d16e7dd9910bcd9bde2c911ed/.github/workflows/dev.yml#L39\n\n## Are there any user-facing changes?\n\n\u003c!--\nIf there are user-facing changes then we may require documentation to be\nupdated before approving the PR.\n\nIf there are any breaking changes to public APIs, please add the `api\nchange` label.\n--\u003e\n\nOriginal-commit: apache/datafusion@c1013ec583b46d55323d8fb173a064c58fa15d2b\n" }, { "commit": "829bd8ad5464eb885e1a4e00afa03c90cd654e0e", "tree": "2b4f647fe7afe7cf5e77c061a3a04354b292c10d", "parents": [ "e9b08ebeb4029d4718d6ac84366478ca57f980e6" ], "author": { "name": "nat-openai", "email": "nat@openai.com", "time": "Tue Oct 27 20:36:53 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 27 20:36:53 2026 +0000" }, "message": "feat: improve Spark from_utc_timestamp compatibility (#25979)\n\n## Which issue does this PR close?\n\n\u003c!--\nWe generally require a GitHub issue to be filed for all bug fixes and\nenhancements and this helps us generate change logs for our releases.\nYou can link an issue to this PR using the GitHub syntax. For example\n`Closes #123` indicates that this PR will close issue #123.\n--\u003e\n\nThis addresses the major incompatibility described in\nhttps://github.com/apache/datafusion-comet/issues/4654 for\n`CometFromUTCTimestamp`. It does not close that issue.\n\nIt\u0027s a follow-up to #19879.\n\n## Rationale for this change\n\n\u003c!--\nWhy are you proposing this change? If this is already explained clearly\nin the issue then this section is not needed.\nExplaining clearly why changes are proposed helps reviewers understand\nyour changes and offer better suggestions for fixes.\n\nPlease explain the problem you are trying to solve in terms of the\nuser-visible\nbehavior, rather than the implementation.\n\nFor example, \"The code in `foo.rs` doesn\u0027t handle nulls\" is a symptom of\nthe\nimplementation. \"COUNT(DISTINCT) returns wrong results when the column\ncontains\nnulls\" is the user-visible problem.\n--\u003e\n\nThe timezone spellings accepted by Spark are different than those\naccepted by Arrow\u0027s parser. As a result, `from_utc_timestamp` in comet\nisn\u0027t safe to use, since it would fail on things like Java short IDs,\nprefixed offsets with second precision, and SystemV-style IDs.\nAdditionally, Arrow will accept offsets beyond the Spark limit of 18\nhours.\n\n## What changes are included in this PR?\n\n\u003c!--\nThere is no need to duplicate the description in the issue here, but it\nis sometimes worth providing a summary of the individual changes in this\nPR.\n--\u003e\n\nThis change improves compatibility by parsing timezone IDs with the\nrules implemented by Spark. As part of doing this, I added\nSparkFromUtcTimestampExpr so that computed timezones evaluated for rows\nwith non-null timestamps, avoiding failing when a timezone is invalid\nbut wouldn\u0027t be required anyway. (Full compatibility with spark\nbehaviour seems quite complex and I haven\u0027t attempted it here.)\n\nIt also adds more documentation recording remaining differences:\n\n- timezone validation and evaluation order can differ from Spark\n- chrono accepts a narrower calendar range than Spark implemented range\n- chrono-tz doesn\u0027t currently perform DST transitions past 2099\n\nThis does not achieve full compatibility with Spark due to those issues,\nbut it makes the region of incompatibility a lot smaller.\n\nIf this approach isn\u0027t the one that folks would prefer, please let me\nknow if there are different avenues you\u0027d rather I explore!\n\nFinally, it\u0027s worth mentioning that this change was LLM-assisted.\n\n## What is the testing strategy for this PR?\n\n\u003c!--\nWe typically require tests for all PRs in order to:\n1. Prevent the code from being accidentally broken by subsequent changes\n2. Serve as another way to document the expected behavior of the code\n\nBriefly describe how this PR is tested, and point to the specific tests\nyou added. For example: \u0027This new feature is covered by the\n`sqllogictest` cases added in `foo.slt`\u0027.\n\nIf this PR does not add tests, explain why. For example, if the change\nis already covered by existing tests, please mention it.\n\nYou should also check the `codecov` bot reply on this PR to confirm the\nchanged code is exercised.\n--\u003e\n\nAdded unit tests in `datafusion/spark/src/function/datetime/timezone.rs`\ncovering accepted spelling, the 18 hour boundary, and SystemV custom\nbehaviour around DST transitions and exception.\n\nAdded SQL logic tests in\n`datafusion/sqllogictest/test_files/spark/datetime/from_utc_timestamp.slt`\ncovering the accepted spellings.\n\n## Are there any user-facing changes?\n\n\u003c!--\nIf there are user-facing changes then we may require documentation to be\nupdated before approving the PR.\n\nIf there are any breaking changes to public APIs, please add the `api\nchange` label.\n--\u003e\n\nYes. The existing Spark `from_utc_timestamp` UDF now accepts additional\nSpark timezone spellings. It also rejects offsets outside Spark\u0027s\n18-hour limit.\n\nThis won\u0027t affect most users since they\u0027d need to have opted into the\nexisting incompatible implementation.\n\n---------\n\nCo-authored-by: Andrew Lamb \u003candrew@nerdnetworks.org\u003e\n\nOriginal-commit: apache/datafusion@36c031242f0607da32cab55e4c27929150908232\n" }, { "commit": "e9b08ebeb4029d4718d6ac84366478ca57f980e6", "tree": "afdc71f98701a38eb2ebc621c6aa15a994e2c85f", "parents": [ "0c32e996ea283ca366e44c86f0fdff7261d057e1" ], "author": { "name": "Matt Butrovich", "email": "mbutrovich@users.noreply.github.com", "time": "Sun Oct 25 22:04:54 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Oct 25 22:04:54 2026 +0000" }, "message": "fix: date_part sub-second parts and result nullability, reusing arrow kernels (#26686)\n\n## Which issue does this PR close?\n\n- Closes #26678.\n- Part of #14738. Its reproducer is a `Duration`, which now returns NULL\ninstead of Postgres\u0027s 0, so the semantics question stays open there.\n- Closes #26685.\n\n## Rationale for this change\n\nThese fixes depend on each other. #26678 and #14738 come from the same\n[NULL-as-zero\nbranch](https://github.com/apache/datafusion/blob/dad9063becc0b10550936425d8b433046c5622e3/datafusion/functions/src/datetime/date_part.rs#L490-L499).\nFixing the #14738 overflow makes `millisecond` and `microsecond` of a\nlong duration return NULL instead of a wrapped value. Without the #26685\nfix, that NULL fails with `declared as non-nullable` on a non-nullable\ninput, so fixing the overflow alone would replace a wrong result with an\nerror.\n\n## What changes are included in this PR?\n\n- `millisecond`, `microsecond` and `nanosecond` share one function that\nreuses arrow\u0027s `date_part` and `cast` where they compute the right\nvalue.\n- Timestamps and times compute these parts in one pass, instead of two\n`date_part` calls and a `binary`.\n- A value that overflows the return type is NULL, as arrow already does\nfor durations.\n- The integer parts of `date_part` and the Spark `date_part` are always\nnullable. `epoch` keeps the input\u0027s nullability.\n- The [dev\nlist](https://lists.apache.org/thread.html/90cqgs9dotp39tcwr5ophvb2ftvjc43p)\nagreed with NULL on overflow and with the nullable results.\n- c920a5b92 and 8d5b5f1a2 add benchmark cases before the changes they\nmeasure, so the benchmark bot can compare against them.\n\n\u003cimg width\u003d\"1600\" height\u003d\"1380\" alt\u003d\"date-part-subsecond-bot\"\nsrc\u003d\"https://github.com/user-attachments/assets/2f5dab7d-6eb5-4a7f-b16e-92a625ff8584\"\n/\u003e\n\nThe benchmark bot measured `Duration` and `Interval` at 3.4x to 17.2x\nfaster, times at 2.0x to 5.1x, timestamps at 1.9x to 2.8x, and\n`Timestamp(Second)` at 1.2x to 1.6x, in both the default and the aligned\nruns.\n\n\u003cdetails\u003e\u003csummary\u003eNumbers\u003c/summary\u003e\n\nSpeedups over c920a5b92 (the bench commit) on 1000 rows, with the bot\u0027s\nthin LTO. The aligned runs add `-align-all-functions\u003d6\n-align-all-nofallthru-blocks\u003d6`. The chart and this table show\n`nanosecond`. The linked runs also have `millisecond` and `microsecond`.\n\n| Case |\n[Neoverse-V2](https://github.com/apache/datafusion/pull/26686#issuecomment-6384580919)\n|\n[aligned](https://github.com/apache/datafusion/pull/26686#issuecomment-6384582980)\n| [Xeon\n8581C](https://github.com/apache/datafusion/pull/26686#issuecomment-6384577309)\n|\n[aligned](https://github.com/apache/datafusion/pull/26686#issuecomment-6384586255)\n|\n|---|---|---|---|---|\n| `nanosecond`, `Duration(ns)`, large values | 12.11x | 12.20x | 13.12x\n| 12.38x |\n| `nanosecond`, `Duration(ns)`, under 2.1 s | 3.86x | 3.94x | 4.87x |\n4.03x |\n| `nanosecond`, `Interval(DayTime)` | 9.60x | 9.58x | 6.31x | 6.40x |\n| `nanosecond`, `Interval(MonthDayNano)` | 6.47x | 6.48x | 8.34x | 8.26x\n|\n| `nanosecond`, `Time64(ns)` | 2.36x | 2.23x | 2.48x | 2.69x |\n| `nanosecond`, `Timestamp(ns)` | 2.01x | 1.98x | 2.33x | 2.26x |\n| `nanosecond`, `Timestamp(ns)` with a timezone | 2.40x | 2.46x | 2.39x\n| 2.75x |\n| `second`, `Timestamp(ns)` (unchanged) | 1.00x | 1.00x | 0.98x | 1.00x\n|\n\nThe unchanged `second` cases range from 0.96x to 1.03x across the four\nruns, so differences in that range are noise.\n\n\u003c/details\u003e\n\n## What is the testing strategy for this PR?\n\n- New cases in\n`datafusion/sqllogictest/test_files/datetime/date_part.slt`.\n- `DESCRIBE` queries for the result nullability in\n`datafusion/sqllogictest/test_files/datetime/date_part.slt` and\n`datafusion/sqllogictest/test_files/spark/datetime/date_part.slt`.\n\n## Are there any user-facing changes?\n\n`EXTRACT(MILLISECOND/MICROSECOND/NANOSECOND)` on `Duration` and\n`Interval` return correct results. A `Duration` whose total in the\nrequested unit overflows the return type returns NULL, as the other\n`Duration` parts already do, instead of a wrapped value or a panic. The\ninteger results of `date_part` and the Spark `date_part` are nullable\neven for non-null input. There are no public API changes.\n\n## AI Disclosure\n\nI wrote this PR with help from Claude but understand and support the\ncore approach, implementation, and test coverage.\n\nOriginal-commit: apache/datafusion@8ea0dbfc341e18ff2f5429d2fca42f70c83b241c\n" }, { "commit": "0c32e996ea283ca366e44c86f0fdff7261d057e1", "tree": "17041f012ddaeedbddd583699ed6e4932b687233", "parents": [ "454a4888e27d01a3378cd6fe245920e7c4fc5122" ], "author": { "name": "Shahzeb Ali", "email": "126076773+Shahzeb-DataEngineer@users.noreply.github.com", "time": "Sat Oct 24 11:05:28 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Oct 24 11:05:28 2026 +0000" }, "message": "refactor(spark): deduplicate bucket computation in width_bucket_interval_mdn_exact (#26489)\n\n## Which issue does this PR close?\n\n- Closes #26307\n\n## Rationale for this change\n\nThe \"only month\" and \"months equals\" branches of\n`width_bucket_interval_mdn_exact` duplicated the bucket-number\ncomputation: the ascending/descending out-of-range checks, the\nzero/non-finite width null guard, and the floor-plus-one bucket clamp.\nDuplicated numeric edge-case handling like this is where a boundary fix\nends up in only one copy.\n\n## What changes are included in this PR?\n\n- Extracted the shared logic into `fn bucket_from_f64(x, l, h,\nnum_buckets, ascending) -\u003e Option\u003ci64\u003e` (`None` for null). Each branch\nnow only computes its projected `(x, l, h)` and calls the helper.\n- No behavior change: `width_bucket_interval_mdn_exact` keeps the same\nout-of-range values (0 / `num_buckets + 1`), the same null cases, and\nthe same clamp.\n\n## What is the testing strategy for this PR?\n\nNo new tests: the existing `width_bucket` module tests\n(`test_width_bucket_interval_mdn_months_only_*`,\n`test_width_bucket_interval_mdn_day_nano_*`, plus the invalid-input\ncases) cover both branches. Verified locally: all 21 tests in the\n`width_bucket` module pass, `cargo fmt` and `cargo clippy -p\ndatafusion-spark --all-targets` are clean.\n\n## Are there any user-facing changes?\n\nNo: internal refactor only, no behavior or API change.\n\nSigned-off-by: Shahzeb Ali \u003c126076773+Shahzeb-DataEngineer@users.noreply.github.com\u003e\nCo-authored-by: Andrew Lamb \u003candrew@nerdnetworks.org\u003e\n\nOriginal-commit: apache/datafusion@92f170cb7ea5ad87bdff8fde89ac7a1541f3b0fa\n" }, { "commit": "454a4888e27d01a3378cd6fe245920e7c4fc5122", "tree": "0d749bee7ecacda2aa7f7f7d6f991a3b5de88a60", "parents": [ "d8fdd56e062ea3df8edbd6aeeb4bb6907560bc7e" ], "author": { "name": "aetos", "email": "aetosdios27@gmail.com", "time": "Thu Oct 22 06:21:24 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 22 06:21:24 2026 +0000" }, "message": "test: run microbenchmarks in extended CI (#25836)\n\n## Which issue does this PR close?\n\nCloses #24868.\n\n## Rationale for this change\n\nMicrobenchmarks can compile successfully but panic when executed.\nRunning their cases in CI catches stale benchmark fixtures and\ninvocation assumptions without collecting performance measurements.\n\n## What changes are included in this PR?\n\n- Add `cargo xtask ci step test microbenchmarks` and a required extended\nCI job. Following the existing workflow convention, it runs in the merge\nqueue, on release-branch pushes, and manually; it skips PR events.\n- Execute Criterion bench targets in test mode with `--no-fail-fast`,\nincluding the physical-plan benchmarks gated by `test_utils`. The\n`ci-bench` profile retains workspace debug assertions and overflow\nchecks with light optimization to keep larger cases practical.\n- Use the checked-in ten-row ClickBench fixture for SQL planner smoke\ntests through `CLICKBENCH_DATA_PATH`. Ordinary performance runs retain\ntheir full-dataset default. The separate SQL benchmark harness, which\nrequires generated datasets, remains covered by its existing jobs.\n- Fix benchmark failures exposed by execution: timestamp timezone\nmetadata, Spark slice input coercion, string return types, and repeat\nrow counts/error assertions. Slice coercion happens before timing the\nUDF.\n- Clarify the Spark ListView benchmark labels and check the repeat\noverflow diagnostic outside the timed loop. Patch two certificate\ndependencies to resolve the security audit failure.\n\n## What is the testing strategy for this PR?\n\n- The full `cargo xtask ci step test microbenchmarks` run passed all 131\nbenchmark targets on upstream `99193cbd0`. After rebasing onto\n`6c1727fcd`, the Spark slice and repeat benchmark smoke tests passed\nagain, including the new labels and overflow diagnostic assertion.\n- `cargo fmt --all` and `cargo clippy --all-targets --all-features -- -D\nwarnings` pass with the pinned Rust 1.99 toolchain on the final rebase.\n- `./ci/scripts/security_audit.sh` passes after patching `rustls-webpki`\nto 0.103.16 and `webpki-root-certs` to 1.0.10.\n- Extended workspace tests were run with the documented features. The\nrun stopped at the existing `sort_with_mem_limit_1_runner` RSS\nassertion: the core integration target had 1,463 passing tests, one\nfailure, and one ignored test. Its RSS monitor reported a zero-byte\nbaseline and a 192.2 MB peak against a 190.7 MB limit. The unchanged\ntest passed in isolation on a clean upstream build, but the full run\nstill fails locally. No tests were skipped and no memory-test code or\nlimits were changed.\n\n## Are there any user-facing changes?\n\nNo query behavior or public API changes. Benchmark contributors gain a\nlocal smoke-test command and automated execution coverage.\n\nOriginal-commit: apache/datafusion@457ccaa56be2af30b3a6c6edd76ba4d155c836dd\n" }, { "commit": "d8fdd56e062ea3df8edbd6aeeb4bb6907560bc7e", "tree": "778e3e40489c7014f454559842ba40c59d16a218", "parents": [ "c5733fefc38b0a769281d9984c136a2a4e760e5f" ], "author": { "name": "Oleks V", "email": "comphead@users.noreply.github.com", "time": "Wed Oct 21 22:47:35 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 21 22:47:35 2026 +0000" }, "message": "feat: support Spark-compatible `stack` function (#26266)\n\n## Which issue does this PR close?\n\n- Part of #15914\n- Related Comet issue:\nhttps://github.com/apache/datafusion-comet/issues/5125\n\n## Rationale for this change\n\nSpark\u0027s `stack(n, expr1, ..., exprk)` separates the values into `n`\nrows. It is the standard way to unpivot in Spark SQL, and Comet falls\nback to Spark for it.\n\n## What changes are included in this PR?\n\n- `stack` in `datafusion-spark`. A ScalarUDF cannot emit rows, so like\n`json_tuple` it returns the `n` rows as a `List\u003cStruct\u003ccol0, col1,\n...\u003e\u003e`, which is the data type of Spark\u0027s Generator. As in Spark, the\nlist is not nullable while its elements and fields are. The caller\n(Comet, or `unnest` in SQL) expands it.\n- The arguments are validated as in Spark\u0027s `Stack.checkInputDataTypes`,\nwith the same messages. `n` is coerced to `INT`. The values of a column\nmust have the same Spark type, and NULLs, Arrow types of one Spark type\n(such as `Utf8` and `Utf8View`, or lists and maps whose field names\ndiffer) and integer literals that fit an `INT` column are cast to it.\n- Primitive columns are copied with a strided loop, and string columns\nof only scalars (such as unpivot keys) are tiled with a few large\n`concat` calls. Other types use `take` over the concatenated values. On\nthe new `stack` bench this is about 3x to 5x faster than `interleave`.\n\n## What is the testing strategy for this PR?\n\n`spark/generator/stack.slt` ports the cases of Spark\u0027s\n`GeneratorFunctionSuite`, `GeneratorExpressionSuite` and\n`TypeCoercionSuite`, and of the `generators`, `table-valued-functions`\nand `join-lateral` golden files. It also covers each kernel path. Unit\ntests cover an empty batch, the row limit and the type widening, which\nSQL cannot reach.\n\n## Are there any user-facing changes?\n\nA new Spark-compatible `stack` function. Unlike Spark, `n` must be an\ninteger literal, so a foldable expression such as `3 - 1` is rejected.\n\nThe rows are materialized per batch, so a call that generates more than\n16777216 rows (`n` times the batch size) fails with a resources error.\nSpark streams the rows and has no limit. With 8192-row batches this\nrejects `n` above 2048 for a column argument.\n\nOriginal-commit: apache/datafusion@6489d1510ce1f38c14491e520d152986a8cb5c5d\n" }, { "commit": "c5733fefc38b0a769281d9984c136a2a4e760e5f", "tree": "f7cced07ed40a295e2d2ed4757b03dbf47191c85", "parents": [ "4b09f5aebcdfafbdfc4762c42fbfd66a3a5a1fa9" ], "author": { "name": "Neil Conway", "email": "neil.conway@gmail.com", "time": "Wed Oct 21 19:26:27 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Oct 21 19:26:27 2026 +0000" }, "message": "chore: Use `value_field` to fetch list element field (#26567)\n\n## Which issue does this PR close?\n\n- N/A\n\n## Rationale for this change\n\nArrow 60 added `value_field()` to `GenericListArray`,\n`GenericListViewArray` and `FixedSizeListArray`. It returns the field\nthat describes the list\u0027s elements.\n\nThis PR adopts this new API more broadly, which enables removing some\nboilerplate and simplifying a few APIs.\n\n## What changes are included in this PR?\n\n* Call `value_field` instead of matching on `data_type` directly\n* Update various APIs that passed both a typed list array along with a\nseparate element field; the latter can be omitted\n\n## What is the testing strategy for this PR?\n\nExisting tests pass.\n\n## Are there any user-facing changes?\n\nNo behavior changes.\n\nOriginal-commit: apache/datafusion@139da57bdae415f6faf3b0d27b1dd1ca520bb0aa\n" }, { "commit": "4b09f5aebcdfafbdfc4762c42fbfd66a3a5a1fa9", "tree": "d52cc00b76c1fd6133f3072dcc13983a88a739c0", "parents": [ "54d933892dd42c2a9a51cae4421eb0d543909e45" ], "author": { "name": "Singleton", "email": "135771543+SingletonYe@users.noreply.github.com", "time": "Fri Oct 16 14:13:08 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Oct 16 14:13:08 2026 +0000" }, "message": "fix: report OneOf signature mismatches as planning errors (#26329)\n\n## Which issue does this PR close?\n\nPart of #20501.\n\nThis is a deliberately narrow slice of that issue, and it is the slice a\nmaintainer previously asked for: #21032 attempted this fix as a large\nchange (it also reworked `sum`/`avg` diagnostics and the public\n`first_value`/`last_value`/`nth_value` arity contracts), was asked to be\nsplit up, and then went stale. This PR is **one behaviour change and\nnothing else**.\n\n## Rationale for this change\n\n`get_valid_types_with_udf` reports \"no `OneOf` branch matched\" with\n`internal_err!`. Every alternative signature failing means the *caller*\npassed the wrong number or types of arguments — a user mistake — yet the\nuser is told:\n\n\u003e This issue was likely caused by a bug in DataFusion\u0027s code. Please\nhelp us to resolve this by filing a bug report in our issue tracker: ...\n\nSo an ordinary SQL typo invites a bug report, and the failure surfaces\nas `Internal error: ...` even though nothing internal went wrong. The\nmaintainer comment on #20070 named exactly this (\"Can try to fix the\ndisplay + internal error reference\").\n\n### Before / after\n\n`SELECT substr(1, 2)`\n\nBefore:\n\n```text\nError during planning: Internal error: Function \u0027substr\u0027 failed to match any signature, errors: Error during planning: Function \u0027substr\u0027 requires String, but received Int64 (DataType: Int64).,Error during planning: Function \u0027substr\u0027 expects 3 arguments but received 2.\nThis issue was likely caused by a bug in DataFusion\u0027s code. Please help us to resolve this by filing a bug report in our issue tracker: https://github.com/apache/datafusion/issues\n```\n\nAfter:\n\n```text\nError during planning: Function \u0027substr\u0027 failed to match any signature, errors: Error during planning: Function \u0027substr\u0027 requires String, but received Int64 (DataType: Int64).,Error during planning: Function \u0027substr\u0027 expects 3 arguments but received 2.\n```\n\nThe message body, the candidate signatures, and the trailing \"No\nfunction matches the given name and argument types ... You might need to\nadd explicit type casts.\" line are all unchanged. Only the `Internal\nerror` kind and the bug-report boilerplate are gone.\n\n## What changes are included in this PR?\n\n- `datafusion/expr/src/type_coercion/functions.rs`: return `plan_err!`\ninstead of `internal_err!` when every `OneOf` branch rejects the\narguments. One line of behaviour change.\n- A unit test covering that failure. It mirrors `substr`\u0027s real\nsignature, so one branch fails on the argument type and the other on the\narity.\n- Ten `sqllogictest` expectations that asserted the old user-visible\ntext, in `array/array_range.slt`, `spark/math/round.slt`, `errors.slt`\nand `math.slt`.\n\n## Are these changes tested?\n\nYes.\n\n- The new unit test fails on `main` and passes with the change; `cargo\ntest -p datafusion-expr --lib` is 282 passed, 0 failed.\n- `cargo fmt --all --check` is clean.\n- I audited every other expectation of this message. The remaining ones\n— `functions.slt`, `aggregate.slt`, `string_literal.slt`,\n`named_arguments.slt`, and one assertion in\n`datafusion/optimizer/src/analyzer/type_coercion.rs` — match on a\nsubstring that does not include the removed prefix, so they are\nunaffected. No `sqllogictest` file asserted the \"This issue was likely\ncaused by a bug...\" boilerplate.\n\nI could not run the full `sqllogictest` suite locally (it needs a full\n`datafusion-core` build), so CI is the verification for the four updated\n`.slt` files.\n\n## Scope\n\nDeliberately left out, to keep this reviewable:\n\n- the message is still verbose and still prefixes every branch error\nwith `Error during planning:` — that is #19004;\n- `errors.join(\",\")` still runs the branch errors together without a\nseparator, which is why the after-message reads `...(DataType:\nInt64).,Error during planning: ...`;\n- the arity phrasing suggested on #20070 (`log expects 1 to 2 arguments,\ngot 3`) is a separate improvement.\n\nHappy to follow up on any of those in separate PRs.\n\nOriginal-commit: apache/datafusion@81824e8230c1189acd2243e1e4ce7aa6e8b38e82\n" }, { "commit": "54d933892dd42c2a9a51cae4421eb0d543909e45", "tree": "bed02b3fcc7ce2a6a474d8ba11d486cb61a1e0f8", "parents": [ "641db3ff21bbb9b6d53d3e0a0701ebb767d64bc0" ], "author": { "name": "Oleks V", "email": "comphead@users.noreply.github.com", "time": "Mon Oct 12 19:18:57 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 12 19:18:57 2026 +0000" }, "message": "perf: pre-size collections and reuse per-iteration buffers (#26192)\n\n## Which issue does this PR close?\n\n- N/A\n\n## Rationale for this change\n\nMany execution paths allocate more than they need. Collections start\nempty although their final size is known, and some temporaries are\nallocated again on every row, batch, partition or column. Each case is\nsmall, but together they add avoidable allocator traffic on hot paths.\n\n## What changes are included in this PR?\n\nTwo mechanical rules applied across the workspace:\n- `with_capacity` where the size is known or tightly bounded by\nin-memory data.\n- Per-iteration temporaries hoisted out of loops and reused with\n`clear()` when they don\u0027t escape the iteration.\n\nFor example, `RepartitionExec` no longer allocates a `Vec` per batch per\noutput partition, `TDigest` merging reuses one buffer instead of a\nfull-size `Vec` per merge step, and hive-partitioned writes no longer\nbuild a `Vec\u003cString\u003e` key per row.\n\n## What is the testing strategy for this PR?\n\nNo behavior change, so existing tests cover it. The workspace `--lib\n--tests --bins` suite and sqllogictest pass locally. Criterion benches\nagainst `main` (`regexp_replace`, `reverse`, `array_slice`,\n`bounded_window`) are neutral to a few percent faster.\n\n## Are there any user-facing changes?\n\nNo.\n\nOriginal-commit: apache/datafusion@9551051f13ee321eb0d41bf437f3718a138e12a2\n" }, { "commit": "641db3ff21bbb9b6d53d3e0a0701ebb767d64bc0", "tree": "b756045d3d4e7df37265ddb84b2b41293b0383cb", "parents": [ "887a3210e54b7f7f89025f7414675261114032c5" ], "author": { "name": "Vova Kot", "email": "volodymyr.kot.ua@gmail.com", "time": "Sun Oct 11 13:22:25 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Oct 11 13:22:25 2026 +0000" }, "message": "fix(spark): derive concat_ws nullability from separator (#26205)\n\n## Which issue does this PR close?\n\n\u003c!--\nWe generally require a GitHub issue to be filed for all bug fixes and\nenhancements and this helps us generate change logs for our releases.\nYou can link an issue to this PR using the GitHub syntax. For example\n`Closes #123` indicates that this PR will close issue #123.\n--\u003e\n\nPart of the work towards\nhttps://github.com/apache/datafusion/issues/15914\n\n## Rationale for this change\n\n\u003c!--\nWhy are you proposing this change? If this is already explained clearly\nin the issue then this section is not needed.\nExplaining clearly why changes are proposed helps reviewers understand\nyour changes and offer better suggestions for fixes.\n\nPlease explain the problem you are trying to solve in terms of the\nuser-visible\nbehavior, rather than the implementation.\n\nFor example, \"The code in `foo.rs` doesn\u0027t handle nulls\" is a symptom of\nthe\nimplementation. \"COUNT(DISTINCT) returns wrong results when the column\ncontains\nnulls\" is the user-visible problem.\n--\u003e\n\n`concat_ws` return value is always nullable in DF, which is incorrect.\nPer upstream Spark implementation, this is only true when separator\nitself is null.\n\n## What changes are included in this PR?\n\n\u003c!--\nThere is no need to duplicate the description in the issue here, but it\nis sometimes worth providing a summary of the individual changes in this\nPR.\n--\u003e\n\nDerive the nullability based on the separator nullability.\n\n## What is the testing strategy for this PR?\n\n\u003c!--\nWe typically require tests for all PRs in order to:\n1. Prevent the code from being accidentally broken by subsequent changes\n2. Serve as another way to document the expected behavior of the code\n\nBriefly describe how this PR is tested, and point to the specific tests\nyou added. For example: \u0027This new feature is covered by the\n`sqllogictest` cases added in `foo.slt`\u0027.\n\nIf this PR does not add tests, explain why. For example, if the change\nis already covered by existing tests, please mention it.\n\nYou should also check the `codecov` bot reply on this PR to confirm the\nchanged code is exercised.\n--\u003e\n\nAdd SLT tests for schema.\n\n## Are there any user-facing changes?\n\n\u003c!--\nIf there are user-facing changes then we may require documentation to be\nupdated before approving the PR.\n\nIf there are any breaking changes to public APIs, please add the `api\nchange` label.\n--\u003e\n\nI believe this doesn\u0027t count as an API change but would like guidance\nfrom a committer.\n\nOriginal-commit: apache/datafusion@454ca08b43fd8bb83362f575750850d5745ddee3\n" }, { "commit": "887a3210e54b7f7f89025f7414675261114032c5", "tree": "76104c6693c42fdb1e4ee67df8654dbbbbccf28d", "parents": [ "0c1e720007beeaff5cb91fb55112e96b01426fb3" ], "author": { "name": "Varun", "email": "tohuya6@gmail.com", "time": "Thu Oct 08 16:01:24 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 08 16:01:24 2026 +0000" }, "message": "fix: keep the running hash for nested dictionaries in Spark xxhash64 (#25813)\n\n## Which issue does this PR close?\n\n- Closes #25799.\n\n## Rationale for this change\n\nSpark `xxhash64` returns different results for the same data depending\non whether a value inside a struct or list is dictionary-encoded. The\ndictionary fast path restarted from the default seed, so earlier\narguments and earlier list elements had no effect.\n\n## What changes are included in this PR?\n\nPorts the fix from apache/datafusion-comet#5757:\n\n- The dictionary fast path is used only when every row has the same\nrunning hash.\n- The fast path starts from that running hash instead of 42.\n\n## What is the testing strategy for this PR?\n\nTwo unit tests ported from Comet, both fail on main:\n\n- `test_dictionary_element_in_list_matches_decoded`: a dictionary inside\na list hashes the same as its decoded values.\n- `test_dictionary_with_nonuniform_seeds_matches_decoded`: rows with\ndifferent running hashes hash the same as the decoded values.\n\nThe existing Spark hash unit tests and sqllogictests pass. Both queries\nfrom the issue now return matching results for dictionary and plain\nvalues.\n\n## Are there any user-facing changes?\n\n`xxhash64` now matches Spark for dictionary-encoded values inside\nstructs and lists. No API changes.\n\nCo-authored-by: tohuya6 \u003c201355151+tohuya6@users.noreply.github.com\u003e\nCo-authored-by: Neil Conway \u003cneil.conway@gmail.com\u003e\n\nOriginal-commit: apache/datafusion@03204c0c84126fcfcc77ba3ef60c59b076d5538f\n" }, { "commit": "0c1e720007beeaff5cb91fb55112e96b01426fb3", "tree": "2caaad7fba0338bffd54a80d967e482d9c202a76", "parents": [ "f9524972e95b17e5d8a6f0763dffc552e7ab3c28" ], "author": { "name": "ranflarion", "email": "ran@flarion.io", "time": "Thu Oct 08 14:03:57 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 08 14:03:57 2026 +0000" }, "message": "fix(spark): return NULL from `char` for a NULL scalar argument (#26126)\n\n## Which issue does this PR close?\n\n- Closes #26125.\n\n## Rationale for this change\n\n`SELECT char(NULL::INT)` fails with `Execution error: The argument must\nbe an Int64 array or scalar.` where Spark returns NULL. NULL rows of an\narray argument already return NULL, only a NULL scalar argument reaches\nthe error.\n\n## What changes are included in this PR?\n\n`spark_chr` returns a NULL `Utf8` scalar for a NULL `Int64` scalar.\n\n## What is the testing strategy for this PR?\n\nA `char(NULL::INT)` case in `spark/string/char.slt`, which fails on main\nwith the error above.\n\n## Are there any user-facing changes?\n\n`char` of a NULL scalar returns NULL instead of an error.\n\nOriginal-commit: apache/datafusion@415f25db4b768cc2090f5740823fbcf561518074\n" }, { "commit": "f9524972e95b17e5d8a6f0763dffc552e7ab3c28", "tree": "2f0b91d12cd1c6c23e1815b5d4fa2ff12c07b2b7", "parents": [ "5cdf57dba5668f72330afeff5117907bc10ecbb3" ], "author": { "name": "Neil Conway", "email": "neil.conway@gmail.com", "time": "Mon Oct 05 00:00:19 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 05 00:00:19 2026 +0000" }, "message": "fix: handle null lists with empty children in Spark\u0027s `shuffle` (#25544)\n\n## Which issue does this PR close?\n\n- N/A\n\n## Rationale for this change\n\nThe Spark-compatible `shuffle` implementation unnecessarily copied the\nfirst child element as a placeholder for each null list. This is wrong\nfor typed-NULL lists with no children, and was redundant work anyway:\nNULL lists can be adequately represented by a repeated offset entry,\nalong with setting the appropriate bit in the validity buffer.\n\n## What changes are included in this PR?\n\n* Fix bug\n* Add unit test\n\n## What is the testing strategy for this PR?\n\nExisting tests pass, new test added.\n\n## Are there any user-facing changes?\n\nNo.\n\nOriginal-commit: apache/datafusion@287b4da0e511473d267a5dc233b38b9836c13c49\n" }, { "commit": "5cdf57dba5668f72330afeff5117907bc10ecbb3", "tree": "1db70a0e8440226cf90a7a52d3864c94f998c1da", "parents": [ "9f98a0ab5a674d1235e742fb19243ffba840a51d" ], "author": { "name": "Kassahun Sanayew", "email": "87040046+KassaSana@users.noreply.github.com", "time": "Sat Oct 03 03:22:13 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Oct 03 03:22:13 2026 +0000" }, "message": "fix: Validate `execution.time_zone` at `SET` time (#25788)\n\n## Which issue does this PR close?\n\n- Part of #17498. This covers `datafusion.execution.time_zone` only.\n\n## Rationale for this change\n\n`datafusion.execution.time_zone` is stored as a plain string, so an\ninvalid value such as `SET TIME ZONE \u003d \u0027Asia/Taipei2\u0027` (or `\u002708:00\u0027`,\n`\u0027+08:00:00\u0027`) is accepted and shown by `SHOW TIME ZONE`. The error only\nappears later, on the first `::TIMESTAMPTZ` cast. Worse,\n`current_date()` and `current_time()` parse the value with `.ok()`, so\nthey silently ignore a bad time zone and return results as if none was\nset.\n\nPostgreSQL and Spark (`spark.sql.session.timeZone` is checked with\n`isValidTimezone` in `SQLConf`) both reject an invalid time zone when it\nis set.\n\nThis picks up #23224 by @Probablism, which was closed as stale, and\nanswers its open review questions:\n\n- **Normalization** (the `timestamps.slt` question): #23224 stored a\nparsed `Tz`, so `+08` came back as `+08:00` and `arrow_typeof(now())`\nchanged. Here the value is checked with `Tz::from_str` but stored\nexactly as written, so the only change users see is that invalid values\nare rejected. `timestamps.slt` is unchanged, and `set_variable.slt` now\nchecks that `SHOW TIME ZONE` returns `+08`.\n- **`AEST` in tests**: `AEST` was never a valid Arrow time zone. The\ntests that used it only passed the string through without parsing it, so\nthey now use `Australia/Sydney`.\n\n## What changes are included in this PR?\n\n- Add `ConfigTimeZone` in `datafusion/common/src/config.rs`. It can only\nbe built through `FromStr`, which checks the value with Arrow\u0027s `Tz`\nparser and keeps the original string. It provides `as_str()` and\n`Display`.\n- Change `ExecutionOptions::time_zone` from `Option\u003cString\u003e` to\n`Option\u003cConfigTimeZone\u003e`. Its `ConfigField` impl parses before\nassigning, following `DFParquetStatistics`, so a rejected value leaves\nthe previous value (or `None`) in place, and `RESET` restores `None`.\n- Update the readers (SQL planner, `now`, `current_date`,\n`current_time`, `to_timestamp*`, `from_unixtime`, Spark `cast` /\n`date_trunc`) to use `as_str()`.\n- Remove `to_timestamp_invalid_execution_timezone_behavior` and\n`to_timestamp_formats_invalid_execution_timezone_behavior`, along with\ntheir now-unused helper. They put an invalid string straight into the\nconfig to test the error at invoke time, and that state can no longer be\nconstructed. The unit test and slt cases below cover the rejection\ninstead.\n\n## What is the testing strategy for this PR?\n\n- `set_variable.slt`: the invalid values `+08:00:00`, `08:00`, `08` and\n`Asia/Taipei2` now fail on `SET` (through both `SET TIME ZONE` and `SET\ndatafusion.execution.time_zone`), and `SHOW TIME ZONE` confirms the\nprevious value is kept. It also checks that `+08` is shown as written.\n- `test_execution_time_zone_validation` in `datafusion-common`: valid\nvalues round-trip exactly; invalid values error and keep the previous\nvalue; an invalid update of an unset value leaves it unset; nested keys\nare rejected.\n- Existing time zone suites pass unchanged (`timestamps.slt`,\n`timestamps_timezone.slt`, `current_*_timezone.slt`,\n`to_timestamp_timezone.slt`, `from_unixtime_timezone.slt`, Spark\ncast/date_trunc).\n\nCommands run (all passing):\n\n- `cargo test -p datafusion-common --lib config::tests` (with and\nwithout `--features parquet`)\n- `cargo test --profile\u003dci --test sqllogictests` (all 523 files)\n- `cargo test -p datafusion-functions --lib datetime`\n- `cargo test -p datafusion-spark --lib function::`\n- `cargo test -p datafusion-sql --lib`\n- `cargo test -p datafusion --test user_defined_integration\nconfig_options`\n- `cargo test -p datafusion-ffi --features integration-tests --test\nffi_udf`\n- `cargo test -p datafusion --test parquet_integration`\n- `cargo fmt --all -- --check`\n- `cargo clippy --all-targets --all-features -- -D warnings`\n\n## Are there any user-facing changes?\n\nYes. `SET TIME ZONE` / `SET datafusion.execution.time_zone` now fails\nimmediately on an invalid time zone instead of failing on a later query,\nor being silently ignored by `current_date()` / `current_time()`.\n\nThis changes the public field type `ExecutionOptions::time_zone` from\n`Option\u003cString\u003e` to `Option\u003cConfigTimeZone\u003e`. Code that reads it can use\n`.as_ref().map(ConfigTimeZone::as_str)`, and code that sets it can use\n`Some(\"UTC\".parse()?)`. Could a maintainer please add the `api change`\nlabel? I\u0027m happy to add a note to the 56.0.0 upgrade guide if that\u0027s\nwanted.\n\ncc @Jefffrey (you reviewed #23224), @kumarUjjawal @alamb (reviewed the\nother #17498 slices)\n\nOriginal-commit: apache/datafusion@b550f59884c47cb03675295d3ef9f35159afc7d5\n" }, { "commit": "9f98a0ab5a674d1235e742fb19243ffba840a51d", "tree": "837d7c78c99f609dbe1ae3417399c625e24bcd7d", "parents": [ "4ff76e3bf4a58b3e8205d7d419ba388f0e31c665" ], "author": { "name": "mrhard9090", "email": "74858909+mrhard9090@users.noreply.github.com", "time": "Wed Sep 30 13:33:30 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 30 13:33:30 2026 +0000" }, "message": "fix: xxhash64 of a NULL struct ignores the struct fields (#25774)\n\n## Which issue does this PR close?\n\n- Closes #25505.\n\n## Rationale for this change\n\nSpark\u0027s `xxhash64` treats a NULL value by leaving the running hash\nunchanged, so `xxhash64` of a NULL struct should be the seed, `42`. When\na struct is NULL but its child arrays still hold values, for example\nafter `nullif(s, s)`, DataFusion hashed those child values anyway:\n\n```sql\nSELECT s IS NULL, xxhash64(s), xxhash64(arrow_cast(NULL, \u0027Struct(\"a\": Int64, \"b\": Utf8)\u0027))\nFROM (SELECT nullif(s, s) AS s FROM (SELECT named_struct(\u0027a\u0027, 1, \u0027b\u0027, \u0027x\u0027) AS s));\n-- true 6158280145716684778 42\n```\n\nThe same NULL gets two different hashes depending on how it was\nproduced.\n\n## What changes are included in this PR?\n\nIn the `DataType::Struct` branch of `create_hashes_internal!`, when the\nstruct array has NULLs, the hashes are saved before the fields are\nhashed, and the saved value is put back for every NULL row. Without\nNULLs the branch works as before and copies nothing.\n\n## What is the testing strategy for this PR?\n\n- New unit test `test_xxhash64_null_struct_ignores_fields`: a struct\nwhose second row is NULL but whose fields still hold values. The NULL\nrow keeps the seed and the valid row is hashed.\n- `spark/hash/xxhash64.slt` gets the query from the issue, which now\nreturns `true 42 42`. Without the change it returns\n`6158280145716684778` for the masked NULL, so the test fails.\n- `cargo test -p datafusion-spark --lib xxhash64` passes, and so do the\n`xxhash64` sqllogictest, `cargo fmt --all` and `cargo clippy -p\ndatafusion-spark --no-deps --all-targets --all-features -- -D warnings`.\n\n## Are there any user-facing changes?\n\n`xxhash64` of a NULL struct now returns the input hash (`42` for a\nsingle argument), matching Spark. There are no API changes.\n\nThis PR was written with help from Claude Opus 5.5. I reviewed the\nchange and ran the tests above myself.\n\n---------\n\nCo-authored-by: mrhard9090 \u003cmoltrax88@gmail.com\u003e\n\nOriginal-commit: apache/datafusion@f56ae979e29f620fa6effae1836cb4b1492d434e\n" }, { "commit": "4ff76e3bf4a58b3e8205d7d419ba388f0e31c665", "tree": "4ae2b07003cbd058799567bc54bc207144e50b25", "parents": [ "7411b612bcec21aaf62b291dbd78f94c8df1d3e2" ], "author": { "name": "Mason", "email": "mason.hall@coralogix.com", "time": "Fri Sep 25 15:58:03 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 25 15:58:03 2026 +0000" }, "message": "Fix panic from `avg(x ORDER BY y)` (and more) (#25711)\n\n## Which issue does this PR close?\n\n\u003c!--\nWe generally require a GitHub issue to be filed for all bug fixes and\nenhancements and this helps us generate change logs for our releases.\nYou can link an issue to this PR using the GitHub syntax. For example\n`Closes #123` indicates that this PR will close issue #123.\n--\u003e\n\n- Closes #.\n\n## Rationale for this change\n\nSeveral aggregate UDFs keep the default implementation of\n`order_sensitivity()` but panic when the `ORDER BY` column is passed in.\n\n\u003c!--\nWhy are you proposing this change? If this is already explained clearly\nin the issue then this section is not needed.\nExplaining clearly why changes are proposed helps reviewers understand\nyour changes and offer better suggestions for fixes.\n\nPlease explain the problem you are trying to solve in terms of the\nuser-visible\nbehavior, rather than the implementation.\n\nFor example, \"The code in `foo.rs` doesn\u0027t handle nulls\" is a symptom of\nthe\nimplementation. \"COUNT(DISTINCT) returns wrong results when the column\ncontains\nnulls\" is the user-visible problem.\n--\u003e\n\n## What changes are included in this PR?\n\nThis change updates these UDFs to return\n`AggregateOrderSensitivity::Insensitive` so that the `ORDER BY` columns\nare ignored.\n\n\u003c!--\nThere is no need to duplicate the description in the issue here, but it\nis sometimes worth providing a summary of the individual changes in this\nPR.\n--\u003e\n\n## What is the testing strategy for this PR?\n\nI added sqllogictests that fail without the fix.\n\n\u003c!--\nWe typically require tests for all PRs in order to:\n1. Prevent the code from being accidentally broken by subsequent changes\n2. Serve as another way to document the expected behavior of the code\n\nBriefly describe how this PR is tested, and point to the specific tests\nyou added. For example: \u0027This new feature is covered by the\n`sqllogictest` cases added in `foo.slt`\u0027.\n\nIf this PR does not add tests, explain why. For example, if the change\nis already covered by existing tests, please mention it.\n\nYou should also check the `codecov` bot reply on this PR to confirm the\nchanged code is exercised.\n--\u003e\n\n## Are there any user-facing changes?\n\nNo\n\n\u003c!--\nIf there are user-facing changes then we may require documentation to be\nupdated before approving the PR.\n\nIf there are any breaking changes to public APIs, please add the `api\nchange` label.\n--\u003e\n\nOriginal-commit: apache/datafusion@e8e41ae958a5e6856ab9b0e7d8fb5deece0a9e91\n" }, { "commit": "7411b612bcec21aaf62b291dbd78f94c8df1d3e2", "tree": "eed8ed013f5cde293bde0414922c888c9a8dd18d", "parents": [ "f1dbbd78e6cb32381e0a3e0e9f6179f87d4603ac" ], "author": { "name": "Advit Arora", "email": "advitarora2@gmail.com", "time": "Tue Sep 22 08:48:42 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 22 08:48:42 2026 +0000" }, "message": "feat(spark): add equal_null (#24827)\n\n## Which issue does this PR close?\n\nNo issue filed. Part of #15914, which tracks the Spark function library.\n\n## Rationale for this change\n\n`equal_null` is missing, and the `misc` module it belongs in had no\nfunctions in it at all. In\nSpark it is an alias: `EqualNull(l, r)` is rewritten to `EqualNullSafe`,\nthe `\u003c\u003d\u003e` operator, so\nit is null-safe equality. Both NULL is true, and the result is never\nNULL:\n\n```sql\nSELECT equal_null(NULL, NULL); -- true\nSELECT equal_null(NULL::int, 1::int); -- false\n```\n\nDataFusion already has that operator as `IS NOT DISTINCT FROM`, so this\nmirrors Spark\u0027s own\nstructure instead of writing a second comparison kernel.\n\n## What changes are included in this PR?\n\n`misc/equal_null.rs` and its registration. `simplify()` rewrites to the\noperator, which is\nSpark\u0027s `replacement` field, and `invoke_with_args` calls `apply_cmp`\nwith the same operator so\nthe function also works with the logical optimizer disabled, as the\ncrate README requires for\nComet.\n\nTwo of the stub\u0027s commented-out queries were malformed, the porting\nscript wrote one cast per\ndistinct `typeof()` key, so a call with two identical literals lost an\nargument. The same\nartifact affects 10 more pairs in 6 other spark files, left alone here.\n\nTwo divergences from Spark are left alone because they belong to the\noperator, not to this\nfunction. Spark treats `-NaN` and `NaN` as equal and DataFusion does\nnot, which is true of\n`IS NOT DISTINCT FROM` generally. Spark also rejects maps at analysis\nsince `MapType` is not\norderable, while `comparison_coercion` here accepts them.\n\n`return_field_from_args` is overridden so the field is not nullable.\n`normalize_float_zero` now\nrecurses into nested children. `compare_op_for_nested` and\n`GroupValuesColumn` both normalize\nthrough it, so the scalar kernels and the two group-by paths agree on\nnested float keys.\n\n## Are these changes tested?\n\nYes, `spark/misc/equal_null.slt` goes from a skipped stub to 27\nassertions covering the truth\ntable, float ordering, columns, arrays, structs, decimals and the arity\nerrors.\n\nReverting `simplify()` to plain `Eq` fails 9 of them, so the file is not\npassing on constant\nfolding. Putting `invoke_with_args` back to a stub fails the two queries\nthat run with the\noptimizer off, and nothing else.\n\n## Are there any user-facing changes?\n\nYes. `equal_null` is new in `datafusion-spark`, and `-0.0` now matches\n`+0.0` inside nested\nvalues, in comparisons and in `GROUP BY`, matching the scalar kernels.\n\nOriginal-commit: apache/datafusion@d42cd854df0f78d95e95946831077db3c7808ced\n" }, { "commit": "f1dbbd78e6cb32381e0a3e0e9f6179f87d4603ac", "tree": "c4ebba430f0707a0262f342fb5a75c06d60186dc", "parents": [ "36df6516e83363704c6ec40eb99c00664f92f115" ], "author": { "name": "Neil Conway", "email": "neil.conway@gmail.com", "time": "Sun Sep 20 13:55:11 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Sep 20 13:55:11 2026 +0000" }, "message": "perf: Size builder capacity using visible array span, not total size (#25502)\n\n## Which issue does this PR close?\n\n- N/A\n\n## Rationale for this change\n\nMany UDFs used the raw size of an input array\u0027s values buffer to reserve\ncapacity in a builder or similar data structure. This can result in\nsignificantly over-allocating space for sliced array inputs. We use the\nnew offset span helpers added in #25481.\n\n`array_element` had a similar but slightly distinct allocation mistake:\nit reserved space for every child element, but it only returns a single\nvalue per input row.\n\nAlong the way, refactor `array_add` to use a shared array math kernel.\n\n## What changes are included in this PR?\n\n* Update ~31 UDFs to size buffer allocations using visible input ranges,\nnot raw input size\n* Add unit tests checking that allocated capacity for various UDFs\nmatches visible input size\n* Reuse the shared array-math kernel for `array_add` and consolidate\ntest fixtures\n\nThe following UDFs are improved:\n\n - `array_add`\n - `array_append`\n - `array_compact`\n - `array_concat`\n - `array_element`\n - `array_normalize`\n - `array_pop_back`\n - `array_pop_front`\n - `array_prepend`\n - `array_remove`\n - `array_remove_all`\n - `array_remove_n`\n - `array_replace`\n - `array_replace_all`\n - `array_replace_n`\n - `array_reverse` (List/LargeList inputs; ListView is more complex)\n - `array_scale`\n - `array_slice` (List/LargeList inputs; ListView is more complex)\n - `array_subtract`\n - `concat` (DataFusion)\n - `concat_ws` (DataFusion)\n - `initcap`\n - `string_to_array`\n - `translate`\n - `concat` (Spark)\n - `shuffle` (Spark)\n - `slice` (Spark)\n - `try_url_decode` (Spark)\n - `unhex` (Spark)\n - `url_decode` (Spark)\n - `url_encode` (Spark)\n\n## What is the testing strategy for this PR?\n\nExisting tests pass. Added new unit tests to check that the behavior of\nvarious UDFs doesn\u0027t change for sliced vs. unsliced inputs; also checked\nthat the allocated capacity for the output array reflects the visible\nspan of the input, not the raw input buffer size.\n\n## Are there any user-facing changes?\n\nNo.\n\nOriginal-commit: apache/datafusion@b8683f3d0ae3f8a2f673afa7cbb9571ac45a96b0\n" }, { "commit": "36df6516e83363704c6ec40eb99c00664f92f115", "tree": "6b8ae3143fac5ce4c8dd28f09071a31b84df075b", "parents": [ "110d1685c5c7315ebc4ab8bf2217cc37c31d044a" ], "author": { "name": "Liang-Chi Hsieh", "email": "viirya@gmail.com", "time": "Sun Sep 20 00:46:12 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Sep 20 00:46:12 2026 +0000" }, "message": "fix: reject LargeList offsets that exceed i32 (#25516)\n\n## Which issue does this PR close?\n\nN/A\n\n## Rationale for this change\n\nSpark map functions accept `LargeList` inputs whose offsets are\nrepresented as\n`i64`. The map construction path converts these offsets to `i32`, but\nthe\nexisting unchecked cast silently wraps oversized values.\n\nFor example, an offset of `2147483648` becomes `-2147483648`. Downstream\ncode\ncan then interpret the wrapped value as an invalid child-array index\ninstead of\nreporting the unsupported input.\n\n## What changes are included in this PR?\n\n- Replace the unchecked `i64` to `i32` conversion with a checked\nconversion.\n- Return an actionable execution error when an offset cannot be\nrepresented as\n `i32`.\n- Add a regression test using a sliced `LargeList` whose first offset\nexceeds\n `i32::MAX`.\n\n## What is the testing strategy for this PR?\n\nAdded `large_list_offsets_do_not_truncate`.\n\nAblation testing confirmed that the test fails with the old\nimplementation,\nwhich returns the truncated offsets `[-2147483648, -2147483647]`, and\npasses\nwith the checked conversion.\n\nAlso verified:\n\n- `cargo fmt --all -- --check`\n- `git diff --check`\n\nThe full test suite is covered by CI.\n\n## Are there any user-facing changes?\n\nYes. Oversized `LargeList` offsets now produce a clear execution error\ninstead\nof being silently truncated. There are no API changes.\n\nOriginal-commit: apache/datafusion@38a1865b7e18889f4645c5b081a44b414af3545c\n" }, { "commit": "110d1685c5c7315ebc4ab8bf2217cc37c31d044a", "tree": "56dbf68813f5838b33c946200bbb22900bf00b7d", "parents": [ "c93275c047f9085635bf9fd67aa851ae070a6cbb" ], "author": { "name": "Neil Conway", "email": "neil.conway@gmail.com", "time": "Sat Sep 19 14:53:53 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Sep 19 14:53:53 2026 +0000" }, "message": "chore: Add helpers for computing `OffsetBuffer` span, length (#25481)\n\n## Which issue does this PR close?\n\n- N/A\n\n## Rationale for this change\n\nCode that works with Arrow arrays often neglects to account for slicing:\nthat is, all of the array\u0027s backing values might not be visible, so code\nshould generally not iterate over values outside the visible range, or\nassume that the raw length of the backing array is equivalent to the\nlogical length of the array.\n\nTo simplify common coding patterns in this area and to encourage correct\nusage, this PR introduces two helper functions:\n\n* `offset_span` returns `(first_offset, value_len)`, the \"visible\" span\ndescribed by an `OffsetBuffer`\n* `offset_span_len` returns `value_len`, the number of values covered by\nan `OffsetBuffer`\n\nThis replaces typical boilerplate required to handle sliced arrays.\n\n## What changes are included in this PR?\n\n* Introduce two new helper functions\n* Adopt those helper functions through DataFusion, replacing\nhand-written offset arithmetic\n* Simplify or eliminate several helpers that did similar things already\n(e.g., `estimate_byte_data_size` in `encoding/inner.rs`)\n* Add unit tests\n\n## What is the testing strategy for this PR?\n\nExisting tests pass; new unit tests added.\n\n## Are there any user-facing changes?\n\nNo. No behavior changes.\n\nOriginal-commit: apache/datafusion@24a7f13f51230def9ceb0eeae920fd3840324e13\n" }, { "commit": "c93275c047f9085635bf9fd67aa851ae070a6cbb", "tree": "b9780a4263005ecdbabf34611370764de4980a64", "parents": [ "fb5dd79d58c67ec2fee61f033243e43e3780c85b" ], "author": { "name": "Neil Conway", "email": "neil.conway@gmail.com", "time": "Fri Sep 18 21:21:54 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 18 21:21:54 2026 +0000" }, "message": "fix: Spark `map_from_arrays`, `map_from_entries` for sliced lists (#25431)\n\n## Which issue does this PR close?\n\n- Closes #25419.\n\n## Rationale for this change\n\nIn `datafusion-spark`, `map_from_arrays` and `map_from_entries` return\nwrong results when their input list array is sliced. The values are\ncorrect, but the keys come from rows outside the slice.\n\n## What changes are included in this PR?\n\n* Fix the shared helper `map_deduplicate_keys` to behave correctly by\nslicing the input key range down to only the currently visible keys\n* Add `debug_assert` to check a useful invariant\n* Update various comments\n* Add unit tests\n* Add SLTs\n\n## What is the testing strategy for this PR?\n\nExisting tests pass, new tests added.\n\n## Are there any user-facing changes?\n\nNo, aside from bugfix.\n\nOriginal-commit: apache/datafusion@6e4d9bae3c63585cfd920d6364c0c438028acac1\n" }, { "commit": "fb5dd79d58c67ec2fee61f033243e43e3780c85b", "tree": "7097632052254f219ced5f27799ca301390fc04d", "parents": [ "1c832e0967e99ba3580a024f8cfc9f9aadd2ddf1" ], "author": { "name": "Jeffrey Vo", "email": "jeffrey.vo.australia@gmail.com", "time": "Sun Sep 13 10:52:14 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Sep 13 10:52:14 2026 +0000" }, "message": "Fix `Numeric` signature coercion to properly handle null types (#24988)\n\n## Which issue does this PR close?\n\n\u003c!--\nWe generally require a GitHub issue to be filed for all bug fixes and\nenhancements and this helps us generate change logs for our releases.\nYou can link an issue to this PR using the GitHub syntax. For example\n`Closes #123` indicates that this PR will close issue #123.\n--\u003e\n\n- N/A\n\n## Rationale for this change\n\n\u003c!--\nWhy are you proposing this change? If this is already explained clearly\nin the issue then this section is not needed.\nExplaining clearly why changes are proposed helps reviewers understand\nyour changes and offer better suggestions for fixes.\n\nPlease explain the problem you are trying to solve in terms of the\nuser-visible\nbehavior, rather than the implementation.\n\nFor example, \"The code in `foo.rs` doesn\u0027t handle nulls\" is a symptom of\nthe\nimplementation. \"COUNT(DISTINCT) returns wrong results when the column\ncontains\nnulls\" is the user-visible problem.\n--\u003e\n\nFor UDFs using `Numeric` signature with more than one argument\n(currently mod \u0026 pmod from datafusion-spark), they cannot handle inputs\n`Null, Int` whereas they accept `Int, Null`. Fixing the coercion logic\nto ensure both are accepted.\n\n## What changes are included in this PR?\n\n\u003c!--\nThere is no need to duplicate the description in the issue here, but it\nis sometimes worth providing a summary of the individual changes in this\nPR.\n--\u003e\n\nModify numeric type coercion handling to ignore null types\n\n## What is the testing strategy for this PR?\n\n\u003c!--\nWe typically require tests for all PRs in order to:\n1. Prevent the code from being accidentally broken by subsequent changes\n2. Serve as another way to document the expected behavior of the code\n\nBriefly describe how this PR is tested, and point to the specific tests\nyou added. For example: \u0027This new feature is covered by the\n`sqllogictest` cases added in `foo.slt`\u0027.\n\nIf this PR does not add tests, explain why. For example, if the change\nis already covered by existing tests, please mention it.\n\nYou should also check the `codecov` bot reply on this PR to confirm the\nchanged code is exercised.\n--\u003e\n\nAdded unit \u0026 SLT tests\n\n## Are there any user-facing changes?\n\n\u003c!--\nIf there are user-facing changes then we may require documentation to be\nupdated before approving the PR.\n\nIf there are any breaking changes to public APIs, please add the `api\nchange` label.\n--\u003e\n\nNo\n\nOriginal-commit: apache/datafusion@e4c4fa43d29d3266b9e7b16a2ec45925f379bfb4\n" }, { "commit": "1c832e0967e99ba3580a024f8cfc9f9aadd2ddf1", "tree": "7ad28d667c66171a827f33a9ad4a5dfed61b0784", "parents": [ "12888a7609e12543e32897a1510687af503eaba3" ], "author": { "name": "Xuanwo", "email": "github@xuanwo.io", "time": "Fri Sep 11 10:34:54 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 11 10:34:54 2026 +0000" }, "message": "chore: check license headers in SQL logic tests (#25182)\n\n## Which issue does this PR close?\n\nNo linked issue; follows up on missing license headers found during\nrelease review.\n\n## Rationale for this change\n\nSeveral SQL logic test files lack ASF license headers, and the existing\nlicense check only selects Rust, Python, and TypeScript files. Include\nSQL logic tests in that check to catch missing headers in future\nchanges.\n\n## What changes are included in this PR?\n\n- Add ASF headers to 18 SQL logic test files, including\n`sort_pushdown.slt`, `datetime/arith_timestamp_duration.slt`,\n`datetime/arith_time_interval.slt`, and `parquet_cdc_config.slt`.\n- Select all `*.slt` files and map the extension to HawkEye\u0027s `script`\ncomment style.\n- Normalize 406 existing headers and their spacing to the canonical\nformat required by HawkEye 7.0.0. SQL statements and expected results\nare unchanged.\n\n## What is the testing strategy for this PR?\n\n- The existing license check covers all 517 SQL logic test files and\npasses with no unsupported files.\n- Temporarily removing a header from both a top-level and a nested SQL\nlogic test makes the CI license check fail; restoring the headers makes\nit pass.\n- Compare all modified SQL logic test bodies against the base commit,\nexcluding leading blank lines: unchanged.\n- Full extended workspace test command passes (11,605 Rust tests passed,\n8 ignored, plus the SQL logic test runner).\n- `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D\nwarnings`, and `./dev/rust_lint.sh`.\n\n## Are there any user-facing changes?\n\nNo. This changes license headers and CI coverage only.\n\nOriginal-commit: apache/datafusion@1fb898093211dc544adb58ae517f32b333273efd\n" }, { "commit": "12888a7609e12543e32897a1510687af503eaba3", "tree": "27af2e3b46426bbf1af48976012af037d86a95b7", "parents": [ "4f3c26f666d5f3abe2de7e3e4da20571927cfcf0" ], "author": { "name": "Zeel Rajodiya", "email": "jeelrajodiyajeel@gmail.com", "time": "Mon Sep 07 00:45:52 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 00:45:52 2026 +0000" }, "message": "feat: Add Spark-compatible `encode` function to datafusion-spark (#21331)\n\n**Rationale**\n\nThe `datafusion-spark` crate is missing the `encode` function. Spark\u0027s\n[`encode(expr,\ncharset)`](https://spark.apache.org/docs/latest/api/sql/index.html#encode)\nconverts a string or binary value into binary using a specified\ncharacter encoding — commonly used in Spark SQL workloads and needed by\nengines built on DataFusion that target **Spark compatibility**.\n\n**What changes are included in this PR?**\n\nAdds `SparkEncode` to `datafusion-spark`\u0027s string functions, emulating\n**Spark 3.5** semantics. It supports **US-ASCII, ISO-8859-1, UTF-8,\nUTF-16, UTF-16BE, UTF-16LE, UTF-32, UTF-32BE, and UTF-32LE**, including\ncommon aliases (`UTF8`, `LATIN1`, etc.) and **case-insensitive**\nmatching. The charset can be a **constant or a per-row column**. Binary\ninput is decoded as **lossy UTF-8** (invalid bytes → U+FFFD) before\nre-encoding, and **unmappable characters are silently replaced with\n`?`**, matching Spark.\n\n**Are these changes tested?**\n\nYes. Coverage lives in `encode.slt` (sqllogictest) and exercises all\ncharsets and aliases, case-insensitive matching, null value/charset\nhandling, **per-row charsets**, binary input\n(`Binary`/`LargeBinary`/`BinaryView`) with lossy UTF-8, `Utf8View`\ninput, and the unsupported-charset error. A Rust unit test covers\nreturn-field nullability.\n\n**Are there any user-facing changes?**\n\nNew `encode` scalar function available when using `datafusion-spark`.\n\n---------\n\nCo-authored-by: Jeffrey Vo \u003cjeffrey.vo.australia@gmail.com\u003e\n\nOriginal-commit: apache/datafusion@fdfb67d05668b576eaf90d329b51ba8e1cb4f571\n" }, { "commit": "4f3c26f666d5f3abe2de7e3e4da20571927cfcf0", "tree": "cf1cb8e67b846146eff1f71118dc3b7aec479dd9", "parents": [ "dd615d844fdbf74f576d0615f8f7d7e5a1425e49" ], "author": { "name": "Amit Vijapur", "email": "vijapuramit@gmail.com", "time": "Sun Sep 06 16:35:07 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Sep 06 16:35:07 2026 +0000" }, "message": "fix(spark): derive pmod\u0027s decimal result type from the declared arguments (#24409)\n\n## Which issue does this PR close?\n\n- Closes #23895.\n\n## Rationale for this change\n\n`pmod` reports a wider decimal type than Spark does. Spark derives the\nresult\ntype of `pmod` with `Pmod.resultDecimalType`, which applies the\n`Remainder`\nrule to the *declared* argument types:\n\n```text\nscale \u003d max(s1, s2)\nprecision \u003d min(p1 - s1, p2 - s2) + scale\n```\n\nFor `pmod(decimal(3,1), decimal(2,1))` Spark reports `decimal(2,1)`, but\nDataFusion reported `Decimal128(3, 1)`.\n\nThe cause is coercion. `SparkPmod` used `Signature::numeric`, which\ncollapses\nboth arguments to a common decimal before `return_type` runs, so the two\nprecisions `return_type` saw were already equal and the rule degenerated\nto the\ninput precision.\n\n## What changes are included in this PR?\n\n- `SparkPmod` moves to `Signature::user_defined` with a `coerce_types`\nthat\nleaves a decimal/decimal argument pair intact, following the precedent\nset by\n `try_sum`. Every other argument combination keeps the coercion\n`Signature::numeric` performed — including its null handling, where a\nnull\nargument is skipped and an all-null call falls back to `Float64` — so\nonly the\n decimal pair changes behaviour.\n- `return_type` applies Spark\u0027s `Pmod.resultDecimalType` rule for\ndecimal\n arguments and is unchanged for everything else.\n- Because Spark\u0027s result type is *narrower* than the dividend, the\noperands\ncannot be cast to it before the remainder is taken without overflowing\nthe\n dividend — `pmod(99.9::decimal(3,1), 2.5::decimal(2,1))` returns\n`decimal(2,1)`, which cannot hold `99.9`. `spark_pmod` therefore widens\nthe\noperands to a common computation type, takes the remainder there, and\nnarrows\n the result afterwards.\n\n### Overflow semantics\n\nThe remainder is bounded by the divisor, but the result type only\ncarries\n`min(p1 - s1, p2 - s2)` integer digits, so the narrowing step can\noverflow when\nthe divisor is wider than the dividend:\n\n```sql\n-- result type decimal(3,1), true value 9999.8\nSELECT pmod(-0.1::decimal(3,1), 9999.9::decimal(5,1));\n```\n\nSpark wraps decimal arithmetic in `CheckOverflow(nullOnOverflow \u003d\n!ansiEnabled)`,\nso the narrowing cast returns NULL in legacy mode and raises under ANSI.\nThe\nwidening cast uses `safe: false` in both modes, since the computation\ntype is\nchosen to fit both operands and a silent NULL there would hide a real\nbug.\n\n### Scope\n\nDeliberately limited to `pmod` over two `Decimal128` arguments, which is\nwhat\n#23895 reports. Three adjacent gaps are left alone and are happy to be\nfollow-ups\nif you would rather see them here:\n\n- **`SparkMod`** has the same bug, since `Remainder.resultDecimalType`\nis the\nsame rule. It is the easier of the two: arrow\u0027s `Op::Rem` already\ncomputes\n`min(p1-s1, p2-s2) + max(s1, s2)`, so `mod` needs only the\n`coerce_types`\npass-through and the matching `return_type`, with no widen/narrow step.\n- **Decimal mixed with integer** still diverges:\n`pmod(2.5::decimal(3,1), 3)`\nreports `Decimal128(21, 1)` where Spark casts INT to `decimal(10,0)` and\n reports `decimal(3,1)`.\n- **`Decimal256`, `Decimal64` and `Decimal32`** pairs fall through to\nthe\n previous behaviour. Spark has no equivalent of the wider types.\n\n## Are these changes tested?\n\nYes.\n\n`datafusion/sqllogictest/test_files/spark/math/pmod.slt` gains:\n\n- four `arrow_typeof` assertions covering equal scales, differing\nprecisions,\n differing scales, and the narrowing case;\n- a value test for `pmod(99.9::decimal(3,1), 2.5::decimal(2,1))`, the\ncase that\n would regress if the operands were narrowed before the remainder;\n- the overflow case above, asserted as NULL in legacy mode and as an\nerror in\n the ANSI block; and\n- null-argument cases pinning the coercion parity described above.\n\n`modulus.rs` gains a unit test for `pmod_decimal_result_type` covering\nthe rule\ndirectly, independent of the planner.\n\nThe existing `pmod` and `mod` value tests are unchanged and still pass.\nVerified\nlocally: `cargo test -p datafusion-spark --all-features` (279 passed),\nall 244\n`spark/` sqllogictest files, `cargo clippy --all-targets --all-features\n-D\nwarnings`, and `cargo fmt --all --check`.\n\n## Are there any user-facing changes?\n\nYes, and it is the point of the fix: `pmod` over two decimals now\nreports the\nsame result type Spark does. Values that fit the Spark result type are\nunchanged. Values that do not fit were previously returned at the wider\ntype and\nare now NULL (legacy) or an error (ANSI), matching Spark. No public API\nchanges.\n\n---------\n\nCo-authored-by: Jeffrey Vo \u003cjeffrey.vo.australia@gmail.com\u003e\n\nOriginal-commit: apache/datafusion@262936eef50727753c47d1d1b3d86dea994c55f0\n" }, { "commit": "dd615d844fdbf74f576d0615f8f7d7e5a1425e49", "tree": "1bb4b918415941b4db40dc69f05a882942810bff", "parents": [ "bf5f6546ba01c091c01adff52c15277a34f8e37c" ], "author": { "name": "Shayan Gh", "email": "98089795+ShayanGho@users.noreply.github.com", "time": "Sun Sep 06 03:38:01 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Sep 06 03:38:01 2026 +0000" }, "message": "fix(spark): accept every integer width in `factorial`, matching Spark\u0027s implicit cast (#24943)\n\n## Which issue does this PR close?\n\n- Closes #24940.\n\n## Rationale for this change\n\nIn Spark mode, `SELECT factorial(5)` fails to plan: an untyped integer\nliteral is `Int64` in\nDataFusion and the function\u0027s signature was `Exact(Int32)`. Spark\u0027s\n`Factorial` declares\n`inputTypes \u003d Seq(IntegerType)` but extends `ImplicitCastInputTypes`, so\na real Spark casts\nTINYINT, SMALLINT and BIGINT arguments to INT before evaluating.\nVerified against\n`pyspark\u003d\u003d4.2.0` under both ANSI settings: every integer width returns\nthe same values.\n\nThe narrow signature was chosen deliberately in #16125 from the\nDatabricks reference, which\ndocuments the parameter as an INTEGER expression. That describes the\ndeclared parameter type;\nthe accepted set is wider because of the implicit cast, which is only\nvisible by running Spark.\n\n## What changes are included in this PR?\n\n- `SparkFactorial` now uses `Signature::coercible` with an implicit\ncoercion from the Integer\ntype class to `Int32`, the same construct `round`, `width_bucket` and\n`bit_get` in this crate\nuse. Planning inserts the cast, so `spark_factorial` still only receives\n`Int32`.\n- `factorial.slt`: replaces the assertion that `factorial(5::BIGINT)`\nmust fail with two blocks: one\naliased row covering the untyped literal, TINYINT, SMALLINT, BIGINT and\na bare NULL, and an\narray-shaped BIGINT query covering a negative value, both range\nboundaries, an above-range value\nand NULL. Every expected value was observed from PySpark 4.2.0, not from\nDataFusion output.\n- Non-integer inputs still fail, now with DataFusion\u0027s standard coercion\nerror. Spark\u0027s wider\nacceptance of STRING / DECIMAL / FLOAT inputs is tracked in #24941, and\nthe ANSI-dependent\n handling of out-of-range integers in #24950 (part of #23929).\n\n## What is the testing strategy for this PR?\n\nThe new `.slt` queries fail on `main` and pass with this change,\nverified by checking out the\npre-fix implementation file and re-running `cargo test --test\nsqllogictests -- spark/math/factorial`:\n\n- with the old signature checked out: both blocks fail with\n `coercion from Int64 to the signature Exact(Int32) failed`.\n- restored: `Progress: 1/1 files completed (100%)`.\n\n`cargo test -p datafusion-spark factorial`, `cargo fmt --all -- --check`\nand\n`cargo clippy -p datafusion-spark --all-targets -- -D warnings` pass.\nThe existing unit tests in\n`factorial.rs` are unchanged.\n\n## Are there any user-facing changes?\n\n`factorial` in `datafusion-spark` now accepts all integer widths. No\nbreaking API change.\n\n🤖 Generated with [Claude Code](https://claude.com/claude-code)\n\n---------\n\nCo-authored-by: Claude Fable 5.1 \u003cnoreply@anthropic.com\u003e\nCo-authored-by: Oleks V \u003ccomphead@users.noreply.github.com\u003e\n\nOriginal-commit: apache/datafusion@a5a525e1d67a6edfb9c29b4d470cf45778e872d6\n" }, { "commit": "bf5f6546ba01c091c01adff52c15277a34f8e37c", "tree": "b3c1840a1e6aa7d02655f2ba89d30ca5d24be116", "parents": [ "c0909aa75b82c558373063fdd97e32597f1f499a" ], "author": { "name": "theirix", "email": "theirix@gmail.com", "time": "Sat Sep 05 23:25:36 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Sep 05 23:25:36 2026 +0000" }, "message": "fix: keep decimal precision for SLT tests (#24934)\n\n## Which issue does this PR close?\n\n- Closes #23835.\n\n## Rationale for this change\n\nProvide full precision for decimals in SLT tests.\nSeparate float formatting (round to 12/15 digits) from decimal\nformatting (avoid rounding)\n\n## What changes are included in this PR?\n\n- For floats, round to 12 digits by default and 15 for Spark\n(historically), via a generic function\n- For decimals, avoid rounding entirely. Emit any trailing zeroes to SLT\noutput, as they are part of decimals. Rely on `format_decimal` directly\n- For Postgres mode with `decimal_to_str` - emit BigDecimal\nrepresentation of numerics directly\n- Remove `big_decimal_to_str` helper with rounding behaviour\n- A few amends to pg_compat tests. I introduced casts in pg_compat tests\nto fit pg_compat mode with explicit types, either double precision or\nnumeric(p,s). Postgres treats literals as numeric and so prints without\nrounding. For some UDF (avg, exp), it was required to cast the output to\nfloat explicitly, because Postgres widens inputs and outputs to decimal.\nIn DataFusion, it produces floats for float inputs.\n\n- Refines #23160 - CC @AdamGS\n\n## What is the testing strategy for this PR?\n\n- An SLT test case for the issue - now it produces the expected decimal\nwith full precision\n- Reworked unit tests to test float and decimal conversion, but not an\nintermediate `big_decimal_to_str`\n- Regenerated SLTs with `--complete` to introduce trailing zeroes - a\nhuge inevitable change\n\n\n## Are there any user-facing changes?\n\nOriginal-commit: apache/datafusion@9442d88c57d4ce2cef1cdcff9eb53a03ef0c6faf\n" }, { "commit": "c0909aa75b82c558373063fdd97e32597f1f499a", "tree": "b79f96ab279242058d78b578a42e3504fb69646a", "parents": [ "c5e086437d0219ee22b382eb828fd2152235c57a" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Sat Sep 05 20:02:51 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Sep 05 20:02:51 2026 +0000" }, "message": "perf: remove per-row String allocation from Spark soundex and quote (#23880)\n\n## Which issue does this PR close?\n\nN/A\n\n## Rationale for this change\n\nSpark\u0027s `soundex` and `quote` both allocated a fresh `String` for every\nrow and\ncollected the results into a `StringArray`.\n\n`soundex` allocated twice per row: once for the code buffer, and again\nfor the\n`format!(\"{soundex_code:0\u003c4}\")` that zero-pads it. `quote` allocated a\n`String`\nsized to the input, then copied the input into it one `char` at a time.\n\nNeither function needs to allocate per row. A soundex code is always\nexactly\nfour ASCII characters, so it fits in a stack buffer. `quote` only ever\nwraps the\ninput and escapes embedded quotes, so it can write straight into the\noutput\nbuffer and copy the runs between quotes rather than character by\ncharacter.\n\n## What changes are included in this PR?\n\n`soundex.rs`:\n\n- `compute_soundex(\u0026str) -\u003e String` becomes `append_soundex(\u0026mut\nStringBuilder, \u0026str)`,\nbuilding the four-character code in a `[u8; 4]` initialised to `b\u00270\u0027` —\nwhich is\n the zero-padding, so the trailing `format!` disappears.\n- Strings that do not start with an ASCII letter are passed through\nunchanged;\npreviously this path called `s.to_string()`, now it appends by\nreference.\n- The `Utf8`/`LargeUtf8` and `Utf8View` entry points share one\n`soundex_impl`\n over `Option\u003c\u0026str\u003e`, pre-sizing the builder at 4 bytes per row.\n\n`quote.rs`:\n\n- `compute_quote(\u0026str) -\u003e String` becomes `append_quoted(\u0026mut\nStringBuilder, \u0026str)`,\nwriting into the builder\u0027s buffer via the `fmt::Write` impl and\nfinalising with\n `append_value(\"\")`.\n- Escaping uses `str::split(\u0027\\\u0027\u0027)` to copy the runs between quotes in\none memcpy\n each, instead of pushing every `char` individually.\n- The builder is pre-sized from the input\u0027s `value_data()` length plus\ntwo bytes\n per row for the surrounding quotes.\n\nOutput is unchanged in both cases.\n\n## Are these changes tested?\n\nExisting coverage pins the behaviour: `spark/string/soundex.slt` and\n`spark/string/quote.slt` assert concrete outputs, including the\nnon-alphabetic\npassthrough, codes shorter than four characters (padding), codes\ntruncated at\nfour, and strings with and without embedded quotes. All 59\n`spark/string`\nsqllogictest files and the 258 `datafusion-spark` unit tests pass.\n\nThe benchmark used below, `datafusion/spark/benches/soundex_quote.rs`,\nis added\nseparately in #23882 so the baseline can be measured on `main` before\nthis\nchange lands. It covers both functions over `Utf8` and `Utf8View` at\n1024 and\n8192 rows with 20% nulls; the `quote` input is a mix of strings with and\nwithout\nembedded quotes so the escaping path is exercised without dominating.\n\n### Benchmarks\n\nCriterion, `apache/main` @ `f1ab86dad` as baseline. Median of the\nreported\nchange interval.\n\n| Benchmark | 1024 | 8192 |\n| --- | --- | --- |\n| `soundex/utf8` | −52.9% | −49.5% |\n| `soundex/utf8view` | −54.5% | −50.2% |\n| `quote/utf8` | −60.7% | −61.5% |\n| `quote/utf8view` | −60.5% | −61.3% |\n\n## Are there any user-facing changes?\n\nNo. Both functions produce byte-identical output; this is purely an\nallocation\nchange.\n\nOriginal-commit: apache/datafusion@be5f2654ea68a38a7cf1ef78d20c70600727264a\n" }, { "commit": "c5e086437d0219ee22b382eb828fd2152235c57a", "tree": "b7384eb203e6888365e4fda91a17b2a97d8df165", "parents": [ "35003999af8c732789310269bafe9350a167408c" ], "author": { "name": "Kosta Tarasov", "email": "33369833+sdf-jkl@users.noreply.github.com", "time": "Fri Sep 04 03:52:52 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 04 03:52:52 2026 +0000" }, "message": "Unify temporal field parsing with DatePart (#24906)\n\n## Which issue does this PR close?\n\n- Closes #22033.\n\n## Rationale for this change\n\nTemporal functions currently parse field names independently, which\nproduces inconsistent user-visible behavior. For example, `date_part`\naccepts aliases such as `mon`, `mons`, `month`, and `months`, while\n`date_trunc` accepts only `month`.\n\nUsing Arrow\u0027s `DatePart` parser as the common representation makes these\naliases consistent across temporal functions and SQL unparsing.\n\n## What changes are included in this PR?\n\n- Use `DatePart::from_str` directly in `date_part` instead of parsing\nthrough `IntervalUnit` and maintaining additional manual mappings.\n- Replace the private `DateTruncGranularity` enum and its string parser\nwith `DatePart`.\n- Restrict `date_trunc` to the `DatePart` variants that represent valid\ntruncation granularities.\n- Update the Spark wrappers to delegate field parsing to the shared\nimplementation while retaining their function-specific semantic\nrestrictions.\n- Map Spark\u0027s Sunday-based `dayofweek` directly to `dow1` instead of\nadding one to the result expression.\n- Use `DatePart` when canonicalizing temporal fields in the SQL\nunparser.\n\n## What is the testing strategy for this PR?\n\nSQL logic tests cover the newly shared aliases, including:\n\n- `date_part`: `QUARTERS`, `ISOWEEK`, `DAYOFYEAR`, and `DAYOFWEEK`\n- `date_trunc`: `mon`, `months`, `qtr`, and `yrs`\n- Spark wrappers: `QUARTERS`, `DAYOFYEAR`, `MONTHS`, and `MINS`\n- Unparser canonicalization of aliases such as `MONS` and `YRS`\n\nThe following checks passed:\n\n- `cargo fmt --all`\n- `cargo clippy --all-targets --all-features -- -D warnings`\n- Extended workspace tests with\n`avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`\n- Spark datetime SQL logic tests\n- `cargo test -p datafusion-spark --lib`\n\n## Are there any user-facing changes?\n\nYes. Temporal functions now consistently accept the field-name aliases\nrecognized by Arrow\u0027s `DatePart` parser. This broadens accepted input\nspellings without changing existing accepted inputs or truncation\nbehavior.\n\nThe Spark wrappers inherit these aliases as well, making them slightly\nmore permissive than Spark\u0027s exact spelling tables.\n\nOriginal-commit: apache/datafusion@6fc7baa7d0ae7b2c10d9faae04c73b2d7561aa29\n" }, { "commit": "35003999af8c732789310269bafe9350a167408c", "tree": "d54697470440086bc1fde03cf5af38258fb0a5fe", "parents": [ "e44933ba2b6eec737f808f08e607a4c34750f889" ], "author": { "name": "Advit Arora", "email": "advitarora2@gmail.com", "time": "Fri Sep 04 03:10:09 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 04 03:10:09 2026 +0000" }, "message": "fix: find_in_set internal error with two LargeUtf8 scalar arguments (#24802)\n\n## Which issue does this PR close?\n\n- No issue filed. `find_in_set.slt` is one of the porting stubs tracked\nby #15914 and the bug fell out of filling it. I can open one if the\nchangelog entry needs a link.\n\n## Rationale for this change\n\n`find_in_set` with two `LargeUtf8` scalar arguments does not return a\nvalue, it trips the planner\u0027s type assertion:\n\n```sql\n\u003e SELECT find_in_set(arrow_cast(\u0027ab\u0027, \u0027LargeUtf8\u0027), arrow_cast(\u0027abc,b,ab,c,def\u0027, \u0027LargeUtf8\u0027));\nInternal error: Assertion failed: result_data_type \u003d\u003d *expected_type: Function \u0027find_in_set\u0027\nreturned value of type \u0027Int32\u0027 while the following type was promised at planning time and\nexpected: \u0027Int64\u0027.\n```\n\n`return_type` goes through `utf8_to_int_type`, which widens to `Int64`\nfor `LargeUtf8`. The three array branches are generic over\n`Int32Type`/`Int64Type` and honour that. The `(Scalar, Scalar)` branch\ncomputed `Some(position as i32)` and passed it to `ScalarValue::from`,\nwhich is always `ScalarValue::Int32`, so it was the one branch that\ncould contradict its own `return_type`. Nothing had executed it:\n`string_literal.slt` covers scalar/scalar for `Utf8` and `Utf8View`\nonly, and the `LargeUtf8` block in `string_query.slt.part` casts\ncolumns, so they land in the array branches.\n\n## What changes are included in this PR?\n\n- Build the scalar result at the width `return_field` promised, with the\nsame `match` on the return type the array branches already use four\ntimes.\n- Fill `spark/string/find_in_set.slt`, previously a stub. Expected\nvalues come from Spark 4.2.0, read off `UTF8String.findInSet` and\nconfirmed on `pyspark\u003d\u003d4.2.0`, not from running DataFusion.\n\nThe `.slt` is the larger half of the diff at 37 queries. Coverage that\nexists nowhere else in the tree: a needle containing a comma, empty and\nconsecutive and trailing-comma elements, no trimming around delimiters,\nfirst occurrence wins on duplicates, and a 20-element list that crosses\n`FIND_IN_SET_LOOKUP_THRESHOLD` so the lookup path from #23460 gets its\nfirst end-to-end test. DataFusion matches Spark on all 37.\n\n## Are these changes tested?\n\nYes, the `LargeUtf8` query above is the regression test. Reverting the\nsource change and rebuilding gives exactly one failure, on that query,\nwith the assertion above.\n\n```\ncargo test --profile\u003dci --test sqllogictests -- spark/string/find_in_set.slt\ncargo test --profile\u003dci --test sqllogictests -- string/ # 65 files\ncargo test --profile\u003dci -p datafusion-functions --lib find_in_set # 6 passed\ncargo clippy --profile ci -p datafusion-functions --lib -- -D warnings\n```\n\nI left `string/string_literal.slt` alone. The new case runs in the same\nCI job, so a second copy next to the `Utf8` block there catches nothing\nextra.\n\n## Are there any user-facing changes?\n\nYes. `find_in_set` over two `LargeUtf8` scalars returns a value instead\nof raising an internal error. No API change.\n\nOriginal-commit: apache/datafusion@9fc7a4d5dec26ff191fb79a3216c32b73d22522e\n" }, { "commit": "e44933ba2b6eec737f808f08e607a4c34750f889", "tree": "6451c290005e4d601aaa3b55e8ab1f1fa7a9e0be", "parents": [ "120a846b66a8afc3e7c23cde13bab0cf2926e7b7" ], "author": { "name": "Emil Ernerfeldt", "email": "emil.ernerfeldt@gmail.com", "time": "Tue Sep 01 06:41:29 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 01 06:41:29 2026 +0000" }, "message": "Enable more clippy lints (#24566)\n\n## Which issue does this PR close?\n\n- Part of #18467.\n\n## Rationale for this change\n\nContinuing #24466: turn on more `clippy::pedantic` lints (opt-out list\nin `Cargo.toml`).\n\n## What changes are included in this PR?\n\nOne commit per lint, each removing its `\"allow\"` line from `Cargo.toml`\nand fixing every site. Review one commit at a time!\n\nLet me know if you disagree with any and I\u0027ll revert it\n\n| Lint | Sites | Fix |\n|\n----------------------------------------------------------------------------------------------------------------------------------\n| ----: |\n--------------------------------------------------------------- |\n|\n[`no_effect_underscore_binding`](https://rust-lang.github.io/rust-clippy/master/index.html#no_effect_underscore_binding)\n| 1 | removed a dead `let _expected \u003d …` binding |\n|\n[`case_sensitive_file_extension_comparisons`](https://rust-lang.github.io/rust-clippy/master/index.html#case_sensitive_file_extension_comparisons)\n| 1 | `Path::extension()` instead of `str::ends_with(\".csv\")` |\n|\n[`cast_ptr_alignment`](https://rust-lang.github.io/rust-clippy/master/index.html#cast_ptr_alignment)\n| 5 | `#[expect]` on FFI tests that downcast to a concrete, aligned type\n|\n|\n[`doc_link_with_quotes`](https://rust-lang.github.io/rust-clippy/master/index.html#doc_link_with_quotes)\n| 25 | quoted doc \"links\" turned into real intra-doc links or backticked\ncode |\n|\n[`ref_option`](https://rust-lang.github.io/rust-clippy/master/index.html#ref_option)\n| 36 | `\u0026Option\u003cT\u003e` params/returns changed to `Option\u003c\u0026T\u003e` |\n|\n[`needless_continue`](https://rust-lang.github.io/rust-clippy/master/index.html#needless_continue)\n| 37 | dropped `continue`s that already ended the loop iteration |\n|\n[`cloned_instead_of_copied`](https://rust-lang.github.io/rust-clippy/master/index.html#cloned_instead_of_copied)\n| 38 | `copied()` instead of `cloned()` for `Copy` types |\n|\n[`unnecessary_semicolon`](https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_semicolon)\n| 185 | dropped semicolons after tail `match`/`if` |\n|\n[`unnecessary_trailing_comma`](https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_trailing_comma)\n| 49 | dropped trailing commas after single-item parens/brackets |\n\nThe `ref_option` commit also changes the signature of the public\n`apply_masking` fn in `datafusion-substrait` (`\u0026Option\u003cMaskExpression\u003e`\n-\u003e `Option\u003c\u0026MaskExpression\u003e`), since it forwards to an internal fn that\nwas flagged.\n\n## Are these changes tested?\n\n`cargo clippy --workspace --all-targets --all-features` reports no\nwarnings. The changes are mechanical and behavior-preserving, so no new\ntests.\n\n## Are there any user-facing changes?\n\nThe `ref_option` commit changes the public `apply_masking` signature in\n`datafusion-substrait`, see above.\n\n---------\n\nCo-authored-by: Claude Fable 5 \u003cnoreply@anthropic.com\u003e\n\nOriginal-commit: apache/datafusion@17547e2b5f0a7a0ebbe466630d9bed1bdd2b4683\n" }, { "commit": "120a846b66a8afc3e7c23cde13bab0cf2926e7b7", "tree": "676186c4f7621bef91dad10c569d65b0f7e79628", "parents": [ "d61e54db0390fa70f0b9afa3be47ff83db87118e" ], "author": { "name": "Emil Ernerfeldt", "email": "emil.ernerfeldt@gmail.com", "time": "Fri Aug 21 08:59:43 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 21 08:59:43 2026 +0000" }, "message": "Enable more clippy lints (#24466)\n\n## Which issue does this PR close?\n\n- Part of #18467.\n\n## Rationale for this change\n\nTurn on all `clippy::pedantic` lints, and do opt-out instead of opt-in.\nThen enable these lints (remove their opt-outs):\n\n## What changes are included in this PR?\n\nOne commit per lint, each removing its `\"allow\"` line from `Cargo.toml`\nand fixing every site. Review one commit at a time!\n\nLet me know if you disagree with any and I\u0027ll revert it\n\n| Lint | Sites | Fix |\n|\n----------------------------------------------------------------------------------------------------------\n| ----: | ------------------------------------------------------- |\n|\n[`collapsible_else_if`](https://rust-lang.github.io/rust-clippy/master/index.html#collapsible_else_if)\n| 1 | flatten `else { if .. }` into `else if` |\n|\n[`range_plus_one`](https://rust-lang.github.io/rust-clippy/master/index.html#range_plus_one)\n| 12 | `a..\u003db` instead of `a..b + 1` |\n|\n[`stable_sort_primitive`](https://rust-lang.github.io/rust-clippy/master/index.html#stable_sort_primitive)\n| 14 | `sort_unstable` where stability cannot be observed |\n|\n[`comparison_chain`](https://rust-lang.github.io/rust-clippy/master/index.html#comparison_chain)\n| 3 | `match Ord::cmp` instead of an `if`/`else if` chain |\n|\n[`should_panic_without_expect`](https://rust-lang.github.io/rust-clippy/master/index.html#should_panic_without_expect)\n| 1 | name the expected panic message |\n|\n[`manual_assert_eq`](https://rust-lang.github.io/rust-clippy/master/index.html#manual_assert_eq)\n| 25 | `assert_eq!`/`assert_ne!` so failures print both values |\n|\n[`needless_for_each`](https://rust-lang.github.io/rust-clippy/master/index.html#needless_for_each)\n| 47 | `for` loops instead of `for_each` |\n|\n[`manual_let_else`](https://rust-lang.github.io/rust-clippy/master/index.html#manual_let_else)\n| 106 | `let .. else` instead of a diverging `match` |\n\n## Are these changes tested?\n\n`cargo clippy --workspace --all-targets --all-features` reports no\nwarnings, and the extended test suite passes. The changes are mechanical\nand behavior-preserving, so no new tests.\n\n## Are there any user-facing changes?\n\nNo.\n\n---------\n\nCo-authored-by: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\nCo-authored-by: Daniël Heres \u003cdanielheres@gmail.com\u003e\n\nOriginal-commit: apache/datafusion@f75c5032af7693cc92583502e57f2afddf140325\n" }, { "commit": "d61e54db0390fa70f0b9afa3be47ff83db87118e", "tree": "7da0b3e28f4a8c765f4bde527f0a2577edfabb4f", "parents": [ "3ccc75503ab366aad70de5e509290042f47cedc7" ], "author": { "name": "Yongting You", "email": "2010youy01@gmail.com", "time": "Thu Aug 20 19:05:39 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Aug 20 19:05:39 2026 +0000" }, "message": "fix: tree-style explain for memory table shows wrong row count (#24511)\n\n## Which issue does this PR close?\n\n\u003c!--\nWe generally require a GitHub issue to be filed for all bug fixes and\nenhancements and this helps us generate change logs for our releases.\nYou can link an issue to this PR using the GitHub syntax. For example\n`Closes #123` indicates that this PR will close issue #123.\n--\u003e\n\n- Closes #.\n\n## Rationale for this change\n\n\u003c!--\nWhy are you proposing this change? If this is already explained clearly\nin the issue then this section is not needed.\nExplaining clearly why changes are proposed helps reviewers understand\nyour changes and offer better suggestions for fixes.\n\nPlease explain the problem you are trying to solve in terms of the\nuser-visible\nbehavior, rather than the implementation.\n\nFor example, \"The code in `foo.rs` doesn\u0027t handle nulls\" is a symptom of\nthe\nimplementation. \"COUNT(DISTINCT) returns wrong results when the column\ncontains\nnulls\" is the user-visible problem.\n--\u003e\nSee reproducer in datafusion-cli, the `rows:1` line in the explain is\nthe wrong result, it should be total rows for the memory table (101)\n```sh\nyongting@Yongtings-MacBook-Pro-2 ~/C/datafusion (main\u003e)\u003e datafusion-cli\nDataFusion CLI v54.1.0\n\u003e create table test_memtable as\nselect *\nfrom generate_series(100);\n0 row(s) fetched.\nElapsed 0.004 seconds.\n\n\u003e explain format tree select * from test_memtable;\n+---------------+-------------------------------+\n| plan_type | plan |\n+---------------+-------------------------------+\n| physical_plan | ┌───────────────────────────┐ |\n| | │ DataSourceExec │ |\n| | │ -------------------- │ |\n| | │ bytes: 65632 │ |\n| | │ format: memory │ |\n| | │ rows: 1 │ |\n| | └───────────────────────────┘ |\n| | |\n+---------------+-------------------------------+\n1 row(s) fetched.\nElapsed 0.001 seconds.\n\n```\n\n## What changes are included in this PR?\n\n\u003c!--\nThere is no need to duplicate the description in the issue here, but it\nis sometimes worth providing a summary of the individual changes in this\nPR.\n--\u003e\nfix the explain for row count\n\n## Are these changes tested?\nsqllogictest\n\u003c!--\nWe typically require tests for all PRs in order to:\n1. Prevent the code from being accidentally broken by subsequent changes\n3. Serve as another way to document the expected behavior of the code\n\nIf tests are not included in your PR, please explain why (for example,\nare they covered by existing tests)?\n--\u003e\n\n## Are there any user-facing changes?\n\n\u003c!--\nIf there are user-facing changes then we may require documentation to be\nupdated before approving the PR.\n\nIf there are any breaking changes to public APIs, please add the `api\nchange` label.\n--\u003e\n\nOriginal-commit: apache/datafusion@10077e8207af7156635f9b8c572152efa50d0ad4\n" }, { "commit": "3ccc75503ab366aad70de5e509290042f47cedc7", "tree": "546c7839a7d5e6d9f983f3c214d87ac4059e5193", "parents": [ "150c013b624b05facfcbc7e136f1e1cf0519bd56" ], "author": { "name": "Goutam Adwant", "email": "8672451+goutamadwant@users.noreply.github.com", "time": "Thu Aug 20 08:29:12 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Aug 20 08:29:12 2026 +0000" }, "message": "refactor: Unify Spark format_string integer numeric assembly (#24480)\n\n## Which issue does this PR close?\n\n- Closes #24425.\n\n## Rationale for this change\n\nSpark `format_string` used separate final-assembly paths for decimal\nintegers and other numeric values. Keeping sign placement, suffix\nplacement, and width padding in multiple paths made their behavior\neasier to drift apart when flag combinations changed.\n\n## What changes are included in this PR?\n\n- Route signed and unsigned decimal integer formatting through the\nexisting `write_numeric_parts` assembly helper.\n- Keep integer conversion, grouping, and sign selection in the integer\nformatter.\n- Preserve the existing unsigned integer policy and the separate\nhexadecimal and octal prefix behavior.\n- Add focused Rust and SQLLogicTest coverage for signed and unsigned\ninteger flags, decimal parentheses, and non-finite float padding.\n\n## Are these changes tested?\n\nYes. The following checks pass:\n\n```text\ncargo fmt --all -- --check\ncargo clippy --all-targets --all-features -- -D warnings\ncargo test -p datafusion-spark --all-features\ncargo test --test sqllogictests -- spark/string/format_string\nRUST_BACKTRACE\u003d1 cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption\n```\n\nThe signed integer expectations were also cross-checked against Spark\n4.2.0 in both ANSI and non-ANSI modes.\n\n## Are there any user-facing changes?\n\nNo. This is an internal refactor that preserves the existing formatting\nbehavior and does not change the public API.\n\nOriginal-commit: apache/datafusion@5091b42191f09e24827a6a14b8653faf7efa2497\n" }, { "commit": "150c013b624b05facfcbc7e136f1e1cf0519bd56", "tree": "57dcccbfa49935dbe20aadfb657c1e8b57fb7fcd", "parents": [ "fed210303d412f774b7bd2ff6737ef4cc7cf8b48" ], "author": { "name": "arsathS", "email": "100337687+arsathS@users.noreply.github.com", "time": "Thu Aug 20 03:07:37 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Aug 20 03:07:37 2026 +0000" }, "message": "feat(spark): add `to_binary` and `try_to_binary` (#24266)\n\n## Which issue does this PR close?\n\n- Part of #15914\n\n## Rationale for this change\n\nSpark\u0027s `to_binary(str[, fmt])` converts a string to binary, choosing\nhow based\non `fmt`: `hex`, `utf-8`/`utf8`, or `base64`. `try_to_binary` is the\nsame\nfunction except that input it cannot convert yields NULL instead of\nraising an\nerror.\n\nNeither is available in the `datafusion-spark` crate, and no core\nDataFusion\nfunction can stand in for them:\n\n- core `decode` accepts only `base64`, `base64pad` and `hex`, so there\nis no\n way to ask it for Spark\u0027s `utf-8` conversion\n- core `decode` always raises on input it cannot convert, so there is\nnothing\n to build `try_to_binary`\u0027s NULL-returning behaviour from\n\n## What changes are included in this PR?\n\n- New `SparkToBinary` and `SparkTryToBinary` UDFs in\n `datafusion/spark/src/function/string/to_binary.rs`\n - `to_binary(str[, fmt])` where `fmt` is a case-insensitive literal of\n `hex`, `utf-8`, `utf8` or `base64`, defaulting to `hex`\n- `hex` decodes two characters per byte, left-padding an odd-length\ninput\n with `0`, matching `Unhex`\n - `utf-8` / `utf8` returns the string\u0027s own UTF-8 bytes\n - `base64` matches Java\u0027s MIME decoder, which Spark uses: the standard\nalphabet, padding optional, and the unused trailing bits of a short\nfinal\n group ignored\n - A NULL value or a NULL `fmt` yields NULL in both functions\n - `to_binary` raises on a malformed value or an unrecognised `fmt`;\n`try_to_binary` returns NULL for both, matching `nullOnInvalidFormat`\n - `fmt` must be foldable, as Spark requires\n- `unhex_scalar` is made `pub(crate)` and reused for the `hex` path\nrather\n than duplicating the decoder\n- Registered in `mod.rs` (`make_udf_function!`, `export_functions!`,\n `functions()`)\n- Filled in the commented-out SLT scaffolds for both functions\n\n## Are these changes tested?\n\nYes.\n- 10 unit tests in `to_binary.rs` (each format, hex as the default,\nempty\ninput, NULL value and NULL format, invalid value, invalid format, column\n input, and a column where one row is invalid)\n- 15 SLT assertions across `spark/string/to_binary.slt` and\n `spark/string/try_to_binary.slt`\n\n## Are there any user-facing changes?\n\nNo. These are new functions in the `datafusion-spark` crate only.\n\n---------\n\nCo-authored-by: Claude Opus 5 \u003cnoreply@anthropic.com\u003e\nCo-authored-by: Jeffrey Vo \u003cjeffrey.vo.australia@gmail.com\u003e\n\nOriginal-commit: apache/datafusion@b428ee6aae269d5d8101002c65638ee176ab2fe0\n" }, { "commit": "fed210303d412f774b7bd2ff6737ef4cc7cf8b48", "tree": "ca44005693b74a6b88dafd675d1cc9812ed0b65b", "parents": [ "6842dafad5aa00ae1376288e3c691a362bb862d6" ], "author": { "name": "Jeffrey Vo", "email": "jeffrey.vo.australia@gmail.com", "time": "Thu Aug 20 01:05:58 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Aug 20 01:05:58 2026 +0000" }, "message": "doc: add implementation guidelines for `datafusion-spark` (#24140)\n\n## Which issue does this PR close?\n\n\u003c!--\nWe generally require a GitHub issue to be filed for all bug fixes and\nenhancements and this helps us generate change logs for our releases.\nYou can link an issue to this PR using the GitHub syntax. For example\n`Closes #123` indicates that this PR will close issue #123.\n--\u003e\n\n- From\nhttps://github.com/apache/datafusion/issues/15914#issuecomment-4324008673\n- Relates to https://github.com/apache/datafusion/issues/21959\n- Relates to https://github.com/apache/datafusion/discussions/22012\n\n## Rationale for this change\n\n\u003c!--\nWhy are you proposing this change? If this is already explained clearly\nin the issue then this section is not needed.\nExplaining clearly why changes are proposed helps reviewers understand\nyour changes and offer better suggestions for fixes.\n\nPlease explain the problem you are trying to solve in terms of the\nuser-visible\nbehavior, rather than the implementation.\n\nFor example, \"The code in `foo.rs` doesn\u0027t handle nulls\" is a symptom of\nthe\nimplementation. \"COUNT(DISTINCT) returns wrong results when the column\ncontains\nnulls\" is the user-visible problem.\n--\u003e\n\nSet some guidelines for how UDFs should be implemented in\n`datafusion-spark`. This is to align us on what we expect from UDFs,\nmaking it easier for contributors \u0026 reviewers alike.\n\nOriginal-commit: apache/datafusion@5e4f3cc89512614d6f403cfedca36b8d1b4804df\n" }, { "commit": "6842dafad5aa00ae1376288e3c691a362bb862d6", "tree": "0e8707b5caab9a482b024c46cf1406caf85e709f", "parents": [ "f6cb81ba860a402981e73f4cc132b26663441042" ], "author": { "name": "Goutam Adwant", "email": "8672451+goutamadwant@users.noreply.github.com", "time": "Wed Aug 19 03:24:54 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 19 03:24:54 2026 +0000" }, "message": "fix(spark): match Spark abs overflow errors (#24156)\n\n## Which issue does this PR close?\n\n- Closes #24145.\n\n## Rationale for this change\n\nWhen ANSI mode is enabled, `datafusion-spark` reports type-specific\nDataFusion errors for integral `abs` overflow. The scalar and array\npaths also produce different messages. Spark 4.2 uses one canonical\n`ARITHMETIC_OVERFLOW` message for both paths.\n\n## What changes are included in this PR?\n\n- Add a Spark-local checked `abs` array kernel so Spark-specific error\ntext does not change core DataFusion behavior.\n- Use the Spark 4.2 `ARITHMETIC_OVERFLOW` message for scalar and array\noverflows.\n- Update the existing Spark SQL logic tests for all signed integer\nwidths and both input shapes.\n\n## Are these changes tested?\n\nYes. The following checks pass:\n\n- `cargo fmt --all -- --check`\n- `cargo clippy -p datafusion-spark --all-targets --all-features -- -D\nwarnings`\n- `cargo test -p datafusion-spark --all-features`\n- `cargo test --profile\u003dci --test sqllogictests -- spark/math/abs.slt`\n- `cargo test --profile\u003dci --test sqllogictests -- math.slt`\n- `RUST_BACKTRACE\u003d1 cargo test --profile ci --exclude\ndatafusion-examples --exclude datafusion-benchmarks --exclude\ndatafusion-cli --workspace --lib --tests --bins --features\navro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`\n\n## Are there any user-facing changes?\n\nYes. With `datafusion.execution.enable_ansi_mode \u003d true`, Spark `abs`\noverflow errors now use Spark 4.2\u0027s canonical `ARITHMETIC_OVERFLOW`\nmessage for scalar and array inputs. There are no public API changes.\n\n---------\n\nSigned-off-by: goutamadwant \u003cworkwithgoutam@gmail.com\u003e\n\nOriginal-commit: apache/datafusion@76a45130a440b3f4238cfcbb211803d90565229a\n" }, { "commit": "f6cb81ba860a402981e73f4cc132b26663441042", "tree": "a634610b04f907452c4e7595643e9ab0354bbe26", "parents": [ "3d2805e4ce9f16c414b9bd0d11fd0afae4639fb8" ], "author": { "name": "Emil Ernerfeldt", "email": "emil.ernerfeldt@gmail.com", "time": "Tue Aug 18 17:12:32 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Aug 18 17:12:32 2026 +0000" }, "message": "Enable more clippy lints (#24322)\n\n## Which issue does this PR close?\n\n- Part of #18467\n- Follow-up to #24066\n\n## Rationale for this change\n\nMore `allow`-by-default clippy lints that simplify code, catch bugs, or\nimprove performance.\nEvery lint here was verified to be outside the default-warn groups, so\nnone is a no-op.\n\n## What changes are included in this PR?\n\n**One commit per lint, including its fixes**, so any single lint can be\nreverted on its own.\nThe first commit is the exception: 19 lints that had zero hits and need\nno code changes.\n\nTwo real bugs fell out of `literal_string_with_formatting_args`, where a\n`{placeholder}` was\nprinted verbatim instead of interpolated:\n* `benchmarks/src/nlj.rs`: `\"NLJ benchmark Q{query_id}\nfailed…\".to_string()`\n* `parquet_advanced_index.rs`: `.expect(\"metadata for file not found:\n{filename}\")`\n\n`fallible_impl_from` also flagged that `From\u003cprotobuf::Constraint\u003e for\nConstraint` panics on a\nmessage with an unset `constraint_mode`. Fixing that needs a breaking\nchange to `TryFrom`, so it\nis only marked with `#[expect]` here.\n\n## Are these changes tested?\n\nCovered by existing tests plus the clippy CI job. I also ran the\nextended test suite locally.\n\n## Are there any user-facing changes?\n\nOne non-breaking signature change: `format_human_display` and a few\nprivate helpers now take `T`\ninstead of `Option\u003cT\u003e` (`clippy::single_option_map`). No public API\nchanges.\n\n---------\n\nCo-authored-by: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\nCo-authored-by: Daniël Heres \u003cdanielheres@gmail.com\u003e\n\nOriginal-commit: apache/datafusion@303ff78ac8948979aa4b2635e366a970cd32d0e4\n" }, { "commit": "3d2805e4ce9f16c414b9bd0d11fd0afae4639fb8", "tree": "95d77837db5e9051bc8bbfd09a42a5d5e4ed6cc6", "parents": [ "1df8b4e70d5a73a60bc96f5c75f15769ae71a479" ], "author": { "name": "kid", "email": "19265318+u70b3@users.noreply.github.com", "time": "Sun Aug 16 01:23:51 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Aug 16 01:23:51 2026 +0000" }, "message": "fix(spark): correct mod ANSI zero divisor and negative zero handling (#23987)\n\n## Which issue does this PR close?\n\n- Closes #23894\n\n## Rationale for this change\n\n`datafusion-spark`\u0027s `mod` routes through the shared `try_rem` helper in\n`datafusion/spark/src/function/math/modulus.rs`, which has two gaps in\nits\nzero-divisor handling (both verified against Spark 3.5.8 – 4.2.0 in the\nissue):\n\n1. **ANSI mode, floating-point divisor.** `try_rem` delegates to Arrow\u0027s\n`rem`\nkernel in ANSI mode, but Arrow only reports division by zero for integer\nand\ndecimal types. Floating-point divisors follow IEEE 754 and quietly\nproduce\n`NaN`, while Spark raises `REMAINDER_BY_ZERO` for a zero divisor of any\n numeric type:\n\n ```sql\n set datafusion.execution.enable_ansi_mode \u003d true;\n SELECT mod(10.5::float8, 0.0::float8); -- NaN, Spark raises\n ```\n\n2. **`-0.0` divisor, both modes.** The legacy path nulls out zero\ndivisors via\n`eq(right, 0)`, but Arrow\u0027s floating-point comparisons use a total order\nin\nwhich `-0.0` is distinct from `0.0`, so a `-0.0` divisor goes\nunrecognised.\n Spark\u0027s `isZero` is a numeric comparison and treats `-0.0` as zero:\n\n ```sql\nSELECT mod(10.5::float8, -0.0::float8); -- NaN, Spark returns NULL\n(legacy)\n ```\n\n## What changes are included in this PR?\n\n`try_rem` now detects zero divisors itself, mirroring the shape #23898\nestablished for `pmod`:\n\n- A new `is_zero` helper counts `-0.0` as zero for the floating-point\ntypes\n (via a `negative_zero` companion, same as #23898).\n- In ANSI mode, any row with a zero divisor raises\n`ArrowError::DivideByZero`\n— the same error Arrow\u0027s `rem` already produces for integers today, so\nthe\nmessage stays uniform across types. The check is masked by the validity\nof\nthe dividend because Spark\u0027s remainder expressions are null intolerant:\na\nNULL dividend short-circuits to NULL before the divisor is validated, so\n `mod(NULL, 0)` must return NULL rather than raise.\n- Both modes substitute NULL for zero divisors before calling Arrow\u0027s\n`rem`,\nso the kernel never sees a zero divisor: legacy mode gets NULLs, and\nANSI\n mode has already raised on the rows that required it.\n\nNote on overlap with #23898: that PR rewrites `pmod` to no longer use\n`try_rem` and adds identical `is_zero`/`negative_zero` helpers. This PR\nis\nindependent of it — `mod` is fixed either way — but whichever lands\nsecond\nshould dedupe the helpers in a rebase. Until #23898 lands, `pmod` also\npicks\nup the `-0.0` and ANSI floating-point zero-divisor fixes through the\nshared\nhelper.\n\nOut of scope: #23897 (reproducing Spark\u0027s exact ANSI error text) is a\nrepository-wide error-message policy question, as noted in that issue.\n\n## Are these changes tested?\n\nYes:\n\n- New unit tests in `modulus.rs`: ANSI floating-point zero divisor\nraises;\n`-0.0` divisor returns NULL in legacy mode and raises in ANSI mode; a\nNULL\ndividend with a zero divisor returns NULL in ANSI mode (integer and\nfloat).\n- New sqllogictest cases in `spark/math/mod.slt` covering the same\nbehavior at\n SQL level.\n- Verified `cargo test -p datafusion-spark`, the `spark/math/mod.slt`\nand\n`spark/math/pmod.slt` sqllogictests, `./dev/rust_lint.sh`, and the\nextended\n workspace suite (ci profile with\n\n`avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`)\n — all green.\n\n## Are there any user-facing changes?\n\nOnly the bug fixes, and only for the Spark `mod`/`pmod` functions: in\nANSI\nmode a floating-point zero divisor now raises instead of returning NaN,\nand a\n`-0.0` divisor is treated as zero in both modes (NULL in legacy mode, an\nerror\nin ANSI mode), matching Spark. No API changes.\n\nOriginal-commit: apache/datafusion@2da78d67ce6dc3e2975906b3ca60e35cd1d9afed\n" }, { "commit": "1df8b4e70d5a73a60bc96f5c75f15769ae71a479", "tree": "2904782e55a99a911ddd6b29df3db925df6b58fe", "parents": [ "d4e97858e999a5bbc830f9f56a912275e8e2563e" ], "author": { "name": "RIchard Baah", "email": "137434454+Rich-T-kid@users.noreply.github.com", "time": "Sat Aug 15 11:06:08 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Aug 15 11:06:08 2026 +0000" }, "message": "Perf: remove extra allocations for hex operations (#24057)\n\n## Which issue does this PR close?\n\n\u003c!--\nWe generally require a GitHub issue to be filed for all bug fixes and\nenhancements and this helps us generate change logs for our releases.\nYou can link an issue to this PR using the GitHub syntax. For example\n`Closes #123` indicates that this PR will close issue #123.\n--\u003e\n\n- Closes #23810.\n\n## Rationale for this change\nThe digest functions hex-encode their output by allocating one String\nper row and then copying each of those into the output array (#23810)\n\u003c!--\nWhy are you proposing this change? If this is already explained clearly\nin the issue then this section is not needed.\nExplaining clearly why changes are proposed helps reviewers understand\nyour changes and offer better suggestions for fixes.\n--\u003e\n\n## What changes are included in this PR?\n\n- This PR reduces per-row allocation and string-conversion overhead in\nthe Spark crypto/hash paths by switching to the buffer-based hex encoder\nAPIs.\n- Updates call sites to re-use vector buffers.\n\n\u003c!--\nThere is no need to duplicate the description in the issue here but it\nis sometimes worth providing a summary of the individual changes in this\nPR.\n--\u003e\n\n## Are these changes tested?\nexisting test cover behavior.\n\u003c!--\nWe typically require tests for all PRs in order to:\n1. Prevent the code from being accidentally broken by subsequent changes\n2. Serve as another way to document the expected behavior of the code\n\nIf tests are not included in your PR, please explain why (for example,\nare they covered by existing tests)?\n--\u003e\n\n## Are there any user-facing changes?\nyes, `encode_bytes()` is being deprecated. This was apart of the public\nAPI\n\u003c!--\nIf there are user-facing changes then we may require documentation to be\nupdated before approving the PR.\n--\u003e\n\n\u003c!--\nIf there are any breaking changes to public APIs, please add the `api\nchange` label.\n--\u003e\n\n---------\n\nCo-authored-by: rich-T-kid \u003crichardbaah@MacBook-Air-de-Richard.local\u003e\n\nOriginal-commit: apache/datafusion@a10d19374ffcb0f24253f819a9e254336561e0bb\n" }, { "commit": "d4e97858e999a5bbc830f9f56a912275e8e2563e", "tree": "eae4e7bc5e3651d049e80e1b5f10c744a7b19bfa", "parents": [ "b7966cfa875c0f4d01712e6c58880dfc1c256134" ], "author": { "name": "Goutam Adwant", "email": "8672451+goutamadwant@users.noreply.github.com", "time": "Fri Aug 14 13:36:59 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 14 13:36:59 2026 +0000" }, "message": "feat(cli): add opt-in Spark function registration (#24179)\n\n## Which issue does this PR close?\n\n- Closes #24146.\n\n## Rationale for this change\n\n`datafusion-cli` does not currently enable the Spark SQL dialect or\nregister the `datafusion-spark` function library, so Spark-compatible\nsyntax and functions cannot be explored interactively. Enabling them by\ndefault could override core DataFusion functions with the same names and\nchange existing CLI behavior.\n\n## What changes are included in this PR?\n\n- Add a `--spark` CLI flag.\n- Select the Spark SQL dialect while constructing the session\nconfiguration.\n- Register Spark-compatible scalar, aggregate, and window functions only\nwhen the flag is enabled.\n- Add CLI integration coverage for both default and opt-in behavior.\n- Document the flag and its function-override behavior in the CLI and\nSpark documentation.\n\n## Are these changes tested?\n\nYes. The following checks pass:\n\n- `cargo test -p datafusion-cli --test cli_integration\nspark_features_require_opt_in`\n- `cargo test -p datafusion-cli --all-targets`\n- `cargo fmt --all -- --check`\n- `cargo clippy --all-targets --all-features -- -D warnings`\n- `./ci/scripts/doc_prettier_check.sh`\n\n## Are there any user-facing changes?\n\nYes. Passing `--spark` enables the Spark SQL dialect, Spark expression\nplanner, and Spark-compatible functions for the CLI session. The option\nis disabled by default, so existing CLI behavior remains unchanged.\n\nOriginal-commit: apache/datafusion@571477c4a7a532fcba67d3e272a3791755dedd79\n" }, { "commit": "b7966cfa875c0f4d01712e6c58880dfc1c256134", "tree": "579ae11e204f54cde9305e6b5c781782545e12a9", "parents": [ "940f8d9b0ecaf77c7ced22f029c54ff856975a18" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri Aug 14 11:31:36 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 14 11:31:36 2026 +0000" }, "message": "fix: correct list field inner type in array functions (#24345)\n\n## Which issue does this PR close?\n\n- Closes #24341\n\n## Rationale for this change\n\nThere is a regression from 54.1.0 to 55.0.0rc2 where the spark function\n`slice` can fail as demonstrated in the updated unit test, and as\ndescribed in the connected issue.\n\n## What changes are included in this PR?\n\nIn the nested functions, get the inner field directly from the input for\narray slicing operations. Also, since it is possible the inner list\ncould be non-nullable emit an empty slice instead of a null child\nelement for the outer lists\u0027s nulls.\n\n## Are these changes tested?\n\nAdded 6 new tests in the SLT suite.\n\n## Are there any user-facing changes?\n\nNone\n\nOriginal-commit: apache/datafusion@9f377c44862747873946e1f28ac27c976b60464d\n" }, { "commit": "940f8d9b0ecaf77c7ced22f029c54ff856975a18", "tree": "be04e9f29157f2775d338664bb4e54f3a1f57793", "parents": [ "aa2732f15381a3535774bbb42a98f277cc5eb314" ], "author": { "name": "dentiny", "email": "dentinyhao@gmail.com", "time": "Fri Aug 14 06:42:16 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 14 06:42:16 2026 +0000" }, "message": "Reduce datafusion-spark dependency footprint (#24351)\n\n## Which issue does this PR close?\n\nN/A\n\n## Rationale for this change\n\n`datafusion-spark` pulls unnecessary crates into its production\ndependency graph, increasing compilation time and artifact size for\ndownstream users.\n\n## What changes are included in this PR?\n\n- Replace compatibility re-exports with the narrower crates that define\n`TableFunction` and `FunctionRegistry`.\n- Remove the unused `crypto_expressions` activation; Spark provides its\nown SHA-1, SHA-2, and CRC32 implementations.\n- Fix the `quote` import so builds without the optional `core` feature\ncontinue to compile.\n\nThese changes reduce the production dependency graph from 279 to 257\npackages. Cargo Machete does not report these dependencies because they\nare referenced in source through re-exports or activated through Cargo\nfeatures.\n\n## Are these changes tested?\n\nYes\n\n## Are there any user-facing changes?\n\nNo. Public APIs and default behavior remain unchanged.\n\nOriginal-commit: apache/datafusion@84a8e6db91d19c203536faa0bbf1466bd5ea692f\n" }, { "commit": "aa2732f15381a3535774bbb42a98f277cc5eb314", "tree": "b56d47d7d2a5eed9b8e1adda5d20275dade4f3fb", "parents": [ "e2b783098b5c8f98f03a0202ac217ef4b8242315" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Tue Aug 11 06:26:34 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Aug 11 13:26:34 2026 +0000" }, "message": "perf: remove per-row String allocations from the Spark url functions (#23884)\n\n## Which issue does this PR close?\n\nN/A\n\n## Rationale for this change\n\nThree allocation problems in the `url` module, all on the per-row path.\n\n**`url_encode`** built a `String` for every row:\n\n```rust\nfn encode(value: \u0026str) -\u003e Result\u003cString\u003e {\n Ok(byte_serialize(value.as_bytes()).collect::\u003cString\u003e())\n}\n```\n\n**`url_decode`** ended with `.map(|parsed| parsed.into_owned())`. Both\n`replace_plus` and `decode_utf8` return a `Cow` that *borrows* when\nthere is\nnothing to rewrite, so `into_owned` allocated a `String` for every row\neven when\nthe value contained no percent-escapes and no `+` at all.\n\n**`parse_url`** built the all-null `key` array for the two-argument form\nby\ncalling `append_null()` once per row against a builder with no capacity.\n\n## What changes are included in this PR?\n\n`url_encode.rs`:\n\n- The three type branches build a pre-sized builder and share an\n`encode_all!`\n loop that clears and refills one scratch `String` per batch.\n- `UrlEncode::encode` is removed; it returned a `Result` whose error arm\nwas\n never constructed and now has no callers.\n\n`url_decode.rs`:\n\n- `decode` returns `Cow\u003c\u0027_, str\u003e` instead of `String`. When\n`replace_plus`\nborrows, the decode borrows straight from the input; when it has already\nallocated (the input contained `+`), owning the decoded form costs\nnothing\n beyond what was already spent.\n- The array paths append the `Cow` to a pre-sized builder rather than\n collecting `Result\u003cStringArray\u003e` from a per-row `String`.\n- **API change:** `spark_handled_url_decode`\u0027s second parameter changes\nfrom\n `impl Fn(Result\u003cOption\u003cString\u003e\u003e) -\u003e Result\u003cOption\u003cString\u003e\u003e` to a new\n`OnDecodeError` enum. The `String` in that signature is what forced the\n allocation. Its only caller is `try_url_decode`, whose closure was\n `Err(_) \u003d\u003e Ok(None)` — exactly `OnDecodeError::Null`.\n\n`parse_url.rs`:\n\n- The `append_null()`-per-row loop becomes\n`new_null_array(\u0026DataType::Utf8, len)`.\n\nNo behaviour change in any of the three.\n\n## Are these changes tested?\n\nExisting coverage pins the behaviour: `spark/url/url_encode.slt`,\n`url_decode.slt`, `try_url_decode.slt`, and `parse_url.slt` assert\nconcrete\nresults including reserved characters, `+` handling, malformed\npercent-encoding\n(which must error for `url_decode` and yield NULL for `try_url_decode`),\nand the\ntwo- versus three-argument `parse_url` forms. All 5 `spark/url`\nsqllogictest\nfiles pass, along with the 258 `datafusion-spark` unit tests.\n\nThe `try_url_decode` unit test is what pins the `OnDecodeError::Null`\npath,\nsince it drives a malformed input through the refactored signature.\n\nBenchmarks are added separately in #23882 so the baselines can be\nmeasured on\n`main` before this lands.\n\n### Benchmarks\n\nCriterion, `apache/main` @ `f1ab86dad` as baseline. Median of the\nreported\nchange interval.\n\n`url_encode`:\n\n| Benchmark | 1024 | 8192 |\n| --- | --- | --- |\n| `url_encode/utf8` | −57.1% | −48.2% |\n| `url_encode/largeutf8` | −56.1% | −49.7% |\n| `url_encode/utf8view` | −56.0% | −46.5% |\n\n`url_decode`, split by whether the input actually needs unescaping:\n\n| Benchmark | 1024 | 8192 |\n| --- | --- | --- |\n| `url_decode/plain_utf8` | −26.4% | −28.8% |\n| `url_decode/plain_utf8view` | −29.0% | −27.2% |\n| `url_decode/escaped_utf8` | −5.4% | −3.6% |\n| `url_decode/escaped_largeutf8` | −4.7% | −3.2% |\n\nThe `plain` rows are the ones that benefit: with nothing to unescape the\ndecoded\nvalue borrows its input. The `escaped` rows still have to allocate, so\nthey gain\nonly the pre-sized builder and the removed intermediate — a few percent,\nas\nexpected.\n\n`parse_url` has no benchmark; its change removes an `append_null()` loop\nthat\nruns once per batch rather than affecting a measured per-row path.\n\n## Are there any user-facing changes?\n\nNo change to SQL behaviour — all three functions return identical\nresults.\n\n`spark_handled_url_decode` is public Rust API and its signature changes\nas\ndescribed above; `OnDecodeError` is new public API on\n`datafusion-spark`.\n\n---------\n\nCo-authored-by: Andrew Lamb \u003candrew@nerdnetworks.org\u003e\nCo-authored-by: Claude Fable 5 \u003cnoreply@anthropic.com\u003e\n\nOriginal-commit: apache/datafusion@66ef2764481e17115da2f5e3ed29125380efac91\n" }, { "commit": "e2b783098b5c8f98f03a0202ac217ef4b8242315", "tree": "310530259449b0c07e5e96c71cf15c6111e40b83", "parents": [ "2dc2fc72c477d90b6482c19be7c8769544eb868c" ], "author": { "name": "Justin O\u0027Dwyer", "email": "justin.odwyer@datadoghq.com", "time": "Tue Aug 11 11:57:29 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Aug 11 09:57:29 2026 +0000" }, "message": "refactor: Refactor numeric sign and padding in Spark format_string (#24115)\n\n## Which issue does this PR close?\n\n\u003c!--\nWe generally require a GitHub issue to be filed for all bug fixes and\nenhancements and this helps us generate change logs for our releases.\nYou can link an issue to this PR using the GitHub syntax. For example\n`Closes #123` indicates that this PR will close issue #123.\n--\u003e\n\n- Closes https://github.com/apache/datafusion/issues/24058\n\n## Rationale for this change\n\n\u003c!--\nWhy are you proposing this change? If this is already explained clearly\nin the issue then this section is not needed.\nExplaining clearly why changes are proposed helps reviewers understand\nyour changes and offer better suggestions for fixes.\n\n--\u003e\n\n`format_decimal` and `format_float` had duplicated logic, and it\u0027d be\neasy to modify one function and forget to change the other, causing\nbugs. This PR forces one source of truth for that logic.\n\n## What changes are included in this PR?\n\n\u003c!--\nThere is no need to duplicate the description in the issue here, but it\nis sometimes worth providing a summary of the individual changes in this\nPR.\n--\u003e\n\nCreated a new helper.\n\n## Are these changes tested?\n\n\u003c!--\nWe typically require tests for all PRs in order to:\n1. Prevent the code from being accidentally broken by subsequent changes\n2. Serve as another way to document the expected behavior of the code\n\nIf tests are not included in your PR, please explain why (for example,\nare they covered by existing tests)?\n--\u003e\n\nYes,\n\n```\ncargo t -p datafusion-spark format_string --lib\n```\n\npasses.\n\n## Are there any user-facing changes?\n\n\u003c!--\nIf there are user-facing changes then we may require documentation to be\nupdated before approving the PR.\n\nIf there are any breaking changes to public APIs, please add the `api\nchange` label.\n--\u003e\n\nNo user facing-changes.\n\nOriginal-commit: apache/datafusion@d3100e2028adfe8560024c40ec3719a2e0fca2ee\n" }, { "commit": "2dc2fc72c477d90b6482c19be7c8769544eb868c", "tree": "16dd388674e783cc597a9d75ea7a03bfdc8383b4", "parents": [ "aea763aa9282bddd290b1a0b9ba10f6c1207619b" ], "author": { "name": "Liang-Chi Hsieh", "email": "viirya@gmail.com", "time": "Sun Aug 09 15:04:59 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Aug 09 07:04:59 2026 +0000" }, "message": "fix: prevent next_day panic on far-future start dates (#24194)\n\n## Which issue does this PR close?\n\n- Closes #23891.\n\n## Rationale for this change\n\n`datafusion-spark`\u0027s `next_day` computed its result by adding the day\noffset to a `chrono::NaiveDate`:\n\n```rust\nSome(Date32Type::from_naive_date(\n date + Duration::days((7 - date.weekday().days_since(day_of_week)) as i64),\n))\n```\n\nWhen the next occurrence of the requested weekday lands past\n`chrono::NaiveDate::MAX` (epoch day `95026236`) — which happens for\nstart dates in the last week of the representable `Date32` range — the\n`NaiveDate + TimeDelta` add **panics** with `NaiveDate + TimeDelta\noverflowed`. (The linked issue reports NULL; the scalar path in fact\npanics.)\n\nSpark\u0027s `DateTimeUtils.getNextDateForDayOfWeek` is pure integer\narithmetic on the epoch day and always produces a value:\n\n```scala\ndef getNextDateForDayOfWeek(startDay: Int, dayOfWeek: Int): Int \u003d {\n startDay + 1 + ((dayOfWeek - 1 - startDay) % 7 + 7) % 7\n}\n```\n\n## What changes are included in this PR?\n\n- Compute the result on the epoch day directly (`days + delta`, via\n`checked_add`) instead of building a `NaiveDate` for the result, so a\nnext occurrence past `NaiveDate::MAX` returns a value instead of\npanicking. The weekday/offset logic is unchanged, so results for all\nin-range dates are identical.\n\nScope note: a *start* day beyond `NaiveDate::MAX` (`\u003e 95026236`) still\nreturns NULL, because the weekday is derived via `NaiveDate`. Extending\nfull `Int`-range parity (computing the weekday arithmetically too) is a\nlarger change left as a follow-up; this PR fixes the panic, which is the\nconcrete reported harm.\n\n## Are these changes tested?\n\nYes.\n\n- Unit test `next_day_handles_far_future_start_dates` covering the two\ncases from the issue (`95026236`/`Mon`, `95026230`/`Tue`), each of which\npanicked before this change.\n- SLT coverage in `spark/datetime/next_day.slt` asserting the same two\ncases (casting the `Date32` result to `Int32`, since these dates are\npast the printable range).\n\n## Are there any user-facing changes?\n\n`next_day` no longer panics for far-future start dates; it returns the\ncorrect epoch day. No API changes.\n\nOriginal-commit: apache/datafusion@354f2d7f17580f0cf68750472e1d6241cfbf6ee1\n" }, { "commit": "aea763aa9282bddd290b1a0b9ba10f6c1207619b", "tree": "3bceb67cc8323c71e1116488de3d9890b0d68db8", "parents": [ "0e490c27afc7112b24dc456ab8caaf03a04ecef9" ], "author": { "name": "Karpagam Balasubramaniam", "email": "kk@trulyh.com", "time": "Wed Aug 05 02:51:25 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 05 09:51:25 2026 +0000" }, "message": "feat: add Spark-compatible atan2 function (#23962)\n\n## Which issue does this PR close?\n\n- Closes #23961\n- Part of #15914\n\n## Rationale for this change\n\nSpark provides [`atan2(exprY,\nexprX)`](https://spark.apache.org/docs/latest/api/sql/#atan2), which\nreturns the angle in radians between the positive x-axis and the point\ngiven by the coordinates (`exprX`, `exprY`). It was not yet implemented\nin `datafusion-spark` — only an auto-generated test stub existed at\n`spark/math/atan2.slt` with its query commented out.\n\n## What changes are included in this PR?\n\n- Add `SparkAtan2` (implementing `ScalarUDFImpl`) in\n`datafusion/spark/src/function/math/atan2.rs`, computing `y.atan2(x)`\nelement-wise via the Arrow `binary` kernel.\n- Register it in `datafusion/spark/src/function/math/mod.rs`.\n- Enable the `atan2.slt` sqllogictest.\n\nThe signature is `exact(Float64, Float64) -\u003e Float64`, following the\n`datafusion-spark` convention of only accepting Spark-supported types.\nNULL in either argument propagates to NULL.\n\nNote: DataFusion core already has an `atan2`, but this is implemented\nself-contained rather than delegating — matching the crate convention,\nwhere 14 of the 15 existing `math` functions reimplement rather than\nwrap core (including `abs`, `ceil`, `floor`, `round`, `rint`, all of\nwhich also exist in core; only `pow` delegates, as a special case). This\nkeeps the Spark function library complete and consistent.\n\n## Are these changes tested?\n\nYes — `datafusion/sqllogictest/test_files/spark/math/atan2.slt` covers:\n- standard angles and all four quadrants (atan2 is quadrant-aware via\nthe signs of both arguments),\n- both axes and the negative x-axis (atan2\u0027s range extends to π, unlike\n`atan`),\n- NULL propagation (either argument) and NULL-beats-Infinity,\n- NaN in every position, including `atan2(NaN, Infinity) \u003d NaN` (for\natan2, NaN propagates even over Infinity — unlike `hypot`, where\nInfinity dominates),\n- all four infinity quadrants and the one-infinite-argument cases,\n- signed zeros (`atan2(-0, -1) \u003d -π`),\n- the array path (including a NULL row).\n\n## Are there any user-facing changes?\n\nYes — adds the Spark-compatible `atan2` scalar function to\n`datafusion-spark`. No breaking changes to public APIs.\n\nOriginal-commit: apache/datafusion@9db5c5de5eb4d3f897dc4c575f505ecd69cdd12a\n" }, { "commit": "0e490c27afc7112b24dc456ab8caaf03a04ecef9", "tree": "8db73d274838dad667b5cc28722f855bb3dae32d", "parents": [ "4df467d91c3111585f83822b0369b272a850bc7e" ], "author": { "name": "Burak Şen", "email": "buraksenb@gmail.com", "time": "Sat Aug 01 01:00:43 2026 +0300" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 31 22:00:43 2026 +0000" }, "message": " refactor: move arrow integer hex dispatch to datafusion-common (#23917)\n\n## Which issue does this PR close?\n\n - Closes #23811.\n\n ## Rationale for this change\n\n Share Arrow integer-to-hex dispatch between `to_hex` and Spark `hex`.\n\n ## What changes are included in this PR?\nmove tohex and integer implementations to common\n\n ## Are these changes tested?\nExisting `to_hex` and spark `hex` tests cover both \n\n ## Are there any user-facing changes?\nNo behavior change but ToHex is public now\n\nOriginal-commit: apache/datafusion@5cf7bff9f8524e732b79d9604325735fc2f4af73\n" }, { "commit": "4df467d91c3111585f83822b0369b272a850bc7e", "tree": "e667828ee21682e2f22e21ce68d0158b11d43a77", "parents": [ "f104c83fd869097d25fcd48e7438bad66e08aaf5" ], "author": { "name": "Floze", "email": "88098863+floze-the-genius@users.noreply.github.com", "time": "Thu Jul 30 17:34:13 2026 +0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 30 13:34:13 2026 +0000" }, "message": "Optimize Spark hex null handling (#23688)\n\n## Which issue does this PR close?\n\n- Closes #23674.\n\n## Rationale for this change\n\nSpark hex encoding preserves input validity exactly, but the byte-array\npath rebuilt the null bitmap row by row and always iterated through\n`Option` values. The input arrays already expose both the original\n`NullBuffer` and an all-valid fast path.\n\n## What changes are included in this PR?\n\n- pass the byte array accessor into `hex_encode_bytes` so the output can\nclone and reuse the input `NullBuffer`\n- split nullable and no-null loops, avoiding per-row validity checks\nwhen the input has no null buffer\n- preserve exact offset, overflow, casing, dictionary, and null\nsemantics\n- add a sliced-array regression that verifies semantic equality and\nbitmap pointer reuse\n- add no-null output coverage and dedicated UTF-8/binary Criterion cases\n\n## Are these changes tested?\n\nYes.\n\n- `cargo test -p datafusion-spark function::math::hex --lib`: 10 passed\n- `cargo fmt --all -- --check`: passed\n- `cargo clippy -p datafusion-spark --all-targets --all-features\n--no-deps -- -D warnings`: passed\n- `git diff --check`: passed\n- benchmark target compiled with `--features core`\n\nThe workspace dependency clippy invocation without `--no-deps` reaches\nan unrelated existing `dead_code` warning in\n`datafusion/physical-expr/src/expressions/dynamic_filters/mod.rs:464`;\nthe changed crate is clean.\n\n### Benchmark\n\nCommand (same warmed build and profile for upstream `main` and this\nbranch):\n\n```bash\nCARGO_BUILD_JOBS\u003d2 \\\nCARGO_PROFILE_BENCH_LTO\u003dfalse \\\nCARGO_PROFILE_BENCH_CODEGEN_UNITS\u003d16 \\\ncargo bench -p datafusion-spark --features core --bench hex -- \\\n \u0027hex_(utf8|binary)_no_nulls\u0027\n```\n\nMean times from the immediate upstream-then-branch sequence:\n\n| case | upstream main | this PR | change |\n| --- | ---: | ---: | ---: |\n| UTF-8, 1,024 rows | 79.895 us | 59.181 us | -25.9% |\n| UTF-8, 4,096 rows | 317.64 us | 244.88 us | -22.9% |\n| UTF-8, 8,192 rows | 509.64 us | 411.27 us | -19.3% |\n| Binary, 1,024 rows | 86.135 us | 40.600 us | -52.9% |\n| Binary, 4,096 rows | 346.16 us | 253.13 us | -26.9% |\n| Binary, 8,192 rows | 635.98 us | 424.78 us | -33.2% |\n\n## Are there any user-facing changes?\n\nNo API or behavior changes. This is an internal performance optimization\nfor Spark-compatible hex expressions.\n\nOriginal-commit: apache/datafusion@6a0e7715498e9e92fa0a879491302088e7f6ba3a\n" }, { "commit": "f104c83fd869097d25fcd48e7438bad66e08aaf5", "tree": "f92799f93404e4b8da6e98e737a1660d52723b4b", "parents": [ "f927994f63bdfcefa1b6d8b162f4c6c6b0b586f8" ], "author": { "name": "Oleks V", "email": "comphead@users.noreply.github.com", "time": "Tue Jul 28 17:52:28 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 29 00:52:28 2026 +0000" }, "message": "feat: support `ansi` for `elt` (#23928)\n\nPart of https://github.com/apache/datafusion/issues/23929\n\n## Rationale for this change\n \nSpark\u0027s `elt` errors on out-of-range indices when ANSI is on, and\nreturns `NULL` when off. DataFusion previously always returned `NULL`\n(there was a `TODO` for this in the source).\n\n ## What changes are included in this PR?\n\n- `elt.rs`: read `enable_ansi_mode` from\n`ScalarFunctionArgs::config_options`; raise `\"The index N is out of\nbounds. The array has M elements.\"` in ANSI mode, keep `NULL` behavior\notherwise. `NULL` index\n always returns `NULL`.\n- `elt.slt`: add missing ANSI-off cases (index 0, negative, large,\n`NULL` index, `NULL` value, vectorized) and a full ANSI-on section with\n`statement error` assertions.\n\n ## Are these changes tested?\n\n Yes. New Rust unit tests and sqllogictest cases cover both modes.\n\n ## Are there any user-facing changes?\n\nWith `enable_ansi_mode \u003d true`, `elt` on invalid indices now errors\ninstead of returning `NULL`. Default behavior unchanged. No API changes.\n\nOriginal-commit: apache/datafusion@ef94a85b85bfc19042cad8c732ad1d78f07b34a5\n" }, { "commit": "f927994f63bdfcefa1b6d8b162f4c6c6b0b586f8", "tree": "753d3c13dd6d825a13be0cbaa026f942af011e88", "parents": [ "ca40c5dbff2301ac25d9183455d30e3fe52c319e" ], "author": { "name": "zhigang", "email": "iamzhigangwang@gmail.com", "time": "Tue Jul 28 13:47:07 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 28 05:47:07 2026 +0000" }, "message": "fix: support parentheses for negative decimal formatting (#23718)\n\n## Which issue does this PR close?\n\n- Closes #23717.\n\n## Rationale for this change\n\nSpark and Java format negative numeric values with parentheses when the\n`(` flag is present. The decimal formatting path always emitted a minus\nsign because it ignored `negative_in_parentheses`, while the\nfloating-point path already handled the flag. This made `format_string`\ninconsistent across numeric input types.\n\n## What changes are included in this PR?\n\n- Add a closing suffix when a negative decimal uses parentheses\nformatting.\n- Include the suffix when calculating width, left alignment, and zero\npadding.\n- Add regression coverage for grouped negative decimals with and without\nan explicit width.\n\n## Are these changes tested?\n\nYes. The following checks pass locally:\n\n- `cargo fmt --all -- --check`\n- `cargo clippy --all-targets --all-features -- -D warnings`\n- `cargo test -p datafusion-spark --lib`\n- The full workspace test command from the contributor guide with\n`avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`\nenabled\n\n## Are there any user-facing changes?\n\nYes. Spark-compatible formatting of negative decimal values now honors\nthe parentheses flag. There are no public API or breaking changes.\n\nOriginal-commit: apache/datafusion@bb75d925cb7a6b8e1c5dc22bead85efc49754fe3\n" }, { "commit": "ca40c5dbff2301ac25d9183455d30e3fe52c319e", "tree": "3069a0655041f41481527230dcb4f0dc2e9005b8", "parents": [ "7f027c40dd16386b544e39d7c51f128bb4c1c284" ], "author": { "name": "Karpagam Balasubramaniam", "email": "kk@trulyh.com", "time": "Sat Jul 25 20:26:54 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Jul 26 03:26:54 2026 +0000" }, "message": "feat: add Spark-compatible hypot function (#23774)\n\n## Which issue does this PR close?\n\n- Closes #23770\n- Part of #15914\n\n## Rationale for this change\n\nSpark provides [`hypot(expr1,\nexpr2)`](https://spark.apache.org/docs/latest/api/sql/#hypot), which\nreturns `sqrt(expr1^2 + expr2^2)` computed without intermediate overflow\nor underflow. It was not yet implemented in `datafusion-spark` — only an\nauto-generated test stub existed at `spark/math/hypot.slt` with its\nquery commented out.\n\n## What changes are included in this PR?\n\n- Add `SparkHypot` (implementing `ScalarUDFImpl`) in\n`datafusion/spark/src/function/math/hypot.rs`, backed by Rust\u0027s\n`f64::hypot` — the same overflow-safe algorithm as Java/Spark\u0027s\n`Math.hypot`.\n- Register it in `datafusion/spark/src/function/math/mod.rs`.\n- Enable the `hypot.slt` sqllogictest.\n\nThe signature is `exact(Float64, Float64) -\u003e Float64`, following the\n`datafusion-spark` convention of only accepting types Spark supports.\nComputation uses the Arrow `binary` kernel so NULL in either argument\npropagates to a NULL result, matching Spark.\n\n## Are these changes tested?\n\nYes — `datafusion/sqllogictest/test_files/spark/math/hypot.slt` covers:\n- scalar Pythagorean triples (`hypot(3, 4)` → 5, `hypot(5, 12)` → 13),\n- double inputs,\n- NULL propagation when either argument is NULL,\n- the array path (including a NULL row),\n- overflow-safety: `hypot(3e200, 4e200)` stays finite, whereas a naive\n`sqrt(a^2 + b^2)` would overflow to `Infinity`.\n\n## Are there any user-facing changes?\n\nYes — adds the Spark-compatible `hypot` scalar function to\n`datafusion-spark`. No breaking changes to public APIs.\n\nOriginal-commit: apache/datafusion@e660118996630bf7173b31a67e97dee97dac1fe4\n" }, { "commit": "7f027c40dd16386b544e39d7c51f128bb4c1c284", "tree": "431fbe7af6f98be3401129b0b46e5ddcbf494629", "parents": [ "53454d972eeb9927bbcb28ea2bcda1f740e50051" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Sat Jul 25 13:09:56 2026 -0600" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Jul 25 19:09:56 2026 +0000" }, "message": "perf: avoid per-row String allocation in Spark bin and char (#23881)\n\n## Which issue does this PR close?\n\nN/A\n\n## Rationale for this change\n\nTwo Spark functions allocated a `String` per row purely to render a\nsmall,\nbounded amount of text:\n\n- `bin` called `format!(\"{value:b}\")` for every row.\n- `char` called `ch.to_string()` for every row — a heap allocation for a\nsingle\n character.\n\nIn both cases the output has a known upper bound (64 binary digits for\nan `i64`,\n4 bytes for a UTF-8 character), so the rendering fits in a stack buffer\nand the\nresult can be appended straight to a pre-sized builder.\n\n## What changes are included in this PR?\n\n`math/bin.rs`:\n\n- `spark_bin` now writes digits right-aligned into a caller-supplied\n`[u8; 64]`\nand returns a `\u0026str` borrowed from it, instead of returning an owned\n`String`.\n- The `collect::\u003cStringArray\u003e()` becomes an explicit loop over a\n `StringBuilder::with_capacity`, sized at 8 digits per row.\n- Negative values still render as their two\u0027s-complement bit pattern,\nmatching\n`{:b}`. The digit loop is a `loop`, not a `while`, so zero renders as\n`\"0\"`\n rather than the empty string.\n\n`string/char.rs`:\n\n- `ch.to_string()` becomes `ch.encode_utf8(\u0026mut encoded)` against a\n`[u8; 4]`\n hoisted out of the loop.\n\nOutput is unchanged in both cases.\n\n## Are these changes tested?\n\nExisting coverage pins the behaviour. `spark/math/bin.slt` asserts\nconcrete\noutput for the cases the rewrite had to get right: zero, negative\nvalues,\n`i64::MIN` (`-9223372036854775808`), `i64::MAX`, and `-2147483648` /\n`-32768` widened from narrower integer types. `spark/string/char.slt`\ncovers\nthe negative-input empty string, the null path, and characters on both\nsides of\nthe ASCII boundary (`char(256)` and above wrap via `% 256`).\n\nAll 119 `spark/math` and `spark/string` sqllogictest files pass, along\nwith the\n258 `datafusion-spark` unit tests.\n\nThe `bin` benchmark used below, `datafusion/spark/benches/bin.rs`, is\nadded\nseparately in #23882 so the baseline can be measured on `main` before\nthis\nchange lands. It covers 1024 and 8192 rows with 20% nulls over two value\ndistributions: small values that render to a handful of digits, and\nfull-range\nvalues that render to the maximum 64. `char` already had\n`datafusion/spark/benches/char.rs` on `main`, so no benchmark change is\nneeded\nfor it.\n\n### Benchmarks\n\nCriterion, `apache/main` @ `f1ab86dad` as baseline. Median of the\nreported\nchange interval.\n\n| Benchmark | 1024 | 8192 |\n| --- | --- | --- |\n| `bin/small` | −75.7% | −73.7% |\n| `bin/wide` | −47.8% | −49.8% |\n\n`char` (1024 rows): −76.1%.\n\n## Are there any user-facing changes?\n\nNo. Both functions produce byte-identical output; this is purely an\nallocation\nchange.\n\nOriginal-commit: apache/datafusion@36c417b6c14244526c81ee5dc4210740ab7b44f8\n" }, { "commit": "53454d972eeb9927bbcb28ea2bcda1f740e50051", "tree": "c0a9ba09eac9f88376c1d3ffa073206cd2aad2b1", "parents": [ "62880f476b5c9d3defa51fdd9b87f42cf14b9417" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Sat Jul 25 08:17:41 2026 -0600" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Jul 25 14:17:41 2026 +0000" }, "message": "refactor: share hex encoding across datafusion-common, functions, and spark (#23766)\n\n## Which issue does this PR close?\n\nN/A\n\n## Rationale for this change\n\nHex encoding was implemented six times across the workspace, at two\ndifferent levels of optimization.\n\nThree copies used a fast byte-pair lookup table:\n\n- `datafusion/spark/src/function/math/hex.rs`\n- `datafusion/functions/src/string/to_hex.rs`\n- `datafusion/functions/src/encoding/inner.rs` (via the `hex` crate)\n\nThe other three used a slower nibble-at-a-time loop pushing one\ncharacter at a time:\n\n- `datafusion/functions/src/crypto/md5.rs`\n- `datafusion/spark/src/function/hash/sha1.rs`\n- `datafusion/spark/src/function/hash/sha2.rs`\n\nBeyond the duplication, this split meant the digest functions were\npaying for a slower encoder\nthan the one already sitting elsewhere in the tree. Consolidating on a\nsingle implementation\nremoves the duplication and moves `md5`, `sha1`, and `sha2` onto the\nfast path.\n\n## What changes are included in this PR?\n\nA new `datafusion_common::utils::hex` module holds the only hex encoder\nin the workspace:\n\n```rust\npub enum HexCase { Lower, Upper }\n\npub fn encode_bytes_into(bytes: \u0026[u8], case: HexCase, out: \u0026mut Vec\u003cu8\u003e);\npub fn encode_bytes(bytes: \u0026[u8], case: HexCase) -\u003e String;\npub fn encode_bytes_to_slice(bytes: \u0026[u8], case: HexCase, out: \u0026mut [u8]);\npub fn encode_u64(v: u64, case: HexCase, buf: \u0026mut [u8; 16]) -\u003e \u0026[u8];\n```\n\nFour entry points rather than one because the call sites have genuinely\ndifferent output needs,\nand forcing them through a single shape would cost an allocation\nsomewhere: `to_hex` writes\nstraight into a `StringArray` values buffer, Spark\u0027s `hex` appends to a\nreused scratch `Vec`,\n`encode` writes into a pre-sized slice, and the digest functions want an\nowned `String`.\n\nMigrated call sites, all producing bit-identical output:\n\n| File | Change |\n| --- | --- |\n| `functions/src/string/to_hex.rs` | local table and two write helpers\ndeleted; eight trait impls collapsed to two macros |\n| `spark/src/function/math/hex.rs` | two nibble tables, two lookup\ntables, `build_hex_lookup` and `hex_int64` deleted |\n| `functions/src/crypto/md5.rs` | local table and `hex_encode` deleted |\n| `spark/src/function/hash/sha1.rs` | local table and inline nibble loop\ndeleted |\n| `spark/src/function/hash/sha2.rs` | local table and `hex_encode`\ndeleted; eight call sites migrated |\n| `functions/src/encoding/inner.rs` | both encode sites moved off the\n`hex` crate |\n\nDeliberately left alone:\n\n- `hex::decode` / `hex::decode_to_slice` in `encoding/inner.rs`, and\nSpark\u0027s `unhex`. The decode\ndirection has different semantics — Spark\u0027s `unhex` left-pads odd-length\ninput, the `hex` crate\ndoes not — so unifying it is a separate question. The `hex` dependency\nstays for those.\n- `ScalarValue`\u0027s binary `Display` impl in `common/src/scalar/mod.rs`,\nwhich writes to a\n `fmt::Formatter` rather than a byte buffer, and is a cold path.\n\nThree details worth a reviewer\u0027s attention:\n\n- The two ancestors of `encode_u64` disagreed on zero: `to_hex` wrote\n`\u00270\u0027` into the caller\u0027s\nbuffer, Spark\u0027s `hex_int64` returned a `\u0027static` `b\"0\"` that never\ntouched it. The shared version\nalways writes into the buffer and returns a subslice of it, so the\nlifetime is uniform. Both\n callers still produce `\"0\"`.\n- Spark\u0027s `hex_encode_bytes` guards large binary input with\n`checked_mul(2)` + `try_reserve`,\nreturning a `DataFusionError` rather than aborting on allocation\nfailure. That guard stays at the\ncall site; `encode_bytes_into` performs no reservation of its own, so\nbehaviour is unchanged.\n- The encoders are `#[inline(always)]`, not `#[inline]`. They are called\nonce per row from two\nother crates, and plain `#[inline]` left them out-of-line across the\ncrate boundary. Measured\ncost of that: +3% on Spark\u0027s byte paths and up to +18% on `to_hex`\u0027s i32\npath, i.e. the\nrefactor was a net regression on those benchmarks until the attribute\nchanged.\n\n## Are these changes tested?\n\nEvery migrated function keeps its existing unit and sqllogictest\ncoverage, which is what pins\nbit-identical output — in particular `spark/hash/sha1.slt` and\n`sha2.slt` assert concrete digest\nstrings for all four SHA-2 bit lengths across both the scalar and array\npaths, and `expr.slt`\ncovers `to_hex` and `md5`.\n\nNew unit tests in `common/src/utils/hex.rs` cover zero, `u64::MAX`,\nsingle-nibble values, the\nodd/even digit-count boundary, two\u0027s complement of negative input, empty\ninput, all 256 byte\nvalues in both cases, appending into a non-empty buffer, and that a\nreused scratch buffer never\nleaks stale digits between calls. Tests cross-check against\n`format!(\"{:x}\")` rather than\nrestating the implementation.\n\nTwo Spark tests changed. `test_hex_int64` now drives `hex_encode_int64`\ninstead of the deleted\nprivate `hex_int64`, keeping all ten cases including `i64::MIN`, `-1`,\nand the uppercase\nexpectations. `test_hex_lookup_table_covers_all_bytes` was deleted — it\nonly cross-checked the raw\nlookup tables, and `encode_bytes_covers_every_byte_value` now does that\nexhaustively through the\npublic API. A test was added for Spark\u0027s lowercase byte path, which\npreviously had no coverage.\n\n### Benchmarks\n\nCriterion, `apache/main` @ `eef101769` as baseline. Median of the\nreported change interval.\n\n`datafusion/functions/benches/to_hex.rs`:\n\n| Benchmark | 1024 | 4096 | 8192 |\n| --- | --- | --- | --- |\n| `i32_random` | −13.8% | −12.8% | −10.2% |\n| `i64_random` | −8.8% | −9.5% | −8.1% |\n| `i64_large_values` | −9.9% | −9.8% | −7.9% |\n\n`scalar_i32` −2.3%, `scalar_i64` −1.8%.\n\n`datafusion/spark/benches/sha2.rs`:\n\n| Benchmark | 1024 | 4096 | 8192 |\n| --- | --- | --- | --- |\n| `array_binary_256` | −20.7% | −18.2% | −19.1% |\n| `array_scalar_binary_256` | −14.6% | −13.2% | −13.1% |\n\n`scalar/size\u003d1` −3.3%.\n\n`datafusion/spark/benches/hex.rs`:\n\n| Benchmark | 1024 | 4096 | 8192 |\n| --- | --- | --- | --- |\n| `hex_int64` | −2.9% | −3.6% | −3.5% |\n| `hex_int64_dict` | −3.2% | −1.5% | −0.2% |\n| `hex_utf8` | −0.2% | −0.3% | +0.7% |\n| `hex_binary` | −0.4% | −0.6% | −0.3% |\n\nThe `hex_utf8` and `hex_binary` paths already used the byte-pair table\nbefore this PR, so they are\nexpected to be flat; they are.\n\n`datafusion/functions/benches/crypto.rs`: `md5_array` −4.0%,\n`md5_scalar` −3.7%. The `sha224` /\n`sha256` / `sha384` / `sha512` cases in that file range from −0.1% to\n+2.2%, but they exercise\n`crypto/basic.rs`, which this PR does not modify and which contains no\nhex encoding — those numbers\nare run-to-run variance, not an effect of this change.\n\nNo number is quoted for Spark `sha1`: it has no benchmark, and its\nchange is the same substitution\napplied to `md5` and `sha2`.\n\n## Are there any user-facing changes?\n\nNo behaviour change — all migrated functions produce byte-identical\noutput.\n\n`datafusion_common::utils::hex` is new public API on\n`datafusion-common`.\n\n---------\n\nCo-authored-by: Andrew Lamb \u003candrew@nerdnetworks.org\u003e\nCo-authored-by: Jeffrey Vo \u003cjeffrey.vo.australia@gmail.com\u003e\n\nOriginal-commit: apache/datafusion@8393fd32b7152779886c2b0176fab12af0cbbe0c\n" }, { "commit": "62880f476b5c9d3defa51fdd9b87f42cf14b9417", "tree": "ee8ba53d26448785a6237d759fea5cde8eac3511", "parents": [ "b17ef831445641ac22492cf7d1b78c76f0612531" ], "author": { "name": "linfeng", "email": "33561138+lyne7-sc@users.noreply.github.com", "time": "Fri Jul 24 20:22:36 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 24 12:22:36 2026 +0000" }, "message": "Remove `GroupsAccumulator::supports_convert_to_state` and require `convert_to_state` (#23489)\n\n## Which issue does this PR close?\n\n\u003c!--\nWe generally require a GitHub issue to be filed for all bug fixes and\nenhancements and this helps us generate change logs for our releases.\nYou can link an issue to this PR using the GitHub syntax. For example\n`Closes #123` indicates that this PR will close issue #123.\n--\u003e\n\n- Closes #23081.\n\n## Rationale for this change\n\n\u003c!--\nWhy are you proposing this change? If this is already explained clearly\nin the issue then this section is not needed.\nExplaining clearly why changes are proposed helps reviewers understand\nyour changes and offer better suggestions for fixes.\n--\u003e\n\nFollowing #23275, all `GroupsAccumulator` implementations now provide\n`convert_to_state`.\n\nThe `supports_convert_to_state` capability flag is therefore no longer\nneeded.\n\n## What changes are included in this PR?\n\n\u003c!--\nThere is no need to duplicate the description in the issue here but it\nis sometimes worth providing a summary of the individual changes in this\nPR.\n--\u003e\n\n- Make `GroupsAccumulator::convert_to_state` a required trait method.\n- Remove `GroupsAccumulator::supports_convert_to_state` and its\nimplementations.\n- Remove the corresponding capability checks from hash aggregation.\n- Simplify skip-partial aggregation to use the required\n`convert_to_state` implementation directly.\n- Add a regression test covering the partial hash aggregation skip path.\n- Document the breaking trait change in the 55.0.0 upgrading guide.\n\n- Remove `FFI_GroupsAccumulator::supports_convert_to_state`. This\nchanges the FFI ABI layout, so providers and consumers must be rebuilt\nagainst DataFusion 55.\n\n## Are these changes tested?\n\n\u003c!--\nWe typically require tests for all PRs in order to:\n1. Prevent the code from being accidentally broken by subsequent changes\n2. Serve as another way to document the expected behavior of the code\n\nIf tests are not included in your PR, please explain why (for example,\nare they covered by existing tests)?\n--\u003e\n\nYes. Added a regression test verifying that skip-partial aggregation\nuses the required `convert_to_state` implementation without a capability\nflag.\n\nExisting physical-plan and FFI tests continue to pass.\n\n## Are there any user-facing changes?\n\n\u003c!--\nIf there are user-facing changes then we may require documentation to be\nupdated before approving the PR.\n--\u003e\n\n\u003c!--\nIf there are any breaking changes to public APIs, please add the `api\nchange` label.\n--\u003e\n\nYes. This is a breaking Rust API change for external `GroupsAccumulator`\nimplementations:\n\n- `convert_to_state` must now be implemented.\n- `supports_convert_to_state` should be removed.\n\nThe migration is documented in the 55.0.0 upgrading guide.\n\nThe `FFI_GroupsAccumulator` layout has changed. FFI providers and\nconsumers must be rebuilt against DataFusion 55 and must not exchange\nthis struct with older major versions.\n\nOriginal-commit: apache/datafusion@3f816137306112f75b412e2806abd24fed0b916f\n" }, { "commit": "b17ef831445641ac22492cf7d1b78c76f0612531", "tree": "6684e17b778f726749c983bfa8562d23f624106f", "parents": [ "1d845ac25d29f55f9d17c23428d90ce2727c1225" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Fri Jul 17 09:18:58 2026 -0600" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 17 15:18:58 2026 +0000" }, "message": "perf: avoid per-row copy in Spark hex byte encoding (#23473)\n\n## Which issue does this PR close?\n\nN/A\n\n## Rationale for this change\n\nThe Spark `hex` function\u0027s byte-encoding path (`hex_encode_bytes`)\nfilled a reusable per-row `Vec` and then copied it into a\n`StringBuilder` for every row. Since each input byte maps to exactly two\nhex characters, the output can be written once directly into a single\nvalue buffer, eliminating the per-row copy.\n\n## What changes are included in this PR?\n\nRewrote `hex_encode_bytes` to write hex digits directly into one growing\nvalue buffer with manually-tracked offsets and a `NullBufferBuilder`,\nconstructing the `StringArray` via `new_unchecked` (the buffer contains\nonly ASCII hex digits, so the existing no-validation behavior is\npreserved). The 256-entry lookup table and all other hex paths are\nunchanged.\n\n## Are these changes tested?\n\nExisting tests (`function::math::hex`) pass unchanged.\n\nBenchmark (`datafusion/spark/benches/hex.rs`):\n\n- hex_binary/1024·4096·8192: ~13.8% / 14.3% / 13.9% faster\n- hex_utf8/1024·4096·8192: ~14.1% / 14.9% / 15.0% faster\n\n## Are there any user-facing changes?\n\nNo\n\nCo-authored-by: Andrew Lamb \u003candrew@nerdnetworks.org\u003e\n\nOriginal-commit: apache/datafusion@d4160119db8c6b348928af100393c23783caf05e\n" }, { "commit": "1d845ac25d29f55f9d17c23428d90ce2727c1225", "tree": "69925da2f02466cdf3cac1f0ebb1fafce5e32362", "parents": [ "6cf2246eafa41f04870ad270b82d7d254d487886" ], "author": { "name": "gstvg", "email": "28798827+gstvg@users.noreply.github.com", "time": "Fri Jul 17 06:50:50 2026 -0300" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 17 09:50:50 2026 +0000" }, "message": "Handle nulls in type coercion of higher-order UDFs, map_extract, spark array_repeat (#23071)\n\n## Which issue does this PR close?\n\n- Closes #23068 .\n\n## Rationale for this change\n\nUnresolved lambda variables and unspecified placeholders report a\n`DataType::Null` data type, which causes an error when used as argument\nof a function that doesn\u0027t handle nulls during type coercion\n\n## What changes are included in this PR?\n\nHandle null in `coerce_single_list_arg` for higher-order functions,\n`map_extract` and in spark `array_repeat`. All other functions (scalar,\nwindow, agg and higher-order) that uses user defined type coercion have\nbeen checked to currently handle null args.\n\n## Are these changes tested?\n\nTest passing `array_filter` with an unresolved lambda variable to\n`Dataframe::with_column` (which indirectly calls\n`value_fields_with_higher_order_udf_and_lambdas` coercion via\n`Projection::try_new`, `projection_schema`, `exprlist_to_fields`,\n`Expr::to_field` ), and sqllogictests\n\n## Are there any user-facing changes?\n\nNo\n\nOriginal-commit: apache/datafusion@fb8fe7ae61efc12c6c09a00d025e35c750e2ee86\n" }, { "commit": "6cf2246eafa41f04870ad270b82d7d254d487886", "tree": "bc6c4fc9f3bfede7bb3e26dcd8df225ccc3738d3", "parents": [ "62bc120246492381bc01d2811c837663ac2f5a0c" ], "author": { "name": "linfeng", "email": "33561138+lyne7-sc@users.noreply.github.com", "time": "Fri Jul 17 02:38:07 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 16 18:38:07 2026 +0000" }, "message": "fix: align dictionary coercion across typed signatures (#23549)\n\n## Which issue does this PR close?\n\n\u003c!--\nWe generally require a GitHub issue to be filed for all bug fixes and\nenhancements and this helps us generate change logs for our releases.\nYou can link an issue to this PR using the GitHub syntax. For example\n`Closes #123` indicates that this PR will close issue #123.\n--\u003e\n\n- Follow-up to #22905.\n- Part of https://github.com/apache/datafusion/issues/19458\n\n## Rationale for this change\n\n\u003c!--\nWhy are you proposing this change? If this is already explained clearly\nin the issue then this section is not needed.\nExplaining clearly why changes are proposed helps reviewers understand\nyour changes and offer better suggestions for fixes.\n--\u003e\n\n#22905 introduced explicit dictionary encoding preservation for\ncoercible function signatures, but dictionary inputs were still handled\ndifferently across `TypeSignatureClass` variants:\n\n| Signature category | Before | After |\n| --- | --- | --- |\n| `Native(...)` | Materialized by default; preserved when explicitly\nrequested | Same default/opt-in contract |\n| Typed non-Native (e.g. `Integer`, `Numeric`, `Binary`) | Retained the\nphysical dictionary type by default | Materialized by default; preserved\nwhen explicitly requested |\n| `Any` | Passed through the original physical input type | Unchanged |\n\nThis PR makes the encoding preservation contract consistent across all\ntyped signature classes: coercion operates on the dictionary value type,\nand the dictionary encoding is restored only when\n`EncodingPreservation::dictionary()` is enabled.\n\nAn audit of the affected built-ins found two functions, Spark hex and\nbitmap_count, that intentionally handle dictionary inputs; both now opt\nin explicitly. Other affected functions generally expect materialized\nvalue arrays, so the new default also avoids cases where signature\nmatching accepted a dictionary but the function implementation rejected\nit at execution time. Functions that continue to materialize dictionary\ninputs do not gain dictionary-aware execution efficiency yet, but they\ncan opt in later if they add support for encoded inputs.\n\n## What changes are included in this PR?\n\n\u003c!--\nThere is no need to duplicate the description in the issue here but it\nis sometimes worth providing a summary of the individual changes in this\nPR.\n--\u003e\n\n- Align dictionary coercion across typed signature classes and preserve\ndictionary encoding when explicitly requested.\n- Explicitly enable dictionary preservation for Spark `bitmap_count` and\nthe binary variant of Spark `hex`.\n- Document the behavior change and migration guidance in the DataFusion\n55.0.0 upgrade guide.\n\n## Are these changes tested?\n\n\u003c!--\nWe typically require tests for all PRs in order to:\n1. Prevent the code from being accidentally broken by subsequent changes\n2. Serve as another way to document the expected behavior of the code\n\nIf tests are not included in your PR, please explain why (for example,\nare they covered by existing tests)?\n--\u003e\n\n- Unit tests cover materialization and preservation for Native,\nnon-Native, and `Any` inputs.\n- SLTs cover `to_hex` materialization and verify that `bitmap_count`\npreserves its dictionary input without an additional cast to `Binary`.\n- Existing Spark `hex` dictionary tests cover its opt-in preservation\nbehavior.\n\n## Are there any user-facing changes?\n\n\u003c!--\nIf there are user-facing changes then we may require documentation to be\nupdated before approving the PR.\n--\u003e\n\n\u003c!--\nIf there are any breaking changes to public APIs, please add the `api\nchange` label.\n--\u003e\n\nThis is a behavioral API change for UDFs using typed non-Native classes\nsuch as `Integer` or `Binary`. UDFs relying on implicit dictionary\npreservation must now enable `EncodingPreservation::dictionary()`\nexplicitly.\n\n`TypeSignatureClass::Any` is unaffected. The upgrade guide has been\nupdated, and this PR should carry the `api change` label.\n\nOriginal-commit: apache/datafusion@5063883381a0c112ae483892bcaf1a2e4cda783f\n" }, { "commit": "62bc120246492381bc01d2811c837663ac2f5a0c", "tree": "b8eec74a581256c73d7fffcde30c91d2d4df449d", "parents": [ "3c19a28dc50c64b639dad5a8a215adf70f1da7fe" ], "author": { "name": "Yongting You", "email": "2010youy01@gmail.com", "time": "Thu Jul 16 14:57:57 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 16 06:57:57 2026 +0000" }, "message": "minor: validate config `meta_fetch_concurrency` when setting it (#23595)\n\n## Which issue does this PR close?\n\n\u003c!--\nWe generally require a GitHub issue to be filed for all bug fixes and\nenhancements and this helps us generate change logs for our releases.\nYou can link an issue to this PR using the GitHub syntax. For example\n`Closes #123` indicates that this PR will close issue #123.\n--\u003e\n\n- Part of https://github.com/apache/datafusion/issues/17498\n\n## Rationale for this change\n\n\u003c!--\nWhy are you proposing this change? If this is already explained clearly\nin the issue then this section is not needed.\nExplaining clearly why changes are proposed helps reviewers understand\nyour changes and offer better suggestions for fixes.\n--\u003e\nThis is logically the same change as #23592, but for another\nconfiguration.\n\n0 is an invalid value for concurrency level, so restricting it to be\nnon-zero\n\n## What changes are included in this PR?\n\n\u003c!--\nThere is no need to duplicate the description in the issue here but it\nis sometimes worth providing a summary of the individual changes in this\nPR.\n--\u003e\n\n## Are these changes tested?\n\n\u003c!--\nWe typically require tests for all PRs in order to:\n1. Prevent the code from being accidentally broken by subsequent changes\n2. Serve as another way to document the expected behavior of the code\n\nIf tests are not included in your PR, please explain why (for example,\nare they covered by existing tests)?\n--\u003e\n\n## Are there any user-facing changes?\n\n\u003c!--\nIf there are user-facing changes then we may require documentation to be\nupdated before approving the PR.\n--\u003e\n\n\u003c!--\nIf there are any breaking changes to public APIs, please add the `api\nchange` label.\n--\u003e\n\nOriginal-commit: apache/datafusion@1adbd84e4f97855acf5e696945490bfe0646246a\n" }, { "commit": "3c19a28dc50c64b639dad5a8a215adf70f1da7fe", "tree": "ec13faba5c8228f8191b8ff13248d007be588d16", "parents": [ "79a62b86d3897e556aa682ee144dfe45bfcf1af0" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Sat Jul 11 11:14:28 2026 -0600" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Jul 11 17:14:28 2026 +0000" }, "message": "perf: avoid intermediate slice allocation in Spark slice function (#23481)\n\n## Which issue does this PR close?\n\n- N/A (small performance improvement)\n\n## Rationale for this change\n\nThe Spark `slice` function\u0027s `calculate_start_end` helper reads the\nlength of each list element in a per-row hot loop via\n`values.value(row).len()`. `GenericListArray::value(row)` materializes a\nnew `ArrayRef` for the sublist on every iteration purely to call\n`.len()` on it. The length is already available from the offset buffer,\nso this allocation is pure overhead.\n\nReplacing it with `values.value_length(row)` reads the length directly\nfrom the offsets with no allocation.\n\nBenchmarked with the existing `datafusion/spark/benches/slice.rs` over\n1M rows:\n\n| case | before | after | improvement |\n|------|--------|-------|-------------|\n| List(Int64), array args | 54.0 ms | 40.0 ms | ~26% faster |\n| List(Int64), scalar args | 88.4 ms | 69.8 ms | ~21% faster |\n\n## What changes are included in this PR?\n\nA single-line change in `datafusion/spark/src/function/array/slice.rs`\nreplacing `values.value(row).len() as i64` with\n`values.value_length(row) as i64`.\n\n## Are these changes tested?\n\nCovered by existing unit tests in\n`datafusion/spark/src/function/array/slice.rs` and slt coverage for the\nSpark `slice` function; behavior is unchanged. The performance impact\nwas measured with the existing `slice` criterion benchmark.\n\n## Are there any user-facing changes?\n\nNo.\n\nOriginal-commit: apache/datafusion@2fbc7582639ee33eeffe83b8b43c354bf05fdd18\n" }, { "commit": "79a62b86d3897e556aa682ee144dfe45bfcf1af0", "tree": "4dcd09f5f65c46e31a960441c0ade2533fc9bd45", "parents": [ "713865e868d8eeae77f2b058c317d80f502fe79a" ], "author": { "name": "Andrew Lamb", "email": "andrew@nerdnetworks.org", "time": "Tue Jul 07 22:35:49 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jul 08 02:35:49 2026 +0000" }, "message": "chore: Update to arrow/parquet 59.1.0 (#23312)\n\n(DRAFT until arrow is updated, I am using this PR to pre-test the\nrelease)\n\n## Which issue does this PR close?\n\n- related to https://github.com/apache/arrow-rs/issues/9878\n## Rationale for this change\n\nUpdate to latest arrow\n## What changes are included in this PR?\n\n\n## Are these changes tested?\n\nYes by CI\n\n## Are there any user-facing changes?\n\nNo API change (this is a minor update of Arrow)\n\nOriginal-commit: apache/datafusion@f34a676302e2320526172705503a5ac8222804ea\n" }, { "commit": "713865e868d8eeae77f2b058c317d80f502fe79a", "tree": "545051aab6ce07a49ebfa4eb8993706ae55f7e45", "parents": [ "e3f8d07e78fe434806bf97c1a394387843b0584a" ], "author": { "name": "Oleks V", "email": "comphead@users.noreply.github.com", "time": "Fri Jul 03 11:47:33 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 03 18:47:33 2026 +0000" }, "message": "spark: support `collect_list` `collect_set` for `windows` execution (#23281)\n\n## Which issue does this PR close?\n\n\u003c!--\nWe generally require a GitHub issue to be filed for all bug fixes and\nenhancements and this helps us generate change logs for our releases.\nYou can link an issue to this PR using the GitHub syntax. For example\n`Closes #123` indicates that this PR will close issue #123.\n--\u003e\n\n- Closes #23261 .\n\n## Rationale for this change\n\nExpand windows coverage to support `collect_set` `collect_list`\nfunctions.\n \n\u003c!--\nWhy are you proposing this change? If this is already explained clearly\nin the issue then this section is not needed.\nExplaining clearly why changes are proposed helps reviewers understand\nyour changes and offer better suggestions for fixes.\n--\u003e\n\n## What changes are included in this PR?\n\n\u003c!--\nThere is no need to duplicate the description in the issue here but it\nis sometimes worth providing a summary of the individual changes in this\nPR.\n--\u003e\n\n## Are these changes tested?\n\n\u003c!--\nWe typically require tests for all PRs in order to:\n1. Prevent the code from being accidentally broken by subsequent changes\n2. Serve as another way to document the expected behavior of the code\n\nIf tests are not included in your PR, please explain why (for example,\nare they covered by existing tests)?\n--\u003e\n\n## Are there any user-facing changes?\n\n\u003c!--\nIf there are user-facing changes then we may require documentation to be\nupdated before approving the PR.\n--\u003e\n\n\u003c!--\nIf there are any breaking changes to public APIs, please add the `api\nchange` label.\n--\u003e\n\nOriginal-commit: apache/datafusion@01c7527be0b318e3517f4cebf150638724e7e5db\n" }, { "commit": "e3f8d07e78fe434806bf97c1a394387843b0584a", "tree": "f63040183f84db1e9d027ff64ecc03f789de7691", "parents": [ "7bc93e4a453320bac264e80ac79c99c3c50ef0d0" ], "author": { "name": "Adam Gutglick", "email": "adamgsal@gmail.com", "time": "Thu Jul 02 22:24:23 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 02 21:24:23 2026 +0000" }, "message": "fix: Handle decimal columns consistently in SLT tests (#23161)\n\n## Which issue does this PR close?\n\n- Closes #23160.\n\n## Rationale for this change\n\nAs described in the issue, different width decimals are currently\nhandled differently, going through inconsistent display path and not\npresenting the full precision.\n\n\n## What changes are included in this PR?\n\n1. Unifies all handling and formatting for the four decimal variants,\nwhich are now all supported. Also includes full display of decimal\nvalues up to the specified precision, which should be deterministic.\n2. I\u0027ve also reran the all tests with `--complete`, which makes a few\nrandom changes around empty whitespaces which I think is good to run\nperiodically (I keep doing that locally in unrelated changes and having\nto `git restore` a bunch of files). Would it make sense to maybe add\nthis check to CI and verify it leaves the worktree clean?\n\n## Are these changes tested?\n\nRan all tests locally\n\n## Are there any user-facing changes?\n\nUsers of `datafusion-sqllogictest` will get different results if they\ndepend on the existing behavior.\n\n---------\n\nSigned-off-by: Adam Gutglick \u003cadamgsal@gmail.com\u003e\n\nOriginal-commit: apache/datafusion@ad7d6ea8ef173e3a9e38dd6340034d249fb8e092\n" }, { "commit": "7bc93e4a453320bac264e80ac79c99c3c50ef0d0", "tree": "50d3c92a30d452a503b55b793e5cbc5d3f902dda", "parents": [ "b2bdb639d8161f9fd96630f3a33ec8efa8c92044" ], "author": { "name": "pchintar", "email": "89355405+pchintar@users.noreply.github.com", "time": "Mon Jun 29 21:30:49 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jun 29 16:00:49 2026 +0000" }, "message": "fix: Preserve integer values in round() for large Int64 and UInt64 inputs (#22697)\n\n## Which issue does this PR close?\n\n- Closes #22696 .\n\n## Rationale for this change\n\n`round()` should not change integer values when the scale is\nnon-negative, since no fractional digits need to be rounded.\n\nCurrently, core `round()` coerces large `Int64` values through\n`Float64`, causing precision loss:\n\n```sql\nSELECT round(arrow_cast(9007199254740993, \u0027Int64\u0027));\n````\n\nBefore/Current Buggy Output:\n\n```text\n9007199254740992.0\n```\n\nExpected:\n\n```text\n9007199254740993\n```\n\nThe Spark-compatible `round()` also fails for `UInt64` values above\n`i64::MAX` even when the scale is non-negative:\n\n```sql\nSELECT round(arrow_cast(18446744073709551615, \u0027UInt64\u0027));\n```\n\nBefore/Current Buggy Output:\n\n```text\nround: UInt64 value 18446744073709551615 exceeds i64::MAX and cannot be rounded\n```\n\n## What changes are included in this PR?\n\n* Preserved integer inputs in core `round()` for non-negative scales\ninstead of routing them through `Float64`.\n* Preserved `UInt64` values in Spark-compatible `round()` when the scale\nis non-negative, avoiding unnecessary `UInt64 -\u003e i64` conversion.\n* Added SQLLogicTest coverage for:\n\n * `Int64` values above `2^53` in core `round()`.\n * `UInt64::MAX` in Spark-compatible `round()`.\n * Both one-argument and two-argument forms.\n\n## Are these changes tested?\n\nYes.\n\n```bash\ncargo fmt --all\ngit diff --check\ncargo test -p datafusion-sqllogictest --test sqllogictests -- spark/math/round.slt\ncargo test -p datafusion-functions round\ncargo test -p datafusion-spark round\n```\n\nI also verified the core regression queries manually:\n\n```sql\nSELECT arrow_typeof(round(arrow_cast(9007199254740993, \u0027Int64\u0027))),\n round(arrow_cast(9007199254740993, \u0027Int64\u0027));\n```\n\nResult:\n\n```text\nInt64 9007199254740993\n```\n\n```sql\nSELECT arrow_typeof(round(arrow_cast(9007199254740993, \u0027Int64\u0027), 2)),\n round(arrow_cast(9007199254740993, \u0027Int64\u0027), 2);\n```\n\nResult:\n\n```text\nInt64 9007199254740993\n```\n\n## Are there any user-facing changes?\n\nNo.\n\nOriginal-commit: apache/datafusion@951b821294ff6d0daeb822d31a0b1952d530ae07\n" }, { "commit": "b2bdb639d8161f9fd96630f3a33ec8efa8c92044", "tree": "602f6fb8357000d00ab17c86482b557783de4e8d", "parents": [ "0bded7cbd75228156441163910f513097aafc27c" ], "author": { "name": "David López", "email": "hola@devel0pez.com", "time": "Sat Jun 27 23:21:49 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Jun 27 21:21:49 2026 +0000" }, "message": "fix(spark): return error from ELT coerce_types when fewer than 2 args (#23164)\n\n## Which issue does this PR close?\n\n- Closes N/A\n\n## Rationale for this change\n\n`SparkElt::coerce_types` validates that ELT receives at least 2\narguments (index + value1), but the error was never returned: it was\nbuilt with `plan_datafusion_err!(...)` as an expression statement whose\nvalue was discarded. Since `DataFusionError` is not `#[must_use]`, the\ncompiler didn\u0027t warn, so the function fell through and continued with\nfewer arguments than required instead of failing with a clear plan-time\nerror.\n\n## What changes are included in this PR?\n\nIn datafusion/spark/src/function/string/elt.rs, the argument-count check\nis wrapped in return Err(plan_datafusion_err!(...)) so the validation\nactually short-circuits when fewer than 2 arguments are provided.\n\n## Are these changes tested?\n\nThis change restores an error path that was previously dropped silently.\nELT\u0027s normal behavior is already covered by the existing tests in\nelt.rs. The invalid-arity branch had no coverage because it was never\nactually exercised.\n\n## Are there any user-facing changes?\n\nELT now returns a plan-time error when invoked with fewer than 2\narguments, instead of silently continuing. No public API changes.\n\nOriginal-commit: apache/datafusion@a59c4f56abae3b80e92dbe3f0cca78ab88378303\n" }, { "commit": "0bded7cbd75228156441163910f513097aafc27c", "tree": "83424f6a5c3d71961317ea75533c700c832fe297", "parents": [ "eaf2a7f2785de51d647b194a461954d0ea414491" ], "author": { "name": "theirix", "email": "theirix@gmail.com", "time": "Sun Jun 21 13:25:23 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Jun 21 12:25:23 2026 +0000" }, "message": "feat: support mixed binary and string types for concat UDFs (#22244)\n\n## Which issue does this PR close?\n\n- Closes #12709.\n\n## Rationale for this change\n\nWhile #21883 introduced binary argument support for the pipe operator,\nthis PR targets three UDFs: `concat`, `concat_ws`, and Spark\u0027s `concat`\nto harmonise all their behaviour.\n\nAfter the first attempt at banning mixed string+concat operations, I\nswitched to allowing them for concat UDFs and the pipe operator - as\nseen in DuckDB, Spark, Databricks. Previously, mixed behaviour was\nallowed in #20787 (not released yet). Thus, no breaking API changes\nsince the klast release\n\n## What changes are included in this PR?\n\n- Added support for binary types (four variants)\n- Allow mixed string/binary operations for UDFs and pipe operator\n- Fixed edge cases when binary-\u003estring type coercion overrode UDF rules,\nso mixed calls were allowed\n- Refactored the three UDFs by extracting duplicate code into shared\nhelpers to keep the logic centralised\n- Refined `concat_ws` behavior for different separator types\n\nThe diff is quite large. Detailed code changes:\n- Added a trait `ConcatBuilder` to abstract string/binary/view/array\noperations\n- Abstracted `StringArray`/`LargeStringArray` into a generic\n`ConcatGenericStringBuilder` - less duplication\n- Introduced mirrored builders for binary types in a new file\n`binaries.rs`\n- Introduced more `ColumnarValueRef` variants to handle nullable and\nnon-nullable binary types\n- Extracted the ColumnarValue -\u003e ColumnarValueRef builder to\n`from_columnar_value` - simplified call sites. Scalar code path stays\nmostly the same\n- Switched from `Signature::variadic` to `Signature::UserDefined` to\nallow different argument types.`Variadic` required every argument\n(including binaries) to be coerced to the same string type, so the UDF\ncannot distinguish between binary and string inputs. It\u0027s a relatively\nuncommon - happy to discuss\n- Simplified Spark\u0027s concat significantly by reusing the `concat`\nimplementation, so it handles only Spark-specific null-handling\n- Moved SLTs from Spark SLT (`spark/concat.slt`) to a generic SLT, so we\ntest both generic and Spark-specific behaviour\n\n\n## Are these changes tested?\n\n- Added more unit tests for previously uncovered major code paths\n- Added more SLTs, especially for type coercion\n\n## Are there any user-facing changes?\n\nNo. Also, reverted a breaking api change for `||` operator\n\n---------\n\nCo-authored-by: Jeffrey Vo \u003cjeffrey.vo.australia@gmail.com\u003e\n\nOriginal-commit: apache/datafusion@a27f030d5829a4460e87dbb8d2a6382c8f9ecd4f\n" }, { "commit": "eaf2a7f2785de51d647b194a461954d0ea414491", "tree": "7fb16400d94a38b153b6619d8adcdb28e2e151a8", "parents": [ "f0ed50222a7c8c1676144566d0ace01630ebc593" ], "author": { "name": "Eduardo Aguilar", "email": "contact@eduaguilar.com", "time": "Fri Jun 19 05:51:16 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jun 19 03:51:16 2026 +0000" }, "message": "bugfix: changed return type of spark\u0027s width_bucket to i64 (#22811)\n\n## Which issue does this PR close?\n- Closes #22602\n\n## Rationale for this change\nThe return type of this function in Spark is int64. It is changed here\nto match Sparks behavior.\n\n## What changes are included in this PR?\nChanged the return type of Spark\u0027s `width_bucket` from i32 to i64.\n\n## Are these changes tested?\nYes, all existing unit tests in `width_bucket.rs` pass. \n\n## Are there any user-facing changes?\n- Spark consumers should expect a different data type when calling the\n`width_bucket` function.\n- No API changes.\n\nOriginal-commit: apache/datafusion@6c2221d3ff630538efbb277cdf68fca794a264eb\n" }, { "commit": "f0ed50222a7c8c1676144566d0ace01630ebc593", "tree": "2b584402bf88bd436eace67dbeef8cddc748b9e5", "parents": [ "b1665a4c0f954bc5218f364c9bc0039776430a08" ], "author": { "name": "Huaijin", "email": "haohuaijin@gmail.com", "time": "Tue Jun 16 05:01:36 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jun 15 21:01:36 2026 +0000" }, "message": "refactor: remove `opt_filter` in `GroupsAccumulator::merge_batch` (#22816)\n\n## Which issue does this PR close?\n\n- Closes #22775.\n\n## Rationale for this change\n\nthe `opt_filter` on `GroupsAccumulator::merge_batch` is a dead\nparameter. Aggregate `FILTER` clauses only apply to raw input rows in\nthe update phase (`update_batch`). `merge_batch` combines already\npre-aggregated states, so there is no per-row filtering to do —\n`opt_filter` is meaningless there.\n\nThe code confirms this:\n- The only production caller (`row_hash.rs`) always passed `None`.\n- Existing implementations already ignored it — e.g. `correlation.rs`\nasserted `opt_filter.is_none()`, and Spark `avg` used `_opt_filter`.\n\n## What changes are included in this PR?\n\n- Removed `opt_filter` from `merge_batch` in the trait and all\nimplementations (built-in aggregates, `physical-expr-common`,\n`functions-aggregate-common`, Spark, and FFI).\n- Updated the trait docs to say `merge_batch` has no `opt_filter`\nbecause filtering happens in the update phase.\n- Changed the group zero-init path in `row_hash.rs` to always use\n`update_batch` with an all-false filter instead of branching to\n`merge_batch`. `update_batch` always takes raw argument types (what\n`aggregate_arguments` provides), and since every row is filtered out the\ndata never matters — this is simpler and more correct.\n- Updated all call sites and tests.\n\n## Are these changes tested?\n\nYes. Existing aggregate tests cover this and were updated to the new\nsignature. The `first_last` tests were adjusted (with comments) to match\nthe merge behavior without a filter, and the FFI and Spark tests were\nupdated too.\n\n## Are there any user-facing changes?\n\nYes — this is a breaking change to the public `GroupsAccumulator` trait:\n`opt_filter` is removed from `merge_batch`. Custom implementations and\ndirect callers must update their signatures.\n\nOriginal-commit: apache/datafusion@c14379b9b528f3b9f98e29f5e020aa06dc182a66\n" }, { "commit": "b1665a4c0f954bc5218f364c9bc0039776430a08", "tree": "5558a756a5e054724f687c1c097641c8644e5271", "parents": [ "d682c3cd41b564a7d9ad315472a6072928660d64" ], "author": { "name": "Ahmed EL.", "email": "xfocus29@gmail.com", "time": "Sat Jun 13 02:40:47 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Jun 13 01:40:47 2026 +0000" }, "message": "fix: preserve Spark next_day whitespace validation (#22720)\n\n## Which issue does this PR close?\n\n- Closes #22717.\n\n## Rationale for this change\n\nSpark does not trim `dayOfWeek` before matching it in `next_day`, but\n`datafusion-spark` currently does. That makes values like `\u0027 MO \u0027`\nsucceed in DataFusion even though Spark treats them as invalid.\n\n## What changes are included in this PR?\n\n- remove the `.trim()` call from `spark_next_day`\n- add a regression test proving whitespace-padded day names are rejected\n\n## Are these changes tested?\n\n- `cargo test -p datafusion-spark\nnext_day_rejects_whitespace_padded_day_names -- --nocapture`\n- `cargo test -p datafusion-spark`\n- `cargo fmt --all --check`\n- `cargo clippy -p datafusion-spark --all-targets --all-features\n--no-deps -- -D warnings`\n\nNote: the broader package clippy invocation still reports an existing\nunused import warning in untouched\n`datafusion/core/src/execution/session_state.rs` on current main.\n\n## Are there any user-facing changes?\n\nBehavior now matches Spark for whitespace-padded `dayOfWeek` inputs in\n`next_day`.\n\n---------\n\nSigned-off-by: xfocus3 \u003cxfocus3@users.noreply.github.com\u003e\nCo-authored-by: xfocus3 \u003cxfocus3@users.noreply.github.com\u003e\nCo-authored-by: Ahmed El amraouiyine \u003cahmed.elamraouiyine@vilavi.fr\u003e\n\nOriginal-commit: apache/datafusion@574a1e6b39acd5b2c521d554863ac50fc0855654\n" }, { "commit": "d682c3cd41b564a7d9ad315472a6072928660d64", "tree": "dea6c2ba3cc17b466f4baed891b7a9c78d1217fc", "parents": [ "452281cc1887f37f5567a4fa7148a0f31f171b1d" ], "author": { "name": "David López", "email": "hola@devel0pez.com", "time": "Sat Jun 13 03:37:17 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Jun 13 01:37:17 2026 +0000" }, "message": "feat(spark): add `concat_ws` with array support (#20928)\n\n## Which issue does this PR close?\n\n- Part of #15914\n\n## Rationale for this change\n\nDataFusion core\u0027s `concat_ws` does not support array arguments. Spark\u0027s\n`concat_ws(sep, ...)` accepts both scalar strings and arrays, expanding\narray elements and skipping nulls. This is needed for Spark\ncompatibility in the `datafusion-spark` crate.\n\n\n## What changes are included in this PR?\n\n- New `SparkConcatWs` UDF in\n`datafusion/spark/src/function/string/concat_ws.rs`\n - Supports `concat_ws(sep, str1, str2, ...)` with scalar strings\n- Supports array arguments: `concat_ws(\u0027,\u0027, array(\u0027a\u0027, \u0027b\u0027), \u0027c\u0027)` →\n`\"a,b,c\"`\n - Null scalars and null array elements are skipped (Spark behavior)\n - Null separator returns NULL\n - Zero value arguments (`concat_ws(\u0027,\u0027)`) returns empty string\n - Supports Utf8, LargeUtf8, Utf8View, List, and LargeList types\n- Registered the function in `mod.rs` (`make_udf_function!`,\n`export_functions!`, `functions()`)\n- Replaced commented-out SLT tests with 14 working test cases covering\nbasic usage, arrays, mixed arguments, nulls, column expressions, and\nedge cases\n\n\n## Are these changes tested?\n\nYes.\n- 7 unit tests in `concat_ws.rs` (basic, null values skipped, null\nseparator, list arrays, list with nulls, mixed scalar+list, multiple\nrows)\n- 14 SLT tests in `spark/string/concat_ws.slt` covering scalars, arrays,\nnulls, column expressions, and edge cases\n\n\n## Are there any user-facing changes?\n\nNo. This is a new function in the `datafusion-spark` crate only.\n\nOriginal-commit: apache/datafusion@e5f7af15a6dafe9e2847a81bae5fddfda5a60c0a\n" }, { "commit": "452281cc1887f37f5567a4fa7148a0f31f171b1d", "tree": "faed7a64dc002f4ec3e05c087ef5788347dd5704", "parents": [ "050a4d2ae545ae182df49c6e4c419c20800709d1" ], "author": { "name": "Andrew Lamb", "email": "andrew@nerdnetworks.org", "time": "Wed Jun 10 14:27:53 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jun 10 18:27:53 2026 +0000" }, "message": "chore: Update to arrow/parquet 59.0.0 (#22744)\n\n## Which issue does this PR close?\n- related to https://github.com/apache/arrow-rs/issues/9110\n\n\n## Rationale for this change\n\nUpdate to latest version of arrow/parquet\n\n## What changes are included in this PR?\n1. Update to arrow/parquet 59.0.0\n2. Adjust code for API differences\n\n## Are these changes tested?\n\nBy CI\n## Are there any user-facing changes?\n\nNew dependency\n\nOriginal-commit: apache/datafusion@dae03ee062b2abf986de8df12ea82fb1578a2d99\n" }, { "commit": "050a4d2ae545ae182df49c6e4c419c20800709d1", "tree": "a6035063300a11b0b06314091232d5d9ebecf4cd", "parents": [ "65b9f2bcd02192aa38c961eb02559a99ad94c54a" ], "author": { "name": "7. Sun", "email": "jhao.sun@gmail.com", "time": "Wed Jun 10 12:49:23 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jun 10 16:49:23 2026 +0000" }, "message": "feat: implement Spark-compatible weekday function (#22740)\n\n## Which issue does this PR close?\n\n- Closes #22599.\n\n## Rationale for this change\n\nImplements the Spark `weekday` function as part of the datafusion-spark\nfunction library (#15914). Spark\u0027s `weekday` returns the day-of-week as\na 0-indexed integer with Monday \u003d 0 .. Sunday \u003d 6, which differs from\nthe existing `dayofweek` (1-indexed, Sunday \u003d 1). This fills a gap for\nSpark-compatibility consumers.\n\n## What changes are included in this PR?\n\n- New `SparkWeekDay` scalar UDF in\n`datafusion/spark/src/function/datetime/weekday.rs`, modeled on the\nsibling `monthname` function.\n- Accepts Date and (implicitly coerced) Timestamp input; returns\n`Int32`.\n- Uses Arrow\u0027s `DatePart::DayOfWeekMonday0` kernel, which is exactly\n0\u003dMonday .. 6\u003dSunday — a direct match for Spark semantics.\n- Null input propagates to null output for both scalar and array paths.\n- Registered in the datetime function `mod.rs` (UDF macro, doc export,\n`functions()` list).\n\n## Are these changes tested?\n\nYes:\n- Rust unit tests: return-field nullability, scalar evaluation (incl.\nnull), and array evaluation.\n- sqllogictest coverage in\n`datafusion/sqllogictest/test_files/spark/datetime/weekday.slt`: scalar\ndates for all 7 weekdays, array input, TIMESTAMP / TIMESTAMP_NTZ / LTZ\ncoercion, null handling, and argument-type / zero-argument error cases.\nThe original PySpark 3.5.5 reference (`weekday(\u00272009-07-30\u0027) \u003d 3`) is\npreserved and validated.\n\n## Are there any user-facing changes?\n\nAdds the new Spark-compatible scalar function `weekday`. No breaking\nchanges.\n\n---\n\nNote: I noticed there is an existing draft PR #22601 for the same\nfunction (last updated 2026-05-28). It appears to be a stale draft, so\nI\u0027ve opened this as a complete, tested implementation — happy to defer\nor collaborate if the draft author is still active.\n\nSigned-off-by: sjhddh \u003c151469562+sjhddh@users.noreply.github.com\u003e\nCo-authored-by: sjhddh \u003c151469562+sjhddh@users.noreply.github.com\u003e\nCo-authored-by: Andrew Lamb \u003candrew@nerdnetworks.org\u003e\n\nOriginal-commit: apache/datafusion@6d4cb31d3a844993ec82279d363783780481a53d\n" }, { "commit": "65b9f2bcd02192aa38c961eb02559a99ad94c54a", "tree": "a94262291cdd820825c601a7fee074c8d66d8a28", "parents": [ "1499c2e443fad93aed817864743c676370aec13a" ], "author": { "name": "Kazantsev Maksim", "email": "kazantsev.maksim.n@gmail.com", "time": "Mon Jun 08 07:49:47 2026 +0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jun 08 03:49:47 2026 +0000" }, "message": "Spark quote function implementation (#22642)\n\n## Which issue does this PR close?\n\n- N/A\n\n## Rationale for this change\n\nAdd new spark function:\nhttps://spark.apache.org/docs/latest/api/sql/index.html#quote\n\n## What changes are included in this PR?\n\n- Implementation\n- SLT tests\n\n## Are these changes tested?\n\nYes, tests added as part of this PR.\n\n## Are there any user-facing changes?\n\nNo, these are new function.\n\n---------\n\nCo-authored-by: Kazantsev Maksim \u003cmn.kazantsev@gmail.com\u003e\n\nOriginal-commit: apache/datafusion@04ef3c7d96e4d13668d8b74e2216b4b3e46c5f15\n" }, { "commit": "1499c2e443fad93aed817864743c676370aec13a", "tree": "0c0c0865cd9a2ee7a477f08d9c6dd4893b077d92", "parents": [ "299d197a07b8a501b80b3459cdd92ca90b4c2e01" ], "author": { "name": "Kumar Ujjawal", "email": "ujjawalpathak6@gmail.com", "time": "Sat Jun 06 01:22:34 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jun 05 19:52:34 2026 +0000" }, "message": "feat: Add Spark SQL parser dialect config (#22529)\n\n## Which issue does this PR close?\n\n\u003c!--\nWe generally require a GitHub issue to be filed for all bug fixes and\nenhancements and this helps us generate change logs for our releases.\nYou can link an issue to this PR using the GitHub syntax. For example\n`Closes #123` indicates that this PR will close issue #123.\n--\u003e\n\n- Closes #21653.\n\n## Rationale for this change\n\nThis lets users combine Spark-compatible functions with Spark SQL\nparsing when they choose the dialect.\n\n\u003c!--\nWhy are you proposing this change? If this is already explained clearly\nin the issue then this section is not needed.\nExplaining clearly why changes are proposed helps reviewers understand\nyour changes and offer better suggestions for fixes.\n--\u003e\n\n## What changes are included in this PR?\n\n - Add `Spark` to the SQL parser dialect config enum.\n - Accept `spark` and `sparksql` as dialect config values.\n - Use Spark dialect for Spark sqllogictests.\n - Keep `with_spark_features()` config-neutral.\n - Update dialect docs and expected config output.\n- Add tests for Spark dialect parsing and Spark SQL execution with Spark\nfunctions.\n\n\u003c!--\nThere is no need to duplicate the description in the issue here but it\nis sometimes worth providing a summary of the individual changes in this\nPR.\n--\u003e\n\n## Are these changes tested?\n\nYes\n\n\u003c!--\nWe typically require tests for all PRs in order to:\n1. Prevent the code from being accidentally broken by subsequent changes\n2. Serve as another way to document the expected behavior of the code\n\nIf tests are not included in your PR, please explain why (for example,\nare they covered by existing tests)?\n--\u003e\n\n## Are there any user-facing changes?\n\nUsers can now set `datafusion.sql_parser.dialect` to `spark` or\n`sparksql`.\n\n\u003c!--\nIf there are user-facing changes then we may require documentation to be\nupdated before approving the PR.\n--\u003e\n\n\u003c!--\nIf there are any breaking changes to public APIs, please add the `api\nchange` label.\n--\u003e\n\n---------\n\nCo-authored-by: Andrew Lamb \u003candrew@nerdnetworks.org\u003e\n\nOriginal-commit: apache/datafusion@7127f8bd79a7ab40cc0189540b912c5415ed6caa\n" }, { "commit": "299d197a07b8a501b80b3459cdd92ca90b4c2e01", "tree": "d50351a1d88483ed6d38b12e6ad2facc1c87b6a0", "parents": [ "c772f59182d680c431321f6bae8513faf148bbec" ], "author": { "name": "kosiew", "email": "kosiew@gmail.com", "time": "Tue Jun 02 19:25:28 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jun 02 11:25:28 2026 +0000" }, "message": "Refactor Spark `format_string` integer conversion dispatch (#22388)\n\n## Which issue does this PR close?\n\n* Closes #22163\n\n## Rationale for this change\n\n`ConversionSpecifier::format` contained substantial duplication across\ninteger `ScalarValue` variants for `%d`, `%x`, `%o`, `%s`, and `%c`\nhandling. Each integer width repeated nearly identical conversion logic,\nmaking the code harder to maintain and increasing the risk of\ninconsistent behavior across integer types.\n\nThis change consolidates integer formatting behavior into shared\ninternal helpers while preserving existing Spark-compatible semantics.\n\n## What changes are included in this PR?\n\n* Introduced a local `IntegerValue` enum to normalize signed and\nunsigned integer handling while preserving width-specific unsigned bit\nbehavior for `%x` and `%o`.\n* Replaced repeated per-variant integer dispatch branches in\n`ConversionSpecifier::format` with a shared `format_integer` helper.\n* Added shared helper methods for:\n\n * decimal formatting\n * unsigned bit formatting\n * `%c` conversion\n * decimal string conversion\n* Added small `macro_rules!` helpers to generate `From\u003cT\u003e for\nIntegerValue` implementations for signed and unsigned integer families,\nreducing repetitive conversion boilerplate while preserving\nwidth-specific unsigned formatting semantics.\n* Added `invalid_integer_conversion` helper to centralize integer\nconversion error generation.\n* Added table-driven regression coverage for integer formatting behavior\nacross:\n\n * signed integer widths\n * unsigned integer widths\n * `%d`, `%x`, `%o`, `%s`, and `%c`\n * null handling behavior\n\n## Are these changes tested?\n\nYes.\n\nAdded `test_integer_formatting_across_widths` covering:\n\n* Signed integer formatting across `Int8`, `Int16`, `Int32`, and `Int64`\n* Unsigned integer formatting across `UInt8`, `UInt16`, `UInt32`, and\n`UInt64`\n* `%d`, `%x`, `%o`, `%s`, and `%c` formatting behavior\n* Null integer formatting behavior\n\n## Are there any user-facing changes?\n\nNo intended user-facing behavior changes. This PR is a structural\nrefactor intended to preserve existing Spark-compatible integer\nformatting semantics.\n\n## LLM-generated code disclosure\n\nThis PR includes LLM-generated code and comments. All LLM-generated\ncontent has been manually reviewed and tested.\n\nOriginal-commit: apache/datafusion@7843ab385bdc90db02a80ccfd410eb74c1aeb0fb\n" }, { "commit": "c772f59182d680c431321f6bae8513faf148bbec", "tree": "f76ce451f4ffca8db5a9eef43431c57dcdef0101", "parents": [ "6f500fa3cf5065fef5cadedd4ec3eb91437709c9" ], "author": { "name": "Krishna Sudarshan J", "email": "75199111+athlcode@users.noreply.github.com", "time": "Mon Jun 01 21:04:49 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jun 01 15:34:49 2026 +0000" }, "message": "added support for MapFromEntries (#21720)\n\n## Which issue does this PR close?\n\nCloses # (none: prerequisite for\nhttps://github.com/apache/datafusion-comet/issues/2706; follow-up to\n#17779 and #19274).\n\n## Rationale for this change\n\nSpark\u0027s default mapKeyDedupPolicy is EXCEPTION, raising\nSparkRuntimeException with error class DUPLICATED_MAP_KEY on duplicate\nmap keys. The existing Spark map_from_entries / map_from_arrays UDFs\nsilently kept the last occurrence, forcing downstream engines like\ndatafusion-comet to fall back to Spark.\n\n## What changes are included in this PR?\n\nDuplicate map keys in Spark map_from_entries, map_from_arrays, and\nstr_to_map now raise [DUPLICATED_MAP_KEY] Duplicate map key {key} was\nfound, matching Spark\u0027s default behaviour and error class.\n\n## Are these changes tested?\n\nYes, via sqllogictest assertions covering the new error across the\naffected Spark map UDFs.\n\n## Are there any user-facing changes?\n\nYes. Duplicate keys now raise [DUPLICATED_MAP_KEY] under the default\npolicy instead of silently collapsing to the last occurrence. No new\nconfig keys, no API changes.\n\n---------\n\nCo-authored-by: Krishna Sudarshan J \u003c75199111+KrishnaSudarshan7@users.noreply.github.com\u003e\n\nOriginal-commit: apache/datafusion@f46a4a431b6c46d1d0e53f00c088002c05adbd95\n" }, { "commit": "6f500fa3cf5065fef5cadedd4ec3eb91437709c9", "tree": "88ab92c21edd5d42b3a9f10c9f5298e85a2bef1e", "parents": [ "32bf63131902c6487368c41e6dcfd675b305130c" ], "author": { "name": "theirix", "email": "theirix@gmail.com", "time": "Mon Jun 01 14:39:21 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jun 01 13:39:21 2026 +0000" }, "message": "fix: wrong precision in a decimal256 log test (#22578)\n\n## Which issue does this PR close?\n\n## Rationale for this change\n\nSpotted wrong usage of decimal precision in `test_log_decimal256_large`\n- passed Decimal128 max precision instead of Decimal256. The test worked\nfine.\n\n## What changes are included in this PR?\n\n- Test fixup\n- Unhardcode the usage decimal precision in non-test code\n\n## Are these changes tested?\n\n- Tests passed\n\n## Are there any user-facing changes?\n\nno\n\nOriginal-commit: apache/datafusion@08db4c8f832f7c9dc8ec6ebfeb168148fe06329e\n" }, { "commit": "32bf63131902c6487368c41e6dcfd675b305130c", "tree": "06d8c863d41931948ffed15682dbb6f260c99820", "parents": [ "4786b627f34f5b2760efc8c86c74134d843a2f1a" ], "author": { "name": "Brijesh Thakkar", "email": "brijeshthakkariitian5126@gmail.com", "time": "Sat May 30 01:14:58 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri May 29 19:44:58 2026 +0000" }, "message": "feat: add SparkPow UDF returning Infinity for pow(0, negative) (#22605)\n\n## Which issue does this PR close?\n\nCloses #22598\n\n## Rationale for this change\n\nIn Apache Spark, the `pow(base, exp)` function follows IEEE 754\nsemantics where raising `0` (or `-0.0`) to a negative exponent yields\npositive `Infinity`.\n\nCurrently, DataFusion\u0027s default core `PowerFunc` mimics PostgreSQL\nbehavior, throwing an explicit error (`\"zero raised to a negative power\nis undefined\"`). To support standard Spark compatibility without\nbreaking core DataFusion expectations, this PR introduces a specialized\n`SparkPow` UDF inside the `datafusion-spark` crate.\n\n## What changes are included in this PR?\n\nThis PR introduces the following changes within the `datafusion-spark`\nintegration crate:\n* **Added `SparkPow` UDF**\n(`datafusion/spark/src/function/math/pow.rs`): Overrides the `Float64`\nexecution path to evaluate `base \u003d\u003d 0.0 \u0026\u0026 exp \u003c 0.0` as `f64::INFINITY`\n(safely catching both `0.0` and `-0.0` due to IEEE 754 equality rules).\n* **Decimal Delegation**: Preserves correctness by delegating non-float\ntypes (like decimals) back to the standard `PowerFunc`, as decimals\ncannot represent infinity.\n* **Function Registration**\n(`datafusion/spark/src/function/math/mod.rs`): Registers the new `pow`\nfunction and establishes `power` as a valid alias.\n* **SQL Integration Tests**\n(`datafusion/sqllogictest/test_files/spark/math/pow.slt`): Updates and\nadds test coverage ensuring `pow(0, -1)`, `power(0, -1)`, and `pow(0.0,\n-1.0)` successfully return `Infinity`.\n\n## Are these changes tested?\n\nYes, the changes are covered via both unit and integration tests:\n1. **Unit Tests**: Added `test_spark_pow_zero_negative_returns_infinity`\nand `test_spark_pow_normal_cases` within `pow.rs` to validate the core\nscalar execution logic.\n2. **Integration Tests**: Extended\n`datafusion/sqllogictest/test_files/spark/math/pow.slt` to verify the\nend-to-end SQL evaluation behavior.\n\n## Are there any user-facing changes?\n\nYes, but only for users utilizing the `datafusion-spark` compatibility\nfeatures. When the Spark dialect/crate is active, evaluating `pow(0,\n\u003cnegative\u003e)` will now return `Infinity` instead of throwing an\nevaluation error. Core DataFusion behavior remains completely unchanged.\n\nOriginal-commit: apache/datafusion@d8c4588285c7db993935292c3f0bae6df3a171d0\n" }, { "commit": "4786b627f34f5b2760efc8c86c74134d843a2f1a", "tree": "ef263466a3cec1bbaa243d48887484f465099ee6", "parents": [ "9450a1842521fdd947fea74f4b379fc1fd4ba7cf" ], "author": { "name": "Lía Adriana", "email": "lia.castaneda@datadoghq.com", "time": "Fri May 29 09:47:56 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri May 29 07:47:56 2026 +0000" }, "message": "refactor: wrap HigherOrderUDFImpl in a concrete HigherOrderUDF struct (#22593)\n\n## Which issue does this PR close?\n\n\u003c!--\nWe generally require a GitHub issue to be filed for all bug fixes and\nenhancements and this helps us generate change logs for our releases.\nYou can link an issue to this PR using the GitHub syntax. For example\n`Closes #123` indicates that this PR will close issue #123.\n--\u003e\n\nPart of https://github.com/apache/datafusion/issues/21172\n\n## Rationale for this change\n\n`HigherOrderUDF` was the only UDF kind defined as a trait that callers\nused directly via `Arc\u003cdyn HigherOrderUDF\u003e`. The other UDFs:\n`ScalarUDF`, `AggregateUDF`, `WindowUDF` — are concrete structs that\nwrap their respective `*Impl trait`, which makes inherent methods like\n`with_aliases` ergonomic to call on the function object. With the\ntrait-only setup, adding aliases to an existing higher-order function\nrequired an extension trait import or a free helper function.\n\nThis PR brings higher order functions in line with the other UDFs so the\nsame `with_aliases` pattern works.\n\n## What changes are included in this PR?\n\n- Rename the `HigherOrderUDF` trait to `HigherOrderUDFImpl`, matching\n`ScalarUDFImpl`/`AggregateUDFImpl`.\nAdd a concrete `HigherOrderUDF` struct wrapping `Arc\u003cdyn\nHigherOrderUDFImpl\u003e`, with the same shape as `ScalarUDF`: new_from_impl,\nnew_from_shared_impl, inner, with_aliases, From\u003cF: HigherOrderUDFImpl\u003e,\nand delegate methods for every trait method.\n`with_aliases` is backed by a private `AliasedHigherOrderUDFImpl`\ndecorator (same pattern as `AliasedScalarUDFImpl`).\n- Update `Expr::HigherOrderFunction`, `FunctionRegistry`, the\n`create_higher_order! `singleton macro, and all consumer files ( across\nseveral crates) to use `Arc\u003cHigherOrderUDF\u003e` instead of `Arc\u003cdyn\nHigherOrderUDF\u003e`.\nExisting impls (`ArrayFilter`, `ArrayTransform`, `ArrayAnyMatch`) now\nimplement `HigherOrderUDFImpl`; their public constructors continue to\nreturn `Arc\u003cHigherOrderUDF\u003e` so external call sites need no changes.\n\nCallers can now write:\n\n`array_filter_higher_order_function().with_aliases([\"filter\"])\n`\n\nexactly like the existing scalar pattern:\n\n`make_array_udf().as_ref().clone().with_aliases([\"array_construct\"])\n`\n\n## Are these changes tested?\n\nCovered by existing tests \n\n## Are there any user-facing changes?\n\nYes, any code referring to `Arc\u003cdyn HigherOrderUDF\u003e` needs to become\n`Arc\u003cHigherOrderUDF\u003e`, and any code that wrote `impl HigherOrderUDF for\nMyHOF` needs to write i`mpl HigherOrderUDFImpl for MyType`. Constructing\na HigherOrderUDF from an impl is HigherOrderUDF::new_from_impl(my_impl)\n(or my_impl.into()).\n\nOriginal-commit: apache/datafusion@34834dc329af52c3c38563f3a62446c6ce1db358\n" }, { "commit": "9450a1842521fdd947fea74f4b379fc1fd4ba7cf", "tree": "d95ec464b5acc7de8e2b2b79179753953c5a69b7", "parents": [ "b900bcf2a89972da3567c8654db3a5f38f4b9248" ], "author": { "name": "Zeel Rajodiya", "email": "jeelrajodiyajeel@gmail.com", "time": "Mon May 25 09:39:05 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon May 25 04:09:05 2026 +0000" }, "message": "feat: Add Spark-compatible `monthname` function to datafusion-spark (#21639)\n\n**Rationale**\n\nThe `datafusion-spark` crate is missing the `monthname` function.\nSpark\u0027s\n[`monthname(date)`](https://spark.apache.org/docs/latest/api/sql/index.html#monthname)\nreturns the **three-letter abbreviated month name** (Jan, Feb, ..., Dec)\nfrom a date or timestamp — commonly used in Spark SQL workloads.\n\n**What changes are included in this PR?**\n\nAdds `SparkMonthName` to `datafusion-spark`\u0027s datetime functions. It\nuses `arrow::compute::date_part(DatePart::Month)` to extract the month\nnumber and maps it to the abbreviated name. The signature accepts\n**Timestamp types** with automatic coercion from Date32/Date64.\n\n**Are these changes tested?**\n\nYes — 6 unit tests covering scalar dates, array dates with nulls, null\nscalars, timestamp microseconds, all 12 months, and return field\nnullability.\n\n**Are there any user-facing changes?**\n\nNew `monthname` scalar function available when using `datafusion-spark`.\n\n---------\n\nCo-authored-by: Andrew Lamb \u003candrew@nerdnetworks.org\u003e\nCo-authored-by: Jeffrey Vo \u003cjeffrey.vo.australia@gmail.com\u003e\n\nOriginal-commit: apache/datafusion@626da1eafdee25396989c4bbf9044e70f7df491a\n" }, { "commit": "b900bcf2a89972da3567c8654db3a5f38f4b9248", "tree": "a2fa9c8590740d7b5729011013a4293db36d1b5a", "parents": [ "e1b58c2c564948aee0ed2edda930dabac8845828" ], "author": { "name": "Yongting You", "email": "2010youy01@gmail.com", "time": "Sat May 23 17:14:20 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat May 23 09:14:20 2026 +0000" }, "message": "chore: Disallow `reserve()` in clippy to prevent panics (#22386)\n\n## Which issue does this PR close?\n\n\u003c!--\nWe generally require a GitHub issue to be filed for all bug fixes and\nenhancements and this helps us generate change logs for our releases.\nYou can link an issue to this PR using the GitHub syntax. For example\n`Closes #123` indicates that this PR will close issue #123.\n--\u003e\n\n- Closes #.\n\n## Rationale for this change\n\n\u003c!--\nWhy are you proposing this change? If this is already explained clearly\nin the issue then this section is not needed.\nExplaining clearly why changes are proposed helps reviewers understand\nyour changes and offer better suggestions for fixes.\n--\u003e\nIn https://github.com/apache/datafusion/pull/22323, we have fixed panic\nbugs by replacing `Vec::reserve()` with `Vec::try_reserve()`\n\nThis PR enforces a clippy rule to disallow `Vec::reserve()`\nproject-wise, to prevent future violations. Also it is easy to ignore\nthis lint with macro, if we can ensure the safety.\n\n## What changes are included in this PR?\n\n\u003c!--\nThere is no need to duplicate the description in the issue here but it\nis sometimes worth providing a summary of the individual changes in this\nPR.\n--\u003e\n\n## Are these changes tested?\n\n\u003c!--\nWe typically require tests for all PRs in order to:\n1. Prevent the code from being accidentally broken by subsequent changes\n2. Serve as another way to document the expected behavior of the code\n\nIf tests are not included in your PR, please explain why (for example,\nare they covered by existing tests)?\n--\u003e\nThey are not testable right now. To trigger the panic, we would need to\nconstruct an Arrow array with more than `i32::MAX` elements, but Arrow\narrays are currently backed by flat vectors.\n\nHowever, if Arrow supports an encoding such as RLE in the future, it may\nbecome possible to represent such large arrays with constant memory. At\nthat point, these code paths would become vulnerable, so I think it is\nbetter to fix them now.\n## Are there any user-facing changes?\n\n\u003c!--\nIf there are user-facing changes then we may require documentation to be\nupdated before approving the PR.\n--\u003e\n\n\u003c!--\nIf there are any breaking changes to public APIs, please add the `api\nchange` label.\n--\u003e\n\nOriginal-commit: apache/datafusion@258c6c2cd9113a10cc755b1c246134a31bf288bb\n" }, { "commit": "e1b58c2c564948aee0ed2edda930dabac8845828", "tree": "e840f634864d2e47251e43a99a4026a72ce29c9e", "parents": [ "dfea3c0ba8e6ca1552fa60bb8562d77ab126cd74" ], "author": { "name": "Oleks V", "email": "comphead@users.noreply.github.com", "time": "Wed May 20 17:48:57 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu May 21 00:48:57 2026 +0000" }, "message": "feat: fix `slice` function on OOB ranges (#22404)\n\n## Which issue does this PR close?\n\n\u003c!--\nWe generally require a GitHub issue to be filed for all bug fixes and\nenhancements and this helps us generate change logs for our releases.\nYou can link an issue to this PR using the GitHub syntax. For example\n`Closes #123` indicates that this PR will close issue #123.\n--\u003e\n\n- Closes #22400 .\n\n## Rationale for this change\n\n\u003c!--\nWhy are you proposing this change? If this is already explained clearly\nin the issue then this section is not needed.\nExplaining clearly why changes are proposed helps reviewers understand\nyour changes and offer better suggestions for fixes.\n--\u003e\n\n## What changes are included in this PR?\n\n\u003c!--\nThere is no need to duplicate the description in the issue here but it\nis sometimes worth providing a summary of the individual changes in this\nPR.\n--\u003e\n\n## Are these changes tested?\n\n\u003c!--\nWe typically require tests for all PRs in order to:\n1. Prevent the code from being accidentally broken by subsequent changes\n2. Serve as another way to document the expected behavior of the code\n\nIf tests are not included in your PR, please explain why (for example,\nare they covered by existing tests)?\n--\u003e\n\n## Are there any user-facing changes?\n\n\u003c!--\nIf there are user-facing changes then we may require documentation to be\nupdated before approving the PR.\n--\u003e\n\n\u003c!--\nIf there are any breaking changes to public APIs, please add the `api\nchange` label.\n--\u003e\n\nOriginal-commit: apache/datafusion@abb943d7cd26182955bc74d784aa7bd1da7a3441\n" }, { "commit": "dfea3c0ba8e6ca1552fa60bb8562d77ab126cd74", "tree": "54e70da416c826521412abfbbc15660919b11e2c", "parents": [ "d1d656518c5b615d559ed5c582472f435ae89794" ], "author": { "name": "kosiew", "email": "kosiew@gmail.com", "time": "Wed May 20 15:12:23 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed May 20 07:12:23 2026 +0000" }, "message": "Refactor Spark `format_string` numeric `%c` conversion dispatch (#22166)\n\n## Which issue does this PR close?\n\n* Part of #22163\n\n## Rationale for this change\n\nThe `%c` conversion path in Spark `format_string` duplicated nearly\nidentical logic across all signed and unsigned integer scalar variants.\nThis duplication increased maintenance overhead and the risk of future\n`%c` fixes being applied inconsistently.\n\nThis change centralizes numeric scalar-to-character conversion in a\nsingle helper while preserving existing behavior and error propagation\nsemantics.\n\n## What changes are included in this PR?\n\n* Added a local `integer_scalar_to_char` helper to centralize integer\nscalar `%c` conversion handling.\n* Routed all non-null integer `%c` conversion paths through the new\nhelper.\n* Removed duplicated `%c` conversion branches from individual integer\nscalar match arms.\n* Preserved existing signed/unsigned conversion semantics by continuing\nto use the existing `signed_to_char` and `unsigned_to_char` helpers.\n* Left all non-`%c` formatting behavior unchanged.\n\n## Are these changes tested?\n\nNo new tests were added in this PR.\n\nThis refactor is intended to preserve existing behavior, and existing\n`%c` formatting tests continue to validate valid and invalid code point\nhandling.\n\nSuggested validation command:\n\n```bash\ncargo test -p datafusion-spark format_char\n```\n\n## Are there any user-facing changes?\n\nNo. This is a maintainability refactor only and is not intended to\nchange observable behavior or error messages.\n\n## LLM-generated code disclosure\n\nThis PR includes LLM-generated code and comments. All LLM-generated\ncontent has been manually reviewed and tested.\n\nOriginal-commit: apache/datafusion@265473ea970b573eff878f74124a6bc87107cfdc\n" }, { "commit": "d1d656518c5b615d559ed5c582472f435ae89794", "tree": "5f7b5ce79ca78e01f1f3e28ef960b24b2f8b40df", "parents": [ "df30b3c0e31247adf9314888e819bde494b01bca" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Tue May 19 20:55:02 2026 -0600" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed May 20 02:55:02 2026 +0000" }, "message": "feat: add Spark-compatible xxhash64 function (#21967)\n\n## Which issue does this PR close?\n\n- Partially addresses https://github.com/apache/datafusion/issues/14044\n\n## Rationale for this change\n\nDonate the `xxhash64` hash function from Comet so that other projects\ncan benefit from it.\nThe function was initially implemented in Comet by @advancedxy.\n\nThis is a continuation of #19627, which has gone stale. To keep the\nchange focused\nand easier to review, this PR adds only `xxhash64`. Murmur3 will follow\nin a separate PR.\n\nThe first commit is the same as the version of #19627 that was\npreviously approved.\n\nThe second commit implements optmizations and bug fixes from the latest\nComet version.\n\nThe third commit is cleanup.\n\n## What changes are included in this PR?\n\n- Add `xxhash64(expr1, expr2, ...)` to `datafusion-spark`.\n- Add Rust unit tests for primitives, boundary values, emoji/CJK\nstrings, float `-0.0`\nnormalization, dictionaries (with and without nulls), `FixedSizeBinary`,\n`Struct`, and\n `List`.\n- Add sqllogictest coverage in\n`datafusion/sqllogictest/test_files/spark/hash/xxhash64.slt`\n with values verified against Spark.\n\n## Are these changes tested?\n\nYes, both Rust unit tests and sqllogictest are included.\n\n## Are there any user-facing changes?\n\nA new `xxhash64` scalar function is available in `datafusion-spark`.\n\nOriginal-commit: apache/datafusion@6b134ddaf5658633fdb4edc6149a934cdaa49309\n" }, { "commit": "df30b3c0e31247adf9314888e819bde494b01bca", "tree": "6205d29b7626c64d074451f5bcbdb0234aa5aff1", "parents": [ "d148e3fa462544685f05d37407d96e01d2a573d5" ], "author": { "name": "Daniël Heres", "email": "danielheres@gmail.com", "time": "Fri May 15 07:51:17 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri May 15 05:51:17 2026 +0000" }, "message": "chore: Update Rust toolchain to 1.95 (#22177)\n\n## Which issue does this PR close?\n\n- Closes: https://github.com/apache/datafusion/issues/22178\n\n## Rationale for this change\n\nUpdate the pinned Rust toolchain to 1.95.0 and keep the workspace\npassing the required formatting and clippy checks with the new\ntoolchain.\n\n## What changes are included in this PR?\n\n- Bump rust-toolchain.toml from 1.94.0 to 1.95.0.\n- Update the contributor docs rust-analyzer example to reference 1.95.0.\n- Apply the minimal clippy and compile fixes needed under Rust 1.95.0.\n\n## Are these changes tested?\n\n- cargo fmt --all\n- cargo clippy --all-targets --all-features -- -D warnings\n\n## Are there any user-facing changes?\n\nNo user-facing runtime changes. This updates the project toolchain and\nrelated developer-facing docs.\n\nOriginal-commit: apache/datafusion@89391b51705415320c0d6b6c934b8e1a34d0837b\n" }, { "commit": "d148e3fa462544685f05d37407d96e01d2a573d5", "tree": "96ac5201b014eab937c1010f13f6f8e3221ddf89", "parents": [ "f3887fd5161dcfffe7a3b52c98dffc3fb62cbe49" ], "author": { "name": "Konstantin Tarasov", "email": "33369833+sdf-jkl@users.noreply.github.com", "time": "Tue May 12 16:12:12 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue May 12 20:12:12 2026 +0000" }, "message": "fix `date_part(\u0027isodow\u0027)` (#22116)\n\n## Which issue does this PR close?\n\n\u003c!--\nWe generally require a GitHub issue to be filed for all bug fixes and\nenhancements and this helps us generate change logs for our releases.\nYou can link an issue to this PR using the GitHub syntax. For example\n`Closes #123` indicates that this PR will close issue #123.\n--\u003e\n\n- Closes #22115.\n\n## Rationale for this change\n\nBug in `date_part` - `isodow` argument logic\n\u003c!--\nWhy are you proposing this change? If this is already explained clearly\nin the issue then this section is not needed.\nExplaining clearly why changes are proposed helps reviewers understand\nyour changes and offer better suggestions for fixes.\n--\u003e\n\n## What changes are included in this PR?\n\n- fix the logic\n- fix the docs\n\u003c!--\nThere is no need to duplicate the description in the issue here but it\nis sometimes worth providing a summary of the individual changes in this\nPR.\n--\u003e\n\n## Are these changes tested?\n\n- Yes, unit tests\n\u003c!--\nWe typically require tests for all PRs in order to:\n1. Prevent the code from being accidentally broken by subsequent changes\n2. Serve as another way to document the expected behavior of the code\n\nIf tests are not included in your PR, please explain why (for example,\nare they covered by existing tests)?\n--\u003e\n\n## Are there any user-facing changes?\n\n- Yes, this PR fixes the logic bug. Docs are updated.\n\u003c!--\nIf there are user-facing changes then we may require documentation to be\nupdated before approving the PR.\n--\u003e\n\n\u003c!--\nIf there are any breaking changes to public APIs, please add the `api\nchange` label.\n--\u003e\n\nOriginal-commit: apache/datafusion@f6a19953ba61e7e27c5781e453f54d3fd47b9b27\n" }, { "commit": "f3887fd5161dcfffe7a3b52c98dffc3fb62cbe49", "tree": "7e673534edda523e71aab79b2335d97399b36760", "parents": [ "122c6aad4cd7c04aaed971fc23b526b52762c1a3" ], "author": { "name": "Neil Conway", "email": "neil.conway@gmail.com", "time": "Mon May 11 22:40:55 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue May 12 02:40:55 2026 +0000" }, "message": "fix: Panic in Spark\u0027s `format_string` for illegal characters (#22077)\n\n## Which issue does this PR close?\n\n- Closes #22076.\n\n## Rationale for this change\n\n`format_string(\"%c\")` in `datafusion-spark` calls\n`char::from_u32.unwrap()` without checking for a `None` return, which\nleads to panicking on invalid inputs (e.g., negative values and illegal\nUnicode codepoints).\n\n## What changes are included in this PR?\n\n* Add error handling for `format_string(\"%c\")`\n* Rust unit tests\n* SLT end-to-end test\n\n## Are these changes tested?\n\nYes.\n\n## Are there any user-facing changes?\n\nNo.\n\nOriginal-commit: apache/datafusion@c94e75486b405fe2525218c135366674ca5f28ed\n" }, { "commit": "122c6aad4cd7c04aaed971fc23b526b52762c1a3", "tree": "cd1c5f20e048edb87aa422eb8bdad20d02902dcd", "parents": [ "aab37a41135f8ec99982de5c96d6123911c98df2" ], "author": { "name": "Neil Conway", "email": "neil.conway@gmail.com", "time": "Fri May 08 07:57:34 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri May 08 11:57:34 2026 +0000" }, "message": "perf: Use `NullBuffer::union_many` (#22070)\n\n## Which issue does this PR close?\n\n- Closes #21880.\n\n## Rationale for this change\n\nNow that `NullBuffer::union_many` has landed in arrow-rs 58.2, we can\nuse it to simplify and speedup various places in DataFusion that were\nunion\u0027ing together \u003e 2 null buffers. In such cases, `union_many` avoids\nthe intermediate allocations that are done by repeated `union` calls.\n\n## What changes are included in this PR?\n\n* Adopt `NullBuffer::union_many` throughout the codebase\n* Along the way, clean up `compute_null_mask` in the Spark crate to be\nmore readable and avoid needlessly padding scalars out to arrays just to\ncompute the null mask\n\n## Are these changes tested?\n\nYes, covered by existing tests.\n\n## Are there any user-facing changes?\n\nNo.\n\nOriginal-commit: apache/datafusion@53517af674e2540805dab0f688f324537ada3220\n" }, { "commit": "aab37a41135f8ec99982de5c96d6123911c98df2", "tree": "78fd611eceea9ac5527c50f39c2e1c5d106f9057", "parents": [ "9383ec081794b7e6639e6cd964e1b940911f1576" ], "author": { "name": "Krishna Sudarshan J", "email": "75199111+athlcode@users.noreply.github.com", "time": "Thu May 07 21:06:13 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu May 07 15:36:13 2026 +0000" }, "message": "feat: support spark compatible floor function (#21933)\n\n## Which issue does this PR close?\nPart of https://github.com/apache/datafusion/issues/15914\n\n- Closes https://github.com/apache/datafusion/pull/20860\n\u003c!--\nWe generally require a GitHub issue to be filed for all bug fixes and\nenhancements and this helps us generate change logs for our releases.\nYou can link an issue to this PR using the GitHub syntax. For example\n`Closes #123` indicates that this PR will close issue #123.\n--\u003e\n\n## Rationale for this change\n\nSpark compatible semantics for `floor` function\n\n\u003c!--\nWhy are you proposing this change? If this is already explained clearly\nin the issue then this section is not needed.\nExplaining clearly why changes are proposed helps reviewers understand\nyour changes and offer better suggestions for fixes.\n--\u003e\n\n## What changes are included in this PR?\n\n Implements SparkFloor UDF with Spark-compatible return types:\n - Float/Integer inputs -\u003e Int64\n - Decimal128(p, s) -\u003e Decimal128(p-s+1, 0)\n\n\n\u003c!--\nThere is no need to duplicate the description in the issue here but it\nis sometimes worth providing a summary of the individual changes in this\nPR.\n--\u003e\n\n## Are these changes tested?\n\nYes . Added unit tests and SLT tests \n\n\u003c!--\nWe typically require tests for all PRs in order to:\n1. Prevent the code from being accidentally broken by subsequent changes\n2. Serve as another way to document the expected behavior of the code\n\nIf tests are not included in your PR, please explain why (for example,\nare they covered by existing tests)?\n--\u003e\n\n## Are there any user-facing changes?\n\n\u003c!--\nIf there are user-facing changes then we may require documentation to be\nupdated before approving the PR.\n--\u003e\n\n\u003c!--\nIf there are any breaking changes to public APIs, please add the `api\nchange` label.\n--\u003e\n\n---------\n\nCo-authored-by: B Vadlamani \u003cfullcircle289@gmail.com\u003e\n\nOriginal-commit: apache/datafusion@015162afac7e8988f431d44bf17b531c21503785\n" } ], "next": "9383ec081794b7e6639e6cd964e1b940911f1576" }