)]}' { "log": [ { "commit": "878e5d3b8d373bca19671af43e49dfb1890f8da2", "tree": "b13492742a600916ae68b71e0e46ed8e2315d42b", "parents": [ "5febeb36e2cac4ad15aab51fe1671957a3ff7dbe" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Thu Oct 08 15:24:00 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 08 15:24:00 2026 -0400" }, "message": "build(deps): cargo update and roll up open dependabot PRs (#1787)\n\n* Cargo update\n\n* build(deps): roll up open dependabot updates\n\nBump GitHub Actions (taiki-e/install-action, astral-sh/setup-uv,\ngithub/codeql-action init+analyze) and uv.lock packages (virtualenv,\nurllib3, tornado, pyjwt, soupsieve). CodeQL init and analyze must be\nbumped together; separately they fail with a config version mismatch.\n\nCo-Authored-By: Claude Opus 5.5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 5.5 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "5febeb36e2cac4ad15aab51fe1671957a3ff7dbe", "tree": "9ec63ecc34abd000da007fb25d3fbb87ee6f15e1", "parents": [ "6c5d9fff14f0df84a657bee1d889bafc9e95a58b" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Thu Oct 08 13:44:29 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Oct 08 13:44:29 2026 -0400" }, "message": "ci: add ty type checker and fix existing type errors (#1786)\n\nAdd ty (pinned to 0.0.84, since it is pre-1.0) to the dev dependency\ngroup, configure it in pyproject.toml to check python/datafusion, and\nrun it in the lint-python CI job and as a local pre-commit hook.\ndatafusion._internal ships no stubs, and pandas/polars are optional\nTYPE_CHECKING-only imports, so they are allowed to stay unresolved.\n\nFix the diagnostics ty reported:\n\n- Make the internal udtf decorator helper require `name`, matching the\n public overloads. `@udtf()` previously passed None into Rust and\n failed with \"\u0027None\u0027 is not an instance of \u0027str\u0027\".\n- Give AggregateUDF.__init__ defaults matching its FFI overload, and add\n the same FFI overload to ScalarUDF and WindowUDF.\n- Add None/non-None overloads to expr_list_to_raw_expr_list and\n sort_list_to_raw_sort_list so callers that unpack the result type\n check.\n- Stop rebinding typed *args and parameters to values of other types.\n- Import warnings.deprecated behind a sys.version_info check and drop\n the unreachable importlib_metadata fallback.\n- Fix smaller annotation mismatches (LogicalPlan.__eq__,\n spark._coerce_i32, CSV file_compression_type).\n\nCo-authored-by: Claude Opus 5.5 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "6c5d9fff14f0df84a657bee1d889bafc9e95a58b", "tree": "e5e209fdb41ffa65c3e0a09b4c58d70c9508442f", "parents": [ "ceb2d8d1eca49985f52e6f8f85055179af3aff5d" ], "author": { "name": "Nuno Faria", "email": "nunofpfaria@gmail.com", "time": "Tue Oct 06 20:26:32 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Oct 06 15:26:32 2026 -0400" }, "message": "fix: Make default sort order nulls last (#1766)\n\n* fix: Make default sort order nulls last\n\n* Add entry to upgrade guide" }, { "commit": "ceb2d8d1eca49985f52e6f8f85055179af3aff5d", "tree": "8b58ca9eda8071182d429ef1037b82a05d766fe0", "parents": [ "a2ddb5531ac8443a4593101f8a801cb4a0088c01" ], "author": { "name": "Rodrigo Stachlewski Palma", "email": "86628053+Rodrigo-Palma@users.noreply.github.com", "time": "Mon Oct 05 10:31:17 2026 -0300" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Oct 05 09:31:17 2026 -0400" }, "message": "fix: forward write_options when write_parquet receives ParquetWriterOptions (#1761)\n\n`write_parquet` takes `write_options`, documents it and declares it in\nthe `@overload` for the `ParquetWriterOptions` form, but that branch\ncalls `write_parquet_with_options(path, compression)` and drops it. The\ndestination accepts the parameter, so everything in\n`DataFrameWriteOptions` (`partition_by`, `single_file_output`,\n`insert_operation`, `sort_by`) was silently ignored for that one\nspelling.\n\nMeasured with the same `DataFrameWriteOptions(partition_by\u003d\"part\")`:\n\n write_parquet(path, ParquetWriterOptions(), write_options\u003dwo)\n -\u003e [\u0027IDuOjvMa3pdEDotb_0.parquet\u0027] not partitioned\n write_parquet_with_options(path, ParquetWriterOptions(), write_options\u003dwo)\n -\u003e [\u0027part\u003da\u0027, \u0027part\u003db\u0027]\n write_parquet(path, \"zstd\", write_options\u003dwo)\n -\u003e [\u0027part\u003da\u0027, \u0027part\u003db\u0027]\n\nNo error and no warning: the files just land in the wrong layout.\n\nThe branch arrived in ef62fa89 (#1169) while `write_options` came\nearlier in #857, so the new delegation path was written without carrying\nthe existing parameter over. The same `if` refuses `compression_level`\nwith an explicit `ValueError`, which shows that arguments incompatible\nwith this branch get rejected on purpose; `write_options` was not\nrejected, only forgotten." }, { "commit": "a2ddb5531ac8443a4593101f8a801cb4a0088c01", "tree": "a116231e06dfa733ea85b13f109a7c2ebcbe831d", "parents": [ "5ef2856f5b02cddcd3d7d3559669d95ef181ccf4" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Sun Oct 04 11:55:41 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Oct 04 11:55:41 2026 -0400" }, "message": "feat: close upstream coverage gaps for DataFusion 55.1.0 (#1763)" }, { "commit": "5ef2856f5b02cddcd3d7d3559669d95ef181ccf4", "tree": "d14d29be7e737ae3be6c1e8f2243b86faed3deac", "parents": [ "22b4e4b1669982ec0bf72590886e13d60f486767" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Tue Sep 29 14:20:55 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 29 14:20:55 2026 -0400" }, "message": "ci: run CI and release workflows on release branches (#1774)\n\nci.yml and release.yml only matched main, so PRs to and merges into\nbranch-* release branches ran no build or tests.\n\nCo-authored-by: Claude Opus 5.5 \u003cnoreply@anthropic.com\u003e" }, { "commit": "22b4e4b1669982ec0bf72590886e13d60f486767", "tree": "f7f9b72949e5373032942c658614f1fef68a4d75", "parents": [ "90f870945437b7e6e33f507d2a59f214ab03a5bf" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri Sep 25 08:18:28 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 25 08:18:28 2026 -0400" }, "message": "Update version number to 55.0.0 (#1762)" }, { "commit": "90f870945437b7e6e33f507d2a59f214ab03a5bf", "tree": "81fd56c8f1c3940dc736b289464c73add2e4a3d4", "parents": [ "516d20df7efca398aeead904d683efc9f8812949" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Thu Sep 24 16:30:46 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 24 16:30:46 2026 -0400" }, "message": "chore: harden the extension API before it ships (#1759)\n\n* chore: harden the extension API before it ships\n\n`SessionContext.with_extensions` and `SessionExtensionComponents` landed\nin #1679 and have not shipped in a release yet. The bundle stack\n(#1738-#1741) reshapes them substantially, and a release would freeze\nthree surfaces in their current form.\n\n`PhysicalOptimizerRuleExportable` was defined in `datafusion.context` and\nnot exported from the package root, so `datafusion.context` would become\nits canonical import path. Move it to `datafusion.extensions` beside the\nrest of the `*Exportable` family, re-export it from `datafusion.context`\nso the old path keeps working, and export it from the package root. The\nmove brings it under `test_extension_api_has_a_doctest`, which drives off\n`extensions.__all__`, so it gains the example it was missing.\n\n`SessionExtensionComponents` was positionally constructible with two\nfields. The stack takes it to nine, three of them pair-shaped. Make\nconstruction keyword-only so every later field addition is additive; no\ncall site in the repository constructed it positionally. This is a new\nconvention rather than a backport, so it has to be applied forward to the\nstack as well.\n\nThe ordering that makes `with_extensions` transactional was stated in\nthree docstrings with no canonical home to point at. Record it under\n`ffi_internals_commit_order` in the contributor guide, and label the\nexisting \"Failure and rollback\" section `extension_bundles_transaction`,\nmatching the names the stack links to.\n\nNo released behaviour changes, so no `api change` label and no\nupgrade-guide entry.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: narrow the extension protocol surface\n\nThe capsule-getter protocols are annotations, never arguments: a bundle\nauthor constructs a SessionExtensionComponents but only ever names\nSessionComponentsExportable in a type hint. Exporting the hints from the\npackage root made this one family the exception among sixteen such\nprotocols, every other one of which is reached through its defining\nmodule.\n\nDrop PhysicalOptimizerRuleExportable, QueryPlannerExportable,\nSessionComponentsExportable, and SessionPlannerExportable from the root\n(__all__ 58 -\u003e 54), keeping SessionExtensionComponents, which is the one\nname a bundle constructs. The three bundle protocols are new in 55.0.0,\nso no import path is lost.\n\nPhysicalOptimizerRuleExportable shipped in 54.0.0 from datafusion.context,\nso its move is a break: context.py now imports it under TYPE_CHECKING only,\nand the upgrade guide records the new path. Nothing else changes for a rule\nauthor -- the protocol is structural and not runtime-checkable, and\nadd_physical_optimizer_rule is untouched.\n\nAlso removes three now-dead autoapi skip entries, repoints three doctests\nthat imported from the root, and fixes the add_physical_optimizer_rule\ncross-reference, which stopped resolving once the class left context.py.\n\ntest_extension_protocols_are_exported_together asserted the premise this\nreverses, so it goes. The five doctests in extensions.py already prove the\nclasses exist there, and SessionExtensionComponents\u0027 own docstring pins\nthe remaining root export.\n\nCo-Authored-By: Claude Opus 5 \u003cnoreply@anthropic.com\u003e\n\n* docs: state the commit-order invariant correctly\n\nThe ordering rule said step 4 \"cannot raise part-way through\", and the\ncomment in `with_extensions` said \"everything above is allowed to raise;\nthis is not\". Neither holds: `_install_extension_planner` runs\n`ffi_query_planner_from_pycapsule` before it calls\n`set_session_query_planner`, so the commit step has fallible work of its\nown.\n\nThe guarantee survives, because that import happens before the write. But\nthe passage is written as a rule for whoever adds the next component kind,\nand as phrased it asks them to preserve a property the code does not have.\nRestate it as what actually holds: every fallible operation, including the\nones inside the commit, completes before the first write.\n\nCo-Authored-By: Claude Opus 5 \u003cnoreply@anthropic.com\u003e\n\n* test: verify the physical optimizer rule docstring example\n\nThe `+SKIP` block in `PhysicalOptimizerRuleExportable` named\n`test_ffi_physical_optimizer_rule_runs_during_planning` as the test that\nruns it for real, but that test never reads the docstring. It is a\nseparately written test that happens to call the same two APIs, so it\ncatches a renamed method or module only by coincidence, and cannot see an\nedit to the docstring at all.\n\nAdd the mirror the convention actually asks for, in the shape of\n`test_with_extensions_docstring_example_still_runs`: parse the live\ndocstring, keep only the skipped statements, drop the skip, and run them.\nOnly `ctx` is supplied, because the skipped statements go on using the\ncontext the runnable block above them opened.\n\nVerified by mutation. Renaming the imported class in the docstring fails\nwith `NameError: name \u0027MyPhysicalOptimizerRule\u0027 is not defined`, and\ndeleting the block fails the `assert examples` guard rather than passing\nvacuously.\n\nCo-Authored-By: Claude Opus 5 \u003cnoreply@anthropic.com\u003e\n\n* refactor: keep bundle protocols out of datafusion.context\n\ndatafusion.context imported QueryPlannerExportable,\nSessionComponentsExportable and SessionPlannerExportable at runtime,\nso they were reachable as datafusion.context.* despite\ndatafusion.extensions being their one home. Move them under\nTYPE_CHECKING and route the runtime isinstance checks through a\nprivate _extensions module alias. The protocols are new in 55.0.0,\nso no released import path is dropped.\n\nAdd a test pinning that all four capsule-getter protocols live only\nin datafusion.extensions, and list PhysicalOptimizerRuleExportable\nin llms.txt.\n\nCo-Authored-By: Claude Opus 5.5 \u003cnoreply@anthropic.com\u003e\n\n* test: pin keyword-only construction; scope the rollback promise\n\nAdd a test that SessionExtensionComponents rejects positional\narguments, so dropping kw_only\u003dTrue fails the suite. Every existing\ncaller passes keywords and would stay green without it.\n\nDrop the absence assertions from the extension protocol import test.\nRe-exporting a name is additive and breaks no caller, so asserting a\nname is missing only adds friction for a later deliberate export.\nKeep the positive check that each protocol imports from\ndatafusion.extensions.\n\nThe contributor guide promised that a raising bundle leaves the\nsession as it was without the carve-out for writes a hook makes to\nthe context it is handed. Add it with a ref to the bundles guide,\nwhich is where the exception is explained.\n\nCo-Authored-By: Claude Opus 5.5 \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "516d20df7efca398aeead904d683efc9f8812949", "tree": "6e99219214a1b982268479c5b7320ae313dbe004", "parents": [ "00529673123d9cc2eed10c127f6a84d60ac5d1af" ], "author": { "name": "Kun Huang", "email": "konjac-h@outlook.com", "time": "Fri Sep 18 01:46:23 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 17 13:46:23 2026 -0400" }, "message": "Add copy button to code blocks in documents (#1744)" }, { "commit": "00529673123d9cc2eed10c127f6a84d60ac5d1af", "tree": "8306e7c1ec778eb0ccc9399879d36a74d67e804d", "parents": [ "c15236fedb5701fdc62970b97a7820d057eead2b" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Wed Sep 16 16:19:44 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 16 16:19:44 2026 -0400" }, "message": "Remove unneccessary copy trait (#1747)" }, { "commit": "c15236fedb5701fdc62970b97a7820d057eead2b", "tree": "abd61f9011f3e632757dc7ba316c5f29985bd01e", "parents": [ "2421da80d8d9f572a62d53642e40a1d867dcfbb3" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Wed Sep 16 14:54:37 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 16 14:54:37 2026 -0400" }, "message": "chore: remove dead indexed_field.rs (#1746)\n\nThe `pub mod indexed_field;` declaration and the PyGetIndexedField class\nregistration were dropped from crates/core/src/expr.rs in b5446efb (#728)\nwhen DataFusion 39 replaced Expr::GetIndexField with the FieldAccessor\ntrait, but the file stayed on disk. It has not been compiled since, and it\nimports GetIndexedField, which no longer exists upstream.\n\nThe nightly pre-commit rust-fmt hook formats files by path, so it keeps\nrewriting the orphan; stable `cargo fmt --check` in CI walks the module\ntree and never sees it. Deleting the file ends that churn.\n\nCloses #1745\n\nCo-authored-by: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "2421da80d8d9f572a62d53642e40a1d867dcfbb3", "tree": "2c66d5463be95a5ef46aea882b399a808a81aeab", "parents": [ "1d629a9d4710ae5b7ed85ea1ea3ad4a1b487c041" ], "author": { "name": "Yifan Chen", "email": "30335308+emecii@users.noreply.github.com", "time": "Wed Sep 16 09:02:33 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 16 12:02:33 2026 -0400" }, "message": "docs: explain S3 object-store configuration for SQL (#1716)\n\n* docs: explain S3 object-store configuration for SQL\n\nSigned-off-by: Yifan Chen \u003cemecii23@gmail.com\u003e\n\n* docs: demonstrate SQL reads through registered object stores\n\n* docs: simplify S3 SQL example wording\n\n---------\n\nSigned-off-by: Yifan Chen \u003cemecii23@gmail.com\u003e" }, { "commit": "1d629a9d4710ae5b7ed85ea1ea3ad4a1b487c041", "tree": "609476c26a915d848d31380701a1146a6d449f5f", "parents": [ "41eadc401c5182887da7bf3c201d03eafd8b0b57" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Mon Sep 14 15:13:26 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 14 15:13:26 2026 -0400" }, "message": "Report physical partitioning and resolve two panics escaping as PanicException (#1720)\n\n* Report physical partitioning, and stop two panics escaping as panics\n\nGroundwork for a multi-library distributed-execution example. Each item here\nis something that example needs and cannot get today.\n\n`ExecutionPlan.output_partitioning` is new. `partition_count` already existed\nbut discards everything except the count, so a driver deciding how to split\nwork across workers could not tell hash-distributed output from merely counted\noutput, nor read the hash keys. It returns a `PhysicalPartitioning`, named to\nkeep it distinct from `datafusion.expr.Partitioning` — that one is the logical\npartitioning `repartition_by_hash` takes as a request, this one is what a built\nplan does. Physical expressions have no Python representation, so the hash keys\nare returned in their displayed form.\n\n`SessionContext.execute` now bounds-checks the partition index. The plan\u0027s\nleaves index their partition vector directly, so an out-of-range index reached\n`MemorySourceConfig` and panicked; the panic was caught as a tokio `JoinError`\nand arrived as `index out of bounds: the len is 2 but the index is 5`, naming\nneither the plan nor the index the caller passed.\n\n`SessionConfig.set` no longer routes through `SessionConfig::set_str`, which\nunwraps. An unknown namespace — `datafusion.runtime.*`, or a config extension\nnot yet installed — aborted with a `PanicException`, which derives from\n`BaseException` and so escapes `except Exception`. `information_schema.\ndf_settings` lists keys in both categories, so replaying settings onto a worker\nhit this first.\n\nTwo docstrings on `ExecutionPlan` claimed that a table registered from record\nbatches cannot be serialized. That is true of `LogicalPlan`, whose\n`try_encode_table_provider` has no arm for one, and false of the physical\nlayer, which inlines the batches: verified by decoding on a context sharing\nnothing with the encoder and executing. A test pins it, since it is what lets a\nworker run a plan the driver encoded.\n\nAlso documents, rather than fixes, the `ForeignExecutionPlan` arm in the\nexample provider\u0027s physical codec. It claims every other library\u0027s nodes, which\nthe extension guide tells authors not to do — but it is load-bearing:\n`EnsureCooperative` runs during a foreign planner\u0027s `create_physical_plan` and\nhands the library back a `ForeignExecutionPlan` wrapping the host\u0027s\n`CooperativeExec`, which has no reachable `try_to_proto`. Narrowing the arm\nmakes 31 of the 51 tests in the query-planner example fail, all on that node.\nThe comment now says so, and says a planner that controls its own physical\noptimizer rules needs no such arm.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Name the execute partition index for what it is, trim the upgrade guide\n\n`SessionContext.execute` took its second argument as `partitions`, which reads\nas a count when it is a single partition index. Rename it to `partition` and\ngive the method a real docstring with a doctest covering both a full sweep over\n`partition_count` and the out-of-range `ValueError`. Every call site in the\nrepo, docs, and examples passes it positionally, so add a short note to the\nupgrade guide for anyone passing it by keyword.\n\nDrop two changelog-shaped sections from the upgrade guide. `output_partitioning`\nis additive and `SessionContext.execute` / `SessionConfig.set` only trade a\npanic for a raise, so neither asks the reader to change anything. The\n`with_extensions` recommendation goes for the same reason; it is advice, and the\nextension guide already carries it under `extension_bundles`.\n\nAlso remove the comment above the partition range check. It explained the check\nby way of a `MemorySourceConfig` panic, which reads as a complaint about\nDataFusion\u0027s leaves. The index arrives straight from Python and no planner has\nseen it, so validating it needs no more justification than any other argument\ncheck at the boundary.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Link the upstream FFI issue, stop teaching execute by counter-example\n\nThe `ForeignExecutionPlan` arm\u0027s comment explained the symptom but left the\nreader no way to find out whether the workaround is still needed. Name the\nupstream umbrella issue, apache/datafusion#25152, and the cascade behind it:\n`FFI_PlanProperties` carries no `scheduling_type`, so `EnsureCooperative`\nreads every foreign leaf as non-cooperative and wraps it, and the resulting\n`ForeignExecutionPlan` then cannot serialize itself. Fixing either half\nretires the arm.\n\nDrop the out-of-range call from `SessionContext.execute`\u0027s doctest. A\ndocstring example shows a reader how to use the method, and this one put a\nwrong call in front of them; the message it asserted is already pinned by\n`test_execute_rejects_an_out_of_range_partition`. The `Raises:` section is\nthe right home for that behaviour, so complete it: a negative or oversized\nindex raises `OverflowError` from the `usize` conversion, not the\n`ValueError` the bounds check produces.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Document what SessionConfig.set actually does\n\nThe docstring was wrong in three ways at once, and the panic fix earlier in\nthis branch is what made it worth opening: it promised \"a new SessionConfig\nobject\" when the method mutates in place and returns self, it mis-indented\nthe `Args` entries so Sphinx rendered them as body text rather than a field\nlist, and it had no `Raises` at all -- so the one behaviour this branch\nchanged, an unknown key raising instead of aborting the interpreter, was\ndocumented only in the Rust source that no user reads.\n\nGive it a truthful `Returns`, a `Raises` covering both an unknown key and an\nunparsable value, and a doctest that reads the option back out through\n`information_schema.df_settings`.\n\nThe `datafusion.runtime.*` trap goes to `configuration.md`, which is where a\nreader is when they need it: those keys appear in `df_settings` but are not\nsettable, so replaying that table verbatim onto a worker fails on the first\nsuch row. The docstring states it in one sentence and points at the guide.\nThat page also now records that the `with_*` methods modify in place, which\nis what makes the chained style it already demonstrates work.\n\nNote the same false `Returns` line appears on every other `with_*` method on\nthis class; correcting those is a separate sweep, not this branch\u0027s business.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Raise ValueError from SessionConfig.set, not a bare Exception\n\nTrading a `PanicException` for an untyped `Exception` only half-fixed the\nproblem: a caller replaying `information_schema.df_settings` still could not\ncatch the failure without swallowing every other error this crate raises,\nand was left matching on message text. A rejected key or an unparsable value\nis an argument error, so it should raise `ValueError` -- the same thing an\nout-of-range partition index gets from `execute` two methods away.\n\nMap through `from_datafusion_error`, which already produces `PyValueError`\nand which `SessionContext.sql` already uses in this file, rather than\npropagating a `PyDataFusionError` and taking its blanket conversion. That\nconversion is deliberately untouched: reclassifying every error out of this\ncrate is a much wider change with its own compatibility story.\n\nThe test can now assert something. It previously ended in\n`assert isinstance(excinfo.value, Exception)` under a\n`pytest.raises(Exception, ...)` that had already proven exactly that, so the\nline could never fail. Assert the type instead, and add a case for a known\nkey with a value of the wrong type, which reaches the same path by a route a\nsettings-replay loop is just as likely to take.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Point output_partitioning at a page that discusses partitioning\n\nThe docstring referred the reader to `distributed_query_engines`, which is\nwrong twice over: that page\u0027s premise is that you do *not* partition by hand\nbecause the engine does it for you, and it documents work that is not yet\nusable from datafusion-python. A reader following the link to find out what\na scheme means landed on a status page for a different road.\n\nNothing under docs/source/ discussed plan partitioning from the caller\u0027s\nside, so give the claim a home on the page that already tells readers to\ncall `repartition` and `repartition_by_hash` to keep their cores busy --\nand that, until now, gave them no way to check whether it worked.\n\nThe new section is worth more than a pointer. `repartition_by_hash(col(\"a\"),\nnum\u003d8)` followed by an aggregation reports `Hash([a@0], 16)`: the optimizer\ndiscarded the requested repartition and inserted its own at\n`target_partitions`, so neither the scheme nor the count is what was asked\nfor. Drop the aggregation and the repartition vanishes entirely, leaving\n`UnknownPartitioning` over the source\u0027s partition count. Both were verified\nagainst a Parquet source, and both are invisible without this accessor,\nwhich is the concrete form of the have-versus-ask distinction the class\ndocstring asserts.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Drop PhysicalPartitioning\u0027s unused inbound conversion\n\n`From\u003cPyPhysicalPartitioning\u003e for Partitioning` had no callers. The type is\na read-only report of what a built plan does, nothing accepts a partitioning\nas an argument, and a Rust caller wanting the `Partitioning` reads it off the\nplan, so there is no inbound direction to support.\n\nRemoving the impl alone left a deprecation warning: pyo3 auto-derives\n`FromPyObject` for a `#[pyclass]` that implements `Clone` and now wants the\nchoice made explicitly. Omission is not the way to decline it, so say\n`skip_from_py_object`, which is what the example crate\u0027s providers already\nuse. Clippy is clean again, and `--all-targets` keeps it that way.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Cover RoundRobinBatch, stop claiming a plan can report Range\n\nThe `scheme` docstring listed four values as though a plan could report any\nof them. Two of the four needed opposite corrections.\n\n`RoundRobinBatch` is reachable, and now tested. It is easy to miss because\nit never survives at the root: the optimizer inserts one only above a source\nwith fewer partitions than `target_partitions` and CPU work above it to\nparallelize, so a single-file Parquet scan under a filter and a grouped\naggregate produces `RepartitionExec: partitioning\u003dRoundRobinBatch(8)` three\nlevels down. The test walks `children` and asserts all three schemes in that\ntree, which also pins that the accessor reads each node\u0027s own partitioning\nrather than the root\u0027s.\n\n`Range` is the other way: it is in the upstream enum, but nothing constructs\none in a physical plan. `RangePartitioning` says optimizer and execution\nsupport is deliberately unimplemented, per\napache/datafusion#22395. Say so, and say why the match arm exists anyway --\nit keeps this getter compiling when that support lands, and dropping it\nwould make the match non-exhaustive.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Tighten three small things around output_partitioning\n\n`scheme` returns one of exactly four strings, so annotate it `Literal`\nrather than `str`. The promise is safe to make: the Rust getter matches\nexhaustively over `Partitioning`, so a new upstream variant is a compile\nerror here before it can be a lie in the type.\n\nBind `output_partitioning` once in the scan half of its test. It was read\nthree times, and each read clones the `Partitioning` and builds a fresh\nwrapper.\n\nValidate the partition index in `execute` before building the\n`TaskContext`, not after. Nothing observable changes; the rejected call just\nstops doing setup work it is about to throw away.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Stop SessionConfig\u0027s constructor aborting on an unknown key\n\n`SessionConfig.set` was fixed to raise instead of panicking, but the\nconstructor still routed its dictionary through `SessionConfig::set`, which\nforwards to `set_str` and unwraps. So\n`SessionConfig({\"datafusion.runtime.memory_limit\": \"unlimited\"})` aborted as a\n`PanicException` -- a `BaseException`, past any `except Exception` -- while the\nsame key passed to `set` raised `ValueError`.\n\nThe constructor is the likelier of the two to meet a bad key: it takes a\n`dict[str, str]`, which is the shape a replayed set of settings arrives in, and\n`information_schema.df_settings` lists eight `datafusion.runtime.*` rows that\ncannot be set from a session config at all.\n\nRoute each entry through `options_mut().set` and map with\n`from_datafusion_error`, so both routes reject the same keys the same way. The\n`ScalarValue::Utf8` wrapper went with it: the parameter is already\n`HashMap\u003cString, String\u003e`, and upstream only called `to_string()` back on it.\n\nWhich entry a dictionary with several bad keys reports is unspecified, since\n`HashMap` iteration order is arbitrary. Documented rather than sorted -- a\ncaller fixes the reported key and runs again.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Say where Range partitioning actually comes from\n\nTwo docstrings claimed no plan reports `Partitioning::Range` because upstream\nsupport was unimplemented. Neither half holds for DataFusion 55:\n`repartition/mod.rs` routes range partitioning through `RangeExpr`, and\n`physical_planner.rs` has a test asserting a planned `RepartitionExec` reports\nit. The comment\u0027s reasoning was also inverted -- the match arm compiles today\nbecause the variant exists today, not in anticipation of it landing.\n\nWhat is true is narrower: nothing in this package\u0027s own API asks for one.\n`repartition` requests round-robin, `repartition_by_hash` requests hash, and\nSQL has no range-repartition syntax. But a plan need not have been built here.\n`datafusion-proto` encodes and decodes physical range partitioning and\n`datafusion-ffi` carries it in both directions, so `ExecutionPlan.from_bytes`\ncan return a plan reporting it, as can an extension library\u0027s query planner.\n\nState that where each reader is: the reachability in the guide beside the rest\nof the scheme discussion, one sentence and a `:ref:` in the docstring. Also\nnote that `hash_expressions` is `None` for `Range`, which leaves the ordering\nand split points reachable only through `repr`.\n\nNo test: reaching `Range` from Python needs a plan built in Rust, and a Rust\ntest here would never run in CI.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Describe expr.Partitioning as what it is, not as an argument\n\nBoth docstrings introducing `PhysicalPartitioning` distinguished it from\n`datafusion.expr.Partitioning` by calling that one \"the partitioning\n`repartition_by_hash` asks for\". No method takes one: `repartition` takes a\ncount and `repartition_by_hash` takes expressions and a count. The type cannot\nbe constructed from Python at all -- `Partitioning()` raises `TypeError` -- has\nno public members, and is only ever handed back by\n`Repartition.partitioning_scheme()`.\n\nDescribe it that way instead: the logical partitioning a `Repartition` node\nrecords. The request-versus-result contrast the docstrings were reaching for is\nreal and worth drawing, so keep it, but attach it to the node that holds the\nrequest rather than to a parameter that does not exist. The Rust comment also\nnotes that the two enums differ, since the logical one has `DistributeBy` and\nno `UnknownPartitioning`.\n\nPin the contrast with a test rather than only asserting it in prose. A\n`repartition_by_hash(num\u003d8)` with nothing above it to consume the redistribution\nis dropped by the optimizer, so the request records Hash into 8 while the built\nplan reports `UnknownPartitioning(2)` -- disagreeing on both scheme and count,\nwhich is the reason the two types stay separate.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Give PhysicalPartitioning equality, and tighten four small things\n\nDrop the guide sentence claiming every config method mutates in place and\nreturns itself. It is true, but every `with_*` docstring still promises \"a new\n`SessionConfig` object\", and a guide that contradicts the API docs it links to\nis worse than a guide that stays quiet. Correcting fifteen docstrings is its\nown change.\n\n`PhysicalPartitioning` gains `__eq__` and `__hash__`, computed structurally\nfrom scheme, partition count and hash expressions. Deliberately not delegated\nto `Partitioning`\u0027s `PartialEq`, whose match lists no `UnknownPartitioning`\narm and so falls through to `false`: two identical `UnknownPartitioning(2)`\nvalues are unequal there. That is defensible for deciding whether a\npartitioning satisfies a distribution requirement, but a non-reflexive\n`__eq__` would be a trap in Python, and `UnknownPartitioning` is what an\nordinary file scan reports. `__hash__` comes along so the class stays usable\nin a set.\n\n`test_output_partitioning_reports_round_robin` asserted set equality over\nevery scheme in the tree, pinning optimizer output that is not the property\nunder test. Membership instead, and 50 rows rather than 2000 -- the round\nrobin appears either way, so the larger file bought nothing.\n\nThe PyO3 parameter is now `partition` to match the wrapper, so\n`ctx.execute(plan, partition\u003d0)` works on the internal binding too, and the\ntest covers the keyword form alongside the positional one.\n\n`from_bytes` had its false memory-table sentence removed but got nothing back,\nso it no longer said that a decoding session need share nothing with the\nencoder -- the property that makes it useful. Restored from the reader\u0027s side.\n\nAlso cover the `OverflowError` a negative index raises, which was documented\non `execute` but never exercised.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* File the SessionConfig tests with the other SessionConfig tests\n\nThe two `SessionConfig.set` tests went into `test_plans.py` because they were\ncommitted alongside the `execute` bounds check, not because they have anything\nto do with plans. `test_context.py` is where `SessionConfig` construction is\nalready covered, and it now also holds the three constructor tests for the\nother half of the same panic defect.\n\nMove them there, ahead of the constructor cases so the method they refer back\nto is read first, and fold the duplicated note about `PanicException` deriving\nfrom `BaseException` into the first of the five. `test_plans.py` keeps its\n`SessionConfig` import for the `with_target_partitions` calls in the\npartitioning tests. No assertion changed.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Import SessionContext in the SessionConfig constructor doctest\n\nThe example used SessionContext but only imported SessionConfig, so it\npassed under --doctest-modules yet failed when copy-pasted.\n\nCo-Authored-By: Claude Fable 5 \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "41eadc401c5182887da7bf3c201d03eafd8b0b57", "tree": "ad90730b55419101697137c77b4f250241c19eb4", "parents": [ "59c1fd1e93705d982b5bb4e9ce37da60a467fff8" ], "author": { "name": "BharatDeva", "email": "bharatdevagir@gmail.com", "time": "Mon Sep 14 12:35:35 2026 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 14 13:35:35 2026 -0400" }, "message": "fix: remove todo from indexed field key (#1667)\n\nCo-authored-by: BharatDeva \u003c278575558+BharatDeva@users.noreply.github.com\u003e" }, { "commit": "59c1fd1e93705d982b5bb4e9ce37da60a467fff8", "tree": "1e7ccf838ac1faf9c2bce3486412474e5f4d5c32", "parents": [ "eeaddab1217e34453eba2641f7c27d695a70f13d" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri Sep 11 18:52:44 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 11 18:52:44 2026 -0400" }, "message": "build(deps): consolidate open Dependabot dependency updates (#1734)\n\nApplies 21 open Dependabot PRs in a single change so CI runs once over\nthe combined result rather than 21 times over lockfiles that each\ninvalidate the next.\n\nCargo (Cargo.lock only; workspace ranges already admit these):\narrow/arrow-array/arrow-schema/arrow-select 59.2.0 -\u003e 59.3.0, uuid\n1.23.3 -\u003e 1.26.1, futures 0.3.32 -\u003e 0.3.34, log 0.4.32 -\u003e 0.4.34,\nasync-trait 0.1.89 -\u003e 0.1.92, serde_json 1.0.150 -\u003e 1.0.151, tokio\n1.52.3 -\u003e 1.53.1, quinn-proto 0.11.14 -\u003e 0.11.16.\n\nPython (uv.lock): tornado 6.5.7 -\u003e 6.5.8, cryptography 49.0.0 -\u003e\n50.0.0, setuptools 82.0.1 -\u003e 83.0.0.\n\nActions: actions/checkout v6 -\u003e v7, actions/setup-python v5/v6 -\u003e v7,\nactions/cache v5 -\u003e v6, taiki-e/install-action v2 -\u003e v2.85.5,\nastral-sh/setup-uv 8.2.0 -\u003e 10.0.1, github/codeql-action init and\nanalyze 4.36.2 -\u003e 4.37.9.\n\nThe codeql-action bumps only pass together: the action rejects a run\nwhere init and analyze are on different versions, so #1700 and #1699\neach fail alone with \"Loaded a configuration file for version\n\u00274.37.9\u0027, but running version \u00274.36.2\u0027\".\n\nThe SHA-pinned actions/checkout in codeql.yml goes to the v7 commit\n3d3c42e5aac5ba805825da76410c181273ba90b1 rather than the stale v6\ncommit #1606 proposed, keeping the pin on the same major version as\nthe tag refs elsewhere.\n\nExcludes #1652 (datafusion 54.0.0 -\u003e 54.1.0), obsolete since main\nmoved to DataFusion 55.1.0.\n\nCo-authored-by: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "eeaddab1217e34453eba2641f7c27d695a70f13d", "tree": "3b523523d879c4df0e9a1c677d7d05e883881311", "parents": [ "7022baaed6d780d0c203b735ec63f6d8361f9969" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri Sep 11 13:18:40 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 11 13:18:40 2026 -0400" }, "message": "Bump DataFusion version to 55.1.0 (#1731)\n\n* Bump DataFusion version to 55.1.0 and remove crates.io patch\n\n* taplo format" }, { "commit": "7022baaed6d780d0c203b735ec63f6d8361f9969", "tree": "b2665cf34c2cdfcd92d7245e839c5e3b7c9270dc", "parents": [ "b6c6f5b94afad5a2ed37cf8bfa4cbf2ed9aa3f9c" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Thu Sep 10 08:59:57 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 08:59:57 2026 -0400" }, "message": "Add method on SessionContext to add all extensions from one library (#1679)\n\n* Add atomic SessionContext.with_extensions API\n\nInstalling FFI extension codecs and query planners by chaining the\nexisting with_* methods can bind task-context providers to intermediate\ncontexts that are later collected, breaking the weak provider reference\nover the FFI boundary. with_extensions creates one destination context,\npasses it to each extension factory so components bind to that exact\ncontext, and installs everything in a single state write.\n\nCo-Authored-By: Claude Fable 5 \u003cnoreply@anthropic.com\u003e\n\n* Add extension-bundle example and with_extensions FFI tests\n\nMyPlannerExtension in the query-planner example crate implements the\n__datafusion_session_extension__ protocol from Rust: it extracts the\ndestination context\u0027s task-context provider, binds fresh observing\ncodecs and a planner to it, and returns SessionExtensionComponents. Its\ncodecs record the max_rows config value resolved through the weak\nprovider, letting tests prove the provider targets the returned context\nrather than the source. Documents with_extensions as the preferred API\nin the FFI guide.\n\nCo-Authored-By: Claude Fable 5 \u003cnoreply@anthropic.com\u003e\n\n* Document and test the context-outlives-DataFrame contract\n\nA DataFrame does not keep its SessionContext alive. FFI components hold\na weak task-context provider, so operations that reach an FFI codec\nafter the context is collected fail with a clean out-of-scope error\nrather than crashing. Lock that behavior in with a test and document\nthe ownership contract in the FFI guide and with_extensions docstring.\n\nCo-Authored-By: Claude Fable 5 \u003cnoreply@anthropic.com\u003e\n\n* Skip private internal methods in wrapper coverage test\n\nSingle-underscore methods on internal pyo3 classes (such as\nSessionContext._install_extensions) are private support methods for the\nPython wrappers and do not require a public wrapper.\n\nCo-Authored-By: Claude Fable 5 \u003cnoreply@anthropic.com\u003e\n\n* Test planner rebinding and codec ids in with_extensions\n\nA codec-only bundle installed on a context that already holds an FFI\nplanner must rebind that planner to the new chains, so the planner\ndecodes through the bundle\u0027s codecs.\n\nCodec ids are derived from the exporting class, so two bundles shipping\nthe same codec class collide and the install is refused. Declaring\n__datafusion_codec_id__ on the object a bundle hands over resolves it,\nand both chains then install.\n\nCo-Authored-By: Claude Fable 5 \u003cnoreply@anthropic.com\u003e\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Fix duplicate attribute docs in SessionExtensionComponents\n\nThe docs build runs Sphinx with --fail-on-warning. SessionExtensionComponents\ndocumented its fields in both a napoleon `Attributes:` section and the dataclass\nclass-body annotations, so autoapi emitted each field twice and the build failed\nwith six \"duplicate object description\" warnings.\n\nMove each field\u0027s description to a per-field docstring under its annotation so\nautoapi renders exactly one entry per field.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Move session extension types into datafusion.extensions\n\nQueryPlannerExportable, SessionExtensionComponents, and\nSessionExtensionExportable describe how an extension library plugs into a\nsession, not how a SessionContext behaves. Give them their own module so\ncontext.py does not keep absorbing the extension surface as it grows.\n\nextensions.py imports SessionContext, the codec protocols, and CapsuleType\nunder TYPE_CHECKING only, so context.py can import from it at runtime\nwithout a cycle. All three names remain importable from datafusion and\ndatafusion.context; QueryPlannerExportable stays out of the top-level\n__all__ as before.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Run the with_extensions docstring example in CI\n\nThe example was marked `+SKIP` because the main suite has no built FFI\nextension to import, which is exactly how such an example rots. Parse the\nstatements out of the live docstring in the query-planner example suite,\ndrop the skip, and execute each one against a real extension bundle. Only\nnames are redirected: `my_extension` resolves to a stand-in combining this\nrepository\u0027s provider codecs and planner, and `SessionContext` supplies the\nconfig that planner reads. A renamed method, a changed signature, or a wrong\nexpected output now fails CI, which already runs this suite.\n\nAlso drop the `extensions` Args entry\u0027s restatement of the type hint and\nsay instead what the hint does not: install order is chain order.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Share the session in with_extensions instead of forking it\n\n`_derive_for_extensions` minted a new `Arc\u003cSessionContext\u003e` via\n`new_with_state(self.ctx.state())`. Every other `with_*` method shares\n`Arc::clone(\u0026self.ctx)`, and `new_with_state` carries the session id over, so\n`with_extensions` returned a second live session claiming the same\n`session_id()` as the source while holding independent `SessionState`.\nConfiguration and the function registry diverged, catalogs stayed shared, and\nboth handles reported the same `__datafusion_codec_id__` — which is\n`session:\u003csession_id\u003e` and exists precisely to distinguish codec chains, so\ninstalling both on a third session was refused as a duplicate id.\n\nThe fork also bought nothing. It was introduced to keep components from binding\nto an intermediate context that could be collected, but there is one\n`Arc\u003cSessionContext\u003e` per session, so no such intermediate exists; deriving one\nis what creates the hazard. Rule 6 of the ffi-capsule-protocol skill already\nsaid to mutate `SessionState` in place rather than derive a replacement.\n\nDelete `_derive_for_extensions` and hand the receiver to the extension\nfactories. `_install_extensions` already returned a handle sharing\n`Arc::clone(\u0026slf.borrow().ctx)`, so removing the fork upstream of it is the\nwhole change. Atomicity is unaffected: both codec chains are built as locals and\nstate is written exactly once, at the end, in `set_session_query_planner`.\n\nReplace `test_with_extensions_provider_targets_returned_context`, which is\nvacuous once the session is shared, with\n`test_with_extensions_shares_the_session_with_the_source`. It asserts matching\nsession ids and that a `SET` issued through the source after installation is\nvisible to the provider the bundle bound. Reintroducing the fork fails it.\n\nUpdate the prose that described the fork-era design: the `with_extensions`\ndocstring and `SessionExtensionComponents` / `SessionExtensionExportable` in\n`datafusion.extensions`, the `with_extensions` and \"What a derived context\nshares\" sections of the FFI guide, the query planner example\u0027s README and\n`extension.rs` comments, and two test docstrings. Note the shared-session\nmechanism in Rule 6 of the skill, since `with_extensions` is where it is\neasiest to get wrong.\n\n`enable_url_table` is once again the only method that mints a second\n`Arc\u003cSessionContext\u003e` for a session; its comment, the FFI guide, and the skill\nnow also record that it forks state while keeping the session id, tracked as a\nbug in #1708.\n\nAlso add the missing doctest to `SessionExtensionComponents` and a pointer to\n`with_extensions` from the upgrade guide, which described only the low-level\ninstall path.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Name a bundle\u0027s bare capsules after the bundle\n\nA codec handed to `with_extensions` as a bare `PyCapsule` fell through to\n`anon:\u003cuuid4\u003e`, an id private to the session that installed it. Plans written\nthrough it are undecodable anywhere else, and `with_extensions` accepts no\n`codec_id\u003d` to override that — so the workaround was to wrap the capsule in an\nobject declaring `__datafusion_codec_id__`, which nothing documented. A\ndistributed engine has to decode its plans in another process, so the shape it\nwould naturally ship — a Rust bundle handing over capsules, as\n`MyPlannerExtension` does — was the one shape that could not work.\n\nThe bundle is the stable name that was missing. It is a plain Python object, so\nits `module.QualName` is library-owned and exactly as stable across processes as\nan exporting codec class\u0027s, which arm 3 of `derive_codec_id` already trusts. The\ncapsule was unnameable only because a capsule carries no type of its own, not\nbecause nothing stable was in reach.\n\nResolve a capsule\u0027s id through the contributing bundle, using `derive_codec_id`\nitself so the bundle inherits the same `__datafusion_codec_id__` escape hatch\nagainst a class rename. The fallback applies only where randomness would have:\nan id declared on the handed-over object, or that object\u0027s own class, still\nwins, so an extension can name a codec directly.\n\nTwo bare capsules of one kind from one bundle collide and are refused. Numbering\nthem by position would be exactly the id `codec.rs` rejects for `anon:` — one\nanother library can mint the same value from — and would break stored plans the\nfirst time the bundle reordered what it returns.\n\n`resolve_codec_id` gains the bundle argument, `_install_extensions` takes\n(codec, bundle) pairs, and the collision message now names both routes to a\ndistinct identity; it previously offered only `codec_id\u003d`, which is unreachable\nfrom `with_extensions`.\n\nCovered in `python/tests/test_context.py`, which reaches every arm without a\nbuilt extension library: the bundle-derived name, an extension pinning its own\nid, an id on the handed-over object winning, an exporting object keeping its\nown, and the two-capsule collision. The cross-FFI case is pinned in the query\nplanner example, where a Rust bundle\u0027s capsules must report\n`datafusion_ffi_query_planner_example.MyPlannerExtension` and no id may be\n`anon:`.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Stop claiming _install_extensions always writes state\n\nThe doc comment said \"the final state is written through this context\u0027s own\n`state_ref()`\", which overstates it. `set_session_query_planner` returns early\nwhen there is no planner to bind, and the codec chains live on the returned\n`PySessionContext` fields rather than in `SessionState` — so a codec-only\ninstall onto a session with no FFI planner writes nothing.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Tidy the loose ends from review of with_extensions\n\nMark `SessionExtensionExportable` `@runtime_checkable` and have\n`with_extensions` check it with `isinstance` rather than `hasattr`, so the\nannotation and the runtime check are the same statement, and callers can ask\nthe question too. Covered by a doctest on the protocol.\n\nReplace the leading-underscore skip in `test_wrapper_coverage` with a named\nallowlist. The pattern also excused `DataFrame._repr_html_`, which a wrapper\ndoes have to provide, so a two-method need was weakening coverage for every\nprivate name. Removing `_install_extensions` from the allowlist fails the test,\nso the entry is load-bearing rather than decorative.\n\nSay in `_CodecOnlyExtension` that retaining the context is what the protocol\ntells real extensions not to do, and that it is kept only so a test can assert\nwhich context the factory was handed.\n\nLet the docstring-example shim in the query planner example accept a config\npositionally, the way the real constructor does. Editing the docstring to\n`SessionContext(config)` now fails as a doctest diff rather than as a\n`TypeError` inside the harness.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Require with_extensions codecs to be objects, not capsules\n\nReverses \"Name a bundle\u0027s bare capsules after the bundle\". Deriving a\ncapsule\u0027s id from the bundle that contributed it reads the identity off the\nwrong object: the bundle is whatever the caller passed to with_extensions, so\nan application that packages several libraries as one bundle of its own\nstamps its identity onto the inner libraries\u0027 codecs. Their pinned\n__datafusion_codec_id__ is discarded and there is nothing the inner library\ncan do about it, since its object never reaches _install_extensions. Nothing\nfails at install time; the mismatch surfaces as an undecodable plan in the\nprocess that reads it, naming an id nobody wrote in source.\n\nSo with_extensions now refuses a bare capsule and names the getter to\nimplement. An id read off the handed-over object is composition-stable by\nconstruction, which the new tests pin at both layers. This also decouples a\ncodec\u0027s wire identity from the bundle\u0027s Python class name, which is what\n__datafusion_codec_id__ exists for, and closes the case where a bundle built\nby a factory function contributed a wire id containing \"\u003clocals\u003e\".\n\nThe low-level methods keep accepting capsules: they take codec_id\u003d, so the\nrandom anon: arm still has an escape hatch. Query planners are unaffected,\ncarrying no wire id.\n\nMyPlannerExtension gains BundledLogicalCodec and BundledPhysicalCodec, small\npyclasses holding the bound FFI codec and declaring pinned ids, as the\nreference shape for a library whose plans leave the process.\n\nAlso, unrelated to the above but adjacent in the docs: with_extensions never\nsaid that a bundle-supplied planner replaces an installed one rather than\nlayering, and the SessionExtensionComponents example that showed a codec was\nfully skipped with undefined names. Both fixed.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Install extension codecs and planners in two phases\n\nA session chains many codecs and dispatches between them by id, so codecs\naccumulate and their order does not affect decoding. A session holds exactly\none query planner, so planners cannot accumulate — they compose by nesting,\neach wrapping the one before it. Collecting both from a single hook forced\nwith_extensions to refuse more than one planner per call, because every factory\nran before anything was installed and so no bundle could see another bundle\u0027s\nplanner to wrap it. Two libraries that each ship a planner could not be\ninstalled together at all, and splitting them across two calls silently\ndiscarded the first.\n\nCodecs now come from __datafusion_session_extension__ and planners from a new\n__datafusion_session_planner__(ctx, fallback), which runs once per bundle in\nargument order after every codec is installed. Each receives the planner built\nso far; wrapping it nests this bundle outside the previous one, so the last\nbundle listed ends up outermost. A bundle implements either hook or both, which\nalso lets a library that ships only an optimizing planner stop returning empty\ncomponents. SessionExtensionComponents loses its query_planner field.\n\nRunning the planner hooks after every codec is installed is what makes a nested\nplanner safe. The rebuild that follows a later codec install reaches only the\noutermost layer, so a fallback captured against a partial chain would stay\nstale; there is now no \"afterwards\" within a call.\n\nAtomicity is unchanged. _install_extension_codecs writes nothing — the chains\nbelong to the returned handle — so phase one is transactional for free, and the\nnest is built in memory with _install_extension_planner performing the single\nsession write after the last hook returns. A hook that raises in either phase\nleaves the caller\u0027s context as it was.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Give the planner example a node only its own codec can carry\n\nThe bundle shipped a planner and codecs, but the halves never met: the planner\nemitted a stock GlobalLimitExec and the codecs delegated everything to the\ndefault codec. So the example asserted by structure that a planner and its\ncodecs belong together without demonstrating why, and no test would have caught\na bundle whose planner emits a node its own codec cannot encode.\n\nDistributedQueryPlanner now wraps its result in a DistributedExec, a type\nprivate to this library, and ObservingPhysicalExtensionCodec claims it by\ndowncast and rebuilds it from its inputs. Nothing else in the session knows the\ntype, which is the reason the two ship as one bundle. The observing codecs stop\nbeing dead weight in the process — they were previously never consulted, and\ndecode_max_rows_seen had no caller.\n\nAlso documents what codec order does and does not control, which building this\nsurfaced. Decoding routes by id and is never order-dependent. Encoding stops at\nthe first codec that claims the node, so a codec claiming a broad category —\nMyPhysicalExtensionCodec claims any ForeignExecutionPlan — takes nodes from any\nlibrary installed after it. The query still succeeds; only the library that\nwrote the bytes changes, which breaks a plan that has to decode elsewhere.\n\nThat gives a bundle two reasons to want different positions for its two halves.\nThe guide now says to contribute each half at its own position with a small\nadapter rather than reordering, since the hooks are independent, and treats the\nlow-level sequence as the last resort it is: it works, but it hands back\nresponsibility for codec-before-planner ordering and leaves a hand-layered\nfallback holding the codecs it captured. No attempt is made to express every\npermutation from one call.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Drop the planner example\u0027s logical codec observer\n\nThe physical observer earns its place now that it claims the bundle\u0027s own\nDistributedExec, but the logical one never did and cannot: it declines all four\nmethods to the default codec, and this library defines no logical extension\nnode for it to claim. The FFI logical codec does not carry arbitrary\nLogicalPlan::Extension nodes anyway, so there is no logical analogue to give\nit. Measuring a query confirms it: every record_task_ctx firing comes from the\nphysical decode path and none from the logical one.\n\nBundledLogicalCodec now wraps DefaultLogicalExtensionCodec directly, which\nkeeps what the logical half actually demonstrated -- a bundle contributing both\ncodec kinds under ids it declares -- and drops 54 lines of trait impl and\nhand-written Debug that fed an accessor nothing could observe.\n\nAlso narrows PlannerObservations::used_fallback back to private. It is read\nonly through MyQueryPlanner::used_fallback in the same module; its neighbours\nneed pub(crate) because extension.rs reads them, and it does not.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Export QueryPlannerExportable and drop the empty-extensions guard\n\nTwo loose ends from review.\n\nQueryPlannerExportable was the only member of the extension protocol family\nleft in the submodule while SessionExtensionComponents,\nSessionExtensionExportable and SessionPlannerExportable were exported from the\npackage root. It types the planner a __datafusion_session_planner__ hook\nreturns, so a bundle author needs it just as much, and one family member\nimporting differently from the rest is a papercut with no upside. Also fixes\ntwo doc references that stopped resolving when these classes moved out of\ncontext.py: a bare :class:`QueryPlannerExportable` and a bare\n:py:class:`SessionExtensionComponents`, both now spelled with their module the\nway the neighbouring datafusion.user_defined references are.\n\nwith_extensions() with no arguments raised instead of installing nothing. That\nput it out of family: every sibling varargs method -- DataFrame.select, filter,\nsort, drop, window -- accepts zero arguments and returns a no-op result, and\nthe two existing \"at least one\" guards in the codebase both cover cases with no\nmeaningful identity element, which this is not. Installing no extensions has an\nobvious answer, and a caller assembling the list from a plugin registry should\nnot have to special-case it being empty. Phase two still runs, so the empty\ncase rebinds an existing FFI planner to unchanged chains; a test pins that the\nplanner survives it.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Describe both extension hooks in the upgrade guide\n\nThe planner-install section still said a bundle exposes\n__datafusion_session_extension__, which stopped being the whole protocol when\nplanners moved to a hook of their own.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Generate heading anchors for h4 so the FFI guide\u0027s links resolve\n\n`myst_heading_anchors` was 3, but the extension-bundles section added in this\nbranch cross-references its own `####` subsections. Sphinx warns\n`\u0027myst\u0027 cross-reference target not found: \u0027when-codec-order-does-matter\u0027` and\nrenders that link as plain text; the docs build does not pass `-W`, so it went\nunnoticed.\n\nBumping to 4 rather than promoting the heading keeps the four subsections\nnested under `### Extension bundles: with_extensions`, where they belong.\nOnly one other `####` heading exists under `docs/source/`.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Skip the planner commit when with_extensions installs nothing\n\n`with_extensions` ended every call with `_install_extension_planner`, which\nrebuilds `SessionState` to rebind an existing FFI planner to this handle\u0027s\ncodec chains. When the call installed no codec there is nothing to rebind\nagainst, so the rebuild is at best churn — and at worst it drags a planner\nthat is sitting on another handle\u0027s codecs onto this one\u0027s, silently undoing\nthat install. `with_python_udf_inlining` already guards its no-op toggle for\nexactly this reason; `with_extensions` now guards the same way.\n\n`test_with_extensions_installing_nothing_leaves_the_planner_alone` covers both\nshapes of \"installed nothing\": no arguments at all, and a bundle whose hooks\nanswer empty. Both fail without the guard, with the planner left on an empty\nlogical chain and the query erroring out.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Reject a lone codec where SessionExtensionComponents wants an iterable\n\n`logical_extension_codecs\u003dcodec` instead of `(codec,)` is the easy mistake to\nmake, and it surfaced as `\u0027MyCodec\u0027 object is not iterable` raised by an\n`extend` call inside `with_extensions` — naming neither the field nor the hook\nthat built the value. `__post_init__` now checks it, so the error lands in the\nextension library\u0027s own frame and says which field is wrong and how to spell\none codec.\n\nIt also normalizes each field to a tuple. The declared type is a tuple and the\nclass is frozen, so a list left in place would be a mutable member of an\nimmutable value, and a generator would be exhausted by the first read. A str\nis refused rather than normalized: it is iterable, so it would otherwise\nbecome a tuple of characters and fail much later as that many bogus codecs.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Widen the with_extensions annotation to both hooks\n\n`with_extensions` accepts a bundle implementing either hook — the runtime check\ntests against both protocols, `test_with_extensions_accepts_a_planner_only_extension`\npins it, and the FFI guide\u0027s `PlannerOf` adapter recommends contributing only the\nplanner half. The annotation named `SessionExtensionExportable` alone, so a type\nchecker rejected the very shape the guide tells authors to write.\n\nTwo doc comments also went stale. `test_with_extensions_no_extensions_keeps_an_installed_planner`\nstill described phase two running and rebinding an existing planner, which the\nno-op guard now skips outright, and `__datafusion_codec_id__` listed a\n`_install_extensions` method that never existed under that name.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Correct three claims in the extension bundle docs\n\nThe `SessionExtensionComponents` doctest took its codec capsule off a\n`SessionContext()` that was dropped on the same line. An FFI codec holds its\ntask-context provider weakly, so that capsule names a session that is already\ngone — the doctest only reads an id back so it passes, but it is the exact\nshape the FFI guide warns against. It now keeps the context in a name.\n\n`SessionPlannerExportable` called returning `fallback` a wrap that \"contributes\nnothing\". It is not: the capsule the first bundle receives wraps the session\u0027s\nplanner for export, so handing it back installs it as a foreign planner and\nevery later plan crosses an FFI boundary that was not there before. `None` is\nthe no-op. Corrected in the protocol docstring, the FFI guide\u0027s canonical\nsection, and the `_PlannerExtension` test helper that repeated the claim.\n\n`__post_init__` walked a written-out list of field names. It now walks\n`dataclasses.fields`, filtered on the `_codecs` suffix so a codec field added\nlater is normalized without anyone remembering to name it, and a future field\nthat is not a codec collection is left alone.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Rename _rebind_query_planner to _export_query_planner\n\nThe method rebinds nothing. It imports whatever a `__datafusion_session_planner__`\nhook returned — an object exposing the getter or a raw capsule — and hands back a\ncapsule, so the next hook receives one either way; its own doc comment already\nsaid \"re-export\". Meanwhile \"rebind\" means something specific and different in\nthis file: rebuilding an installed planner against a handle\u0027s codec chains, which\nis what `set_session_query_planner` does. Freeing the word keeps the two apart.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Say which codec chains each extension hook\u0027s context carries\n\n`__datafusion_session_planner__` was documented as receiving a context that\ncarries the final codec chains, and `MyPlannerExtension` relies on exactly that\nwhen it takes the host\u0027s codecs off `ctx` instead of minting its own. The other\nside was never stated: `__datafusion_session_extension__` runs before anything\nis installed, so its `ctx` is the same session with the chains the receiver\nalready had — missing this call\u0027s codecs, including the bundle\u0027s own.\n\nBoth hooks hand back a valid task-context provider, which is what components\nactually need, so the difference only bites an author who reads codec chains off\nthe context. Recorded on `SessionExtensionExportable`, in the FFI guide\u0027s\ntwo-phase section, and in Rule 2 of the capsule-protocol skill.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Make the planner-hook tests assert what their names claim\n\n`test_with_extensions_threads_the_planner_through_in_order` never checked\nan order: it asserted each hook recorded one fallback and that the second\nwas not None, both of which hold for a host that ran the hooks backwards.\n`test_with_extensions_skips_a_planner_hook_returning_none` asserted only\nthat downstream ran, while its comment claimed the skipped hook had not\nbecome downstream\u0027s fallback.\n\n`_PlannerExtension` now takes an optional shared list the hooks append\nthemselves to, so order is observable. The threading test asserts that\nlist, plus that the second hook\u0027s fallback is not the object the first was\nhanded -- the host re-exports every return value before passing it on. A\ncapsule is opaque from Python, so that cannot separate a re-export of the\nfirst planner from a fresh read of the session\u0027s; the comment says so and\npoints at the FFI suite\u0027s `test_with_extensions_nests_planners_in_argument_order`,\nwhich pins the nesting by asserting the outer planner delegated.\n\nThe skip test records the skipped hook\u0027s fallback too, and asserts both\nhooks ran, that downstream was handed a different capsule, and that the\nresulting context still queries.\n\nEach new assertion was mutation-tested: reversing the planner loop, dropping\nthe `if supplied is None: continue`, and replacing the re-export with a\nstraight pass-through each fail exactly one of these tests.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Correct three more claims in the extension bundle docs\n\n`QueryPlannerExportable` said `session` is the\n`datafusion.context.SessionContext` the planner is being installed on. It\nis not. The capsule getters are called from Rust and receive the PyO3\ncontext, so `isinstance(session, SessionContext)` is False -- while its\nrepr reads `datafusion.SessionContext`, because the pyclass declares\n`module \u003d \"datafusion\"`. It carries every capsule getter and\n`__datafusion_codec_id__`, which is all the protocol needs, so the fix is\nto say duck-type it rather than to change what is passed. The two bundle\nhooks are the exception and do receive the wrapper, since `with_extensions`\ndispatches them from Python; the ffi.md section on capsule getters now\ndraws the same distinction.\n\nThe `with_extensions` `Raises:` section listed ValueError for colliding\ncodec ids only. A getter returning a capsule of the wrong kind also raises\nit -- `Expected name \u0027datafusion_query_planner\u0027 in PyCapsule, instead got\n\u0027datafusion_logical_extension_codec\u0027` -- which\n`test_with_extensions_rejects_bad_codec_capsule` already pins.\n\nThe `datafusion.extensions` module docstring said phase two runs the\nplanner hook \"once per bundle\". Once per bundle that implements it; a\nbundle implements either hook or both.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: add a user-facing extensions page and split distributing-work\n\nThe user guide had no page for someone who installs an extension library\nand wants to run queries with it. `with_extensions` was described only in\na 123-line docstring and in the contributor-guide FFI page, which opens by\nexplaining that Rust has no stable ABI — the wrong altitude for a reader\nwho just wants their queries to run somewhere else.\n\nAdds `user-guide/extensions.md`: what an extension library is, which kinds\nregister directly versus needing `with_extensions`, the two failure modes\nthat actually bite (a collected context, a version mismatch), and how to\ncheck what a session was taught. It names no capsule, no ABI, and no codec\nid; the only occurrences of `FFI` and `TaskContextProvider` are inside the\nerror string a reader would be searching for.\n\nSplits `distributing-work.md` into a directory. The page was entirely about\npickling expressions to worker pools and treated query-level distribution\nas two stubs at the end, so nothing on the site connected `with_extensions`\nto distribution at all — which is the road a data scientist is actually\nlooking for. The index now asks who owns the partitioning decision and\nroutes accordingly; `query-engines.md` carries the missing bridge and\nabsorbs the two upstream work-in-progress sections.\n\nWires `sphinx-reredirects`, which has been a declared dependency since\n#1578 without ever being enabled, so the old `distributing-work.html` URL\nkeeps working. Also fixes three pointers that went stale in the MyST\nmigration and still named `.rst` files, a malformed `ref:` role in\nudf-and-udfa.md that rendered as literal text, and a doubled \"the\" in\ndata-sources.md.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: split the FFI guide into one section per audience\n\n`contributor-guide/ffi.md` had grown to 795 lines serving three different\nreaders at once, filed under a section whose index says it is for people\ncontributing to this repository. An engineer at delta-rs or a distributed-engine\nvendor is neither a contributor nor an end user; they consume a published,\nversioned protocol, and the only description of it lived behind a heading\ntelling them the page was not for them.\n\nAdds a third top-level section, `extension-guide/`, so the sidebar reads User\nGuide / Extension Guide / Contributor Guide — one per audience. It carries the\n`(ffi)\u003d` label, so every existing reference keeps resolving, including the\n`:ref:`ffi`` inside `context.py`\u0027s docstring that ships in the wheel.\n\nThe maintainer-facing rationale moves to `contributor-guide/ffi-internals.md`:\nthe weak-`Arc` scheme, why repairing an orphaned provider cannot work, why\nplanner codec rebinding is one level deep, and the two upstream issues. An\nextension vendor should not be reading \"that is a bug rather than a design, do\nnot copy the pattern\" halfway down their integration guide.\n\nThe PyO3 `frozen` policy moves to `contributor-guide/pyo3-guidelines.md`. It is\nproject review policy with nothing to do with FFI, and extracting it repairs a\nprose bug: it had been spliced into the middle of \"Implementation Details\", so\nthe sentence \"If you were interfacing with a library that provided the above\n`FFI_TableProvider`\" resumed 46 lines after the snippet it referred to. Those\ntwo halves are rejoined on `capsule-protocol.md`.\n\nFills the coverage gap the split exposed. The codebase exports 18 capsule\ngetters and the old page documented 7; the remaining 11 appeared on no page\nthat even listed them. The section index now carries a table of all 18, and\n`table-providers.md` and `functions.md` document the catalog family, table\nfunctions, physical optimizer rules, and extension options for the first time.\n\nCorrects the argument rule while moving it. The old section asserted that\ncapsule getters \"receive the SessionContext they are being installed on\", which\nis true for the codec, planner, and table-function hooks but not for the\ncatalog family: `CapsuleGetterArg::LogicalCodec` passes the host\u0027s logical\ncodec as a bare capsule, and `__datafusion_table_provider__` gets a session\nfrom `SessionContext.register_table` but a codec capsule from\n`Schema.register_table`. Nothing breaks, because every implementation passes\nthe argument to `ffi_logical_codec_from_pycapsule`, which handles both — so the\nrule is now stated by capability rather than by type, and the upgrade guide\nsays which hooks changed.\n\nAlso converts the guide\u0027s five in-page heading links to labelled `{ref}`\ntargets, since heading anchors rot silently on rewording, and redirects the old\n`contributor-guide/ffi.html` URL.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: give each docstring claim one canonical home\n\nThe extension docstrings and the FFI guide had grown the same six claims 3-5\ntimes each, in wording that had already started to drift. `with_extensions` was\n123 lines — the longest docstring in the package by 30 — and most of it argued\na design rather than stating a contract.\n\nRoutes each recurring claim to one home and leaves a one-line pointer\nelsewhere. Session sharing and context lifetime move onto the `SessionContext`\nclass docstring, since they are properties of the type and that is exactly why\nfive methods had each reworded them. The `None`-vs-`fallback` contract moves\nonto `SessionPlannerExportable`, since it is a return-value contract of one\nmethod. The two-phase rationale, the objects-not-capsules argument, and the\ncodec-order argument stay in the guide, which owns the \"why\".\n\nThe duck-type-the-session rule moves from `QueryPlannerExportable` to\n`LogicalExtensionCodecExportable`, which is already the designated `session`\nreference for that family and where the codec protocols were pointing for it\nanyway. It also gains the fact the old text was missing: across the protocol\nthis argument is not always a session, so duck-typing it is not a style\npreference.\n\nNot every docstring shrank. `__datafusion_query_planner__` was 4 lines with no\nexample while `set_query_planner` told callers to capture the fallback through\nit, so it grew to 25. Several others grew by gaining the `Args`/`Returns`/\n`Raises` sections they were missing, and by trading `+SKIP` examples for\nrunnable ones — a `SessionContext` satisfies the capsule-getter protocols, so\nits own exported capsule stands in for a library\u0027s without a build step. The\ntotal across these sixteen docstrings is roughly flat, at 577 lines before and\n629 after; what changed is that the rationale left and the contract arrived.\n\nThe 21 lines of `dataclasses.fields` rationale on\n`SessionExtensionComponents.__post_init__` become comments in the method body.\nThat is the fix rather than touching `autoapi_options`: `__post_init__` is a\n*special* member, so dropping `private-members` would not hide it, and dropping\n`special-members` would delete the entire `__datafusion_*__` reference surface.\n`conf.py` now records why that setting is deliberately left alone, and\nde-duplicates the four re-exported `extensions` classes the way it already did\nfor `DataFrame` and `SessionContext`.\n\nAdds `python/tests/test_docstrings.py`, which is what would have caught the\n123-liner: a 95-line ceiling with no waiver list, a doctest-presence check over\nthe extension protocol, and a check that a docstring naming the guide actually\nlinks it. The last two each found a real defect on first run.\n\nAlso fixes what this branch made newly load-bearing in the serialization\nsurface: `plan.py`\u0027s single-backtick RST rendered `LogicalExtensionCodec` and\nfriends as italics rather than links into the new API, and the `global_ctx()`\nfallback in `Expr.from_bytes` now silently yields a context with no extension\ncodecs, which before this branch lost only registrations.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: point the non-Sphinx consumers at the new pages\n\nFour links in the two example READMEs pointed into `docs/source/.../ffi.md`\nwith heading anchors. Sphinx does not link-check those, so deleting the page\nwould have left them silently 404ing. They now use published URLs, which also\nfixes a second problem: a relative path into `docs/source` only renders on\ngithub.com and is broken for anyone reading the README from crates.io, an\nsdist, or a vendored copy. `grep -rn \"docs/source\" examples/` is now empty.\n\n`.ai/skills/ffi-capsule-protocol/SKILL.md` named the deleted page as \"where the\ntruth is\", and AGENTS.md sends agents to that skill before they touch a capsule\ngetter. It now points at the specific extension-guide pages and separately at\n`ffi-internals.md`.\n\n`llms.txt` filed the whole subject under \"Optional\" as \"extending the Python\nbindings\" — the wrong shelf for the headline feature of a major release, since\nthat section means \"skippable on a tight context budget\". It gains an\nExtensions and distribution section, `datafusion.extensions` and\n`datafusion.ipc` in the API list, both FFI example crates, and the corrected\n`distributing-work` URL.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: fix extension-guide review findings, rename phase-one bundle hook\n\nSix findings from a read-through of the new extension guide, plus the API\nrename one of them turned into.\n\nThe \"process local tokens\" note sat in the guide index with nothing around it to\nexplain what a token was or why the reader should care. It moves to a new\n`extension_codec_durable_metadata` section in `codecs.md`, where the reader is\nalready thinking about what goes in a payload: what to encode, then what the\nexamples do instead, and the three consequences that follow from parking live\nobjects in a process-local map — no double decode, no fan-out, and a leak for\nany plan that never reaches a decoder. The checklist item now points there\ninstead of at the guide index.\n\nThe hook reference loses its `Capsule name` column, which restated\n`datafusion_\u003cthing\u003e` for every row when the naming rule already derives it, and\ngains a `Contributes` column instead. The argument column stays: those four\nvalues are protocol, not a signature, and only 4 of the 18 hooks have a Python\ndefinition to link at all — the rest are host-side imports, so links into Rust\nsource would rot faster than the table. Staleness is handled by\n`test_hook_reference_table_lists_every_hook` instead, which greps `crates/` and\n`python/datafusion/` for `__datafusion_*__` and diffs the set against the table\nrows. Verified it fails when a row is dropped.\n\n`capsule-protocol.md` described `abi_stable`, which datafusion-ffi no longer\nuses. It now describes `stabby` and the part that is not stabby: `FFI_Option`\nand `FFI_Result` are datafusion-ffi\u0027s own, because stabby\u0027s require\n`T: IStable` and the `FFI_*` structs hold self-referential function pointers.\nThe conversion example converts to `Arc\u003cdyn TableProvider\u003e` rather than naming\n`ForeignTableProvider`, since the `From` impl compares library markers and\nreturns the original `Arc` when both sides are the same library. Three other\nsnippets on that page had gone stale with it: `FFI_TableProvider::new` with\nthree arguments, `PyCapsule::new_bound`, and a receiving snippet whose variable\nwas named `codec`.\n\nIn `table-providers.md` all five `Registered with` cells now render\n`Receiver.method`, so the schema row reads `Catalog.register_schema` rather\nthan a bare dotted path, with one sentence on reaching a `Catalog` first.\n`Other session components` was in `functions.md`, where an optimizer rule and a\nconfig struct are neither functions nor tables; it becomes its own page,\n`other-components.md`, carrying the `extension_other_hooks` label so the index\nrows still resolve.\n\nThree guide pages named individual tests, which invites exactly the divergence\nthe reference is supposed to prevent. They name the suite now.\n\nFinally the phase-one bundle hook. `__datafusion_session_extension__` reused\nthe name of the whole thing it is a hook on — `with_extensions` takes\nextensions and `SessionExtensionExportable` is the bundle protocol — while its\nsibling `__datafusion_session_planner__` is named for its content. It is now\n`__datafusion_session_components__`, matching both its sibling and the\n`SessionExtensionComponents` it returns, which leaves room for the UDF and\nprovider fields that will join the codec fields later. The protocol class\nfollows it to `SessionComponentsExportable`, since that file\u0027s convention is\none class per hook name. Neither name has shipped, so the upgrade guide needs\nno before-and-after; it introduces both hooks as new in this release and names\nthe new one.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* test: match the hook table against dispatch sites, not raw text\n\n`test_hook_reference_table_lists_every_hook` scanned every byte of\n`crates/**.rs` and `python/datafusion/**.py` for `__datafusion_*__`, so\ncomments and docstrings counted as evidence a hook exists. A doc-comment\ncontrasting a hook with one that was removed, or naming a hypothetical,\nwould have had to be deleted or added to the guide\u0027s table, and neither\nis right.\n\nCount sites instead. On the Rust side a site is a string literal holding\nnothing but the hook name -- what `hasattr`, `getattr`, and\n`call_capsule_getter` are handed -- or a `fn` of that name, which is a\nhook the host implements itself. Error strings that merely embed a name\nno longer count; each already sits beside a real lookup.\n\nOn the Python side, read the syntax tree rather than the text: a method\nbeing defined, an attribute being accessed, or a string standing alone.\nA docstring is one string node holding the whole docstring, so prose\ndrops out without a rule of its own.\n\nThe dispatched set is unchanged at 18, still matching the table exactly.\nVerified in both directions: a comment naming a removed hook now passes,\nwhile adding a real lookup for an undocumented name still fails.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Fable 5 \u003cnoreply@anthropic.com\u003e" }, { "commit": "b6c6f5b94afad5a2ed37cf8bfa4cbf2ed9aa3f9c", "tree": "9c5ad85b68cc69e4343f57056ce59cb68cd8b526", "parents": [ "e6c75870a2c465aae4787605f91a9dff47b0e911" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri Sep 04 14:07:52 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 04 14:07:52 2026 -0400" }, "message": "Make extension codecs composable (#1678)\n\n* Make extension codecs composable\n\nInstalling a logical or physical extension codec now prepends it to a\ncodec chain instead of replacing the prior codec. The most recently\ninstalled codec is consulted first, falling through codec by codec to\nthe default codec. This lets multiple independent extension libraries\ninstall codecs on the same session, and removes the codec registration\nordering requirement between libraries.\n\nChain dispatch treats a codec error as \"not mine\". Encoding runs each\ncodec against a scratch buffer so failed attempts leave no partial\nbytes, and treats Ok-with-no-bytes (encode by name) as no opinion so\nlater codecs still get a chance. When every codec fails, the errors\nare aggregated so the owning codec\u0027s diagnostic is not masked by the\ndefault codec\u0027s generic error.\n\nAlso preserves the python_udf_inlining setting when installing a\ncodec; previously it was silently reset to enabled.\n\nDocuments the remaining planner constraint: a session holds one query\nplanner, layering is explicit via fallback capsules, and codecs must\nbe installed before exporting or chaining planners because a planner\ncapsule captures the codecs at export time.\n\nCo-Authored-By: Claude Fable 5 \u003cnoreply@anthropic.com\u003e\n\n* test: port codec Rust tests to pytest\n\nThe Rust tests added for the composable codec work never ran: CI invokes\n`cargo fmt` and `cargo clippy --all-targets` but no `cargo test`, so the\ntests compiled and were never executed. Rather than add a `cargo test`\njob — which would also require feature-gating `pyo3/extension-module`,\nsince the test binary cannot link on Linux while it is unconditional —\nmove the coverage to pytest, matching this repository\u0027s practice of\ntreating the user-facing Python surface as the first line of defense.\n\nRemove both `#[cfg(test)]` modules from crates/core/src/codec.rs and\nreplace them as follows:\n\n- Four wire-header round-trip tests and the Python-minor-mismatch test\n were already covered by existing cases in test_pickle_expr.py.\n- `strip_errors_on_too_old_version` asserted nothing: it returns early\n because WIRE_VERSION_MIN_SUPPORTED equals WIRE_VERSION_CURRENT.\n- The unsupported-wire-version and Python-major-mismatch cases move to\n test_pickle_expr.py, patching the header in place inside the encoded\n protobuf. The patches preserve length so the outer message stays\n parseable and the bytes reach the codec.\n- The three truncated-header cases are dropped. Truncation changes the\n payload length and breaks the protobuf framing, so they fail before\n reaching the header check and cannot be expressed from Python.\n- The codec-chain tests move to the FFI example suite, which exercises\n the same chain through the real FFI boundary.\n\nMyLogicalExtensionCodec gains an optional token overriding the byte\nprefix it stamps on encoded table providers. Two instances with distinct\ntokens own disjoint slices of the wire format, which is what makes chain\nordering and fall-through observable from Python.\n\nThe ported inlining test asserts encode and decode behavior rather than\nthe `python_udf_inlining()` getter the Rust test checked. This is a\nstronger assertion: the getter is preserved even when a composed\nPython-aware codec re-inlines a UDF that the outer strict codec declined\nto inline, so the original test could not have caught that path.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: record the Python-first testing preference in AGENTS.md\n\nNew coverage should land as a doctest example or a pytest case. Agents have\nbeen adding Rust tests that CI never executes: no workflow invokes\n`cargo test`, and `cargo clippy --all-targets` only compiles the test code.\nWrite down that constraint, along with the reason a `cargo test` job is not a\ntrivial addition, so the tradeoff does not have to be rediscovered.\n\nAlso point at the FFI example suites, which are easy to overlook when judging\nwhether behavior is reachable from Python.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: dispatch chained codecs by identity, not by trial\n\nDecoding walked the codec chain and took the first codec that returned\nOk. That is unsound, and upstream has already been bitten by it:\napache/datafusion#16980 records ComposedPhysicalExtensionCodec decoding\nan encoded parquet file as csv because the payload happened to parse\nwith an earlier codec, and #16986 fixed it by recording the encoder.\nProtobuf carries no type identity, so two codecs whose leading field\nnumbers and wire types line up decode each other\u0027s payloads cleanly.\nA byte-prefix convention does not help: the natural implementation is\n`Message::decode(buf)`, which has no prefix to check and cannot decline.\n\nPayloads written by a chained codec now carry an envelope naming the\ncodec that wrote them, and decoding consults exactly that codec. The\nenvelope is applied and stripped inside PythonLogicalCodec /\nPythonPhysicalCodec, so third-party codecs are unmodified and never see\nit -- they receive the bare bytes they wrote.\n\nKeyed on a stable identity rather than chain position. Position is\nsound for Ballista and datafusion-distributed because their codec lists\nare pinned -- a compile-time constant in one, a fixed entry plus\nappended user codecs rebuilt from the same startup code in the other.\nA datafusion-python chain is assembled by user Python across sessions\nthat share no struct, and a library shipping a codec cannot know its own\nindex, so position would silently name the wrong codec whenever two\nsessions install in different orders.\n\nIdentity is derived and asks nothing of existing libraries: an explicit\ncodec_id, else __datafusion_codec_id__, else the exporting class\u0027s\nmodule and qualified name. A bare PyCapsule exposes nothing stable --\nevery capsule reports the same type -- so it gets a session-local id and\na pointed error if its payloads reach an unrelated session. Installing\ntwo codecs under one id is rejected at install time rather than\nresolving to whichever entry came first.\n\nCodecs now append rather than prepend, so encoding is claimed by the\nfirst codec installed that wants the object. Installing a library can\nonly claim objects nothing else claimed; it can never take over an\nexisting library\u0027s objects, and it never renumbers ids that older\npayloads reference.\n\nTwo payloads are deliberately left unframed. The terminal codec writes\nbare, so a session with no extension codecs is byte-identical to a build\nwithout chaining. And an encode that writes nothing stays empty: an\nempty fun_definition is DataFusion\u0027s encode-by-name signal, and framing\nit would set the field, permanently skipping the registry lookup the\ndecoder does first and breaking codecs that reconstruct a function from\nits name alone (from_proto.rs, the `None \u003d\u003e ctx.udf(..).or_else(..)`\narm). That arm is also why an empty buffer still consults every codec:\nthere are no bytes to tag. It is not the hazard tagging removes -- the\nquestion asked is \"do you own the function named x\", which is\nname-scoped, and a disagreement needs two libraries claiming one name.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* test: pin the by-name decode path, and document identity dispatch\n\nAdds NameOnlyUdfCodec to the FFI example: a codec owning functions that\nare fully described by their names, so try_encode_udf writes nothing and\ntry_decode_udf rebuilds the function from the name with no registry\nentry. DataFusion supports that shape directly -- an empty\nfun_definition sends the decoder to the FunctionRegistry first and the\ncodec second (from_proto.rs, the `None \u003d\u003e ctx.udf(..).or_else(..)` arm).\n\nNothing covered that arm before, and it is the path most at risk from a\nplausible change: wrapping every chained encode in the identity envelope\nwould make an empty payload non-empty, set fun_definition, and skip the\nregistry lookup permanently. That breaks ordinary by-name round trips as\nwell as codecs like this one, and no other test would notice. The\ndecoding session here deliberately never registers the function, so only\nthe codec can supply it.\n\nExposes logical_extension_codec_ids() and physical_extension_codec_ids()\non SessionContext. The ids are the dispatch keys a payload names, so\nlisting them answers \"which library owns this plan\" and \"can this\nsession decode it\" -- and they are what a decode failure reports.\n\nDocumentation rewritten around identity rather than ordering:\n\n- ffi.md states that codecs need no changes, why rejecting foreign\n payloads is not something a codec can reliably do, the identity\n resolution ladder, and the two payloads left unframed with the reason\n for each.\n- Both example READMEs drop the \"most recently installed is consulted\n first\" language, which described dispatch that no longer exists.\n- upgrade-guides.md gains a section for the behaviour change: codecs\n compose rather than replace, no codec-side change is required, and the\n wire format changes only for sessions that install one.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: record codec chain dispatch in the FFI capsule protocol skill\n\nThe skill already triggers on \"any FFI_* export that asks for ... an\nextension codec\", but had nothing to say about how a chain decides which\ncodec handles a payload -- the part of this area most likely to be got\nwrong, and the part with a rationale that is not visible from the code.\n\nRule 8 records it: dispatch by identity rather than by trying codecs\nuntil one returns Ok, with the upstream incident that settles the\nquestion (apache/datafusion#16980, a Parquet payload decoded as CSV) and\nthe protobuf reason a byte-prefix convention cannot fix it.\n\nIt also records why we key on identity where Ballista,\ndatafusion-distributed, and upstream all key on chain position. Their\nlists cannot disagree -- one is a compile-time constant, the other a\npinned entry plus user codecs rebuilt from the same startup code. Ours is\nassembled by user Python, and to_bytes/from_bytes puts two independently\nbuilt sessions on either end of one payload. Copying the upstream design\nhere reintroduces the same silent mis-decode from the other direction,\nwhich is exactly the kind of thing an agent reading only upstream would\ndo.\n\nAnd it records the two payloads that are never framed, because framing\nthe empty one is a plausible tidy-up that breaks by-name decoding\nsilently. Points at NameOnlyUdfCodec as the guard.\n\nExtends the trigger to cover codec.rs dispatch, which the old wording\ndid not obviously reach, and documents the optional\n__datafusion_codec_id__ hook on the two codec Protocols -- where Rule 5\nsays such things belong.\n\nReviewed the other four skills: check-upstream, make-pythonic,\naudit-skill-md, and the user-facing datafusion_python skill are all\nunaffected, none of them touching codec dispatch.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: move codec dispatch rationale out of the skill and into the code\n\nRule 8 restated, less precisely, what the docstrings in codec.rs already\nsay: identity dispatch, why not position, the two unframed payloads,\nappend-and-first-claim-wins. A skill earns its place by carrying what\nyou cannot learn from the file you are about to edit -- Rule 1 greps a\nfamily across files, Rule 3 governs code in an extension author\u0027s own\nrepo, Rule 5 is a four-file process checklist. \"How the dispatch in\ncodec.rs works\" is what codec.rs is for, and duplicating it there just\ncreates a second copy to keep in sync.\n\nTwo pieces were worth keeping, and neither is a rule.\n\nThe apache/datafusion#16980 citation moves to `chain_decode`, next to\nthe code someone would be editing when tempted to delete the envelope\nand walk the chain instead. It is the concrete evidence that makes the\nwarning land, and it was the one thing the skill had that the code did\nnot.\n\nRule 5 gains a clause: changing what a codec puts on the wire is as\nbreaking as changing a getter signature, and easier to miss because\nnothing fails to compile. That is genuinely cross-file process, which is\nwhat the skill is for.\n\nAlso enriches ChainEntry\u0027s docstring with the datafusion-distributed\nhalf of the \"why not position\" argument, which previously named only\nBallista and upstream. Both consumers matter: the point is that their\ncodec lists structurally cannot disagree and ours can.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: give a capsule-installed codec an id no other session can mint\n\nA codec installed from a bare PyCapsule has nothing stable to derive an\nidentity from, so it was tagged `anon:{chain_length}`. That is a position,\nnot an identity: every session numbers from the same end, so two unrelated\nsessions that each install one capsule both mint `anon:0`. A payload written\nby the first was then handed to the second session\u0027s codec — the positional\ndispatch this chain design exists to avoid, reintroduced in the one case\nthat has no derivable id.\n\nThe failure was quiet. A codec offered bytes it does not recognise falls\nthrough to its own inner default codec, so the error came back as\n`LogicalExtensionCodec is not provided` and named neither codec, instead of\nthe intended \"encoded by extension codec X, which is not installed on this\nsession\" with the `codec_id\u003d` hint.\n\nMint a random id per install instead, reusing the `Uuid` idiom already in\n`context.rs`. The chain clones the id along with the codec, so a payload\nstill decodes anywhere in the installing session\u0027s lineage; everywhere else\nit now fails with the pointed error. `derive_codec_id` no longer needs the\nchain length, so that parameter goes away.\n\nAlso fixes two docs that described the pre-identity-dispatch design: the\n`codec.rs` module doc claimed codecs are consulted most-recently-installed\nfirst (it is install order, and decoding does not walk the chain at all) and\nasked downstream codecs to reject foreign payloads (an encode-side contract\nonly — a payload only ever reaches the codec whose id it carries), and a\nphysical codec test docstring said a second install prepends.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: identify a session installed as a codec by its session, not its class\n\nInstalling one context\u0027s codec stack on another session derived the\nidentity from the class, and every `SessionContext` shares one class. So\nevery session reported `datafusion.context.SessionContext`: two of them\ncould not coexist on one target, and a payload written through one\nresolved to the other on decode, failing as\n`LogicalExtensionCodec is not provided` from the stranger\u0027s own inner\ndefault codec. Same shape as the positional `anon:{n}` collision, but via\nan id that looks portable, so the error carried no hint either.\n\n`SessionContext` now declares `__datafusion_codec_id__` carrying its\nsession id. That goes through the existing resolution arm for an object\npinning its own identity, so one implementation covers both the Python\nwrapper and the internal object and `derive_codec_id` keeps its four\ndocumented arms. Handles derived from one session report the same id, so\ninstalling two of them on one target is refused — their payloads would be\nindistinguishable on decode.\n\nFraming is unchanged. A session-exported codec is opaque across the FFI\nboundary, so the outer session cannot enumerate what is inside it and the\nenvelope naming its own entry is the only handle it has; nesting costs\nabout 50 bytes per hop and only arises when composing sessions, which a\nlibrary exporting its own codec class never does. What that route needed\nwas a usable identity, not fewer frames.\n\nDocuments the lifetime coupling that comes with it: imported codecs\nresolve their task context against the source session and stop working\nwhen it is dropped, so this composes sessions rather than copying codecs\nout of one.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* test: pin the class-derived codec id, and the escape hatch from it\n\nA codec that declares no identity is named `module.QualName`, and that\nstring goes on the wire in front of every payload it writes. Nothing\nasserted it, so renaming `MyLogicalExtensionCodec` or moving its module\nwould have silently changed the wire format with a green suite.\n\nPins it twice: against the literal, so a rename has to come here and be\nacknowledged, and against `__module__`/`__qualname__`, so the literal\ncannot drift away from what the code actually derives.\n\nAlso covers the reason `__datafusion_codec_id__` exists, which had no\ntest either. A plan encoded by a codec under its old class name decodes\non a session that only knows the new one, and the payload carries the\npinned id rather than either class name. Under the class-derived default\nthose are two different ids and the plan is undecodable.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: aim the codec documentation at its readers\n\nThe codec documentation added in this PR drifted into narrating the\nimplementation. An extension author cannot act on the name of a private\nRust type, and a caller installing a library\u0027s codec does not need the\nenvelope layout or the argument for why trial decoding is unsound.\n\nRemoved from `ffi.md`: the paragraph naming `Python{Logical,Physical}Codec`\nas the thing that wraps payloads, the protobuf-ambiguity argument with the\nupstream postmortem, the note on why the bare-capsule id is random, and the\ntwo-unframed-payloads section, which opened by addressing whoever is\n\"changing this code\". What each reader can act on stays: implement your\ncodec as though yours is the only one, do not write defensive prefix checks,\nhow identity is derived, when to pass `codec_id\u003d`, that install order does\nnot affect decoding, and that a codec may encode a function by name alone.\nAll of the removed rationale already lives in `crates/core/src/codec.rs`,\nwhich the section now points to for anyone changing the framing.\n\nTrimmed `with_logical_extension_codec` to what a caller acts on, with a\n`:ref:` to the FFI guide for the rest, and added the `ValueError` on a\nduplicate id, which the longer version buried. `with_physical_extension_codec`\nnow delegates in one sentence instead of restating the same paragraphs, which\nhad already begun to drift.\n\nReworded the internal vocabulary out of the upgrade guide and\n`logical_extension_codec_ids`: \"terminal codec\", \"writes unframed\", and\n\"identity envelope\" name internals a reader cannot see. Dropped the\nexplanation of why a session id rather than a class name identifies a\ncontext installed as a codec, keeping the constraint that follows from it.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: state the one case where a codec sees bytes it may not own\n\nThe FFI guide told codec authors not to guard against another library\u0027s\npayloads, on the grounds that such a check \"simply never fires\". That was\nwrong twice over. The check executes on every decode, and its reject branch\nis reachable: an empty payload carries no identity to route on, so\n`chain_resolve_by_name` offers it to every installed codec in turn, and a\ncodec asked about a function it does not own rejects — correctly.\n\nThat exception is worth documenting on its own terms, because an author who\ntrusts the \"you only ever see your own payload\" guarantee can index into an\nempty `buf` and panic. `try_decode_udf` and its aggregate and window\nsiblings can be called with an empty buffer and another library\u0027s `name`, so\nthe guide now says to decide from `name` and not to assume `buf` is\nnon-empty. `name_only_codec.rs` in the FFI example already does exactly\nthat, so the guide and the worked example agree.\n\nThe advice against defensive checks is gone rather than reworded. An author\nwho already writes one keeps it and nothing breaks; an author who does not\nneeds nothing. What remains is the portability case: a codec that also ships\nto hosts dispatching by position or by trial may still want its own guard,\nand keeping one costs nothing here.\n\nTwo claims in codec.rs conflated a marker check with relying on decode\nfailure. A marker an encoder writes and a decoder checks is reliable — it is\nwhat the envelope itself does. What cannot be relied on is\n`MyMessage::decode(buf)` declining someone else\u0027s bytes, and what cannot be\nrequired is third-party codecs all honouring such a convention. Both are now\nstated as what they are.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: call a codec id an id, and say what one is\n\nThe codec documentation used \"identity\" as a term of art without ever\ndefining it, while the API it describes calls the same thing `codec_id`.\nA reader met an abstract noun, had to guess what it referred to, and then\nhad to map it onto the parameter they actually type.\n\nThe FFI guide now defines it once, where the surrounding text has already\ndescribed the mechanism: datafusion-python records which codec wrote each\npayload, and that record is the codec\u0027s id — a short string stored inside\nthe plan, which has to name the same codec in the process that decodes as\nit did where the plan was written. That last clause is the reason ids\nexist and is the part a reader needs.\n\nReplaced the derivation-order list with the three cases where a reader has\nto do something: two instances of one class collide and raise `ValueError`,\na bare `PyCapsule` gets an id private to its session, and a class you\nintend to rename wants `__datafusion_codec_id__`. Each names the symptom\nand the fix. The order in which an id is resolved answered a question\nnobody asks, and the sub-clause about how a `SessionContext` names itself\ndescribed an id nobody types. The session-composition paragraph loses its\nid explanation, keeping the lifetime coupling that is the real hazard.\n\nNet 34 lines to 26 in that part of the guide, with a definition it did not\nhave before. The same substitution follows through the upgrade guide, the\n`SessionContext` and protocol docstrings, and both example READMEs, since\nleaving the user-facing pages on the old vocabulary would have been worse\nthan the original.\n\nUntouched: \"shared-library identity\" earlier in the guide, which predates\nthis work and refers to distinct DataFusion library markers.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: plain language for the two codec id prefixes\n\nBoth doc comments stacked several clauses per sentence and stated the\nrejected alternatives abstractly, which a reviewer reported as hard to\nparse. Same content, plainer register: the anonymous prefix now says\nthe suffix is a fresh UUID, matching what derive_codec_id mints, and\nspells out the anon:0 collision instead of describing it as sessions\nnumbering from the same end.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* test: count decodes, not just encodes, on the composed chain\n\ntest_ffi_logical_codec_composes_with_later_install asserted the encode\ncount and then only that the round-trip produced equal batches. The\nround-trip result alone does not distinguish the codec decoding the\nbytes it wrote from the table being resolved some other way, so the\ndecode side of the chain was unpinned. Count it too.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: describe the composed-codec test in install order\n\nThe docstring for test_composed_codecs_with_query_planner said the\nextra codecs decline everything and encoding falls through to the\nprovider codecs, which describes the reverse of what happens. The\nprovider codecs are installed first and chain_encode walks in install\norder, so they claim their objects before the extra codecs are\nconsulted; decoding dispatches by id and does not consult them at all.\nSay that, and say what the test asserts: the extra pair changes\nnothing, where replace semantics would have broken the round trip.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Fable 5 \u003cnoreply@anthropic.com\u003e" }, { "commit": "e6c75870a2c465aae4787605f91a9dff47b0e911", "tree": "f42c93fd03251555615efa7b10b4a6a18f119ec5", "parents": [ "2f528d2d53a550ba5434466e32af78b8c5bfac82" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri Aug 28 16:21:27 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 28 16:21:27 2026 -0400" }, "message": "Add FFI query planner support (#1677)\n\n* Add FFI query planner support\n\nAI Disclosure: This code was written in part by an AI agent.:\n\n* Add three-library FFI planner example\n\nAI Disclosure: This code was written in part by an AI agent.:\n\n* Update FFI query planner integration\n\nAI Disclosure: This code was written in part by an AI agent.:\n\n* add rat\n\n* fix: install FFI test wheels from nested artifact paths\n\nThe FFI test wheel artifact now bundles two projects, so upload-artifact\npreserves a `\u003cproject\u003e/dist/` prefix instead of placing the wheels at the\nartifact root. The install step globbed `wheels/*.whl`, which no longer\nmatched them, so the FFI wheels were silently skipped and the FFI unit\ntests failed with `ModuleNotFoundError: No module named\n\u0027datafusion_ffi_example\u0027`.\n\nInstall the recursive `find` results instead of re-globbing.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: address review of FFI query planner support\n\nCollapse the two duplicated planner-install blocks into a single\n`ctx_with_rebound_planner`. A derived context shares the existing\n`SessionContext` when there is no foreign planner to rebind, and forks\nonly when one is installed, since the FFI codecs capture the context\nthey are built against.\n\nDocument what that fork shares. Catalogs, tables, and the runtime\nenvironment stay shared; registered functions, configuration, and the\noptimizer rule lists are snapshotted. The caveat lands on all four\nderivation methods and on a new contributor-guide subsection, with\ntests covering both halves.\n\nExplain why `RuntimeAwareQueryPlanner` exists at all. Upstream\u0027s\n`ForeignQueryPlanner` is the consumer-side adapter that lets an\n`FFI_QueryPlanner` satisfy the `QueryPlanner` trait, which is what makes\na planner from another shared library installable in a `SessionState`.\nIts trait method receives only a `\u0026LogicalPlan` and a `\u0026dyn Session`, so\nit has nowhere to obtain a runtime handle and passes `None`.\n\nThroughout datafusion-ffi each library attaches its own runtime to the\nobjects it exports, so a producer-side wrapper can enter that runtime\nbefore running its own library\u0027s code. A provider owned by another\nlibrary keeps its owner\u0027s runtime even when it travels through our\ncatalog, because `FFI_TableProvider::new_with_ffi_codec` unwraps a\n`ForeignTableProvider` back to the original handle and discards the\nruntime passed alongside it. `session_runtime` is that same rule applied\nto the session: `FFI_SessionRef` is our object and every callback on it\nruns our code.\n\nIt matters for what those callbacks hand back. A plan produced by our\nown planner returns as `FFI_ExecutionPlan::new(plan, runtime)`, and\n`execute` enters that runtime before calling into the plan; the same\nholds for our physical optimizer rules and for tables we own rather than\nre-export. The delegation case this type exists for is exactly that\nshape. A foreign planner falling back to our planner through\n`__datafusion_query_planner__` receives a plan whose execution needs our\nruntime, and datafusion-python owns that runtime as a process global\nwhile the Python thread calling in carries no ambient one.\n\nThe same reasoning is why `__datafusion_query_planner__` re-exports\nthrough the adapter rather than unwrapping to the inner handle. A\nconsumer reaching us through `ForeignQueryPlanner` calls with `None`, so\nthe adapter is what restores our handle on the way back out. Unwrapping\nwould save a planning-time round trip and silently drop it.\n\nIn the planner example, match the two real spellings of the row-limit\nconfig key exactly instead of by suffix, and validate after both lookup\npaths so the fallback cannot accept `max_rows \u003d 0`. The key appears\ntwice because rebuilding a `ConfigOptions` across the FFI boundary\nparks every foreign extension inside a single `FFI_ExtensionOptions`,\nitself namespaced under `datafusion_ffi`.\n\nAlso declare `requires-python \u003d \"\u003e\u003d3.10\"` on the provider example to\nmatch the `abi3-py310` feature it builds against, and link both example\nREADMEs to the contributor guide rather than restating its caveats.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: drop the runtime adapter and fix exported capsule lifetimes\n\nRemove `RuntimeAwareQueryPlanner`. It existed to re-attach our Tokio\nhandle to the session we hand to a foreign planner, on the reasoning that\n`ForeignQueryPlanner` passes `session_runtime: None`. That handle turns\nout to have no reachable path: the query planner FFI exchanges serialized\nbytes rather than plan handles, a provider owned by another library keeps\nits own runtime because `FFI_TableProvider::new_with_ffi_codec` unwraps a\n`ForeignTableProvider` back to the original handle, and we execute on our\nown runtime regardless. Setting the handle to `None` left every test\npassing. Codec rebinding now downcasts upstream\u0027s `ForeignQueryPlanner`\ndirectly, which also stops `__datafusion_query_planner__` adding a second\nlayer, since `new_with_ffi_codecs` already unwraps that type. The\n`datafusion-session` dependency is no longer needed in crates/core.\n\nKeep the exporting session alive for codecs handed out in a PyCapsule.\n`FFI_TaskContextProvider` stores its provider in a `Weak`, so a capsule\nstopped working as soon as the `SessionContext` that produced it went out\nof scope. That made the natural spelling of the documented fallback\npattern fail:\n\n fallback \u003d ctx.__datafusion_query_planner__()\n ctx \u003d ctx.with_query_planner(MyPlanner(fallback\u003dfallback))\n\nRebinding `ctx` dropped the exporter and planning then failed with\n\"TaskContextProvider went out of scope over FFI boundary\". Both Python\ncodecs gained an opt-in `exported_session`, set only by the three capsule\ngetters. The keep-alive lives in the inner codec because the consumer\nclones the FFI handle out of the capsule and `clone` clones the inner\ncodec\u0027s `Arc`, so a capsule-scoped keep-alive would die too early. It is\ndeliberately opt-in: the same codecs are also attached to providers and\ncatalogs that end up back inside the session, where a strong reference\nwould close a `SessionContext -\u003e SessionState -\u003e query planner -\u003e FFI\ncodec` cycle. Both structs now implement `Debug` by hand, because\n`SessionContext` is not `Debug`.\n\nAdd two example tests. One drives a plan containing `RepartitionExec`,\nwhich spawns Tokio tasks as it runs, through all three libraries, so the\ncodecs are exercised on a multi-node plan rather than a bare scan. The\nother layers a planner on top of the session\u0027s existing planner using the\ncapsule captured beforehand, which is the delegation pattern upstream\nprescribes; `Session::create_physical_plan` cannot be used for this,\nbecause it dispatches through the installed planner and recurses.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* add override for datafusion version to pre-release testing of upstream fix\n\n* remove unintentionally committed files\n\n* Empty commit to trigger CI\n\n* fix: keep the example planner\u0027s exported task context alive\n\n`FFI_TaskContextProvider` downgrades the provider it is given to a\n`Weak`, so building one inline in `__datafusion_query_planner__` left the\ncapsule carrying a provider that was already dropped by the time it\nreturned. Every codec callback through that capsule would have failed\nwith \"TaskContextProvider went out of scope over FFI boundary\". The\nexample did not notice because it ships the default codecs and no custom\nextension nodes, so `try_decode` is never reached.\n\n`MyQueryPlanner` now owns the context and hands out clones of it. The\n`QueryPlanner` the capsule carries holds a reference too, so the capsule\nstays usable even when the Python object that exported it is dropped\nfirst.\n\nDocument the distinction the inline construction obscured. The\n`TaskContextProvider` supplied at export time backs the exporting\nlibrary\u0027s own codec callbacks, decoding that library\u0027s nodes in its own\nregistry. It is unrelated to the `\u0026dyn Session` that later arrives at\n`create_physical_plan`, which belongs to the host, and it could not be\nderived from that session in any case, since the codecs are built before\nany session exists.\n\nRename `PlannerConfig` to `MyPlannerConfig` to match `MyQueryPlanner`.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* test: cover which session a foreign codec decodes against\n\nThe example codecs restore objects from a process-local token registry and\nnever read the `TaskContext` their FFI decode callbacks are handed, so\nwhich session that context belongs to was untestable. The token path\nignores the registry entirely, which is why an empty\n`SessionContext::new()` has served as the exported provider without\nanyone noticing.\n\nBoth codecs now accept `require_udf_on_decode`. When set, every decode\ncall resolves that scalar function out of the task context it was given\nand fails with the session id if it is absent, which makes the answer\nobservable. Each codec registers a marker function on the context it\nexports, so a name owned by the codec\u0027s library and a name owned by the\nhost can be told apart.\n\nFour tests use it. The two library-local cases pass: a foreign codec\nresolves against the session its own library supplied. The two\nhost-registered cases are `xfail(strict\u003dTrue)`, because a function\nregistered on the host with `register_udf` is not visible to a foreign\ncodec\u0027s decode callback at all. A fifth pins the current error so the\nfailure mode stays legible. Strict xfail means the pair will announce\nitself if the upstream design changes.\n\nDocument the rule this establishes, and correct the surrounding section:\n`with_query_planner` rebuilds a foreign planner against the session that\nwill run the query, so the provider a planner library supplies is\nreplaced on that path. Codecs installed through\n`with_logical_extension_codec` and `with_physical_extension_codec` keep\nthe provider their own library exported, which is the case these tests\nexercise.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat: pass the session to the capsule getters that need it\n\n`FFI_QueryPlanner::new` and `FFI_{Logical,Physical}ExtensionCodec::new`\nask an extension library for a `TaskContextProvider`, and a planner for\ntwo codecs on top of that. A library has none of those. Both examples\nanswered with `Arc::new(SessionContext::new())`, an empty session that\nresolves nothing, held weakly by `FFI_TaskContextProvider` and therefore\nalso a lifetime hazard.\n\nThe table provider protocol already solved this: the host calls\n`__datafusion_table_provider__(session)` and the library takes what it\nneeds off the session. Do the same for the other three getters.\n`__datafusion_query_planner__`, `__datafusion_logical_extension_codec__`,\nand `__datafusion_physical_extension_codec__` now receive the\n`SessionContext` they are being installed on. A codec takes the task\ncontext provider from it; a planner takes both codecs and uses\n`new_with_ffi_codecs`, which needs no provider at all. Neither example\nconstructs a `SessionContext` any more.\n\nDecode callbacks consequently resolve against the session running the\nquery. The two `xfail(strict\u003dTrue)` tests from the previous commit now\npass unmodified: a scalar function registered on the host with\n`register_udf` is visible inside a decode callback executing in another\nlibrary, for both the logical and physical codec. A negative control\nkeeps the check honest, and a further test covers a function registered\nafter the codec was installed, since the provider is a live handle rather\nthan a snapshot.\n\n`PySessionContext` gains an `ancestors` list. A foreign codec is built\nagainst the session current at the time it is installed and holds it\nweakly, so installing a foreign planner afterwards — which forks — would\nstrand the codec once the Python name is rebound. The keep-alive lives on\n`PySessionContext` rather than on the codec because nothing reachable\nfrom a `SessionContext` reaches a `PySessionContext`, so it cannot close\na cycle. What it does not paper over is the fork itself: a function\nregistered after the fork is not visible to a codec bound to the session\nbefore it, which is the existing derived-context caveat seen from the\ncodec\u0027s side, and is covered by a test.\n\n`SessionContext` accepts and ignores the argument on all three getters,\nso a session satisfies the same protocol a library implements and\n`ctx.__datafusion_query_planner__()` keeps working for the delegation\npattern. Calling a stale getter that takes no session now reports an\nincompatible-library error naming the method, matching what\n`table_provider_from_pycapsule` does.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: record the FFI capsule protocol as a convention\n\nThe session-passing rule was already settled for four getters and\ndocumented in the 52.0.0 upgrade guide, but nothing pointed an agent or a\nnew contributor at it before they wrote a fifth. Write it down where it\nwill be found.\n\nAdd the 55.0.0 upgrade guide entry this branch owes. Changing\n`__datafusion_logical_extension_codec__` and\n`__datafusion_physical_extension_codec__` to take a session breaks every\nextension library implementing them, so it needs before/after Rust in the\nsame shape as the 52.0.0 entry.\n\nCorrect `user-guide/io/table_provider.md`. It still showed the pre-52.0.0\nsignature with no session and a `PyCapsule::new_bound` call, so the one\npage a reader is most likely to find contradicted the convention.\n\nAdd `.ai/skills/ffi-capsule-protocol/`. Its description is written as a\ntrigger rather than a task, because the existing skills are all things to\nrun on request and a convention read as one would be skipped. It leads\nwith enumerating the family, which is the step that makes the rest\nunnecessary.\n\nPoint `CLAUDE.md` at it, since that file loads unconditionally and a skill\nonly helps once someone goes looking. Also note that `docs/temp/` is\ngitignored build output that `grep -r` surfaces with stale copies, and\nrequire an upgrade guide section alongside the `api change` label, so a\nbreaking change forces a visit to the file that records the conventions.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: put skill frontmatter before the license header\n\nEvery `.ai/skills/*/SKILL.md` opened with the ASF header and only then the\nYAML frontmatter, which has to be the first thing in the file. The result\nwas that no skill\u0027s `description` was readable: the skill listing showed\n`\u003c!---` for all of them, so the field meant to say when a skill applies\nsaid nothing. `skills/datafusion_python/SKILL.md` already had the right\norder and was the model to follow.\n\nMove the header below the frontmatter in all four. Apache RAT still\napproves each file — it looks for the license anywhere, not at the top —\nverified with rat 0.13.\n\nThis matters most for the new `ffi-capsule-protocol` skill, whose\ndescription is written as a trigger condition rather than a task name.\nThe existing skills are all tasks to run on request, so a convention that\nhas to be read *before* writing code is easy to filter out while skimming\nfor something to invoke. Note the distinction in the skills section of\n`AGENTS.md`.\n\nThen remove what that makes redundant. `AGENTS.md` had grown a copy of the\nskill\u0027s opening grep and a summary of its central rule. Two copies of one\nconvention, with the more discoverable copy free to drift, is exactly the\nfailure this branch already fixed in `user-guide/io/table_provider.md`.\n`AGENTS.md` now says only when to look and where; the skill owns the\nprocedure. The `docs/source` versus `docs/temp` note moves the other way,\nout of the skill and into `AGENTS.md`, where it applies to everything\nrather than to this one protocol.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Update temporary DF version with corrections in FFI\n\n* feat: rebind foreign codecs when a planner install forks the session\n\nInstalling a foreign query planner writes to `SessionState`, and\n`with_query_planner` must not modify its receiver, so it forks. A foreign\ncodec holds an `FFI_TaskContextProvider` pointing at the session it was\ninstalled on, and until now the fork could not move it: passing a new\nprovider to `FFI_LogicalExtensionCodec::new` was silently discarded\nwhenever the codec was already foreign. The fork rebound only its own\nouter wrapper, so decode callbacks in the extension library kept\nanswering from the pre-fork registry, and the pre-fork session had to be\nretained or the weakly held provider dangled.\n\napache/datafusion#24722 fixes the discard; those constructors now adopt\nthe provider on the already-foreign path. Repoint the patch at the branch\ncarrying it and rebind both codecs onto the fork.\n\nVerified the branch carries everything already pinned rather than trusting\nthe commit graph, which reports the two as diverged: across 3811 files the\nonly differences are the four constructors from the fix, and\n`datafusion/ffi/src/session/mod.rs` is byte-identical, so the\n`create_physical_plan` codec fix arrives as its branch-55 backport.\n\n`ancestors` and its helpers are deleted. They existed only to keep the\npre-fork session alive for a codec that could not be moved off it, and a\ncodec bound to the running session needs no such anchor.\n\nThree tests, replacing two that were weaker than they looked. One\nregisters a function on the fork after the codec was installed on its\nparent and resolves it, which is the direct evidence the rebind happened;\nit failed before this change. One installs a planner twice and asserts the\nfirst context still cannot resolve a function registered only on the\nsecond, covering the clone-before-adopt half — a rebind that mutated the\nshared handle would pass the first test and fail this one. The third keeps\nthe live-handle case. The test it replaces required a name registered\nnowhere, so it passed for the same reason as the negative control and\nnever exercised a fork at all.\n\nNote the version floor in `Cargo.toml` rather than raising it now: the\npatched branch still reports 55.0.0, so the requirement can only move to\n55.1.0 when the patch section is removed. Building against 55.0.0 without\nthe patch would compile and silently skip the rebind.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: restore base64 0.23.1 in Cargo.lock\n\nRegenerating the lock against the patched DataFusion fork silently\ndowngraded base64 from 0.23.1 to 0.23.0. Nothing requires the older\nversion -- neither the fork nor upstream 55.0.0 constrains it -- so this\nwas incidental churn from the lockfile refresh, not a resolution result.\n\nRestores the checksum main already had and re-points the three\ndependents (datafusion-common, datafusion-functions, parquet). No other\ndependency moves; cargo metadata --locked still resolves cleanly.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: preserve session id across a planner fork\n\nInstalling a foreign query planner forks the session state, and the fork\nwas minting a new session id. SessionStateBuilder::new_from_existing\ndrops the id and build() replaces it with a fresh UUID, while\nSessionContext had already cached the original into a field of its own\nback at new_with_state. Overwriting the state in place afterwards left\nthe two disagreeing: session_id() returned the pre-fork id, every\nTaskContext handed to a foreign codec carried a different one.\n\nNothing in DataFusion core keys on the session id beyond debug logging,\nso this broke no in-tree behavior. It matters at the FFI boundary, where\nsession id equality is the idiom for \"which session is this codec bound\nto\", and for extension libraries correlating host-side and worker-side\nstate. Upstream hit the same case in SessionContext::enable_url_table\nand preserves the id explicitly, guarded by preserve_session_context_id.\n\nPassing the id through the builder makes the fork, its state, and its\nTaskContexts agree, which is what the derived_parts doc comment and the\nFFI contributor guide already claimed.\n\nVerified by reading the id out of a decode callback via the example\ncodec\u0027s require_udf_on_decode error path -- the only way to observe the\nstate-side id from Python -- with and without the fix.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: preserve session id in add_physical_optimizer_rule\n\nSame drift just fixed in derived_parts, but on a path that never forks.\nadd_physical_optimizer_rule rebuilds SessionState through\nSessionStateBuilder::new_from_existing and writes it straight back into\nthe caller\u0027s own session, so the fresh id build() mints replaces the one\nSessionContext had already cached at construction. The session the user\nis holding then reports one id from session_id() and a different one\nfrom every TaskContext it hands out, with no derivation to explain it.\n\nReproduced against a foreign codec, reading the id back out of a decode\ncallback: identical setup differing only by an add_physical_optimizer_rule\ncall went from MATCH to DRIFT, and back to MATCH with the id threaded\nthrough the builder.\n\nThis is the last new_from_existing call site in the crate.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* test: guard the session id a codec decodes against\n\nThe two session id fixes had no regression guard. A Python-level\nassertion cannot provide one: session_id() reads a copy SessionContext\ncaches at construction, which stayed correct through both bugs. The id\nthat actually moved was the one inside the TaskContext handed to a\nforeign codec\u0027s decode callback, which nothing exposed.\n\nGive the example codecs a TaskContextProbe that records it. This\nreplaces the bare AtomicUsize the require_udf_on_decode support used, so\nthe counter and the session id are recorded together, and the id is\nrecorded on every decode rather than only when a function was requested.\n\nThree tests, all against the codec-side id rather than session_id():\na fork agrees with its codecs, add_physical_optimizer_rule does not move\nthe id, and a two-deep fork chain leaves both halves on the parent\u0027s id.\n\nConfirmed non-vacuous: with both fixes reverted all three fail and the\nother 17 tests pass; with only the derived_parts fix restored, exactly\nthe add_physical_optimizer_rule test still fails.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: stop reporting any getter TypeError as an outdated library\n\ncall_capsule_getter rewrote every TypeError from a capsule getter into\n\"Incompatible libraries ... Upgrade the library providing this object\",\nand dropped the original. Only an arity mismatch means the library is\nout of date. An extension author whose own getter raised a TypeError --\na bad cast, a wrong argument to something it called -- was told the\nerror was a version problem and lost the error that would have located\nit.\n\nThe two are distinguishable without guessing at message text: an arity\nmismatch is raised by the call machinery before the getter\u0027s frame\nexists, so no frame unwinds and no traceback is attached, while an error\nfrom the body carries one. Verified to hold for both pure-Python and\npyo3-compiled getters, which is the case that matters here since\nextension libraries are compiled.\n\nAlso chains the original as __cause__ on the paths that do report an\nupgrade, so the arity error stays readable.\n\nTests cover all three outcomes. Confirmed non-vacuous: dropping the\ntraceback check fails only the inside-the-getter test, dropping\nset_cause fails only the upgrade test.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix!: remove physical_codec_from_pycapsule\n\nThe from_pycapsule! macros call the getter with no arguments. That is\ncorrect for __datafusion_physical_optimizer_rule__ and\n__datafusion_task_context_provider__, which take no session, but\n__datafusion_physical_extension_codec__ now takes the session it is\nbeing installed on, so this helper was the one member of the family left\nspeaking the old protocol.\n\nNothing in the tree called it, but datafusion-python-util is published\nby `cargo publish --workspace`, so it was still reachable. Against an\nupdated codec it raised a bare TypeError, bypassing the ImportError that\nnames the method. Against an outdated one it succeeded and produced a\ncodec resolving names against the wrong session -- the silent failure\nthe rest of this work exists to prevent.\n\nRemoving it is a breaking change to that crate, but the crate already\nbreaks this release: ffi_logical_codec_from_pycapsule gained its session\nparameter. A compile error pointing at the replacement beats a helper\nthat quietly binds to nothing.\n\nCallers move to ffi_physical_codec_from_pycapsule, which passes the\nsession, plus (\u0026ffi).into() where an Arc\u003cdyn PhysicalExtensionCodec\u003e is\nwanted -- what crates/core already does.\n\nDocuments both helper changes in the 55.0.0 upgrade guide, which until\nnow covered only the __datafusion_*__ method signatures and not the Rust\nhelpers the same authors call.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: check the FFI major version on every importer that can\n\nOnly ffi_query_planner_from_pycapsule validated the version a capsule\nreported. The codec and table provider importers dereference a foreign\nstruct through the same `unsafe { data.as_ref() }` and were happy to\naccept one built against a different DataFusion.\n\nExtracts the planner\u0027s inline check into check_ffi_version and applies\nit to the logical codec, physical codec, and table provider importers as\nwell. The helper is pub so extension libraries writing their own\nimporters can use it.\n\nTwo things the symmetry cannot reach, both now documented where someone\nwould look:\n\nFFI_TaskContextProvider, FFI_TableProviderFactory, and\nFFI_ExtensionOptions carry no version field, so their importers cannot\ncheck. The from_pycapsule!/try_from_pycapsule! macros are #[macro_export]\nand generic over the FFI type, so requiring a version field there would\nbreak downstream users holding one of those three; they stay unchecked\nand their doc comment now says to call check_ffi_version directly.\n\nThis is a diagnostic, not a soundness guarantee, and the helper says so:\n`version` is not the first field on any of these structs, so reading it\nalready assumes the local layout. It turns the realistic failure -- a\nlibrary compiled against a different DataFusion -- into a clear error\ninstead of undefined behaviour on first use, which is what\ndatafusion_ffi::version is documented to be for.\n\nVerified all four sites are wired by inverting the comparison and\nconfirming each one fires from the test suites.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: say where to relax the FFI version check\n\nExact equality is only right while datafusion_ffi::version tracks the\ncrate\u0027s semver major, which it does today, so the number moves on every\nmajor release whether or not the ABI changed. If a version span later\nbecomes compatible, a maintainer needs to know that this one body holds\nthe whole policy -- callers pass a value and no decision -- and that\nrelaxing it at a call site would reintroduce the split the helper was\nadded to remove.\n\nAlso records the likelier resolution: if the ABI is stable but version\nstill follows the crate major, upstream\u0027s compatibility marker is wrong\nfor every consumer, so the fix belongs there rather than in a local\nrange policy.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: route every capsule getter through call_capsule_getter\n\nThe table provider and table function importers each carried their own\ncopy of the TypeError-to-ImportError mapping, predating\ncall_capsule_getter and never folded into it. Both therefore missed the\ncorrection it since received: they rewrote a TypeError raised inside a\ncorrectly-signed getter into \"upgrade your library\", and discarded the\noriginal.\n\nThree copies of one mapping, two of them stale, is the reason to have\none. Both now call the shared helper, so they pick up the traceback\ndiscrimination and the __cause__ chain, and any later correction reaches\nall three by construction.\n\nTheir messages named DataFusion 52.0.0. The shared message names the\nmethod that refused the argument instead, which points at the specific\nhook rather than a release, and the upgrade guide carries the version\ndetail. call_capsule_getter is now pub, with a doc comment saying to use\nit rather than calling getattr directly.\n\nTests cover both outcomes on both paths. Verified against the previous\nbuild that they are non-vacuous: before this change the raises-inside\ncase produced the same misleading ImportError as the old-signature case.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: explain the dropped \u0026 in the codec migration snippet\n\nThe before and after snippets pass the task context provider\ndifferently, by reference in one and by value in the other, with nothing\nsaying why. Read as a diff it looks like a typo in one of them, and a\nreader correcting it would be puzzled when both versions compile.\n\nBoth are valid: the parameter is impl Into\u003cFFI_TaskContextProvider\u003e,\nwhich is satisfied by \u0026Arc\u003cdyn TaskContextProvider\u003e and by\nFFI_TaskContextProvider itself, and the latter is what\nffi_task_context_provider_from_pycapsule returns. The argument changes\nbecause the provider now comes from the session instead of a field,\nwhich is the point of the migration.\n\nThe contributor guide shows only the post-migration form, so it needs no\nequivalent note.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: spell out the token registry lifecycle in the example codecs\n\nThe README already describes these as one-shot registries that consume\neach token during decoding, but the source comments did not, and the\nsource is what someone reuses the pattern from. The existing comment\nwarned that the registry is process-local without saying that a decode\nremoves its entry, which is the constraint most likely to bite.\n\nDocuments both consequences on the registry accessors, where the\nmechanism lives, with a pointer from each struct doc:\n\n- Decode consumes the token, so the same encoded bytes cannot be decoded\n twice. Fine here because every plan is encoded immediately before the\n one decode that consumes it, but it rules out replaying a stored plan,\n retrying a decode, or fanning one plan out to several readers.\n- An encode that never reaches a decoder leaks for the life of the\n process. Normal operation does not: encode and decode counts balance\n exactly across repeated queries, which is what makes remove-on-decode\n the right trade here rather than a leak on every call.\n\nComments only.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: resolve a planner fallback when it is installed, not constructed\n\nMyQueryPlanner::new imported its fallback immediately, with no session\nto pass, so the fallback\u0027s getter was called with no arguments. That\nworks for a SessionContext, whose getter takes the session optionally,\nand for a raw capsule, which has no getter at all. It fails for another\nforeign planner, which implements the same protocol this type does and\nrequires the argument -- and layering on another planner is the case a\ndistributed engine actually needs. The docstring claimed fallback \"takes\nanything exporting __datafusion_query_planner__\", which was not true.\n\nHolds the Python object instead and imports it in\n__datafusion_query_planner__, where the session is in hand and can be\nforwarded. All three fallback kinds now work.\n\nDeferring also removes a footgun rather than adding one. Passing a\nSessionContext now delegates to whichever planner it holds at install\ntime, and since with_query_planner calls the getter before installing,\nthe context still reports its previous planner, so wrapping a context in\na planner installed on that same context does not recurse.\n\nArc\u003cPy\u003cPyAny\u003e\u003e rather than Py\u003cPyAny\u003e because pyo3 0.29 gates Py: Clone\nbehind the py-clone feature, and this type derives Clone. Matches how\nPythonTableFunctionCallable holds its callable.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: accumulate planner observations instead of overwriting them\n\nforeign_session, foreign_provider, and foreign_plan were written with\nstore, so each one described only the most recent plan. Their accessors\nare named foreign_*_observed, which asks whether the thing was ever\nseen, and the tests assert them after running more than one query. The\nexisting tests passed by luck.\n\nReproduced: after scanning a foreign provider and then running\nSELECT 1, foreign_provider_observed goes from True back to False.\n\nWrites them with fetch_or so a later plan cannot retract what an earlier\none observed. plan_calls already accumulated, used_fallback only ever\nstores true so it was already cumulative, and last_max_rows is\ndeliberately last-wins as its name says. Documents that split on the\nstruct, since it is the kind of thing that gets \"tidied\" back.\n\nConfirmed non-vacuous: with store restored, exactly the new test fails\nand the other 22 pass.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Update rev for upstream datafusion to pre-release of 55.1.0\n\n* refactor!: install the query planner in place instead of forking\n\n`with_query_planner` derived a new `SessionContext` to install a planner,\non the grounds that the receiver must not be modified. That mints a fresh\n`Arc\u003cSessionContext\u003e` allocation, and every FFI handle in play is bound to\nan allocation rather than to the logical session: `FFI_TaskContextProvider`\nholds its provider weakly, and a registered catalog provider upgrades that\nhandle on every `supports_filters_pushdown` and every `scan`.\n\nSo the natural `ctx \u003d ctx.with_query_planner(planner)` dropped the session\na foreign catalog had been registered on, and the next query failed with\n`TaskContextProvider went out of scope over FFI boundary`. Reproduced with\na `MyCatalogProvider` registered before the install and a `WHERE` clause to\nforce pushdown during logical optimization.\n\nRebinding cannot cover this. It reaches the codecs `PySessionContext` holds\nin its own fields; a codec embedded in a registered `FFI_CatalogProvider` —\nand in every `FFI_SchemaProvider` and `FFI_TableProvider` minted from it —\nhas no Python-side handle. Nor can a codec retain the session that built\nit: codecs are routinely handed to a provider that is registered straight\nback into that session, closing `SessionContext -\u003e catalog -\u003e FFI provider\n-\u003e FFI codec -\u003e SessionContext`.\n\nInstall in place instead, writing `SessionState` back through `state_ref()`\nexactly as `add_physical_optimizer_rule` already did. A session keeps one\n`Arc\u003cSessionContext\u003e` for life, so no handle is ever orphaned and the bug\ncannot occur. This deletes the fork and everything that existed to repair\nit: `ancestors`, `rebound_{logical,physical}_codec`, `exported_session` on\nboth codecs, and the `exported_ffi_*` builders.\n\nThe query planner lives in `SessionState`, so it belongs to the session\nrather than to a handle on it. `with_query_planner(planner) -\u003e\nSessionContext` therefore becomes `set_query_planner(planner) -\u003e None`,\nmatching `add_physical_optimizer_rule`.\n\nThe 55.1.0 pin may no longer be needed — its stated reason in Cargo.toml is\nthe rebinding this removes — but that is left alone pending a check of the\nrest of the PR.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* test: install codecs in the provider-survival test\n\nThe test registered a foreign catalog provider and no codecs, so it could\nnever finish: once the planner install stopped orphaning the provider, the\nquery got past filter pushdown and then failed at plan serialization with\n`LogicalExtensionCodec is not provided`. That is the same unrelated failure\n`test_query_planner_requires_provider_codec` already covers, and it would\nmask a dangling handle rather than expose one.\n\nInstall both provider codecs, and fold the codec-install-after-planner case\nin as a parameter rather than a near-duplicate test. Both orderings write\n`SessionState` — one installs the planner, the other rebuilds it against a\nnew codec — so both exercise the path that must not replace the session\u0027s\n`Arc\u003cSessionContext\u003e`.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: keep the UDF inlining setting when installing a codec\n\n`with_logical_extension_codec` and `with_physical_extension_codec` built\nthe replacement wrapper with `Python{Logical,Physical}Codec::new`, whose\nconstructor defaults `python_udf_inlining` to true. Installing a codec on\na context that had opted out therefore turned inlining back on without\nsaying so:\n\n ctx \u003d SessionContext().with_python_udf_inlining(enabled\u003dFalse)\n ctx \u003d ctx.with_logical_extension_codec(codec) # inlining silently back on\n\nThat matters beyond a stale flag. Inlining is what embeds a cloudpickled\ncallable in the wire format, and it is opt-out precisely because that is\nnot portable across interpreters and not something every deployment wants\nto ship. A codec install is not a request to change it.\n\nCarry the receiver\u0027s setting across instead. Both new tests fail on the\nprior build with `DFPYUDF` reappearing in the blob, and the paired\n`..._preserves_inlining_when_enabled` case pins the default-on direction so\nthe fix cannot degenerate into hard-coding it off.\n\nThe physical case is covered in `test_plans.py` rather than alongside the\nlogical one: `Expr.to_bytes` only routes through the logical codec, so an\nassertion there would pass with the physical bug still present. It takes an\n`ExecutionPlan` to observe.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: route the last seven capsule getters through call_capsule_getter\n\n`call_capsule_getter` claimed every capsule getter went through it, so the\nmapping from a refused argument to a diagnosable error would live in one\nplace. Seven sites still called `getattr(...).call0()` or `.call1(...)`\ndirectly, so the claim was false and four of them — catalog provider,\nschema provider, catalog provider list, table provider factory — still\nhanded an out-of-date extension library a bare `TypeError`.\n\nThose four take the host\u0027s logical extension codec rather than the session,\nwhich is why they could not simply be passed through as they stood: the\ndiagnostic would have told a catalog author their method \"must accept the\nSessionContext\", pointing them at the wrong parameter. Carry the argument\nand its description together in a `CapsuleGetterArg` so one diagnostic can\nserve getters that take a session, getters that take a codec, and getters\nthat take nothing. `Option\u003c\u0026Bound\u003cPyAny\u003e\u003e` still converts into it, so the\ndocumented `*_from_pycapsule` helper signatures are unchanged.\n\nThe three zero-argument getters (scalar, aggregate, window UDF) route\nthrough as well. Nothing can be refused there, but the rule is easier to\nfollow with no exceptions to remember.\n\nAlso add `validate_pycapsule` to `table_provider_from_pycapsule` and\n`ffi_logical_codec_from_pycapsule`, the two extraction sites that lacked\nit. This is not redundant with `pointer_checked`, despite appearances:\n`pointer_checked` bottoms out in CPython\u0027s `PyCapsule_GetPointer`, whose\nerror is the fixed string `PyCapsule_GetPointer called with incorrect\nname` and names neither the expected capsule nor the one received. Say so\nin a doc comment so it does not get \"simplified\" away later.\n\nDrop the unused `datafusion-proto` dependency from the query planner\nexample while here.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: correct the pin rationale and derived-context caveats\n\nThe Cargo.toml comment justified the pre-release pin by \"the FFI codec\nrebinding in `PySessionContext::derived_parts`\", a symbol that no longer\nexists. That rebinding was not removed, it moved into\n`set_session_query_planner`, which every `with_*` method calls. It depends on\n`FFI_QueryPlanner::new_with_ffi_codecs` unwrapping a `ForeignQueryPlanner` and\nreplacing its codecs, a swap that is a silent no-op before 55.1.0\n(apache/datafusion#24722). So the pin is still required, not droppable.\n\nThe `with_*` methods rebuild the installed planner on the *shared* session, so\nthe rebind takes effect even when the returned context is discarded.\n`with_python_udf_inlining` additionally claimed \"the original session is\nunchanged\", which the rebuild contradicts; it is the context\u0027s own codec\nsettings that are unchanged.\n\nAlso:\n- Note that the arity-vs-body TypeError split in `call_capsule_getter` holds\n only because the call originates in Rust. A Python-level shim between the\n host and the getter would supply a traceback and silently disable it.\n- Document the new FFI major-version gate in the 55.0.0 upgrade guide. Table\n providers previously performed no such check, so a mismatched extension\n library that used to load now raises ImportError.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* test: pin the planner rebind and guard the planner example on log warnings\n\n`test_a_discarded_derived_context_still_rebinds_the_planner` covers the\nsurprising half of the shared-session rebuild: a codec installed through a\ncontext that is then thrown away still binds to the session\u0027s planner. A fresh\ncodec instance makes it observable, since the planner encodes the outbound\nlogical plan with whichever codec it holds. Verified non-vacuous -- a codec\nbuilt but never installed reports zero encode calls.\n\nThe query planner example had no conftest, so it ran without the autouse\nfail-on-log-warning handler the provider example uses, despite calling\n`pyo3_log::init()` for the same reason. Copied verbatim; the suite passes\nunder it with no allowlist needed.\n\nTwo table provider tests called the deprecated `register_table_provider`,\nwhich is a one-line forwarder to `register_table`, so they reached the same\ncapsule path while emitting DeprecationWarning.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* chore: drop the stub uv.lock from the query planner example\n\nIt locked nothing -- a single editable entry for the crate itself and no\ndependencies. Nothing consumed it either: CI runs both example suites with\n`uv run --no-project`, and the older datafusion-ffi-example has no lock file\nat all. The codespell skip list in pyproject.toml matches on a bare `uv.lock`\nglob, so it still covers the repository root lock.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: scope the planner codec rebind to one layer\n\nInstalling a codec rebuilds the installed planner against it, but the\nrebuild reaches exactly one `ForeignQueryPlanner`. A planner that resolved\na fallback at install time keeps that fallback\u0027s codecs, and neither side\ncan repair it: the host has no handle past the first layer, and the planner\nlibrary cannot re-derive codecs at plan time because `FFI_QueryPlanner`\nholds them by value and `Session` exposes no accessor for the host\u0027s\ncurrent ones. Tracked upstream in apache/datafusion#24762.\n\nThe examples cannot demonstrate it. Their fallback lives in the same cdylib\nas its wrapper, and `From\u003c\u0026FFI_QueryPlanner\u003e` short-circuits on a matching\n`library_marker_id`, so a same-library hop never serializes. Measured: a\nlayered planner produces the same codec traffic as a flat one.\n\nWhat is demonstrable is that the session\u0027s planner tracks whichever handle\nwrote it last, so re-installing a planner from the original handle rebinds\nthe session back to that handle\u0027s codecs rather than picking up a codec\ninstalled through a derived one. Pinned by a new test as the sequel to\n`test_a_discarded_derived_context_still_rebinds_the_planner`.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: skip the planner rebind when inlining is unchanged\n\n`with_python_udf_inlining` rebuilds the handle\u0027s codecs and rebinds the\nsession\u0027s planner to them, which is state shared with every other handle on\nthat session. Asking for the setting a context already has changes nothing,\nso it should not pay that side effect: a defensive no-op toggle on `ctx`\notherwise drags the planner back onto `ctx`\u0027s codecs and silently undoes a\ncodec installed through another handle.\n\nReturning the existing codecs is observationally equivalent to the rebuild\notherwise -- it wraps the same inner codec in a fresh `Python*Codec` -- so\nthe guard is only visible through that side effect. The new test fails\nwithout it with `assert 0 \u003e 0`.\n\nAlso pins the divergence the rebind creates. The planner carries the codecs\nof whichever handle installed it last; every other path on a context uses\nthat context\u0027s own codec field. Those can be different handles, and then\n`Expr.to_bytes(ctx)` and `ctx.sql(...)` encode with different codecs on the\nsame `ctx`. Stated as a rule in the FFI guide rather than left implicit in\nthe description of the mechanism.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: route the last three capsule getters through the helper\n\n`call_capsule_getter` documents itself as having no exceptions to remember,\nand had three. The `from_pycapsule!` and `try_from_pycapsule!` macros each\nhand-rolled the getattr/call0 prologue, and `SessionConfig::with_extension`\ncalled `__datafusion_extension_options__` directly. All three take no\nargument, so with `CapsuleGetterArg::None` the helper returns the original\nerror untouched and the substitution is behavior-identical.\n\nThis is what makes the claim true rather than aspirational: the grep in the\nFFI capsule protocol skill now turns up no bare call sites.\n\n`with_extension` also gains the `validate_pycapsule` check every other\nextractor has, so a mismatched capsule is named instead of raising CPython\u0027s\nfixed \"called with incorrect name\". It keeps its own `hasattr` precheck,\nwhose AttributeError is more useful than the helper\u0027s pass-through, and\ntakes no version check because `FFI_ExtensionOptions` carries no version\nfield.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* chore: drop the unused datafusion-proto dep from the util crate\n\nIts only use was `physical_codec_from_pycapsule`, whose `dyn\nPhysicalExtensionCodec` output type came from that crate. That helper is\ngone, replaced by `ffi_physical_codec_from_pycapsule`, which returns the FFI\ntype and leaves the conversion to the caller. Nothing in `crates/util/src`\nreferences `datafusion_proto` any more, and it is not re-exported, so\nextension libraries depending on this crate are unaffected.\n\nThe package stays in Cargo.lock; crates/core still uses it.\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "2f528d2d53a550ba5434466e32af78b8c5bfac82", "tree": "48d00ed7a1d094e6e77ef3bf25c159ae2820deef", "parents": [ "2bd80eca6151917e02721dbb7025cfad6fcd74aa" ], "author": { "name": "Andy Grove", "email": "agrove@apache.org", "time": "Wed Aug 26 12:48:22 2026 -0600" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 26 12:48:22 2026 -0600" }, "message": "chore: use DataFusion 55.0.0 from crates.io (#1695)\n\nNow that DataFusion 55.0.0 is published to crates.io, remove the\n[patch.crates-io] entries that pointed at the 55.0.0-rc3 git tag." }, { "commit": "2bd80eca6151917e02721dbb7025cfad6fcd74aa", "tree": "cb80595f648c19c1d311de45572e20ac4de151a9", "parents": [ "38422e440bae361d46520978fb4decb3fe2ae97e" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri Aug 14 19:46:43 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 14 19:46:43 2026 -0400" }, "message": "chore: Bump release candidate version to 55.0.0rc3 (#1686)\n\n* Bump release candidate version\n\n* spark slice function now respects inner field name" }, { "commit": "38422e440bae361d46520978fb4decb3fe2ae97e", "tree": "f4e728bad775bfbcc9493b954533fd00fb6c7857", "parents": [ "c577c0f41785978918aa1683f6020adab448cc5b" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Thu Aug 13 11:38:50 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Aug 13 11:38:50 2026 -0400" }, "message": "Bump to 55.0.0 rc2 (#1685)" }, { "commit": "c577c0f41785978918aa1683f6020adab448cc5b", "tree": "2e05becbb00519b1aafbfbdf91873b2a4c9c7dd1", "parents": [ "cc2ec5c620f7fce1dab05ed72ef6627a4348a6ee" ], "author": { "name": "hakunamatata-sb", "email": "121852557+hakunamatata-sb@users.noreply.github.com", "time": "Sun Aug 09 18:20:53 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Aug 09 08:50:53 2026 -0400" }, "message": "[PyLimit] adding skip and fetch with datafusion expr support (#1674)" }, { "commit": "cc2ec5c620f7fce1dab05ed72ef6627a4348a6ee", "tree": "801affb962829f631dd7bd9bbdaf9dee02ef52fd", "parents": [ "4702915c4f019a251809d5d7d9d2014e7d7c4251" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Thu Aug 06 10:52:41 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Aug 06 10:52:41 2026 -0400" }, "message": "chore: Update to latest main on upstream DataFusion (#1662)\n\n* Update to main on the upstream datafusion to prepare for DF55 release\n\n* Deprecate singular external table location\n\nAI Disclosure: This code was written in part by an AI agent.:\n\n* Fix tests for updated DataFusion behavior\n\nAI Disclosure: This code was written in part by an AI agent.:\n\n* create table requires a location\n\n* Remove unused Python token from Substrait deserialization\n\nAI Disclosure: This code was written in part by an AI agent.:\n\n* test: cover external table locations binding\n\nAI Disclosure: This code was written in part by an AI agent.:" }, { "commit": "4702915c4f019a251809d5d7d9d2014e7d7c4251", "tree": "bcb0cd5a65d85a1c1abc7e85cb07869ad72ac86e", "parents": [ "7eaef922de85e62ae9c5624a18c60f529fee9429" ], "author": { "name": "BharatDeva", "email": "bharatdevagir@gmail.com", "time": "Mon Aug 03 03:41:32 2026 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 03 16:41:32 2026 +0800" }, "message": "Add is_nan aliases (#1643)\n\nAdds datafusion.functions.is_nan(...) as an alias for isnan(...) and adds Expr.is_nan() as the corresponding expression method. Existing isnan() behavior is unchanged.\n\nThe PR also adds coverage for both the function-level alias and the expression method." }, { "commit": "7eaef922de85e62ae9c5624a18c60f529fee9429", "tree": "91ebe91cefe5ce84f007f3373adf7a6b21508f8d", "parents": [ "3d9583ca85c375f6797401b88ddde079a0c30319" ], "author": { "name": "BharatDeva", "email": "bharatdevagir@gmail.com", "time": "Mon Aug 03 03:30:57 2026 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 03 16:30:57 2026 +0800" }, "message": "fix: tighten scalar UDF return type hints (#1642)\n\nThis PR tightens the scalar UDF annotations in python/datafusion/user_defined.py:\n\nthe callable return type is bounded to pa.Array\nthe scalar UDF constructor accepts return_field: pa.Field\nthe decorator form accepts return_field: pa.DataType | pa.Field" }, { "commit": "3d9583ca85c375f6797401b88ddde079a0c30319", "tree": "94e3fdd1e299cc29b8cba2ecda2c0951bd0f39a7", "parents": [ "4906d391beb0d224c034e60809db72172f0244fe" ], "author": { "name": "Wonseok Yang", "email": "onestone.yang@gmail.com", "time": "Fri Jul 31 04:13:14 2026 +0900" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 30 15:13:14 2026 -0400" }, "message": "docs: fix bootstrap instructions in contributor guide (#1618)\n\n* docs: add missing cd step in contributor guide bootstrap\n\n* docs: fix maturin develop flag typo (-uv -\u003e --uv)" }, { "commit": "4906d391beb0d224c034e60809db72172f0244fe", "tree": "cbaa77144831ba41c8fe62f68b212a1a6056f39a", "parents": [ "5cc32306a8435ab7fb0b55216d97e0084bc1b9d1" ], "author": { "name": "Jared Yu", "email": "qzyu999@gmail.com", "time": "Thu Jul 30 12:12:21 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 30 15:12:21 2026 -0400" }, "message": "Add object_store parameter to register/read file methods for thread-safe credentials (#1625)\n\n* Add object_store parameter to register/read file methods\n\nAdd an optional object_store parameter to register_parquet, read_parquet,\nregister_csv, read_csv, register_json, read_json, register_avro, read_avro,\nregister_arrow, and read_arrow methods.\n\nWhen provided, the store is automatically registered for the URL scheme\nand host parsed from the path, removing the need to manually call\nregister_object_store separately. This enables thread-safe cloud\ncredential handling without os.environ mutation.\n\nIncludes unit tests (mock-based) for URL parsing logic and integration\ntests using LocalFileSystem for end-to-end validation.\n\nCloses #1624\n\n* Fix file:// URL handling in _register_object_store_for_path\n\n- Allow empty netloc for file:// scheme (e.g. file:///tmp/data.parquet)\n- Require netloc (host/bucket) only for non-file schemes (s3, gs, az, https)\n- Add doctest +SKIP to register_parquet example referencing nonexistent file\n- Use Path.as_uri() in end-to-end tests for portable file:// URL construction\n- Add tests for file:// acceptance and scheme-without-host rejection\n\n* Refactor tests to use pytest.mark.parametrize per reviewer feedback\n\nReplace class-based test structure with flat parametrized functions\nto match the repository\u0027s testing conventions.\n\n---------\n\nCo-authored-by: Tim Saucer \u003ctimsaucer@gmail.com\u003e" }, { "commit": "5cc32306a8435ab7fb0b55216d97e0084bc1b9d1", "tree": "2f7c344581c6c294ebdf2b32dfd32871e0618854", "parents": [ "6d16ecd4bd26b597553d7f7d321745a286bcf18f" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Tue Jul 28 15:15:29 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 28 15:15:29 2026 -0400" }, "message": "Use allowlisted Rust toolchain action revision (#1661)\n\n* Use allowlisted Rust toolchain action revision\n\nAI Disclosure: This code was written in part by an AI agent.:\n\n* Set rust toolchain so developer and CI get a consistent experience\n\n* minor clippy fixes" }, { "commit": "6d16ecd4bd26b597553d7f7d321745a286bcf18f", "tree": "8826d2ce1e4ceebc190de0901b396d8987e8e021", "parents": [ "699e00ef15686550aaec576987a691b38c7d625f" ], "author": { "name": "kosiew", "email": "kosiew@gmail.com", "time": "Wed Jul 29 00:45:57 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 28 12:45:57 2026 -0400" }, "message": "Update pinned Rust toolchain action revision (#1635)\n\n* fix: replace rust-toolchain action with rustup commands for toolchain setup\n\n* fix: pin dtolnay/rust-toolchain sha\n\n* pin dtolnay/rust-toolchain sha" }, { "commit": "699e00ef15686550aaec576987a691b38c7d625f", "tree": "3e56fc98bc9d25296914f1fe3c38da44e0f4636f", "parents": [ "5ce35f8993151fe8cd8c4fb7865aee102752bc5b" ], "author": { "name": "Fangchen Li", "email": "fangchen.li@outlook.com", "time": "Tue Jul 14 00:55:21 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 14 15:55:21 2026 +0800" }, "message": "fix: no panic on empty partition in register_record_batches (#1627)\n\n* fix: no panic on empty partition in register_record_batches\n\n`register_record_batches` read the schema via unchecked `partitions.0[0][0]`,\nwhich panics when a partition contains no record batches (e.g. the batches of\nan empty pyarrow table, whose `to_batches()` returns `[]`).\n\nTake the schema from the first available batch across all partitions and return\na clear error when there is none, mirroring the empty-table handling added for\n`from_arrow_table` in #613. A non-empty later partition now also works instead\nof panicking on an empty leading one.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: address review feedback\n\n- Use find_map(|p| p.first()) instead of flatten().next() to make the\n \"first batch in any partition\" intent clearer (per @kosiew).\n- Extend regression test to also assert the empty outer partition list\n case ([]) raises the same ValueError.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "5ce35f8993151fe8cd8c4fb7865aee102752bc5b", "tree": "4ab97026ba135ca6748405f4b6ae9d99f7698d79", "parents": [ "5b341d36ff14433be7989bdb0061dfc0ae2eca10" ], "author": { "name": "Adrian Garcia Badaracco", "email": "1755071+adriangb@users.noreply.github.com", "time": "Fri Jul 10 01:11:05 2026 +0000" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 10 09:11:05 2026 +0800" }, "message": "fix: clone parquet-testing into `parquet` to match submodule path in verify script (#1617)\n\n`verify-release-candidate.sh` clones the parquet-testing repository into a\n`parquet-testing` directory, but the git submodule path declared in\n`.gitmodules` is `parquet`, and every parquet-based test reads its data from\n`parquet/data/...` (e.g. `python/tests/test_io.py`, `test_store.py`,\n`test_context.py`). The arrow-testing clone one line above already correctly\nuses the submodule path (`testing`).\n\nThis is currently latent because the script\u0027s `python3 -m pytest` invocation is\ncommented out, so the wrong directory is never exercised. Fixing the path is a\nprerequisite for enabling the test run during release verification.\n\n\nClaude-Session: https://claude.ai/code/session_01Pj5DVU7MaammM2nfHh1ZRG\n\nCo-authored-by: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "5b341d36ff14433be7989bdb0061dfc0ae2eca10", "tree": "d17205bc85f6386cbe51bed10474aec4b532f4e6", "parents": [ "21b52998cb5606f7700b0ba672314c5350ead2ca" ], "author": { "name": "Aurélien Pupier", "email": "apupier@ibm.com", "time": "Thu Jul 02 21:49:25 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 02 15:49:25 2026 -0400" }, "message": "Restrict trigger push branch for GitHub Workflow (#1619)\n\nFeature branches rarely need their own CI runs: the code is already\ntested when a pull request is opened against a release branch. If the\npush trigger has no branch restriction and pull_request is also\nconfigured, every push to a branch with an open PR runs the workflow\ntwice: once for the push and once for the PR synchronisation.\n\nAlways give the push trigger an explicit list of branches: this stops\nbranches created from a release branch from inheriting its workflow\nruns.\n\nsee https://cwiki.apache.org/confluence/pages/viewpage.action?pageId\u003d430408443#GitHubActionsRecommendedPractices-Restrictthepushtriggertospecificbranches\n\nSigned-off-by: Aurélien Pupier \u003capupier@ibm.com\u003e" }, { "commit": "21b52998cb5606f7700b0ba672314c5350ead2ca", "tree": "c581a030c9389651d955551d14f69275fe3516cc", "parents": [ "5118a8bccfbe93cdfe481bcd6fad131c5faf4d00" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Mon Jun 29 07:24:01 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jun 29 07:24:01 2026 -0400" }, "message": "Update version to 54.0.0 (#1610)\n\n* Bump version number. Generate changelog.\n\n* add symlinks to rat exclusion list" }, { "commit": "5118a8bccfbe93cdfe481bcd6fad131c5faf4d00", "tree": "76b39226739852a5b20ce17095947c9347a8d2b7", "parents": [ "a4cf887a0317dde6ca8241134c2c248a951d4fe6" ], "author": { "name": "Liang-Chi Hsieh", "email": "viirya@gmail.com", "time": "Mon Jun 29 14:12:55 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jun 29 14:12:55 2026 +0800" }, "message": "Fix non-portable `find -iname` in verify-release-candidate.sh (#1613)\n\nThe SNAPSHOT check used `find -iname \u0027Cargo.toml\u0027` without a starting\npath. GNU find tolerates this, but BSD find (macOS) rejects it with\n`find: illegal option -- i`. The error makes the piped `xargs grep`\nreceive no input and return non-zero, so the `if` silently evaluates\nfalse and the SNAPSHOT check is skipped on macOS rather than running.\n\nAdd the explicit `.` starting path so the check works on both GNU and\nBSD find.\n\nCo-authored-by: Isaac" }, { "commit": "a4cf887a0317dde6ca8241134c2c248a951d4fe6", "tree": "450a808b4beb54012960eec9a1f86d5a4b7c7a38", "parents": [ "b6e5d714cac2cc6a1f26828536002071956783f9" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Thu Jun 25 14:10:05 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jun 25 14:10:05 2026 -0400" }, "message": "remove 3.13 freethreaded builds (#1609)" }, { "commit": "b6e5d714cac2cc6a1f26828536002071956783f9", "tree": "67ecebea5bd3fb99d5c53fd53c85e7ee71067a62", "parents": [ "4e223ff09a70c0d54d7f658bc35cf66eb0b724df" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Thu Jun 25 13:32:34 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jun 25 13:32:34 2026 -0400" }, "message": "chore: resolve audits after DF54.0.0 update to skill (#1608)\n\n* docs: sync datafusion_python SKILL.md with current API\n\nApply findings from a SKILL.md audit against the current public API:\n\n- Pitfall #2: remove the incorrect claim that `.between(...)` needs\n `lit()`. Its bounds auto-wrap, so `col(\"a\").between(1, 10)` is correct.\n Update the BETWEEN examples and SQL-to-DataFrame row to drop the\n redundant `lit()`.\n- Pitfall #6: arithmetic on aggregates inside `aggregate(...)` is fixed in\n datafusion-python 54; annotate the pitfall as applying to 53 and earlier.\n- Spark divergence table: drop the `substring` row. Both namespaces are\n 1-indexed, so listing it in a \"behaves differently\" table is misleading.\n- Document APIs not previously covered: `Expr.try_cast`, `DataFrame.cast`\n mapping form, DataFrame-level `fill_null`, `SessionContext.read_batch` /\n `read_batches`, `array_contains`, and `DataFrame.col`/`column` for join\n disambiguation.\n\nCo-Authored-By: Claude Opus 4.8 \u003cnoreply@anthropic.com\u003e\n\n* Resolve segfault in datafusion-python 54 with pyarrow 24\n\n---------\n\nCo-authored-by: Claude Opus 4.8 \u003cnoreply@anthropic.com\u003e" }, { "commit": "4e223ff09a70c0d54d7f658bc35cf66eb0b724df", "tree": "ee45a9eb454eda7c70db69a79fc29af8b2265a78", "parents": [ "5b7ad645bde69f380bb5387100fde5f72804fded" ], "author": { "name": "kosiew", "email": "kosiew@gmail.com", "time": "Mon Jun 22 16:02:34 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jun 22 16:02:34 2026 +0800" }, "message": "Add deprecation warnings for Expr passed to confirmed literal-only function arguments (#1605)\n\n* feat: add deprecation warning for Expr passed to literal-only args\n\n- Introduced shared `_warn_if_expr_for_literal_arg` in `functions/__init__.py`\n- Added `DeprecationWarning` for the following methods when `Expr` is passed as argument:\n- `encode(..., encoding\u003dExpr)`\n- `decode(..., encoding\u003dExpr)`\n- `digest(..., method\u003dExpr)`\n- `arrow_cast(..., data_type\u003dExpr)`\n- `arrow_try_cast(..., data_type\u003dExpr)`\n- `arrow_metadata(..., key\u003dExpr)`\n\ntest: update tests to check for warnings\n\n- Implemented tests in `test_functions.py` to ensure:\n- Warning is raised for `Expr` form\n- No warning for native literal form\n\n* fix(tests): resolve E501 line length issue in test_functions.py\n\n* feat: consolidate warning helpers and update temporal function calls\n\n- Collapsed warning helpers into a single function `_warn_if_expr_for_literal_arg` in `python/datafusion/functions/__init__.py`.\n- Updated callers of temporal functions `_date_part` and `_date_trunc` in `python/datafusion/functions/__init__.py`.\n- Modified the digest behavior test in `python/tests/test_functions.py` to use native method strings.\n- Updated the encode/decode behavior test to use native \"base64\" in `python/tests/test_functions.py`.\n\n* docs: update function examples in common-operations user guide\n\n- Replaced `string_literal(\"Float64\")` with `\"Float64\"` in examples.\n- Replaced `str_lit(\"Int32\")` with `\"Int32\"` in examples.\n- Removed unused `string_literal` and `str_lit` imports." }, { "commit": "5b7ad645bde69f380bb5387100fde5f72804fded", "tree": "62d5ef79ff5b3cd3c0db8fe1bd510987f6c283ef", "parents": [ "ac436976c818af890b675b6497ecd725a3609eda" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Wed Jun 17 18:21:53 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jun 17 18:21:53 2026 -0400" }, "message": "feat: accept native Python literals on date/time functions (#1563)\n\nWiden date/time scalar function signatures to accept native Python\n``str``/``int`` literals alongside ``Expr``:\n\n- ``date_bin``: ``stride``, ``source``, ``origin`` accept ``Expr | str``.\n- ``make_date``, ``make_time``: components accept ``Expr | int``.\n- ``to_date``, ``to_time``, ``to_timestamp``, ``to_timestamp_{millis,\n micros,nanos,seconds}``, ``to_unixtime``: ``*formatters`` accept\n ``Expr | str``.\n\nAdd ``coerce_to_expr_list`` public helper in ``datafusion.expr`` mirroring\n``coerce_to_expr`` / ``ensure_expr_list`` for variadic call sites.\n``date_bin`` uses ``Expr.string_literal`` directly because its planner\ncoerces ``Utf8`` (not ``Utf8View``) literals to ``Interval``/``Timestamp``.\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "ac436976c818af890b675b6497ecd725a3609eda", "tree": "3d3d6b67a6c71b8051b4f6eb3f4de89b440813ea", "parents": [ "2d074713d1505256ed92a07e77c18ee43713a303" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Wed Jun 17 18:16:13 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jun 17 18:16:13 2026 -0400" }, "message": "chore: update rust dependencies (#1604)\n\n* cargo update\n\n* fix broken test on main" }, { "commit": "2d074713d1505256ed92a07e77c18ee43713a303", "tree": "7e54074583f8eed5c34e746a9c7d88a645fe2889", "parents": [ "55fd2c925c95711ad7222a5a48b6691d482d6056" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Wed Jun 17 13:03:04 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jun 17 13:03:04 2026 -0400" }, "message": "feat: expose spark-compatible functions (#1564)\n\n* feat: expose Spark-compatible functions (#1482)\n\nAdd `datafusion.functions.spark` module exposing the upstream\n`datafusion-spark` crate\u0027s UDF/UDAF library (~87 functions across string,\nmath, datetime, hash, array, aggregate, bitwise, bitmap, conditional,\ncollection, conversion, json, map, url categories).\n\nFor DataFrame use, import the typed Python wrappers from\n`datafusion.functions.spark`. For SQL use, call\n`SessionContext.enable_spark_functions()` to register the Spark UDFs by\nname (overriding DataFusion built-ins of the same name with their Spark\nsemantics — NULL-propagating `concat`, 1-indexed `substring`, HALF_UP\n`round`, etc.).\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* chore: drop unused borrow_deref_ref allows\n\nSeven `#[allow(clippy::borrow_deref_ref)]` attributes on module\ndeclarations in `crates/core/src/lib.rs` had become stale — the only\nremaining lint hit was a redundant `\u0026*x.as_str()` pattern in\n`parse_file_compression_type`. Rewriting that call to\n`\u0026x.unwrap_or_default()` lets every allow come off, removing noise that\nnew modules were copying without need.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: tighten spark_functions macros via expr_fn\n\nSwitch most spark wrappers from UDF-direct path (which forced\n`spark_udf_fixed!(name, fn_category::name, args...)` repetition) to a\n`spark_expr_fn!` macro that mirrors the existing `expr_fn!` macro in\n`functions.rs`, so calls collapse to `spark_expr_fn!(sha2, arg1\nbit_length);`.\n\nUDF-direct retained for genuinely variadic functions whose upstream\n`expr_fn` wrappers were generated with a single-`Expr` arm by\n`export_functions!` (concat, array, xxhash64, parse_url family, etc.) so\nthat the Python side keeps its `*args` ergonomics.\n\nAggregates collapse the same way via `spark_aggregate!` mirroring\n`aggregate_function!`. Net 173 lines removed.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: clarify spark functions cover DataFrame API too\n\nThe intro wording implied \"SQL functions\" only; the same wrappers are the\nprimary entry point for the DataFrame API as well.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: rewrite spark DataFrame intro for users\n\nReplace API-speak (\"Import the submodule\", \"Returned values are Expr\ninstances that compose\") with a concrete description of where users can\nactually drop these calls.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: defer spark function list to API reference\n\nHand-maintained category list would drift from the actual module as\nupstream `datafusion-spark` adds/removes functions. Replace with a\npointer to the AutoAPI-generated reference, which renders from the\nmodule itself.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* test: replace spark doctest skips with verified examples\n\n38 wrappers carried `# doctest: +SKIP` because outputs weren\u0027t verified at\nauthoring time. Run each with concrete inputs, capture actual outputs, and\ninline the values so the doctests execute and stay correct.\n\nCovers datetime (20), URL (5), bitmap (3), map (3), and remaining hash,\nJSON, math, string, conversion, and format_string cases. Net new doctest\ncoverage: 65 examples now run that were skipped before; total skipped\nacross the suite drops from 53 to 12.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor(spark): rename function params to match pyspark\n\nAlign positional parameter names in `functions.spark` with pyspark.sql.functions:\n- aggregate first positional → `col` (avg, try_sum, collect_list, collect_set)\n- unary `arg` → `col` across math/string/byte/datetime helpers\n- multi-arg renames: array_contains (col, value), array (*cols), shuffle (col),\n array_repeat (col, count), slice (x, start, length), shiftleft/right/rightunsigned\n (col, numBits), add_months (start, months), date_add/sub (start, days),\n date_diff (end, start), date_trunc (format, timestamp), time_trunc (unit, time),\n trunc (date, format), next_day (date, dayOfWeek), from/to_utc_timestamp\n (timestamp, tz), sha2 (col, numBits), xxhash64 (*cols), map_from_arrays\n (col1, col2), width_bucket (v, min, max, numBucket), substring (str, pos, len),\n concat (*cols), elt (*inputs), is_valid_utf8/make_valid_utf8 (str)\n\nBodies updated to reference the new names; positional callers unaffected.\nThis finishes Category 1 / Category 4 (spark-side BOTH-bucket) renames from\nPYSPARK_ALIGNMENT_PLAN.md PR 1.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat(spark): make pyspark-optional params optional\n\nMatch pyspark\u0027s optional-parameter surface in the spark namespace:\n- make_dt_interval, make_interval: all parts default to zero (int32 0 / lit 0.0)\n- str_to_map: pair_delim defaults to \u0027,\u0027, key_value_delim defaults to \u0027:\u0027\n- round: scale defaults to 0 (HALF_UP rounding to nearest integer)\n- shuffle: accepts `seed` kwarg for pyspark parity; raises NotImplementedError\n for non-None values until the Rust binding supports it\n- like, ilike: accept `escapeChar` for pyspark parity; same NotImplementedError\n guard; first positional renamed `string` → `str` to match pyspark\n\nceil/floor `scale\u003d` deferred — the underlying Rust expr_fn is single-arg.\n\nAdded a module-level `_ZERO_I32` literal to avoid rebuilding the pyarrow\nint32 zero scalar on every call.\n\nTests: positional-compat coverage for aggregates (`spark.avg(col)` etc.),\ndefaults-omitted cases for the optional-arg functions, and\nNotImplementedError cases for `shuffle(seed\u003d)` and `like/ilike(escapeChar\u003d)`.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor(spark): reshape varargs to match pyspark signatures\n\nReplace generic ``*args`` with explicit pyspark-style signatures:\n- json_tuple(col, *fields) — first positional is the JSON expr\n- format_string(format, *cols) — `format` is the printf template; a plain\n ``str`` is auto-promoted to a literal\n- parse_url(url, partToExtract, key\u003dNone) — `key` is optional and only\n meaningful with ``partToExtract\u003d\u0027QUERY\u0027``\n- try_parse_url(url, partToExtract, key\u003dNone) — same shape\n- url_decode(str), try_url_decode(str), url_encode(str) — single-argument\n forms (multi-arg calls were always semantically wrong)\n\nTests cover the three-arg parse_url path and the plain-str format_string\nauto-promotion.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(skills): cover the new spark function namespace\n\n`functions.spark` mirrors `pyspark.sql.functions` and now ships on this\nbranch. Update every skill that references the function surface:\n\n- skills/datafusion_python/SKILL.md (user-facing): add an import\n reference, a Core Abstractions row, and a \"Spark-Compatible Functions\"\n subsection listing coverage by category, the SQL-vs-DataFrame usage\n (`enable_spark_functions`), and the divergent-semantics table\n (concat NULL, round HALF_UP, trunc) so callers know which namespace\n to pick.\n- .ai/skills/check-upstream/SKILL.md: new area for the `datafusion-spark`\n crate with the coverage policy (parity with pyspark, extras allowed\n when positional pyspark calls still work). Hygiene check also now\n spans `functions/spark.py`\u0027s `__all__`.\n- .ai/skills/audit-skill-md/SKILL.md: add `functions.spark` to the\n surface table and a `spark-functions` scope so this audit also\n validates the new subsection and divergent-semantics table.\n- .ai/skills/make-pythonic/SKILL.md: explicit scope note that the\n spark namespace is a deliberate pyspark mirror — generic native-type\n coercion does not apply there. Path references updated to the new\n `functions/__init__.py` module layout.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(skills): drop references to PYSPARK_ALIGNMENT_PLAN.md\n\nThe plan file is a working document, not a committed artifact, so skills\nmust not point at it. Inline the one substantive reference (the\n\"deferred to follow-up PRs\" callout in make-pythonic) and drop the\ncross-cutting pointer from check-upstream.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(skills): make-pythonic also targets functions.spark\n\nPrevious guidance said to skip the spark namespace entirely. That was\nwrong: the spark namespace should also feel pythonic — it just carries\nthe extra constraint that every signature must remain compatible with\npyspark.sql.functions (parameter names, positional order, accepted input\ntypes). Pythonic widenings like `Expr → Expr | int` are on-brand there\nbecause pyspark itself accepts the int form.\n\nRewrite the scope section to spell out the compatibility rules (keep\nparameter names/order; widen input types, never narrow; extra kwargs\ndefault to None) and extend \"How to Identify Candidates\" to include\n`functions/spark.py`.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(skill): point at spark __all__ instead of enumerating\n\nEnumerating spark functions in the user-facing skill duplicates the\n__all__ list in python/datafusion/functions/spark.py and will drift the\nmoment a new function lands or is renamed. Replace the per-function\nlisting with a category summary and a discovery snippet that queries\nthe actual __all__ at runtime, which is the authoritative source.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(spark): use isoformat in spark_cast doctest\n\npyarrow tzinfo repr differs across versions (\u003cUTC\u003e vs\nzoneinfo.ZoneInfo(key\u003d\u0027UTC\u0027)), breaking the doctest on some platforms.\nisoformat is stable across versions.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(spark): fix map_from_entries doctest to use the right function\n\nThe example called map_from_arrays, so it never exercised\nmap_from_entries. Build an array-of-struct input and call the\ndocumented function.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(spark): list Spark-compatible aggregates in aggregations guide\n\nAdd avg, try_sum, collect_list, and collect_set under a dedicated\nSpark-Compatible Functions entry, with a note that the\ndatafusion.functions.spark namespace mirrors Spark semantics and may\ndiffer from the like-named built-ins. Adds a (spark-functions) anchor\nfor the cross-reference.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(spark): align if_ doctest with the single-value accessor style\n\nUse the same lit-based single-row select and [0].as_py() accessor as\nthe other wrappers instead of the lone to_pylist() call.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat(spark): accept native Python literals for literal-friendly args\n\nAudit the functions.spark namespace against pyspark.sql.functions and\nwiden arguments that pyspark types as a non-column literal so callers can\npass bare int/float/str instead of wrapping in lit():\n\n- int args: array_repeat count, slice start/length, shiftleft/shiftright/\n shiftrightunsigned numBits, sha2 numBits, round scale, substring pos/len,\n width_bucket numBucket\n- int32-coerced args (binding requires int32): add_months months,\n date_add/date_sub days, space n, make_dt_interval/make_interval parts\n- float args: modulus/pmod operands; make_*_interval secs\n- str args: next_day dayOfWeek, date_trunc/trunc format, date_part field,\n from_utc_timestamp/to_utc_timestamp tz, spark_cast type_str,\n json_tuple *fields\n- Any: array_contains value, if_ if_true/if_false\n\nArguments that pyspark types as ColumnOrName (str means column name, not a\nliteral) are left as Expr to avoid diverging from pyspark semantics:\nilike/like pattern, parse_url partToExtract/key, str_to_map delimiters,\nbit_get pos, time_trunc unit.\n\nAlso rename str_to_map\u0027s delimiter params to pairDelim/keyValueDelim to\nmatch pyspark exactly (they were pair_delim/key_value_delim).\n\nAdd a coercion test matrix and update docstring examples to show the\nnative-literal calling convention.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat(spark): accept column-name str for ColumnOrName args\n\nFor arguments that pyspark types as ColumnOrName, a bare str means a\ncolumn name (not a literal). Widen these to Expr | str and resolve a str\nto a column reference via _to_raw_expr, matching pyspark semantics:\n\n- ilike/like pattern\n- parse_url/try_parse_url partToExtract and key\n- str_to_map pairDelim/keyValueDelim\n- bit_get pos\n- time_trunc unit\n\nDocument the column-name behavior in each docstring and add a test\nconfirming a bare str resolves to a per-row column value.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: correct grammar in file_compression_type error message\n\n\"must one of\" → \"must be one of\".\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "55fd2c925c95711ad7222a5a48b6691d482d6056", "tree": "66354d0d6c3cb39aca20068a4651d2f33ac6f649", "parents": [ "c0ac93b68fff606aed570aed09d72d749ba79bae" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Tue Jun 16 18:13:07 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jun 16 18:13:07 2026 -0400" }, "message": "docs: convert reStructuredText sources to MyST markdown (#1579)\n\n* docs: convert restructuredText sources to MyST markdown\n\nPhase 2 of the documentation-site refresh. Run `rst2myst convert` over\nevery human-authored .rst file under docs/source/ and remove the\noriginals. The result:\n\n- 33 .rst files become 33 .md files (user guide, contributor guide,\n index, links).\n- Headings, paragraphs, hyperlinks, code blocks, admonitions, and\n toctree directives all map cleanly to MyST syntax.\n- Cross-reference anchors round-trip through MyST as `(label)\u003d`\n blocks. The converter kebab-cased the labels (e.g. `(io-csv)\u003d`),\n but every `{ref}` target in the corpus still uses the underscore\n form from the original RST (`{ref}\\`CSV \u003cio_csv\u003e\\``) and so do the\n Python docstrings that AutoAPI pulls in. Rewrite the anchors back\n to the underscore form so the existing references resolve.\n- 86 `{eval-rst}` blocks remain — they all wrap `.. ipython::`\n directives, which have no first-class MyST equivalent. They render\n identically and don\u0027t block the build.\n\nconf.py changes:\n\n- Enable `colon_fence` and `deflist` MyST extensions (rst-to-myst\n emits these on a few files, particularly execution-metrics.md).\n- Keep `.rst` in `source_suffix` even though no human-authored RST\n remains: sphinx-autoapi generates RST under autoapi/ at build time\n and Sphinx needs the suffix registered to parse it.\n\nAGENTS.md: update the two .rst paths called out under \"Aggregate and\nWindow Function Documentation\" to point at the .md equivalents.\n\nVerified by building locally — `build succeeded`, no warnings, all\ninternal cross-references resolve, the ipython examples on the\nlanding page and basics page still execute.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: fix Apache license header format in converted markdown files\n\nRST-to-MD conversion emitted MyST `%` comment syntax with blank line\nbetween each header line, which renders as visible text. Replace with\ncanonical `\u003c!--- ... --\u003e` HTML comment block matching upstream\napache/datafusion and this repo\u0027s existing markdown files.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: fix broken cross-reference links in distributing-work\n\nThe RST -\u003e MyST conversion left two intra-page links as undefined\nreference-style links, which CommonMark renders as literal bracketed\ntext (no Sphinx warning, so the --fail-on-warning build still passed).\nPoint both at the auto-generated heading anchors instead.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: execute examples via myst-nb; native tables and validated refs\n\nRemoves the last RST-syntax islands from the converted MyST markdown so\nthe docs are markdown-native for both human and LLM authors.\n\nExecutable examples (A): replace IPython.sphinxext.ipython_directive with\nmyst-nb. The 83 `{eval-rst}` + `.. ipython:: python` blocks become native\n`{code-cell} ipython3` blocks, and the 14 pages that carry them gain\njupytext/kernelspec front matter so myst-nb runs them. conf.py routes .md\nthrough myst-nb with nb_execution_mode\u003d\"force\" and\nnb_execution_raise_on_error\u003dTrue, so a failing example now fails the build.\n\nmyst-nb gives each page its own kernel instead of the IPython directive\u0027s\nsingle namespace shared across all documents in build order. That isolation\nsurfaced expressions.md, which only ever worked by inheriting `col`/`lit`\nfrom an earlier-built page — it now imports them itself. It also changes the\nexecution working directory to each page\u0027s own folder, so build.sh symlinks\nthe example data next to every page that reads it by relative name and\nregisters the python3 kernel; CI now calls build.sh so it matches local.\n\nTables (B): the 3 `.. list-table::` directives become GFM markdown tables.\n\nCross-references (C): the two intra-page links in distributing-work.md that\nthe conversion left as undefined markdown references (and that built green\nwhile rendering literal brackets) become `{ref}` roles backed by explicit\n`(label)\u003d` targets, so a future break fails the build instead of shipping\nsilently.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: render DataFrame cell outputs as text, not the HTML widget\n\nmyst-nb prefers a cell\u0027s `_repr_html_` over its text repr. A datafusion\nDataFrame\u0027s HTML repr is a Jupyter-oriented widget — inline styles plus an\ninjected \u003cscript\u003e — that renders at the wrong width in the docs theme.\n\nSet nb_mime_priority_overrides so the html builder prefers text/plain. The\n35 cells that end in a bare DataFrame now show the same readable ASCII\ntable the old IPython directive produced, with no per-cell `.show()` edits\nand no dependence on the package-generated HTML staying theme-compatible.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(aggregations): use .alias() on grouping(), drop obsolete workaround\n\napache/datafusion#21411 is resolved — `.alias()` now works directly on a\n`grouping()` expression. Removed the note describing the limitation and the\nwith_column_renamed workaround in the rollup and grouping_sets examples,\naliasing the grouping columns inline instead. Verified on the current\nbranch: the aliased aggregates execute and produce the named columns.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: use a dark-mode variant of the logo\n\nThe header logo was the same SVG in both color modes; the light-colored\nwordmark was hard to read on the dark theme. Point the theme\u0027s image_dark\nat a new original_dark.svg whose wordmark uses light strokes.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: restore right-hand on-this-page TOC, collapsible\n\nThe theme refresh emptied secondary_sidebar_items, dropping the\non-this-page table of contents that the previous site showed. Bring it\nback on the right, wrapped in a native \u003cdetails\u003e so readers can fold it\naway on the longer guide pages. Adds a custom page-toc-collapsible\nsecondary-sidebar template and styles the \u003csummary\u003e toggle (no JS).\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: let readers hide the right TOC sidebar for full-width content\n\nFollow-up to restoring the on-this-page TOC: \"collapsible\" should hide the\nentire right-hand frame, not just fold the list. Replace the \u003cdetails\u003e\nwrapper with a floating toggle button (toc-toggle.js) that hides the whole\nsecondary sidebar via a body class; the flex article container then\nreclaims the width (its 60em cap is lifted while hidden). The preference is\nremembered across pages in localStorage, and the button is suppressed below\nthe theme\u0027s breakpoint where the sidebar is already collapsed.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix(deps): pin typing-extensions to one version so uv.lock parses in CI\n\nAdding the myst-nb docs stack pulled a newer typing-extensions only on\nPython \u003c 3.11, splitting it into two locked versions. Our own\n`typing-extensions; python_full_version \u003c \u00273.13\u0027` dependency then spanned\nthat split, which uv recorded as a multi-version edge without a `version`\nfield — a form older uv builds (the one in CI\u0027s pinned setup-uv) reject\nwith \"missing source field but has more than one matching package\".\n\nAdd a [tool.uv] constraint-dependencies pin of typing-extensions\u003e\u003d4.15.0\nso it resolves to a single version across all supported Pythons, removing\nthe fork and the under-specified edge. Relocked; uv lock --locked is clean\nand no multi-version package has a marker-only edge.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(deps): drop pickleshare and explicit ipython from docs group\n\nBoth were only needed by the old IPython.sphinxext.ipython_directive,\nwhich myst-nb replaced. pickleshare (IPython %store, abandoned 2018) has\nno remaining consumer. ipython is now pulled transitively by ipykernel\nand myst-nb, so the explicit floor is redundant. Relocked.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Apply reviewer\u0027s suggestion to fix CI error\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "c0ac93b68fff606aed570aed09d72d749ba79bae", "tree": "46d37d40b06b7436b4a3cfc0ce41a86b2fba0e73", "parents": [ "fd15b03ecc227957b752429bf5b7c56c0f139f1e" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Mon Jun 15 16:23:05 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jun 15 16:23:05 2026 -0400" }, "message": "feat: expose arrow_field, arrow_try_cast, cast_to_type, with_metadata (#1568)\n\n* feat: expose arrow_field, arrow_try_cast, cast_to_type, with_metadata\n\nAdds Python bindings for five scalar functions from\ndatafusion::functions::expr_fn that were not previously surfaced:\n\n- arrow_field: returns a struct describing an expression\u0027s Arrow field\n (name, data_type, nullable, metadata).\n- arrow_try_cast: like arrow_cast but yields NULL on cast failure.\n- cast_to_type / try_cast_to_type: casts a value to the type of a\n reference expression. These are exposed as a single Python entry\n point cast_to_type(value, type_ref, *, try_cast\u003dFalse); the kwarg\n switches between the strict and try variants.\n- with_metadata: attach Arrow field metadata; the inverse of\n arrow_metadata. Accepts a dict[str, str] for ergonomics.\n\nUpdates skills/datafusion_python/SKILL.md to list the new functions\nand documents the cast_to_type kwarg behavior.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: collapse try_cast_to_type into cast_to_type kwarg\n\nThe previous commit exposed cast_to_type and try_cast_to_type as two\nseparate pyo3 bindings and unified them in the Python wrapper via a\ntry_cast kwarg. That left try_cast_to_type in datafusion._internal\nwithout a matching public Python name, breaking\ntest_datafusion_missing_exports.\n\nMove the dispatch into the rust binding: cast_to_type now takes a\ntry_cast kwarg and selects between functions::expr_fn::cast_to_type\nand try_cast_to_type internally. Only one pyo3 binding is registered,\nso the wrapper-coverage check passes and the Python entrypoint is\nunchanged.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat: accept pyarrow DataType in arrow_try_cast\n\nMirrors arrow_cast: arrow_try_cast now accepts `pa.DataType` in addition\nto `str` and `Expr`. Adds `Expr.try_cast(pa.DataType)` PyO3 binding for\nthe pyarrow-type routing path.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: guard with_metadata against empty dict and empty keys\n\nEmpty `metadata` dict now returns the input expression unchanged\n(previously bubbled an opaque DataFusion error about minimum arg\ncount). Empty keys raise `ValueError` to match the docstring contract.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: assert full struct shape in arrow_field doctest\n\nPrevious doctest set metadata on the input field but only checked the\nname — the metadata setup was dead. Now the example asserts the full\nreturned struct (name, data_type, nullable, metadata) so the demo\nshows what the function actually produces.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* test: add unit tests for arrow_try_cast, arrow_field, cast_to_type, with_metadata\n\nMirrors the existing test_arrow_cast pattern. Covers:\n- arrow_try_cast: string-syntax, pa.DataType, and null-on-failure paths\n- arrow_field: full returned struct shape (name, data_type, nullable, metadata)\n- cast_to_type: type-from-expr happy path and try_cast\u003dTrue null behavior\n- with_metadata: round-trip through arrow_metadata, empty-dict no-op, and\n empty-key ValueError\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* test: parameterize arrow cast / try_cast tests\n\nFolds the previous four cast tests (arrow_cast + arrow_try_cast × str\n+ pyarrow target type) into a single parameterized test that runs both\nfunctions across all five target-type variants. Collapses the two\ncast_to_type tests (happy path + try_cast\u003dTrue) into one parameterized\ntest, and parameterizes arrow_try_cast null-on-failure over both\ntarget-type syntaxes. 7 test functions, 19 cases — net less code, same\ncoverage.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: point cast_to_type at arrow_cast for static target types\n\nAdds a one-line cross-reference so users with a known target type\nreach for arrow_cast / arrow_try_cast instead of building a sentinel\nexpression to feed cast_to_type.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: split cast_to_type into cast_to_type and try_cast_to_type\n\nReplace the try_cast bool flag with separate cast_to_type and\ntry_cast_to_type functions, matching upstream DataFusion and the\narrow_cast / arrow_try_cast pair. Also drop the redundant data_type\nparametrization on test_arrow_try_cast_null_on_failure, since the\nstr-vs-pyarrow distinction is already covered by test_arrow_cast_variants.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "fd15b03ecc227957b752429bf5b7c56c0f139f1e", "tree": "417451c057c4702b287f4427a16dd3a194ff35ed", "parents": [ "3256138ecc57de2666df704909a371919ff6fbfd" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Mon Jun 15 14:01:13 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jun 15 14:01:13 2026 -0400" }, "message": "feat: expose array_compact, array_normalize, cosine_distance, inner_product (#1567)\n\n* feat: expose array_compact, array_normalize, cosine_distance, inner_product\n\nAdds Python bindings for four scalar functions from\ndatafusion::functions_nested::expr_fn that were not previously surfaced:\n\n- array_compact / list_compact: drop NULLs from an array.\n- array_normalize / list_normalize: L2-normalize a numeric array.\n- cosine_distance: 1 - cosine_similarity(a, b).\n- inner_product: dot product of two numeric arrays.\n\nImplementation routes each through the existing array_fn! macro in\ncrates/core/src/functions.rs, mirroring the other functions_nested\nwrappers. Python wrappers in python/datafusion/functions.py follow the\nestablished pattern with doctest examples; list_* aliases use the\none-line + See Also form per project convention.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: clarify array_normalize and cosine_distance docstrings\n\nExpand both docstrings with plain-English definitions, worked examples,\nranges, use cases, and behavior on edge cases (zero vector → NULL,\nlength-mismatched inputs fail). Adds a zero-vector example to\narray_normalize and an orthogonal-vector example to cosine_distance.\nUpdates the list_normalize alias summary to match.\n\nCo-Authored-By: Claude \u003cnoreply@anthropic.com\u003e\n\n* test: add alias-equivalence and length-mismatch tests for array fns\n\nPin the contracts the doctests don\u0027t cover:\nlist_compact/list_normalize must produce the same output as their\narray_* primaries, and cosine_distance/inner_product must reject\nlength-mismatched inputs at execution time.\n\n* feat: expose dot_product alias for inner_product\n\nMatch upstream DataFusion SQL alias surface (inner_product UDF\nregisters `dot_product` in its alias list). Also expand\n`inner_product` docstring with NULL/length-mismatch behavior to\nmatch peer distance fns added in this PR.\n\nCo-Authored-By: Claude Opus 4.7 \u003cnoreply@anthropic.com\u003e\n\n* test: fold dot_product alias check into parametrized test\n\nGeneralize test_array_function_aliases to accept multi-column data so\nthe dot_product/inner_product alias case fits, dropping the standalone\ntest_dot_product_alias_matches_inner_product.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "3256138ecc57de2666df704909a371919ff6fbfd", "tree": "7ffc97b66a3033f2455f77d81a331331cb6d2f12", "parents": [ "cd7506ad645bf7f3c01796cbb16ec5821d56d389" ], "author": { "name": "kosiew", "email": "kosiew@gmail.com", "time": "Sat Jun 13 23:40:08 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Jun 13 23:40:08 2026 +0800" }, "message": "Deprecate `Expr` temporal part arguments in date extraction and truncation functions (#1587)\n\nAdded a shared helper to emit a DeprecationWarning when an Expr is passed to a literal control argument.\n\nUpdated the implementations of:\n\ndate_part\ndatepart\nextract\ndate_trunc\ndatetrunc\nto warn when part is provided as an Expr.\n\nRefactored alias functions (datepart, extract, and datetrunc) to delegate through internal helper functions so warnings reference the user-facing function name correctly and avoid duplicate warning behavior.\n\nUpdated existing temporal function tests to use the preferred string-literal form for part.\n\nAdded targeted tests verifying:\n\nExpr inputs emit DeprecationWarning for date_part, datepart, and extract.\nExpr inputs emit DeprecationWarning for date_trunc and datetrunc.\nString literal inputs continue to work without emitting deprecation warnings." }, { "commit": "cd7506ad645bf7f3c01796cbb16ec5821d56d389", "tree": "10445a59b69eba5d2c06297cec0891dfbabe5c0c", "parents": [ "43395a46aa2597b8090248ecd9a10c601ebec0bf" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri Jun 12 16:45:40 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jun 12 16:45:40 2026 +0200" }, "message": "build(deps): batch dependabot dependency updates (#1589)\n\nCombine open Dependabot updates into a single PR.\n\nGitHub Actions:\n- github/codeql-action 4.35.4 -\u003e 4.36.2 (#1585)\n- astral-sh/setup-uv 8.1.0 -\u003e 8.2.0 (#1584)\n\nCargo:\n- chrono 0.4.44 -\u003e 0.4.45 (#1583)\n- log 0.4.30 -\u003e 0.4.32 (#1582)\n- uuid 1.23.1 -\u003e 1.23.2 (#1565)\n\nPython (uv.lock):\n- pyarrow 22.0.0 -\u003e 23.0.1 (#1580)\n- idna 3.10 -\u003e 3.15 (#1552)\n- pytest 8.3.4 -\u003e 9.0.3 (#1542)\n- pyjwt 2.10.1 -\u003e 2.12.0 (#1540)\n- pygments 2.19.1 -\u003e 2.20.0 (#1539)\n- requests 2.32.3 -\u003e 2.33.0 (#1538)\n- urllib3 2.3.0 -\u003e 2.7.0 (#1537)\n- pynacl 1.5.0 -\u003e 1.6.2 (#1536)\n- cryptography 44.0.0 -\u003e 46.0.7 (#1535)\n\nCo-authored-by: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "43395a46aa2597b8090248ecd9a10c601ebec0bf", "tree": "553f644b0f04690547ee8ad293a095bb4e091e78", "parents": [ "43df9f7b954c00cc40bd257317926eddb6d4092e" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri Jun 12 16:17:20 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jun 12 16:17:20 2026 +0200" }, "message": "Remove patch now that 54 is released (#1588)" }, { "commit": "43df9f7b954c00cc40bd257317926eddb6d4092e", "tree": "88fa1d4dcd8da96c01a58e6bb4bc505258a48849", "parents": [ "407298f37d9b5ccb4439cbe77ee148bf95e52de9" ], "author": { "name": "kosiew", "email": "kosiew@gmail.com", "time": "Sun Jun 07 21:22:06 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Jun 07 15:22:06 2026 +0200" }, "message": "feat(dataframe): update group_by to accept None and normalize to empty list (#1581)\n\n- Updated `group_by` method to accept `None` and normalize it to an empty list.\n- Improved docstring for clarity.\n- Added regression test in `test_dataframe.py` to verify that `None` equals an empty list.\n- Updated documentation to mention that `group_by\u003dNone` is now supported." }, { "commit": "407298f37d9b5ccb4439cbe77ee148bf95e52de9", "tree": "de8b9b774e25944b8bce9916ef5d975faf54f3f4", "parents": [ "bfa14f4ffa879c83acfab2f1d480d9ed474baf7d" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Sun Jun 07 09:19:37 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Jun 07 15:19:37 2026 +0200" }, "message": "Improve documentation site layout (#1578)\n\n* docs: refresh theme — pydata-sphinx-theme 0.16, top navbar, dark mode\n\nBump pydata-sphinx-theme 0.8.0 -\u003e 0.16 to enable the modern navbar slot\nAPI and dark/light theme switcher. Configure top navbar with logo,\nnav links, GitHub icon, and theme switcher in conf.py. Drop the custom\ndocs-sidebar.html override and the layout.html block that silenced the\nnavbar — both predate the slot API and conflict with the new theme.\nStrip CSS overrides that fought the old theme (--pst-header-height: 0,\nnavbar-brand sizing) and add a dark-mode variant for the inline code\ncolor and table-stripe shading. Fix the stale github_repo\n(\"arrow-datafusion-python\" -\u003e \"datafusion-python\") so future Edit-on-\nGitHub links resolve. Bump copyright year and project name.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: collapse navbar to section landing pages\n\nPrevious structure dumped every top-level toctree entry from index.rst\ninto the navbar, producing eight items including external URLs (\"Github\nand Issue Tracker\", \"Rust\u0027s API Docs\", ...) that wrapped to two lines\neach. Introduce user-guide/index.rst and contributor-guide/index.rst as\nsection landing pages with nested toctrees, then point index.rst at just\nthose two plus autoapi/index. The navbar now reads \"User Guide\",\n\"Contributor Guide\", \"API Reference\" — three single-line entries. Move\nthe external links into the index.rst body where they\u0027re discoverable\nwithout crowding navigation.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: restore external links lost in navbar restructure\n\nAdd Examples and Rust API as text links in the top navbar via the\npydata-sphinx-theme external_links option. Nest the code-of-conduct\nlink inside the Contributor Guide toctree so it appears alongside the\nother contributor pages. Drop the duplicate \"Further reading\" bullet\nlist from the landing page now that every link has a permanent home.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: render Rust API link as docs.rs icon next to GitHub\n\nMove the Rust API docs entry from external_links to icon_links and use\nthe fa-brands fa-rust gear mark. Now sits next to the GitHub icon in\nnavbar_end with matching visual weight instead of a wider text link.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: render sidebar nav on landing page\n\nThe default pydata-sphinx-theme sidebar-nav-bs starts at the current\ntop-level section, so the root index — which has no parent section —\nends up with an empty sidebar. The theme\u0027s layout also explicitly\nfilters sidebar-nav-bs out of the sidebar list when suppress_sidebar_\ntoctree() returns true (which it does for root pages), so simply\noverriding sidebar-nav-bs.html in templates doesn\u0027t help.\n\nAdd a sidebar-globaltoc.html template that calls Sphinx\u0027s toctree()\nglobal directly to render the full document tree, and wire it through\nhtml_sidebars under a name the theme\u0027s suppress filter doesn\u0027t strip.\nLanding page now shows User Guide / Contributor Guide / API Reference\nin the sidebar with the current section expanded on inner pages.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: render expandable chevrons in sidebar nav\n\nSwitch the sidebar toctree call from toctree() to generate_toctree_html\nwith collapse\u003dFalse, so nested \u003cul\u003es render into the DOM for every\nbranch. The pydata-sphinx-theme JS then wraps them in \u003cdetails\u003e with\nfa-chevron-down toggles, matching the datafusion-comet sidebar where\neach section with children can be expanded inline. show_nav_level\u003d1\nkeeps deeper levels collapsed on first load.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: expand sidebar to show level 2 entries by default\n\nBump show_nav_level 1 -\u003e 2 so the landing-page sidebar opens with\nUser Guide / Contributor Guide / API Reference already expanded to\ntheir immediate children. Deeper levels remain collapsed behind\nchevrons so the sidebar stays scannable.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: add Links sidebar section for external references\n\nRestore the \"Links\" sidebar heading that the previous site had —\nGitHub and Issue Tracker, Rust API Docs, Code of Conduct, Examples.\nImplemented as a second hidden toctree with :caption: Links so the\npydata-sphinx-theme sidebar renders the heading above the four\nexternal URLs. Drop Code of Conduct from the Contributor Guide\ntoctree since it now lives under Links instead.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: consolidate external URLs into a single Links nav item\n\nReplace the second hidden toctree (which expanded each external URL\ninto its own navbar entry) with a dedicated links.rst landing page,\nand add a single \"links\" entry to the main toctree. Top navbar now\nshows User Guide / Contributor Guide / API Reference / Links — four\nitems, no wrapping. Clicking Links opens the page that lists GitHub,\nRust API Docs, Code of Conduct, and Examples.\n\nDrop the external_links Examples entry from conf.py since the same\nURL now lives on the Links page.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: add favicon matching the main datafusion site\n\nDrop in the same favicon.svg the main datafusion.apache.org site\nuses (just the Apache DataFusion mark, no wordmark) and wire it\nthrough html_favicon. Browsers and bookmarks now show the project\nicon instead of the generic Sphinx page glyph.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: address Copilot review feedback on sidebar config\n\nTwo small follow-ups from the Copilot reviewer on #1578:\n\n- Append .html to the html_sidebars entry. Sphinx\u0027s Jinja loader\n resolves both \"sidebar-globaltoc\" and \"sidebar-globaltoc.html\" to\n the same template, but the explicit form is closer to the spelling\n in the Sphinx docs and is harder to misread.\n- Update the inline comment in sidebar-globaltoc.html that still\n claimed show_nav_level\u003d1 after we bumped it to 2 in conf.py. Now\n describes the variable wiring instead of hard-coding a number that\n has to be kept in sync with conf.py.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "bfa14f4ffa879c83acfab2f1d480d9ed474baf7d", "tree": "3e1a6711138678f3e5ee8bbe94cbb6681662bb96", "parents": [ "23062f78ad55a7121517c5ab00742503e95c7342" ], "author": { "name": "Daniel Mesejo", "email": "mesejoleon@gmail.com", "time": "Fri Jun 05 18:09:17 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jun 05 12:09:17 2026 -0400" }, "message": "refactor(context): deduplicate register/read option-building logic (#1479)\n\n* refactor(context): deduplicate register/read option-building logic\n\nExtract shared helpers (convert_partition_cols, convert_file_sort_order,\nbuild_parquet/json/avro_options, convert_csv_options), standardize path\ntypes to \u0026str, and remove redundant intermediate variables.\n\n* refactor(context): accept PathBuf for path arguments in register/read methods\n\nChange path parameters from \u0026str to PathBuf in all register/read methods\n(register_listing_table, register_parquet, register_json, register_avro,\nregister_arrow, read_json, read_parquet, read_avro, read_arrow) so callers\ncan pass either a Python str or a pathlib.Path object. For register_csv and\nread_csv, which take \u0026Bound\u003cPyAny\u003e to handle lists, extract path elements as\nPathBuf rather than String for the same reason.\n\nAdd a path_to_str helper that converts PathBuf to \u0026str, returning an explicit\nerror for non-UTF-8 paths rather than silently corrupting them.\n\nAdd build_arrow_options helper to deduplicate register_arrow/read_arrow\noption-building logic, consistent with the existing parquet/json/avro helpers.\n\nCo-Authored-By: Claude Sonnet 4.6 \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Sonnet 4.6 \u003cnoreply@anthropic.com\u003e" }, { "commit": "23062f78ad55a7121517c5ab00742503e95c7342", "tree": "3457b0f2adce40f8adbd74d1f6adf5bc899bf863", "parents": [ "4a7761736f57376a9c769efd46a8b34e7f46b8f0" ], "author": { "name": "Nuno Faria", "email": "nunofpfaria@gmail.com", "time": "Fri Jun 05 17:02:21 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jun 05 12:02:21 2026 -0400" }, "message": "fix: Skip `fork` and `forkserver` on `win32` (#1566)\n\n* fix: Skip fork and forkserver on win32\n\n* Fix fmt" }, { "commit": "4a7761736f57376a9c769efd46a8b34e7f46b8f0", "tree": "378608186bf4d7d717970316835aca43c87768fc", "parents": [ "d021e6afa8e08bee42fb9673ba811352c464bdf7" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri Jun 05 12:01:53 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jun 05 12:01:53 2026 -0400" }, "message": "feat: expose SessionContext.copied_config and parse_capacity_limit (#1570)\n\nAdds two small additions to SessionContext that mirror upstream:\n\n- copied_config(): returns a copy of the active SessionConfig wrapped\n in the existing SessionConfig Python class. Useful when callers want\n to seed a new context from another context\u0027s settings, or inspect\n the current configuration without sharing mutable state.\n\n- parse_capacity_limit(config_name, limit): static helper that parses\n size strings like \"100M\", \"1.5G\", \"512K\", or \"0\" into a byte count.\n Useful when configuring a RuntimeEnvBuilder from human-friendly\n inputs. Wraps SessionContext::parse_capacity_limit; the deprecated\n parse_memory_limit is intentionally not exposed.\n\nThree other items from the same gap cluster (runtime_env,\ncopied_table_options, the deprecated parse_memory_limit) are not\nincluded here. The first two would require wrapping new Rust types\n(RuntimeEnv, TableOptions) whose surface is much larger than the\naccessors themselves; the third is deprecated upstream. Those are\nfiled as separate follow-up issues.\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "d021e6afa8e08bee42fb9673ba811352c464bdf7", "tree": "35dc626766ec9a8179a34206ed6bc35e442b4c92", "parents": [ "af388667f8d8583abdd634fa343252ddb4a0da16" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Thu Jun 04 07:57:47 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jun 04 07:57:47 2026 -0400" }, "message": "feat: import user-defined physical optimizer rules over FFI (#1557)\n\n* feat: user-defined OptimizerRule and AnalyzerRule from Python\n\nExpose `SessionContext.add_optimizer_rule` and\n`SessionContext.add_analyzer_rule` symmetric with the existing\n`remove_optimizer_rule`. Each accepts a Python subclass of the new\n`datafusion.optimizer.OptimizerRule` / `AnalyzerRule` ABCs.\n\nImplementation:\n\n* New `crates/core/src/optimizer_rules.rs` wraps user Python instances\n in `PyOptimizerRuleAdapter` / `PyAnalyzerRuleAdapter`, which\n implement the upstream `OptimizerRule` / `AnalyzerRule` traits.\n* `OptimizerRule.rewrite(plan)` returns `None` for \"no change\" or a\n new `LogicalPlan`. The adapter maps that to\n `Transformed::no` / `Transformed::yes` so the upstream optimizer\u0027s\n fixed-point loop terminates correctly.\n* `AnalyzerRule.analyze(plan)` must always return a `LogicalPlan`;\n returning `None` surfaces a `DataFusionError::Execution` naming the\n offending rule.\n* The upstream `\u0026dyn OptimizerConfig` / `\u0026ConfigOptions` arguments are\n not surfaced to Python in this MVP; rules that need configuration\n should capture it at construction time (for example by holding a\n `SessionContext` reference) or be implemented in Rust.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat: import FFI physical optimizer rules; drop Python logical rules\n\nReplace the Python-defined OptimizerRule/AnalyzerRule approach with FFI-imported physical optimizer rules.\n\nThe Python logical-rule approach could observe plans but not transform them: there are no Python constructors for LogicalPlan node variants, so a rule could only return None or the input plan unchanged. The audience for custom rules also overlaps strongly with people who can write Rust.\n\nDataFusion exposes no FFI bridge for the logical OptimizerRule/AnalyzerRule traits, but it does export FFI_PhysicalOptimizerRule for the physical PhysicalOptimizerRule trait. This commit imports those instead.\n\nChanges:\n\n* Remove crates/core/src/optimizer_rules.rs, python/datafusion/optimizer.py, python/tests/test_optimizer.py, and the SessionContext.add_optimizer_rule / add_analyzer_rule methods. remove_optimizer_rule is unchanged (pre-existing).\n* New crates/core/src/physical_optimizer.rs reads a __datafusion_physical_optimizer_rule__ capsule and converts it via Arc\u003cdyn PhysicalOptimizerRule\u003e::from(\u0026FFI_PhysicalOptimizerRule).\n* SessionContext gains a physical_optimizer_rules constructor argument. Upstream offers no API to add physical rules to a live context, so they are appended to the builder at construction time only.\n* The datafusion-ffi-example crate gains MyPhysicalOptimizerRule, a counter-backed rule used by _test_physical_optimizer_rule.py to prove the rule fires over FFI during physical planning.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: type physical_optimizer_rules with an Exportable Protocol\n\nReplace the `list[Any]` hint on the SessionContext `physical_optimizer_rules` argument with a `PhysicalOptimizerRuleExportable` Protocol, matching the existing `TableProviderExportable` / `*Exportable` pattern used for other FFI-capsule objects.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: reference PhysicalOptimizerRuleExportable in SessionContext docstring\n\nPoint the `physical_optimizer_rules` argument docs at the new\n`PhysicalOptimizerRuleExportable` Protocol instead of describing the duck type inline.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: move FFI capsule detail to PhysicalOptimizerRuleExportable\n\nThe PyCapsule / FFI_PhysicalOptimizerRule mechanics describe the Protocol, not the SessionContext constructor. Move that detail onto PhysicalOptimizerRuleExportable and leave the constructor argument docs focused on behavior.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: drop redundant comment in SessionContext constructor\n\nRemove the explanatory comment about FFI bridge availability; the same information already lives on PhysicalOptimizerRuleExportable.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: drop module-level doc comment from physical_optimizer\n\nSibling FFI-import modules (udf, udaf, catalog, table) carry no module-level docs, and the rst-style markup did not match Rust conventions. The function doc comment already states intent.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: import physical optimizer rule via from_pycapsule! macro\n\nReplace the hand-written crates/core/src/physical_optimizer.rs with a `from_pycapsule!` invocation in the util crate, matching `physical_codec_from_pycapsule` and the other FFI capsule importers. The macro already handles the hasattr/getattr/cast/validate/pointer_checked sequence and the infallible `Arc::from(\u0026FFI)` conversion, so the dedicated module is no longer needed.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: trim PhysicalOptimizerRuleExportable docstring\n\nDrop the sentence about logical-rule FFI availability; it is background, not type-hint information, and keeps the Protocol docstring in line with the other *Exportable hints.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Minor refactor\n\n* refactor: register physical optimizer rules via live add method\n\nDrop the `physical_optimizer_rules` constructor argument on\n`SessionContext` and replace it with `add_physical_optimizer_rule`,\nmatching the existing `register_*` shape on the same class. The new\nmethod rebuilds the session state via `SessionStateBuilder::new_from_existing`\nso previously registered tables, UDFs, and catalogs are preserved.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* test: drop redundant FFI physical optimizer rule export test\n\nCoverage subsumed by test_ffi_physical_optimizer_rule_runs_during_planning,\nwhich exercises the same capsule export via add_physical_optimizer_rule.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "af388667f8d8583abdd634fa343252ddb4a0da16", "tree": "769758bc6940a3e523c79057b4b1fc1f5c67f1ad", "parents": [ "3d4c56c0757ddc0372ced03ffa97a13ea50d1bd8" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri May 29 18:04:24 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri May 29 18:04:24 2026 -0400" }, "message": "feat: expose lambda and higher-order array functions (#1561)\n\n* feat: expose lambda and higher-order array functions\n\nAdd a Pythonic API for DataFusion\u0027s higher-order array functions and the\nlambda expressions they consume.\n\n- Rust: lambda_, lambda_var, array_transform, and array_any_match pyfunctions,\n plus a ResolveLambdaVariables analyzer rule so expression-builder plans\n (which emit unresolved lambda variables) resolve before optimization.\n- Python: array_transform / array_any_match (with list_transform, any_match,\n list_any_match aliases) accept either a Python callable or an explicit\n lambda built with lambda_ / lambda_var. Callables are introspected so their\n parameter names become the lambda parameters.\n- Tests and docs (expressions guide + agent skill), noting v1 limits: lambda\n expressions are not serializable, and SQL arrow syntax needs the DuckDB\n dialect.\n\n* test: fold lambda tests into pytest parameterization\n\nCombine the eight higher-order function result tests into a single\nparametrized test_higher_order_function_results, and the two to_lambda\nrejection tests into test_to_lambda_rejects_invalid_arg. Each case keeps\na readable id via pytest.param.\n\nCo-Authored-By: Claude \u003cnoreply@anthropic.com\u003e\n\n* feat: expose array_filter higher-order function\n\nAdd array_filter, the remaining lambda-based higher-order array function\nin DataFusion (alongside the already-exposed array_transform and\narray_any_match). Includes the list_filter alias matching upstream, tests,\nand documentation in the expressions guide and skill.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: emphasize lambda terminology, trim skill lambda section\n\nLead user-facing array-lambda docs with \"lambda function\" instead of\n\"higher-order function,\" which is less recognizable to users. Drop the\nalias list, serialization caveat, and DuckDB-dialect note from the skill\nto keep it lean; those details already live in the docstrings.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: broaden SQL lambda dialect coverage\n\nOther dialects (ClickHouse, Snowflake, Databricks) also enable lambda\nparsing via sqlparser-rs. Document the full set and recommend the\n``lambda x: x`` keyword form, since DuckDB will drop the ``x -\u003e x``\narrow form in v2.1. Parametrize the SQL test over the four dialects\nusing the keyword syntax.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude \u003cnoreply@anthropic.com\u003e" }, { "commit": "3d4c56c0757ddc0372ced03ffa97a13ea50d1bd8", "tree": "8f2db8f7e47dbb7331eeb46cbb3273b9e5b54f0c", "parents": [ "0840763851e47a40676dc47b29f5f3a62b0127d7" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri May 29 16:53:19 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri May 29 16:53:19 2026 -0400" }, "message": "feat: create free-threaded python wheels (#1553)\n\n* Initial commit for free threaded python support\n\n* ci: use uvx to run maturin in native wheel builds\n\nThe free-threaded matrix entries skip `uv sync` to avoid resolving\nproject dependencies against cp313t/cp314t (many dev deps lack\nfree-threaded wheels), so `uv run --no-project maturin` failed on\nmacOS/Windows with \"Failed to spawn: `maturin`\". Switch to\n`uvx maturin@1.8.1`, which runs maturin in an isolated tool env\nindependent of the project venv and matches the pin used by\nmaturin-action for manylinux builds.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* ci: resolve free-threaded interpreter path explicitly on Windows\n\nmaturin\u0027s `--interpreter python3.14t` fails on Windows because the\nfree-threaded build ships as plain `python.exe` (no `tN` suffix). Look\nup `sys.executable` of the python on PATH (which actions/setup-python\nprepends with the free-threaded install), assert\n`Py_GIL_DISABLED \u003d\u003d 1` so a misconfigured PATH can\u0027t silently build a\nGIL wheel, and normalize backslashes to forward slashes so the path\nsurvives re-expansion in the downstream `run:` line.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* build: enable PyO3 generate-import-lib for Windows free-threaded wheels\n\nWindows free-threaded Python does not expose `abiflags` in sysconfig,\nso PyO3\u0027s default Windows linkage path fails with \"A python 3\ninterpreter on Windows does not define abiflags in its sysconfig ಠ_ಠ\"\nwhen building cp31Xt wheels. Enabling the `generate-import-lib` PyO3\nfeature switches Windows builds to a generated import library\n(provided by the `python3-dll-a` crate) that does not depend on a\nfully populated sysconfig. It is a no-op on macOS and Linux and is\ncompatible with the existing `abi3` feature.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* ci: bump maturin to 1.13.3 for Windows free-threaded support\n\nmaturin 1.8.1 errors out on Windows free-threaded interpreters with\n\"A python 3 interpreter on Windows does not define abiflags in its\nsysconfig\" even when given a valid `python.exe`. Newer maturin\nreleases handle the missing abiflags gracefully for cp31Xt builds.\nBump both the `uvx maturin@` pin used for native macOS/Windows wheels\nand the `maturin-version` passed to PyO3/maturin-action for the\nmanylinux containers.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* ci: standardize wheel build job names as \"\u003cOS\u003e \u003carch\u003e (\u003ctag\u003e)\"\n\nThe mac/Windows matrix shared a single name template that prepended\n\"macOS arm64 \u0026 Windows\" to every entry, which got truncated in the\nGitHub UI sidebar and made it hard to tell macOS and Windows runs\napart. Rename all wheel build jobs to the same pattern so the OS,\narchitecture, and python tag are visible at a glance:\n\n- Linux x86_64 / arm64\n- macOS arm64 / x86_64\n- Windows x86_64\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* taplo fmt\n\n* build: move pygithub to release group to fix free-threaded wheel builds\n\npygithub pulls in cryptography via pyjwt[crypto]. cryptography 44.0.0\nships only abi3 wheels, which free-threaded interpreters cannot use, so\nuv builds it from sdist; its bundled PyO3 0.23.2 caps at Python 3.13 and\nfails on 3.14t. pygithub is only used by the manual release changelog\nscript, so move it out of the dev group into a new release group.\n\u0027uv sync --dev\u0027 (used by CI test jobs) no longer drags in cryptography.\n\n* ci: pin uv venv to setup-python interpreter for free-threaded jobs\n\nPassing a bare version like \u00273.13t\u0027 to \u0027uv venv --python\u0027 let uv fall\nback to a different system interpreter (3.12), creating a venv whose ABI\ndid not match the downloaded cp313t wheel and failing the install. Use\nthe python-path output from setup-python so the venv uses exactly the\ninterpreter that was set up.\n\n* taplo fmt\n\n* ci: set UV_PYTHON so uv sync keeps the free-threaded interpreter\n\nPinning only \u0027uv venv --python\u0027 was not enough: \u0027uv sync\u0027 ignores the\nexisting .venv, runs its own interpreter discovery, and recreated the\nvenv with the system 3.12, again mismatching the cp313t wheel. Set\nUV_PYTHON to the setup-python interpreter for the install and test\nsteps so every uv command (venv, sync, pip, run) uses it.\n\n* ci: run tests from the .venv, not the bare setup-python interpreter\n\nSetting UV_PYTHON on the test step pointed \u0027uv run --no-project pytest\u0027\nat the setup-python interpreter, which has no pytest installed, causing\n\u0027Failed to spawn: pytest\u0027. UV_PYTHON is only needed in the install step\nto build the .venv with the right interpreter; the test step must use\nthat .venv. Drop UV_PYTHON from the test step.\n\nCo-Authored-By: Claude \u003cnoreply@anthropic.com\u003e\n\n* ci: install datafusion wheel into the activated .venv\n\nSetting UV_PYTHON as a step env split the install across two\nenvironments: \u0027uv sync\u0027 populated .venv while \u0027uv pip install\u0027 targeted\nthe bare setup-python interpreter, so the datafusion wheel never landed\nin .venv and \u0027import datafusion\u0027 failed under pytest. Pin the\ninterpreter at \u0027uv venv --python\u0027, activate the venv, and pass --active\nto \u0027uv sync\u0027 so sync and pip install both target the same .venv.\n\nCo-Authored-By: Claude \u003cnoreply@anthropic.com\u003e\n\n* ci: point uv at the venv interpreter by path for free-threaded jobs\n\nActivating the venv and passing --active still let \u0027uv sync\u0027 run its own\ninterpreter discovery, which skips free-threaded builds and re-picked the\nsystem 3.12, recreating .venv and breaking the cp313t/cp314t wheel\ninstall. Pass the venv\u0027s own interpreter (.venv/bin/python) explicitly to\n\u0027uv sync\u0027, \u0027uv pip install\u0027, and \u0027uv run\u0027 so every step stays in the\nfree-threaded environment created by \u0027uv venv\u0027.\n\nCo-Authored-By: Claude \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "0840763851e47a40676dc47b29f5f3a62b0127d7", "tree": "c3b74841a5eadb2061f4d45f0f328f1889425a51", "parents": [ "744dd23ff3bfeeafbf532d71c909e0aedb7b2194" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri May 29 13:30:39 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri May 29 13:30:39 2026 -0400" }, "message": "feat: accept distinct kwarg on sum and avg (#1556)\n\n* feat: accept distinct kwarg on sum and avg\n\nUpstream exposes `sum_distinct` / `avg_distinct` / `count_distinct` as\nsibling functions that call the same underlying UDAF with\n`distinct: bool \u003d true`. The Rust binding side already routes\n`distinct\u003dSome(true)` through the aggregate builder for `sum`, `avg`,\nand `count` — but only `count` exposed the kwarg on the Python wrapper.\n\nAdd `distinct: bool \u003d False` to `sum()` and `avg()` mirroring the\nexisting `count()` signature, and update SKILL.md so the check-upstream\naudit does not re-flag the three upstream `*_distinct` shortcuts as\ngaps. The plan emitted by `sum(col, distinct\u003dTrue)` matches what\nupstream\u0027s `sum_distinct(col)` builds.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* test: fold sum/avg distinct tests into parameterized aggregation test\n\nMove the standalone test_sum_distinct_kwarg and test_avg_distinct_kwarg\nfrom test_functions.py into the existing test_aggregation::test_aggregation\nparameterization, matching how distinct is already covered for median,\narray_agg, count, and bit_xor.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: clarify distinct kwarg on sum and avg\n\nDrop the unhelpful \"upstream avg_distinct/sum_distinct shortcut\"\nreference in favor of describing the actual behavior.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: note sum/avg distinct argument-order breaking change\n\ndistinct is inserted before filter on sum and avg for consistency with\nthe other aggregate functions, breaking positional filter callers. Add a\nDataFusion 54.0.0 upgrade-guide entry covering the migration.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Update docs/source/user-guide/upgrade-guides.rst\n\nCo-authored-by: Nick \u003c24689722+ntjohnson1@users.noreply.github.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\nCo-authored-by: Nick \u003c24689722+ntjohnson1@users.noreply.github.com\u003e" }, { "commit": "744dd23ff3bfeeafbf532d71c909e0aedb7b2194", "tree": "f136efd90adc2e879b5c6c7ee0868af5b455bf3e", "parents": [ "7df58e531bb949785cae3cca488a1ae55cb6d478" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri May 29 09:25:16 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri May 29 09:25:16 2026 -0400" }, "message": "chore: remove unused PyConfig (#1485)\n\n* Remove unused Config class that could not be connected to a SessionContext\n\nCloses #322.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Minor correction after merge main\n\n---------\n\nCo-authored-by: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "7df58e531bb949785cae3cca488a1ae55cb6d478", "tree": "76528a4da5662f5fbd027a176bdbc24652f60f54", "parents": [ "987228300b1c52215b4bb10a1cd5781c40648fbe" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri May 29 08:09:42 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri May 29 08:09:42 2026 -0400" }, "message": "feat: pass calling SessionContext to Python UDTF callbacks (#1555)\n\n* feat: pass calling SessionContext to Python UDTF callbacks\n\nDataFusion 53 added `TableFunctionImpl::call_with_args(TableFunctionArgs)`\nwhere `TableFunctionArgs` carries both the positional expression\narguments and the calling `\u0026dyn Session`. The pure-Python UDTF path\npreviously discarded everything but the exprs.\n\nThread the session through when the user callback\u0027s signature opts in\nby declaring a `session` keyword parameter (or `**kwargs`). At call\ntime we downcast the `\u0026dyn Session` to its canonical `SessionState`\nimpl and build a fresh `SessionContext` over the same Arc-shared state,\nexposed to Python as a `datafusion.SessionContext` wrapper. Existing\ncallbacks whose signatures do not declare `session` continue to be\ncalled with the positional expression arguments only — no behavior\nchange for current users.\n\nNote: a UDTF body cannot drive a fresh `ctx.sql(...).collect()` on the\npassed-in session because the outer SQL execution already holds the\ntokio runtime. Use the session for metadata access (catalogs, UDF\nlookups, config) rather than nested DataFrame collection.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: clarify py_session_from_session downcast is defensive\n\nThe doc comment implied a foreign FFI session was a real input. No\ncurrent path reaches a pure-Python UDTF with a non-SessionState\nsession: the SQL planner and __call__ both hand a SessionState, and a\nForeignSession would only arrive via FFI-export of the UDTF, which\ndatafusion-python does not do. Reword to state the guard is defensive\nand rewrap the error string.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: opt-in UDTF session injection via with_session flag\n\nReplaces signature sniffing with an explicit ``with_session\u003dTrue`` kwarg\non ``TableFunction`` / ``udtf``. Avoids name-based detection footguns\n(positional-only ``session`` params, accidental ``**kwargs`` opt-in,\nshadowing by unrelated params) and makes author intent visible at\nregistration. Also documents the feature in the UDTF user guide.\n\nRust field renamed ``accepts_session`` -\u003e ``inject_session_on_call`` to\nmatch the Python-side opt-in semantics.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: reject with_session\u003dTrue for FFI UDTFs and qualify mutation docs\n\nRaise TypeError when with_session\u003dTrue is combined with an FFI-exported\ntable function (one exposing __datafusion_table_function__). The Rust\nFFI branch does not consult the flag, so it would silently be dropped;\nguard both TableFunction.__init__ and the udtf() convenience entry.\n\nQualify the doc claim that mutations through the injected session\npropagate to the caller: registry mutations do (shared Arc registries),\nbut config changes do not (SessionConfig is cloned). Mirror the caveat\nin TableFunction.__init__ per the user-guide caveats convention.\n\nCo-Authored-By: Claude Opus 4.7 \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "987228300b1c52215b4bb10a1cd5781c40648fbe", "tree": "a7c9ee949f08ca706a0b57e80d9ffc4edc689a74", "parents": [ "0a9ca68ade8bbe06aaf67928bb8edc65670baa24" ], "author": { "name": "kosiew", "email": "kosiew@gmail.com", "time": "Fri May 29 17:42:10 2026 +0800" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri May 29 17:42:10 2026 +0800" }, "message": "Export `to_datafusion_err` from the util crate root (#1487)\n\n* Re-export error helpers from crate root\n\nPublicly re-export curated error helpers, including\nto_datafusion_err, from the crate root. Add a\nregression test in crates/util/tests/root_exports.rs\nto ensure correct functionality in an integration-test\ncontext.\n\n* Restrict re-exported items in lib.rs\n\nMake only to_datafusion_err publicly re-exported from the crate\nroot. Keep PyDataFusionError and PyDataFusionResult as private\nimports for internal use, enhancing encapsulation and reducing\nexposure of non-essential components.\n\n* feat: remove unnecessary blank line in lib.rs to improve code formatting\n\n* rm root_exports.rs" }, { "commit": "0a9ca68ade8bbe06aaf67928bb8edc65670baa24", "tree": "8c9a9acdca0b95cfad42961c19ffe43adc5d3063", "parents": [ "baec559b0a7c85934338d6da80ffbe538004f4d4" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Thu May 28 17:09:33 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu May 28 17:09:33 2026 -0400" }, "message": "docs: user guide + runnable examples for distributing expressions (#1547)\n\n* docs: user guide page + runnable examples for distributing expressions\n\nWraps up the Expr-pickle work with the user-facing material:\n\n* docs/source/user-guide/io/distributing_work.rst — new user guide\n page covering the multiprocessing, Ray, and datafusion-distributed\n patterns. Includes the Security section that is the canonical home\n for the cloudpickle / pickle.loads threat model.\n* docs/source/user-guide/io/index.rst — toctree entry.\n* examples/multiprocessing_pickle_expr.py — runnable example: a\n Pool.map of a closure-capturing UDF across processes, with worker\n context registration in the initializer.\n* examples/ray_pickle_expr.py — Ray actor analogue.\n* examples/datafusion-ffi-example/python/tests/_test_pickle_strict_ffi.py\n — exercises the strict-mode refusal end to end against an FFI\n capsule scalar UDF (kept under the FFI example crate because the\n test needs that crate\u0027s compiled artifacts).\n* examples/README.md — index entries for the new files.\n\nAlso tightens three docstrings that previously duplicated the\nsecurity warning so they point at the canonical Security section\ninstead:\n\n* PythonLogicalCodec::with_python_udf_inlining (rustdoc): one-line\n summary plus a relative pointer to distributing_work.rst and the\n upstream Python pickle module security warning.\n* SessionContext.with_python_udf_inlining: one-sentence summary plus\n :doc: link to the user guide.\n* datafusion.ipc module docstring: cross-reference to the user guide\n for the full pattern.\n\nThe crate-level codec.rs module rustdoc also updates \"pure-Python\nscalar UDFs\" to \"scalar / aggregate / window UDFs\" now that all three\nare covered.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: document Python-version and import portability caveats for inline UDFs\n\nReviewer feedback on the Expr-pickle PRs (#1544) asked that the\ncloudpickle portability caveats be discoverable on the user-facing\npage, not only in docstrings. The distributing_work.rst page is the\ndesignated canonical home for the distribution story, so add them here:\n\n* New \u0027Portability requirements for inline Python UDFs\u0027 subsection\n covering the matching-Python-minor-version requirement and the\n by-value vs by-reference import-capture rule (imported modules must\n be importable on the worker).\n* Qualify the \u0027fully portable\u0027 Python-UDF bullet to point at the new\n requirements.\n* Cross-reference the new subsection from the closure-capture note.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: restore version-byte and cloudpickle-cache rustdoc wording\n\nTwo codec.rs docstrings were reworded in PR4 in ways that dropped\ninformation:\n\n* try_encode_python_scalar_udf: restore the `DFPYUDF` family prefix +\n version byte description of the payload framing (PR4 had collapsed it\n to `DFPYUDF1` prefix, dropping the version-byte mention).\n* cloudpickle cached-handle comment: restore \"The encode/decode helpers\n above\" wording.\n\n* docs: fix reversed tuple order in multiprocessing example docstring\n\nThe \u0027Worker layout\u0027 docstring described tasks as `(expr, label)` but\nthe code builds and unpacks them as `(label, expr)`. Correct the doc\nto match.\n\n* Respond to first batch of reviewer comments\n\n* docs: relocate and restructure distributing-work guide\n\nMove the page from user-guide/io/ to the top level of user-guide/ — distributing work is a runtime/operational concern, not a file-format topic, and the shorter \"Distributing work\" title fits the sidebar cleanly.\n\nRestructure the body to lead with the practical worker-setup pattern instead of the four-slot SessionContext taxonomy. The taxonomy survives at the bottom as a reference subsection; the worker-init example and portability rules now reach the reader before they need it. Also addresses reviewer NIT: wrap the `if __name__ \u003d\u003d \"__main__\":` guidance in a `.. note::` admonition and link to the Python multiprocessing docs.\n\nAdd a header paragraph to each runnable example pointing to the user-guide page so a reader who jumps straight to the example gets the surrounding context.\n\nCo-Authored-By: Claude Opus 4.7 \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "baec559b0a7c85934338d6da80ffbe538004f4d4", "tree": "d18a0c7b5b9385d49cd65dd5f8f68040c6042259", "parents": [ "23f9179ad08189637f88218a8ea77a1222262abc" ], "author": { "name": "BharatDeva", "email": "bharatdevagir@gmail.com", "time": "Thu May 28 14:28:47 2026 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu May 28 15:28:47 2026 -0400" }, "message": "fix: type scalar UDF returns as Arrow arrays (#1528)\n\nCo-authored-by: BharatDeva \u003c278575558+BharatDeva@users.noreply.github.com\u003e" }, { "commit": "23f9179ad08189637f88218a8ea77a1222262abc", "tree": "36f85a95e9618d5cb3f0d1c1e2e9649270dc8c3c", "parents": [ "56b1ceaae2a5023c6b3238999dc55fcee781aa24" ], "author": { "name": "BharatDeva", "email": "bharatdevagir@gmail.com", "time": "Thu May 28 08:34:43 2026 -0500" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu May 28 09:34:43 2026 -0400" }, "message": "docs: document null-handling function arguments (#1527)\n\nCo-authored-by: BharatDeva \u003c278575558+BharatDeva@users.noreply.github.com\u003e" }, { "commit": "56b1ceaae2a5023c6b3238999dc55fcee781aa24", "tree": "c8a4a4da02eceee853e6e63f96d11aa7fa7887ff", "parents": [ "081325afe3ac97c6d2ef793a352a26a2634d2738" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Wed May 27 13:22:19 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed May 27 13:22:19 2026 -0400" }, "message": "Bump DataFusion to 1321d60 (54.0.0) (#1562)\n\nUpdate the pinned DataFusion git rev to\n1321d60cc37ee487d1e7ce7f501357c3236b2542, which is DataFusion 54.0.0.\n\nBump the workspace dependency requirements from 53 to 54 so the\n[patch.crates-io] git overrides actually bind (cargo only applies a\npatch when its version satisfies the dependency requirement), and\nrefresh Cargo.lock accordingly.\n\nAdapt to the 54 API:\n- Remove the DatasetExec::apply_expressions override; apply_expressions\n is no longer a member of the ExecutionPlan trait.\n- factorial now errors on negative input, so take abs() before\n applying factorial in the parametrized expr test and update the\n expected values.\n\nCo-authored-by: Claude \u003cnoreply@anthropic.com\u003e" }, { "commit": "081325afe3ac97c6d2ef793a352a26a2634d2738", "tree": "060f3a8b66790c1b6d3f53426274d54b2fe9d6e5", "parents": [ "fa021fef203c9194747b9ebf1c1f526867d407b1" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Tue May 26 15:05:53 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue May 26 15:05:53 2026 -0400" }, "message": "feat: Python UDFs: per-session inlining toggle and strict refusal setting (#1546)\n\n* feat: per-session Python UDF inlining toggle + sender ctx + strict refusal\n\nAdds a per-session toggle that turns inline Python UDF encoding on or\noff, plus the supporting plumbing to make it usable through\npickle.dumps.\n\nCodec layer:\n * PythonLogicalCodec / PythonPhysicalCodec gain a python_udf_inlining\n bool (default true) and a with_python_udf_inlining(enabled) builder.\n Each try_encode_udf{,af,wf} short-circuits to inner when the toggle\n is off; each try_decode_udf{,af,wf} that recognizes a DFPY* magic\n on a strict codec returns a clean Execution error instead of\n invoking cloudpickle.loads. The refusal message names the UDF and\n the wire family so an operator can see at a glance whether to\n re-encode the bytes or register the UDF on the receiver.\n\nSession layer:\n * PySessionContext::with_python_udf_inlining(enabled) returns a new\n session whose stacked logical + physical codecs both carry the\n toggle. The Arc\u003cSessionState\u003e is cloned (cheap), only the codec\n pair is rebuilt, so registrations and config stay attached.\n * SessionContext.with_python_udf_inlining(*, enabled) is the Python\n wrapper. enabled is keyword-only because positional booleans at\n the call site read as opaque.\n\nSender-side context:\n * datafusion.ipc gains set_sender_ctx / get_sender_ctx /\n clear_sender_ctx thread-locals. Expr.__reduce__ now consults\n get_sender_ctx() to pick the codec for outbound pickles, which is\n the only path through which a strict session affects pickle.dumps\n (the protocol calls __reduce__ with no arguments). Without a\n sender context the default codec is used.\n\nTests:\n * test_pickle_expr.py picks up TestPythonUdfInliningToggle (covers\n both directions of the toggle plus the explicit-ctx fast path),\n TestWorkerCtxLifecycle (set/clear/threading), and\n TestSenderCtxLifecycle.\n * New test_pickle_multiprocessing.py + helpers exercise the full\n driver -\u003e worker round-trip on a multiprocessing.Pool with set_*_ctx\n installed in the worker initializer.\n * CI workflow gets a 30-minute timeout-minutes backstop so a hung\n pickle worker can\u0027t block the matrix indefinitely.\n\nUser-guide docs and the runnable examples land in PR4 of this series.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* update uv lock\n\n* docs: clarify Python UDF inlining docstring; drop unresolved :doc: refs\n\nRewrite with_python_udf_inlining docstring for readability and remove\nreferences to /user-guide/io/distributing_work, which does not exist\nyet. Keep security warning inline as a .. warning:: Security block,\nmatching the existing pattern in Expr.to_bytes / from_bytes /\n__reduce__. The central doc will land in a follow-on PR.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: add doctest examples for sender ctx + UDF inlining toggle\n\nPer CLAUDE.md, every Python function needs a docstring example.\nAdds examples to with_python_udf_inlining, set_sender_ctx,\nclear_sender_ctx, and get_sender_ctx. Also clarifies that\nwith_python_udf_inlining returns a new SessionContext and leaves\nthe original unchanged, matching the with_logical_extension_codec\npattern.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: address review nits for UDF inlining toggle + sender ctx\n\n* codec: strict refusal routes through `read_framed_payload` so\n malformed inline bytes surface their own diagnostic; the\n \"inlining is disabled\" message now fires only when the payload\n would have decoded.\n* codec: add summary line above `PythonPhysicalCodec::with_python_udf_inlining`\n cross-link for rustdoc rendering.\n* expr: hoist `get_sender_ctx` import to module top; note that\n `__reduce__` also drives `copy.copy` / `copy.deepcopy`.\n* context: accept `with_python_udf_inlining` positionally or as\n kwarg (drop `*,`).\n* tests: replace size-ratio heuristic with semantic check for the\n `DFPYUDF` family prefix; switch single-batch closure test to\n `pool.apply`.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: keyword-only inlining flag, skip GIL on prefix mismatch\n\n- `SessionContext.with_python_udf_inlining` now keyword-only (`*, enabled`)\n to match the documented call style and the existing doctests/tests.\n- `refuse_if_inline` and the three `try_decode_python_*` decoders short-\n circuit on a `starts_with(family)` check before `Python::attach`, so\n plans whose UDFs are not Python-defined no longer pay a GIL acquisition\n per decode call. Semantics preserved: `strip_wire_header` already\n returns `Ok(None)` when the prefix does not match.\n- `datafusion.ipc` module docstring wraps the `set_sender_ctx` example in\n `try`/`finally` and notes that the thread-local holds a strong\n reference to the installed `SessionContext` until cleared.\n\nCo-Authored-By: Claude Opus 4.7 \u003cnoreply@anthropic.com\u003e\n\n* Add dev dependency\n\n* Add testing for CI failure\n\n* Additional debugging for mp tests in CI\n\n* Set path for workers\n\n* more path updates for unit tests\n\n* test(pickle): remove multiprocessing CI debug instrumentation\n\nMultiprocessing forkserver/spawn hang was diagnosed and fixed: workers\ncould not import `tests._pickle_multiprocessing_helpers` because\n`pytest --import-mode\u003dimportlib` does not add the test parent dir to\n`sys.path`. The fix (appending the parent dir to `sys.path` so it is\ninherited by mp workers without shadowing the installed `datafusion`\nwheel) is retained. This commit drops the diagnostic scaffolding that\nwas added to identify the hang point:\n\n- `_diag` + per-import / per-task log writes to /tmp\n- `snapshot_processes` and the `threading.Timer` that captured worker\n state mid-hang\n- `diag_init` Pool initializer\n- \"Dump multiprocessing diagnostic log\" CI step\n\nPre-existing infrastructure is kept: per-test `@pytest.mark.timeout(120)`\n(backed by `pytest-timeout` dev dep) and the job-level\n`timeout-minutes: 30` backstop on the test matrix.\n\nCo-Authored-By: Claude Opus 4.7 \u003cnoreply@anthropic.com\u003e\n\n* Shorten rust side docstring since it\u0027s duplicative of the exposed python docstring\n\n* docs: clarify strict-mode refusal message and to_bytes inlining docs\n\nAddress PR review feedback:\n\n- codec.rs: rewrite strict-refusal error to present the two real\n remediations (sender re-encode by-name + receiver register; or\n receiver enables inlining, accepting cloudpickle risk) instead of\n bundling registration with both-side inlining.\n- expr.py: qualify to_bytes docstring so Python UDF self-contained\n behavior is conditional on with_python_udf_inlining being enabled.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: clarify with_python_udf_inlining enabled arg is required\n\nReword docstring to drop misleading \"(the default)\" claim. The\n`enabled` parameter is keyword-only and required — there is no\nargument default. Note instead that fresh sessions inline UDFs\nuntil the toggle overrides them (a session-level default, not an\nargument default).\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: demonstrate strict-mode refusal in with_python_udf_inlining docstring\n\nReplace placeholder isinstance check with a doctest that registers\na Python UDF, encodes an expression on the default session, then\nshows the strict session refusing to decode the inline payload.\nExercises the actual behavior the toggle controls.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: convert sender-ctx example to executable doctest\n\nReplace the code-block in the ipc module docstring that demonstrated\nset_sender_ctx with a doctest that actually runs. Worker-init example\nremains a code-block since it documents a Pool-initializer pattern\nthat does not fit naturally into a doctest.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: use \u0027thread-local sender context\u0027 as adjectival phrase\n\nBare \u0027thread-local\u0027 as a noun reads ambiguously next to the\n_local.ctx attribute name. Hyphenate as adjective with explicit\n\u0027sender context\u0027 noun so the referent is unambiguous.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: drop trailing clear_sender_ctx from set_sender_ctx example\n\nThe trailing cleanup call was test hygiene, not API teaching, and\nrisked implying callers must always pair set with clear. Adjacent\nclear_sender_ctx and get_sender_ctx doctests are self-contained\n(they explicitly set or clear before asserting), so removing the\ncleanup line does not affect doctest outcomes.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "fa021fef203c9194747b9ebf1c1f526867d407b1", "tree": "17fb28c1801c45c0b31022291191c84e8a2d0264", "parents": [ "f43830480b754900b93a30c36e6280d0ce0577f1" ], "author": { "name": "Nick", "email": "24689722+ntjohnson1@users.noreply.github.com", "time": "Tue May 26 11:49:45 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue May 26 11:49:45 2026 -0400" }, "message": "Add details on caching to skill (#1521)" }, { "commit": "f43830480b754900b93a30c36e6280d0ce0577f1", "tree": "584a6d50fa243d6fbe1c8863a2a77d371d887230", "parents": [ "dac9ec6230dba8717d8a0d27de19a141600486b1" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Tue May 26 11:48:57 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue May 26 11:48:57 2026 -0400" }, "message": "feat: expose variety of features from DF54 update (#1554)\n\n* refactor: migrate FFI example table function to call_with_args\n\nDataFusion 53 deprecated `TableFunctionImpl::call(args: \u0026[Expr])` in\nfavor of `call_with_args(args: TableFunctionArgs)`. `PyTableFunction`\nwas migrated in 5a64b0d; this brings the FFI example along so it no\nlonger relies on the deprecated entry point.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat: type SessionContext codec setters with exportable Protocols\n\nPR #1541 introduced `with_logical_extension_codec` /\n`with_physical_extension_codec` setters typed as `codec: Any`. The Rust\nextractors accept either a raw `PyCapsule` or any object exposing\n`__datafusion_logical_extension_codec__` /\n`__datafusion_physical_extension_codec__`.\n\nAdd `LogicalExtensionCodecExportable` / `PhysicalExtensionCodecExportable`\nProtocols in `python/datafusion/user_defined.py` (matching the existing\n`ScalarUDFExportable` pattern) and tighten both setter signatures to\n`Protocol | _PyCapsule`. Pure typing change; no runtime behavior diff.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat: accept variadic field path in get_field\n\nUpstream exposes both `get_field(expr, name)` and\n`get_field_path(expr, [names...])`, but both ultimately call the same\nscalar UDF with a base expression plus one or more name args. Collapse\nthe Python surface into a single variadic `get_field(expr, *names)`\nthat accepts either a one-step lookup or a path of names, dispatching\nthrough a single Rust binding.\n\nNote in `.ai/skills/check-upstream/SKILL.md` that `get_field_path` is\ncovered by the variadic form so future audits do not flag it as a gap.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat: SessionContext.read_batches / read_batch\n\nWrap upstream `SessionContext::read_batches`, which materializes a\nDataFrame directly from a sequence of `RecordBatch`es without\nregistering a named table. The single-batch convenience\n`SessionContext.read_batch` is implemented in pure Python by calling\n`read_batches([batch])`, so the Rust side only needs the one binding.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat: SessionContext UDF lookup helpers\n\nExpose `udf(name)` / `udaf(name)` / `udwf(name)` lookups symmetric with\nthe existing `register_udf` / `register_udaf` / `register_udwf` setters,\nplus `udfs()` / `udafs()` / `udwfs()` for enumerating registered\nfunction names. Looked-up functions come back as the same\n`ScalarUDF` / `AggregateUDF` / `WindowUDF` wrappers users already get\nfrom registration, so they can be called as expressions or re-registered\ninto a different session.\n\nReturns Vec\u003cString\u003e from the list helpers (sorted) rather than the raw\nHashSet upstream returns, so calling code gets a stable ordering.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* bump pre-commit so it stops failing CI checks\n\n* test: drop xfail on timestamp[s] parquet roundtrip\n\npyarrow.parquet promotes timestamp[s] to timestamp[ms] on write (apache/arrow#41382),\nso the read array never matched the input. Cast the expected array to timestamp[ms]\nin test_simple_select to assert DataFusion reads what Arrow actually stored.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* test: capture deprecation warning in repr_rows conflict case\n\nDataFrameHtmlFormatter(repr_rows\u003d..., max_rows\u003d...) fires the deprecation\nwarning before raising ValueError, but pytest.raises does not catch warnings.\nThe escaping warning surfaced in every pytest run. Wrap the call in both\npytest.raises and pytest.warns so the warning is asserted, not leaked.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(udf): document SessionContext UDF lookup with worked examples\n\nAdd Examples docstrings (doctest) for `udf` / `udaf` / `udwf` / `udfs` /\n`udafs` / `udwfs` that demonstrate the lookup pattern, including a\nlate-binding example where the function name comes from configuration.\nAdd tests covering config-driven dispatch and built-in UDAF / UDWF\nlookup so the documented patterns are exercised end-to-end.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor(udf): raise KeyError on UDF/UDAF/UDWF lookup miss\n\n`SessionContext.udf` / `udaf` / `udwf` previously surfaced upstream\n`DataFusionError::Plan` as a generic exception whose message (\"There\nis no UDF named ...\") is set by DataFusion and can drift between\nreleases. Pre-check membership via `udfs()` / `udafs()` / `udwfs()`\nand raise `PyKeyError` on miss so callers get the Pythonic\ndict-style lookup behavior and tests are no longer coupled to the\nupstream wording.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor(udf): add _from_internal classmethod to UDF wrappers\n\n`SessionContext.udf` / `udaf` / `udwf` previously constructed wrapper\nobjects by calling `__new__` directly and writing the private `_udf`\n/ `_udaf` / `_udwf` attribute from outside the owning module. Three\nnear-identical blocks coupled `context.py` to wrapper internals.\n\nAdd a `_from_internal` classmethod on each wrapper that takes an\nalready-constructed `df_internal` handle and returns a wrapper\nwithout re-running `__init__`. The lookup methods now collapse to a\nsingle call, the `__new__` bypass is documented on the wrapper class\nitself, and renaming the private field is a one-spot edit.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: widen SessionContext.read_batches to accept any iterable\n\nThe underlying PyArrow FFI extractor for `Vec\u003cRecordBatch\u003e` requires a\nPython `list`, so the previous `list[pa.RecordBatch]` annotation was\naccurate but unnecessarily strict. Accept any\n`Iterable[pa.RecordBatch]` on the Python side and materialize to a\nlist before crossing the FFI boundary so callers can pass generators,\ntuples, or other iterables without manual conversion.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(context): trim codec docstrings, reference Exportable protocols\n\nDrop prose restatement of the type union for `with_logical_extension_codec`\nand `with_physical_extension_codec`. Keep the dunder name (not visible from\nthe type hint) and cross-link the `LogicalExtensionCodecExportable` /\n`PhysicalExtensionCodecExportable` protocols so Sphinx resolves them.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(udf): drop return-type cross-refs in udf/udaf/udwf docstrings\n\nThe `:py:class:` link back to the wrapper class shadowed the return type\nannotation and risked drifting if the class were moved. Replace with a\nplain backtick literal; surrounding contract prose is unchanged.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(functions): use F alias in get_field doctest\n\nThe doctest namespace already imports `datafusion.functions as F`,\nmaking `F.named_struct` / `F.get_field` shorter than the\n`dfn.functions.*` form.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "dac9ec6230dba8717d8a0d27de19a141600486b1", "tree": "2c4c4327e94248d28aeb205cfb44d8596b0849d7", "parents": [ "afaeccbf27623e983bba594f466024d1f52c5a0f" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Wed May 20 18:58:32 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed May 20 18:58:32 2026 -0400" }, "message": "feat: enable pickling for Python aggregate and window UDFs (#1545)\n\n* feat: inline encoding for Python aggregate and window UDFs\n\nExtends the PythonLogicalCodec / PythonPhysicalCodec inline encoding\nintroduced for scalar UDFs to also cover Python-defined aggregate and\nwindow UDFs. The cloudpickle tuple shape per family is:\n\n DFPYUDA (agg) (name, accumulator_factory, input_schema_bytes,\n return_schema_bytes, state_schema_bytes,\n volatility_str)\n DFPYUDW (window) (name, evaluator_factory, input_schema_bytes,\n return_schema_bytes, volatility_str)\n\nSame wire-framing as scalar (family magic + version byte + cloudpickle\nblob), same schema serde (arrow-rs native IPC), same cached cloudpickle\nhandle. The agg state schema is encoded as a full IPC schema so the\npost-decode UDF reports the same names + nullability + metadata as the\nsender — relevant for accumulators whose StateFieldsArgs consumers key\noff names rather than positional DataType.\n\nRequired restructuring two existing UDF impls so the codec can grab\nthe Python callable directly:\n\n* udaf.rs: replaces create_udaf + AccumulatorFactoryFunction closure\n with a named PythonFunctionAggregateUDF that stores the Py\u003cPyAny\u003e\n accumulator factory. Synthesizes state_{i} field names when the\n Python constructor passes only Vec\u003cDataType\u003e; from_parts preserves\n the full state schema on the decode side.\n* udwf.rs: renames MultiColumnWindowUDF -\u003e PythonFunctionWindowUDF,\n drops the PartitionEvaluatorFactory PtrEq wrapper, stores the\n Py\u003cPyAny\u003e evaluator directly. PartialEq and Hash get the same\n pointer-identity fast path + debug-log exception handling already\n on PythonFunctionScalarUDF.\n\nUser-facing surface:\n\n* AggregateUDF.name and WindowUDF.name properties (parallel to the\n ScalarUDF.name shipped in PR1).\n* Existing UDAF/UDWF construction paths are unchanged.\n\nThe per-session with_python_udf_inlining toggle, sender-side context,\nstrict refusal, and user-guide docs land in PRs 3-4 of this series.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat: restore pub UDAF/UDWF helpers and document inline encoding\n\nRe-export `to_rust_accumulator`, `to_rust_partition_evaluator`, and\n`PythonFunctionWindowUDF` (with a `MultiColumnWindowUDF` alias) by\npromoting `udaf` and `udwf` to `pub mod` so prior downstream Rust\nconsumers keep their API surface after the inline-encoding refactor.\n\nAdds an end-to-end window UDF pickle round-trip test that runs the\ndecoded evaluator over a real session, mirroring the aggregate test.\n\nDocuments the cloudpickle-based shipping behavior of Python aggregate\nand window UDFs in the user-guide aggregations and windows pages.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: address PR #1545 review feedback\n\n- Fix CountAcc.merge in pickle test: sum over states[0] (partition\n counts), not over the list of state fields. The prior implementation\n only added partition 0\u0027s count when merging across partitions.\n- Drive test_agg_udf_evaluates_after_roundtrip with a two-batch\n DataFrame so merge actually runs and the round-tripped state-field\n schema is exercised end-to-end.\n- Correct PY_AGG_UDF_FAMILY / PY_WINDOW_UDF_FAMILY doc comments and the\n aggregate block comment to reference \"return schema bytes\" rather\n than \"return type\" / \"return_type_bytes\" so the docs match the actual\n on-wire layout.\n- Keep `udaf` and `udwf` modules private (matching `udf`) and\n selectively re-export the helpers downstream Rust consumers rely on\n (`to_rust_accumulator`, `to_rust_partition_evaluator`,\n `PythonFunctionWindowUDF`, `MultiColumnWindowUDF`) instead of\n exposing the whole module surface.\n- Rename codec helpers `*_agg_udf` -\u003e `*_udaf` and `*_window_udf` -\u003e\n `*_udwf` for naming consistency with the Python public aliases.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "afaeccbf27623e983bba594f466024d1f52c5a0f", "tree": "88bff2377b13d0acef907ac81871225f7e27fcf1", "parents": [ "8ba06e4147122962f356e11b73175379473f75a7" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Tue May 19 09:31:30 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue May 19 09:31:30 2026 -0400" }, "message": "feat: enable pickling of most Expr except udaf and udwf (#1544)\n\n* feat: pickle support for Expr via inline scalar UDF encoding\n\nAdds Python-aware encoding to PythonLogicalCodec/PythonPhysicalCodec\nso a ScalarUDF defined in Python travels inside the serialized\nexpression (cloudpickled into fun_definition) instead of needing a\nmatching registration on the receiver. With that in place, Expr gains\n__reduce__ + classmethod from_bytes(buf, ctx\u003dNone) so pickle.dumps /\npickle.loads work end-to-end on expressions built from col, lit,\nbuilt-in functions, and Python scalar UDFs.\n\nWire format is framed as \u003cDFPYUDF magic, version byte, cloudpickle\ntuple\u003e; the version byte lets a too-new/too-old payload surface a\nclean Execution error instead of an opaque cloudpickle unpack\nfailure. Schema serde is via arrow-rs\u0027s native IPC (no pyarrow\nround-trip). Cloudpickle module handle is cached per-interpreter\nthrough PyOnceLock.\n\nWorker-side context resolution lives in a new datafusion.ipc module:\nset_worker_ctx / get_worker_ctx / clear_worker_ctx plus a private\n_resolve_ctx helper consulted by Expr.from_bytes. Priority is\nexplicit ctx \u003e worker ctx \u003e global SessionContext. FFI UDFs still\ntravel by name and require the matching registration on the\nreceiver\u0027s context.\n\nAggregate and window UDF inline encoding, the per-session\nwith_python_udf_inlining toggle, sender-side context, and the\nuser-guide docs land in follow-on PRs.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(pickle): add cloudpickle security warnings, docstring examples, edge-case tests\n\nInline `.. warning::` blocks on `Expr.to_bytes`, `Expr.from_bytes`, and\n`Expr.__reduce__` so the cloudpickle / arbitrary-code-execution caveat is\nvisible at the public API surface in advance of the user-guide page that\nlands in PR 4.\n\nAdd doctest-style `Examples:` blocks to `datafusion.ipc` functions\n(`set_worker_ctx`, `clear_worker_ctx`, `get_worker_ctx`, `_resolve_ctx`),\n`ScalarUDF.name`, and the new `Expr` pickle methods, per CLAUDE.md.\n\nTighten `Expr.__reduce__` return annotation to\n`tuple[Callable[[bytes], Expr], tuple[bytes]]`.\n\nTests: multi-arg UDF round-trip (covers synthetic `arg_{i}` schema-field\nloop in the codec) plus malformed-bytes paths through `Expr.from_bytes`.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* as_any no longer in api\n\n* feat(pickle): stamp Python (major, minor) in UDF wire header\n\ncloudpickle bytecode is not portable across Python minor versions —\na payload produced on 3.11 fails to load on 3.12 with an opaque\nmarshal/unpickle error. Embed the sender\u0027s (major, minor) in the\nDFPYUDF wire header and reject mismatches at decode time with an\nactionable error that names both versions, instead of letting the\nfailure surface from inside cloudpickle.loads.\n\nHeader layout becomes:\n DFPYUDF (7) | version (1) | py_major (1) | py_minor (1) | cloudpickle\n\nExtend the Security warnings on Expr.to_bytes / from_bytes /\n__reduce__ with a Portability section covering the cross-version\nconstraint and cloudpickle\u0027s by-value/by-reference behavior (the\ncallable inlines bytecode and closure cells, but imported names\ntravel by reference and must be importable on the receiver). Add\na matching Serialization model note to the datafusion.ipc module\ndocstring.\n\nNew tests:\n - codec::wire_header_tests: py-major/minor mismatch, truncated\n py-version bytes, round-trip with py-version\n - test_pickle_expr::test_cross_version_error_message: patches the\n py_minor byte inside an emitted payload and asserts the error\n message identifies the version mismatch\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "8ba06e4147122962f356e11b73175379473f75a7", "tree": "8238b487946e3b83109ea7bf5d27a6d25aad1cb8", "parents": [ "baef8f00e1de6a086fa1814c98e65d216fb16277" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Mon May 18 11:51:36 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon May 18 11:51:36 2026 -0400" }, "message": "Update datafusion dependency to latest in preparation for DF54 (#1532)\n\n* feat: upgrade upstream DataFusion 53 → main (pre-54)\n\nBump workspace deps to apache/datafusion@3d06bedc (git pin) in\npreparation for the 54.0.0 release. Workspace package version moves\nto 54.0.0 to track the upstream major convention.\n\nCompile fixes:\n- Drop as_any impls (trait now has Any as supertrait) and use the\n upstream-provided downcast_ref helper on dyn trait objects.\n- Reconcile FFI provider From conversions to drop redundant `+ Send`\n on Arc\u003cdyn ...\u003e bounds.\n- Cast/TryCast: data_type → field.data_type() (FieldRef rename).\n- Stub match arms for new Expr::HigherOrderFunction / Lambda /\n LambdaVariable and ScalarValue::ListView / LargeListView variants;\n proper exposure deferred to PR 3 audit.\n- DatasetExec: partition_statistics returns Arc\u003cStatistics\u003e; add\n required apply_expressions trait method.\n- Suppress TableFunctionImpl::call deprecation pending call_with_args\n refactor that needs Session plumbing.\n\nUser-facing test updates for upstream behavior changes:\n- median / approx_median / approx_percentile_cont now return Float64.\n- String functions (concat_ws, lower, upper, repeat, reverse,\n split_part, translate) return StringView when given StringView.\n- overlay appends past end-of-string rather than replacing the input.\n- arrays_zip / list_zip struct field names \"c0\"/\"c1\" → \"1\"/\"2\".\n- Filter on mismatched cast types now errors (was 0 matches).\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat: expose DataFrame.alias and tidy public API after DF53→54 audit\n\nCompanion to the upstream DataFusion 53 → main bump. The\ncheck-upstream audit (PR 3 of dev/release/upstream-sync.md) surfaced a\nsmall set of trivial wins; this commit ships them.\n\nTrivial wins:\n- DataFrame.alias(name) — wraps the logical plan in a SubqueryAlias.\n- functions.__all__: add `instr` and `position` (both were defined as\n public defs but missing from `__all__`, so they didn\u0027t show up in\n `from datafusion.functions import *` or generated docs).\n- top-level `datafusion.__all__`: re-export `TableProviderFactory` and\n `TableProviderFactoryExportable` (previously only reachable via the\n `datafusion.catalog` submodule).\n\nNon-trivial gaps surfaced by the audit (DataFrame.registry,\ninto_*/task_ctx, SessionContext extensibility surface, distinct-aware\naggregate variants, TableFunctionImpl::call_with_args migration, FFI\nProtocol pipeline gaps) are deferred — each warrants its own design\nand PR.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* taplo fmt\n\n* Update unit test to go along with https://github.com/apache/datafusion/pull/22133\n\n* docs: demonstrate alias via self-join in DataFrame.alias example\n\nPrior example called alias(\"t\") then to_pydict(), which did not show\nthe qualifier effect. Replace with a self-join that uses col(\"l.val\")\nand col(\"r.val\") so the disambiguation behavior is visible.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat: wrap higher-order, lambda, and lambda-variable Expr variants\n\nDataFusion 54 introduces Expr::HigherOrderFunction, Expr::Lambda, and\nExpr::LambdaVariable. PyExpr::to_variant previously errored on each\nwith py_unsupported_variant_err. Add PyHigherOrderFunction, PyLambda,\nand PyLambdaVariable wrappers, register them in the expr pymodule and\nre-export from python/datafusion/expr.py, and dispatch to_variant to\nthe new wrappers.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat: wire rex_type and rex_call_operands for new Expr variants\n\nMap HigherOrderFunction and Lambda to RexType::Call; LambdaVariable to\nRexType::Reference. In rex_call_operands return the args for\nHigherOrderFunction, the body for Lambda, and self for LambdaVariable\n(mirroring Column). In rex_call_operator return the underlying UDF\nname for HigherOrderFunction and the literal \"lambda\" for Lambda.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat: support LargeList/ListView/LargeListView in map_from_scalar_to_arrow\n\nThese ScalarValue variants all wrap Arc\u003c...Array\u003e, exposing the outer\nDataType via Array::data_type(), so we can mirror the existing\nScalarValue::List arm instead of returning PyNotImplementedError. This\nmakes Expr.types() work for plans that round-trip through SQL or proto\nwhere these scalar variants surface.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: switch PyTableFunction to non-deprecated call_with_args\n\nDataFusion 53.0.0 deprecated TableFunctionImpl::call in favor of\ncall_with_args(args: TableFunctionArgs), which threads a Session\nreference alongside the exprs. Implement call_with_args on\nPyTableFunction (delegating to the FFI variant\u0027s call_with_args, or\nignoring the session for the pure-Python variant which doesn\u0027t use it)\nand have __call__ build a TableFunctionArgs from the global session.\nDrops both #[allow(deprecated)] attributes.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* build: revert workspace version to 53.0.0 and move DF overrides to [patch.crates-io]\n\nThe workspace version was prematurely bumped to 54.0.0 in the\nDF53→pre-54 upgrade. Restore it to 53.0.0 until we are actually\nready to cut the 54 release.\n\nThe same change had moved every datafusion-* dependency from a\ncrates.io version constraint to a direct git dep in\n[workspace.dependencies]. Switch them back to \"version \u003d \\\"53\\\"\" and\nmove the git rev overrides into [patch.crates-io] so the published\nmanifest will be patch-free.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* taplo format\n\n* test: sort FFI test results by partition key before equality compare\n\nMulti-partition `collect()` returns batches in execution-scheduling\norder, which is non-deterministic and differs between local and CI\nrunners. Sort by the first value of column 0 (unique per partition in\neach affected test) so the expected/actual comparison is stable.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Bump datafusion main commit\n\n* test: cover new DF54 expr wrappers, catalog factories, and DataFrame.alias\n\nAdd module-metadata checks for HigherOrderFunction, Lambda, LambdaVariable\nand the top-level TableProviderFactory / TableProviderFactoryExportable\nre-exports, plus a self-join regression test exercising the new\nDataFrame.alias() qualifier-based selection path.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "baef8f00e1de6a086fa1814c98e65d216fb16277", "tree": "53d7aaca6eeebed0d0f6662343b9b04a4dba6222", "parents": [ "55870ff30c4a1f086e0b63de434ebf8b674ac110" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri May 15 09:00:21 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri May 15 09:00:21 2026 -0400" }, "message": "Add support for logical and physical codecs (#1541)\n\n* feat: unify logical + physical proto codec stack via SessionContext\n\nIntroduces a single composable codec layer that every serializer reads\nfrom the session, replacing the hardcoded `DefaultLogicalExtensionCodec`\n/ `DefaultPhysicalExtensionCodec` calls scattered across PyLogicalPlan,\nPyExecutionPlan, and the Rust-wrapped Python provider plumbing.\n\nKey changes:\n\n* New `PythonLogicalCodec` and `PythonPhysicalCodec` (crates/core/src/codec.rs)\n wrap any inner `LogicalExtensionCodec` / `PhysicalExtensionCodec`. Both\n share a `DFPYUDF1` magic-prefix path for in-band cloudpickle encoding\n of Python scalar UDFs, so an `ExecutionPlan` / `PhysicalExpr`\n referencing a Python `ScalarUDF` round-trips through either layer.\n Magic-prefix registry table (DFPYUDF1 in use; DFPYUDA1 / DFPYUDW1 /\n DFPYPE1 reserved) documented in the module header.\n\n* `PySessionContext` stores `Arc\u003cPythonLogicalCodec\u003e` and\n `Arc\u003cPythonPhysicalCodec\u003e` directly. FFI wrappers are built on demand\n via `ffi_logical_codec()` / `ffi_physical_codec()` for capsule export\n and downstream `RustWrappedPy*` consumers. Adds\n `__datafusion_physical_extension_codec__` getter +\n `with_physical_extension_codec` setter (symmetric with the logical\n pair).\n\n* `PyLogicalPlan.to_proto` / `from_proto` renamed to `to_bytes` /\n `from_bytes`, now reading the session\u0027s logical codec. `to_proto` /\n `from_proto` survive as deprecated thin wrappers emitting\n `DeprecationWarning`.\n\n* `PyExecutionPlan` gains the same `to_bytes` / `from_bytes` rename +\n deprecated aliases, plus `__datafusion_execution_plan__` capsule\n getter and `from_pycapsule` (ported from poc_ffi_query_planner).\n\n* New `PyPhysicalExpr` class with `to_bytes` / `from_bytes` /\n `from_pycapsule` / `__datafusion_physical_expr__`. `from_bytes`\n takes an input pyarrow Schema for column-reference resolution.\n\n* `datafusion-python-util` gains `from_pycapsule!` /\n `try_from_pycapsule!` macros + `physical_codec_from_pycapsule`,\n `task_context_from_pycapsule`, `create_physical_extension_capsule`\n (ported from poc_ffi_query_planner).\n\n* `PythonFunctionScalarUDF` exposes `func()`, `input_fields()`,\n `return_field()`, `volatility()`, `from_parts()` accessors needed\n by the codec.\n\nPython wrapper updates: `LogicalPlan` / `ExecutionPlan` add\n`to_bytes` / `from_bytes` + deprecate `to_proto` / `from_proto`;\n`ExecutionPlan` adds capsule getter + `from_pycapsule`; new\n`PhysicalExpr` wrapper class exported from the top-level package;\n`SessionContext` exposes the physical codec capsule + setter.\n\nTest coverage in python/tests/test_plans.py: round-trip via new API,\ndeprecation warnings on old API, capsule protocol getters,\nsession-routed codec on both layers.\n\n`PyLogicalPlan` PyCapsule protocol is intentionally not added —\n`datafusion-ffi` does not expose `FFI_LogicalPlan`, so there is no\nstable cross-crate shape to publish. Round-tripping a `LogicalPlan`\ngoes through `to_bytes` / `from_bytes` only.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* test: FFI-example integration tests for codec + plan capsule APIs\n\nAdds four downstream-crate fixtures in `datafusion-ffi-example` so the\nnew PR1 surface can be tested with the same FFI-handoff pattern used\nfor table providers, UDFs, etc. Existing tests prove the API exists;\nthese tests prove it composes with code that lives in another crate.\n\nNew Rust types in `examples/datafusion-ffi-example/src/`:\n\n* `MyLogicalExtensionCodec` — delegates to\n `DefaultLogicalExtensionCodec` and bumps atomic counters on the UDF\n encode/decode entry points. Exported via\n `__datafusion_logical_extension_codec__`. Installed onto a session\n with `ctx.with_logical_extension_codec(my_codec)`.\n* `MyPhysicalExtensionCodec` — mirror for `PhysicalExtensionCodec`.\n* `MyExecutionPlan` — wraps a one-column `EmptyExec`, exposes\n `__datafusion_execution_plan__`. Lets the receiver consume an\n `ExecutionPlan` capsule that did not originate in\n datafusion-python.\n* `MyPhysicalExpr` — wraps `Literal(Int32(42))`, exposes\n `__datafusion_physical_expr__`. Same FFI handoff for physical\n expressions.\n\nNew tests:\n\n* `_test_logical_extension_codec.py` — codec installs cleanly, the\n session re-exports its capsule, and `try_encode_udf` fires on the\n user codec when serializing a plan that references a `ScalarUDF`.\n The decode counterpart is a round-trip check rather than a counter\n assertion: when the UDF is in the receiver\u0027s function registry,\n `parse_expr` resolves by name before consulting the codec.\n* `_test_physical_extension_codec.py` — symmetric.\n* `_test_execution_plan.py` — parametrized over typed-class vs\n raw-capsule input; verifies `ExecutionPlan.from_pycapsule` consumes\n the downstream capsule.\n* `_test_physical_expr.py` — same for `PhysicalExpr.from_pycapsule`.\n\nAPI changes forced by the new tests:\n\n* `PyLogicalPlan.to_bytes`, `PyExecutionPlan.to_bytes`,\n `PyPhysicalExpr.to_bytes` now accept an optional `ctx` parameter.\n When supplied, encoding routes through the session\u0027s installed\n codec instead of a fresh default. `ctx\u003dNone` preserves the previous\n default-codec behavior used by the deprecated `to_proto` shims.\n* The util `from_pycapsule!` / `try_from_pycapsule!` macros now\n validate the capsule name via `pointer_checked(Some(c\"...\"))`\n rather than `pointer_checked(None)`. The latter rejects named\n capsules outright with CPython\u0027s \"incorrect name\" error.\n* `SessionContext.with_logical_extension_codec` and\n `with_physical_extension_codec` now wrap the returned internal\n context in `SessionContext` so the result has the full Python\n surface. The pre-existing logical setter was returning a raw\n internal object that lacked `sql()` and friends.\n\n`examples/datafusion-ffi-example/Cargo.toml` gains `datafusion` and\n`datafusion-proto` workspace dependencies for the new Rust impls.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: tighten PR1 scope to codec plumbing only\n\nReview feedback pass. PR1 is now strictly the composable codec layer +\nsession routing + class-method serialization API. Anything that\ntouches actual Python UDF inline encoding or Python expression\nwrapping moves to PR2 alongside the pickle work.\n\nDropped:\n\n* `encode_python_scalar_udf` / `decode_python_scalar_udf` helpers\n from `crates/core/src/codec.rs`, along with cloudpickle and pyarrow\n imports. The wrapper codecs now delegate every method to `inner`.\n `DFPYUDF1` magic constant is kept (marked `dead_code` for now) as a\n reservation so PR2 has a single definition site.\n* `udf.rs` reverted to pre-PR1 shape. The codec no longer needs\n `func()` / `input_fields()` / `volatility()` / `from_parts()`\n accessors. Re-added by PR2 when scalar-UDF inlining lands.\n* `PyPhysicalExpr` class + Python wrapper + `__init__` export +\n `MyPhysicalExpr` FFI fixture + `_test_physical_expr.py`. No\n consumer in PR1 or PR2 plan documents; symmetry with\n `PyExecutionPlan` is not enough to justify the surface area.\n* Rust-side `PyLogicalPlan::to_proto` / `from_proto` and\n `PyExecutionPlan::to_proto` / `from_proto` deprecated wrappers.\n The deprecation lives entirely in the Python wrapper layer, which\n emits `DeprecationWarning` and forwards to `to_bytes` /\n `from_bytes`. Less Rust duplication.\n* `PythonLogicalCodec::with_default_inner` /\n `PythonPhysicalCodec::with_default_inner` — redundant with\n `impl Default`. Logic moved into `Default::default`.\n* `PySessionContext::default_logical_codec` /\n `default_physical_codec` helpers. Inlined as\n `Arc::new(PythonLogicalCodec::default())` at the three call sites.\n\nTests (root: 1076, FFI example: 36) all green after the cuts.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* remove unuseful code comments\n\n* docs: rewrite codec module comments around purpose, not PR sequence\n\nThe previous doc-block framed PythonLogicalCodec / PythonPhysicalCodec\nin terms of \"PR1 delegates, PR2 will add encoding\" — useful for\nreview, useless for someone reading the code later.\n\nReframed in terms of what the codecs exist to do: encode Python-side\nplan references (pure-Python UDFs, etc.) into the proto wire format\nso plans can cross process boundaries without the receiver having to\npre-register every callable. The wrappers sit at the top of the\nsession\u0027s codec stack and delegate non-Python encoding to a\ncomposable inner codec.\n\nMagic-prefix registry table loses the \"reserved\" column. Doc still\nnotes that the in-module impls currently delegate and that\nencoder/decoder hooks land alongside the corresponding Python-side\nserialization work.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat(codec): forward every LogicalExtensionCodec /\nPhysicalExtensionCodec method to inner\n\nPythonLogicalCodec previously only overrode the four required methods\non the trait plus the scalar UDF pair, so the default trait impls\n(returning \"LogicalExtensionCodec is not provided\") shadowed any\ndownstream FFI codec for file formats, aggregate UDFs, and window\nUDFs. A user installing their own codec via\n`SessionContext.with_logical_extension_codec(...)` would silently\nlose access to its `try_*_file_format`, `try_*_udaf`,\n`try_*_udwf` implementations.\n\nForward every trait method to `inner` so the user-installed codec is\nfully reachable. Same change on the physical side, including\n`try_*_expr`, `try_*_udaf`, `try_*_udwf` — the corresponding\nPython-aware paths can layer on later by intercepting before\ndelegation.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: tighten codec dispatch test docstrings\n\nThe previous docstrings claimed the tests verify \"PythonLogicalCodec\ndelegates non-Python UDFs to the inner codec.\" That\u0027s\nforward-looking — the codecs currently delegate every UDF\nunconditionally, so the test would behave identically for Python and\nnon-Python UDFs.\n\nRewrite to describe what the test actually proves: the dispatch chain\n`PyLogicalPlan.to_bytes -\u003e session.logical_codec -\u003e PythonLogicalCodec\n-\u003e FFI -\u003e user impl` (and the physical mirror) forwards correctly,\nobservable via the user codec\u0027s atomic counter incrementing after one\nencode pass.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor(ffi-example): MyExecutionPlan emits real data via\nMemorySourceConfig\n\nWas a one-column `EmptyExec` stub useful only as a capsule-handoff\ntarget. Promoted to a minimal reference impl that a downstream Rust\ncrate can copy when exposing a custom `ExecutionPlan` to\ndatafusion-python: configurable `num_rows`, produces a single batch\nof sequential `Int32` values under column `value`, wrapped in\n`DataSourceExec` via `MemorySourceConfig::try_new_exec`. Header\ncomment explains the typical use case (remote backend, streaming\nsource, synthetic data generator) and the\n`__datafusion_execution_plan__` capsule shape downstream crates\nshould follow.\n\nTest asserts the schema-bearing plan survives the FFI hop: a\n`DataSourceExec` arrives with the expected partitioning and no\nchildren. Schema details are not surfaced through the FFI display\npath (only the wrapping `ForeignExecutionPlan` name + inner plan\nname appear), so the test does not assert the column name.\n\n`to_bytes` round-trip of an FFI-imported plan is not exercised:\nencoding requires a physical codec that knows how to serialize\n`ForeignExecutionPlan`, which the default codec does not. A\ndownstream user round-tripping such a plan must install their own\ncodec via `with_physical_extension_codec`. Documented in the test\nfile rather than asserted on.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: drop dormant ExecutionPlan PyCapsule round-trip\n\n`PyExecutionPlan::from_pycapsule` and the matching\n`__datafusion_execution_plan__` exporter have no consumer in this\nrepo, on the POC `poc_ffi_query_planner` branch, or on any sibling\nbranch (`testing/datafusion-distributed`, `testing/ffi-library-marker`,\n`tmp/ffi-with-codecs`). The pair was wired up speculatively for FFI\nplan handoff that no Python code path actually performs today.\n\nDrop the whole capsule round-trip for `ExecutionPlan`:\n\n* Rust `PyExecutionPlan::from_pycapsule` and\n `__datafusion_execution_plan__`.\n* Python `ExecutionPlan.from_pycapsule` and\n `__datafusion_execution_plan__` wrappers.\n* `MyExecutionPlan` FFI fixture + `_test_execution_plan.py` + lib.rs\n registration. Was solely a test fixture for the dropped path.\n* `test_execution_plan_pycapsule_protocol` in `python/tests/test_plans.py`.\n\n`PyExecutionPlan.to_bytes` / `from_bytes` survive — they encode\nthrough the session\u0027s physical codec and have real coverage.\nCapsule round-trip can be re-added when a concrete consumer\n(distributed worker, bridge library) lands.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat: PyExpr.to_bytes / from_bytes via session logical codec\n\nMirrors PyLogicalPlan / PyExecutionPlan: encode through the session\u0027s\ninstalled `LogicalExtensionCodec` (or a default-inner\n`PythonLogicalCodec` when no `ctx` is supplied), decode against the\nsession\u0027s function registry + codec via `parse_expr`.\n\nRust side calls `datafusion_proto::logical_plan::to_proto::serialize_expr`\nand `from_proto::parse_expr`. Python wrapper threads an optional\n`SessionContext` through.\n\nTests cover the session-routed roundtrip and the no-ctx default-codec\nencode path. Adds a third consumer of `session.logical_codec()`\nalongside `PyLogicalPlan` and the codec dispatch tests in the FFI\nexample, broadening coverage of the codec stack.\n\nThis is the last piece of the PR1 codec surface — follow-up pickle\nwork (`Expr.__reduce__`, worker-scoped context, multiprocessing) can\nbuild on this without bundling the byte-level serialization API.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* test(ffi-example): assert codec roundtrip restores plan output\n\nPR review feedback: weak `is not None` checks let regressions slip\npast. Mirror python/tests/test_plans.py — logical compares\n`df.collect() \u003d\u003d round_trip.collect()`; physical compares\n`str(original) \u003d\u003d str(restored)`.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "55870ff30c4a1f086e0b63de434ebf8b674ac110", "tree": "3b3c62ab32911e96aa773ee6374555ca40c0423c", "parents": [ "db22a9240e5832d9a88b74c640e3b60abfbe52c1" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Wed May 13 10:28:12 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed May 13 10:28:12 2026 -0400" }, "message": "build(deps): combined dependabot bumps (Cargo + workflows) (#1534)\n\nCombines the following dependabot PRs:\n\nCargo:\n- tokio 1.50.0 -\u003e 1.52.3 (#1530)\n- arrow-array 58.1.0 -\u003e 58.3.0 (#1522, also picks up 58.3.0)\n- arrow-schema 58.1.0 -\u003e 58.3.0 (#1523, also picks up 58.3.0)\n- datafusion 53.0.0 -\u003e 53.1.0 (#1515)\n- datafusion-catalog 53.0.0 -\u003e 53.1.0 (#1513)\n- datafusion-ffi 53.0.0 -\u003e 53.1.0 (#1512)\n- datafusion-proto 53.0.0 -\u003e 53.1.0 (#1511)\n- datafusion-common 53.0.0 -\u003e 53.1.0 (#1510)\n- mimalloc 0.1.48 -\u003e 0.1.50 (#1514)\n- uuid 1.23.0 -\u003e 1.23.1 (#1508)\n- rustls-webpki 0.103.10 -\u003e 0.103.13 (#1506)\n- rand 0.9.2 -\u003e 0.9.4 (#1495)\n\nGitHub Actions:\n- github/codeql-action 4.32.5 -\u003e 4.35.4 (#1531)\n- astral-sh/setup-uv 7.3.1 -\u003e 8.1.0 (#1500)\n\nEach individual PR was passing CI before this combined bump.\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "db22a9240e5832d9a88b74c640e3b60abfbe52c1", "tree": "995fb0241ff6c61d7ab14234e8068b92c71ee47a", "parents": [ "13b2c47b0d5e348cea24b9264e87fd67666c56f1" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Wed May 06 15:06:53 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed May 06 15:06:53 2026 -0400" }, "message": "docs: add upstream sync process documentation (#1524)\n\n* docs: add upstream sync process documentation\n\nDocument the three-PR workflow used to sync to a newer upstream\napache/datafusion version: bump crate deps + fix breakage, consolidate\ntransitive deps, then fill API and documentation gaps via\n/check-upstream. Cross-reference from dev/release/README.md.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: add audit-skill-md skill\n\nNew AI agent skill at .ai/skills/audit-skill-md/SKILL.md to keep the\nuser-facing skills/datafusion_python/SKILL.md in sync with the public\nPython API. Audits SessionContext, DataFrame, Expr, and functions\nsurfaces for new APIs not covered, stale mentions, examples that drifted\nfrom idiomatic style, and missing version notes. Wired into PR 3 of the\nupstream sync workflow documented in dev/release/upstream-sync.md.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: verify upstream sync completed before release\n\nAdd a checklist item to \"Preparing the main Branch\" pointing release\nmanagers at dev/release/upstream-sync.md so the crate bump, dependency\nconsolidation, and /check-upstream and /audit-skill-md passes are\nconfirmed done before the release branch is cut.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: scope upstream sync cargo update to datafusion family\n\nReplace `cargo update -p datafusion` with an explicit multi-`-p`\ninvocation listing every `datafusion-*` workspace dependency, so PR 1\nof the upstream-sync workflow refreshes only the datafusion family\nand leaves other transitives for PR 2 to consolidate.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: correct datafusion-* pin location in upstream sync\n\nPR 1 step 1 incorrectly stated downstream `datafusion-*` crates are\npinned in `crates/core/Cargo.toml`. Pins live in the root\n`[workspace.dependencies]`; per-crate manifests inherit via\n`workspace \u003d true`. Reword step 1 to point at the right file.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: restore workspace.package version bump in upstream sync\n\nPR 1 step 1 must also bump `[workspace.package].version` because the\n`datafusion-python` major version tracks the upstream `datafusion`\nmajor. The previous reword dropped that instruction. Reinstate it\nalongside the `[workspace.dependencies]` updates.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: align audit-skill-md description with body version phrasing\n\nFrontmatter description referenced \"requires upstream DataFusion vX\",\nbut the body of the skill settles on the `datafusion-python NN` form\n(consistent with the package/upstream-major equivalence). Switch the\ndescription to match so the skill speaks one language end to end.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: fold make-pythonic step into PR 3 of upstream sync\n\nAudit-skill-md documents the order\n`/check-upstream` -\u003e `/make-pythonic` (optional) -\u003e `/audit-skill-md`,\nbut PR 3 of the upstream-sync workflow only listed the first and last.\nInsert the make-pythonic pass as step 3 so signatures get aligned\nbefore the SKILL.md audit, avoiding example churn. Drops the orphan\ntrailing paragraph in favor of inline guidance on when to defer\nlarger reshapes to their own PR.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: drop literal Cargo.toml version from audit-skill-md inputs\n\nReplace literal `version \u003d \"53.0.0\"` example with a pointer to the\n`[workspace.package]` field plus an `NN.0.0` placeholder so the skill\nprose does not drift each major bump.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "13b2c47b0d5e348cea24b9264e87fd67666c56f1", "tree": "14579478ae62f6a4e95898402a17c5392dd7722b", "parents": [ "c657dad97349c1113e843e3e15bb41f865e65a97" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Sun May 03 10:52:10 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun May 03 10:52:10 2026 -0400" }, "message": "Update user documentation for AI agent skill usage (#1505)\n\n* docs: publish SKILL.md on the docs site via myst include\n\nAdds a new `skill` page that embeds the repo-root `SKILL.md` through the\nmyst `{include}` directive, so the agent-facing guide lives on the\npublished docs site without duplication. The page is wired into the\nUser Guide toctree. Implements PR 4a of the plan in #1394.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: publish llms.txt at docs site root\n\nAdds `docs/source/llms.txt` in llmstxt.org schema: a short description\nplus categorized links to the agent skill, user guide pages, DataFrame\nAPI reference, and example queries. `html_extra_path` in `conf.py`\ncopies it verbatim to the published site root so it resolves at\n`https://datafusion.apache.org/python/llms.txt`. Implements PR 4b of\nthe plan in #1394.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: add write-dataframe-code contributor skill\n\nAdds `.ai/skills/write-dataframe-code/SKILL.md`, a contributor-facing\nskill for agents working on this repo. It layers on top of the\nuser-facing repo-root SKILL.md with:\n\n- a TPC-H pattern index mapping idiomatic API usages to the query file\n that demonstrates them,\n- an ad-hoc plan-comparison workflow for checking DataFrame translations\n against a reference SQL query via `optimized_logical_plan()`, and\n- the project-specific docstring and aggregate/window documentation\n conventions that CLAUDE.md already enforces for contributors.\n\nImplements PR 4c of the plan in #1394.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: add audit-skill-md skill\n\nAdds `.ai/skills/audit-skill-md/SKILL.md`, a contributor skill that\ncross-references the repo-root `SKILL.md` against the current public\nPython API (functions module, DataFrame, Expr, SessionContext, and\npackage-root re-exports). Reports two classes of drift:\n\n- new APIs exposed by the Python surface that are not yet covered in\n the user-facing guide, and\n- stale mentions in the guide that no longer exist in the public API.\n\nThe skill is diff-only — it produces a report the user reviews before\nany edit to `SKILL.md`. Complements `check-upstream/`, which audits in\nthe opposite direction (upstream Rust features not yet exposed).\n\nImplements PR 4d of the plan in #1394.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: enrich RST pages with demos relocated from TPC-H rewrite\n\nMoves the illustrative patterns that #1504 removed from the TPC-H\nexamples into the common-operations docs, where they serve as\npattern-focused teaching material without cluttering the TPC-H\ntranslations:\n\n- expressions.rst gains a \"Testing membership in a list\" section\n comparing `|`-compound filters, `in_list`, and `array_position` +\n `make_array`, plus a \"Conditional expressions\" section contrasting\n switched and searched `case`.\n- udf-and-udfa.rst gains a \"When not to use a UDF\" subsection\n showing the compound-OR predicate that replaces a Python-side UDF\n for disjunctive bucket filters (the Q19 case).\n- aggregations.rst gains a \"Building per-group arrays\" subsection\n covering `array_agg(filter\u003d..., distinct\u003dTrue)` with\n `array_length`/`array_element` for the single-value-per-group\n pattern (the Q21 case).\n- Adds `examples/array-operations.py`, a runnable end-to-end\n walkthrough of the membership and array_agg patterns.\n\nImplements PR 4e of the plan in #1394.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: wire new contributor skills and plan-comparison diagnostic into AGENTS.md\n\n- List the three contributor skills (`check-upstream`,\n `write-dataframe-code`, `audit-skill-md`) under the Skills section so\n agents know what tools they have before starting work.\n- Document the plan-comparison diagnostic workflow (comparing\n `ctx.sql(...).optimized_logical_plan()` against a DataFrame\u0027s\n `optimized_logical_plan()` via `LogicalPlan.__eq__`) for translating\n SQL queries to DataFrame form. Points at the full write-up in the\n `write-dataframe-code` skill rather than duplicating it.\n\n`CLAUDE.md` is a symlink to `AGENTS.md`, so the change lands in both.\n\nImplements PR 4f of the plan in #1394.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: rename aggregations.rst demo df to orders_df to avoid clobbering state\n\nThe \"Building per-group arrays\" block added in the previous commit\nreassigned `df` and `ctx` mid-page, which then broke the\nGrouping Sets examples further down that share the Pokemon `df`\nbinding (`col_type_1` etc. no longer resolved). Rename the demo\nDataFrame to `orders_df` and drop the redundant `ctx \u003d SessionContext()`\nso the shared state from the top of the page stays intact.\n\nVerified with `sphinx-build -W --keep-going` against the full docs\ntree.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: replace raw SKILL.md include with a human-written AI-assistants page\n\nThe previous approach embedded the repo-root `SKILL.md` on the docs\nsite via a myst `{include}`. That file is written for agents -- dense,\nskill-formatted, and not suited to a human browsing the User Guide. It\nalso relied on a fragile `:start-line:` offset to strip YAML\nfrontmatter.\n\nReplace it with `docs/source/ai-coding-assistants.md`, a short\nhuman-readable page that mirrors the README section added in #1503:\nwhat the skill is, how to install it via `npx skills` or a manual\npointer, and what kinds of things it covers. `SKILL.md` stays at the\nrepo root as the single source of truth; agents fetch the raw GitHub\nURL directly.\n\n`llms.txt` is updated to point its Agent Guide entry at\n`raw.githubusercontent.com/.../SKILL.md` and to include the new\nhuman-readable page as a secondary link. The User Guide toctree now\nreferences `ai-coding-assistants` in place of the removed `skill`\nstub.\n\nVerified with `sphinx-build -W --keep-going` against the full docs\ntree.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: drop redundant assistants list in ai-coding-assistants intro\n\nThe introduction and the \"Installing the skill\" section both enumerated\nthe same set of supported assistants. Drop the intro copy; the list\nthat matters is next to `npx skills add`, where it answers \"what does\nthis command actually configure?\"\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: convert ai-coding-assistants page from markdown to rst, shorten title\n\nEvery other page in `docs/source/user-guide` and the top-level\n`docs/source` is written in reStructuredText; the lone `.md` page was\nan inconsistency. Rewrite in rst so the ASF header matches the rest of\nthe tree, cross-references can use `:py:func:` roles if we ever add\nany, and myst is no longer required just to render this one page.\n\nAlso shorten the page title from \"Using DataFusion with AI Coding\nAssistants\" to \"Using AI Coding Assistants\" -- it already sits under\nthe DataFusion user guide so the product name is redundant.\n\nVerified with `sphinx-build -W --keep-going`.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: drop audit-skill-md skill\n\nThe skill as written pushed for every public method to be mentioned\nin `SKILL.md`, which is the wrong goal. `SKILL.md` is a distilled\nagent guide of idiomatic patterns and pitfalls, not an API reference\n-- autoapi-generated docs and module docstrings already provide full\nper-method coverage. An audit pressing for 100% method coverage would\nbloat the skill file into a stale copy of that reference.\n\nThe two checks with actual value (stale mentions in `SKILL.md`, and\ndrift between `functions.__all__` and the categorized function list)\nare small enough to be ad-hoc greps at release time and do not\nwarrant a dedicated skill.\n\nAlso remove references to the skill from `AGENTS.md` and the\n`write-dataframe-code` skill\u0027s \"Related\" section.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: drop write-dataframe-code skill\n\nA separate PR covers the same contributor-facing material (TPC-H\npattern index, plan-comparison workflow, docstring conventions),\nso this skill is redundant. Remove the skill directory and the\ncorresponding references in `AGENTS.md`, including the\nplan-comparison section that pointed at it.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: show Parquet pushdown plan diff in \"When not to use a UDF\"\n\nThe previous version of the section asserted that a UDF predicate\nblocks optimizer rewrites but did not show evidence. Replace the two\n`code-block` examples with an executable walkthrough that writes a\nsmall Parquet file, runs the same filter two ways, and prints the\nphysical plan for each.\n\nThe native-expression plan renders with three annotations on the\n`DataSourceExec` node that the UDF plan does not have:\n\n- `predicate\u003dbrand@1 \u003d A AND qty@2 \u003e\u003d 150` pushed into the scan\n- `pruning_predicate\u003d... brand_min@0 \u003c\u003d A AND ... qty_max@4 \u003e\u003d 150`\n for row-group pruning via Parquet footer min/max stats\n- `required_guarantees\u003d[brand in (A)]` for bloom-filter / dictionary\n skipping\n\nThe UDF form keeps only `predicate\u003dbrand_qty_filter(...)`: the scan\nhas to materialize every row group and call the Python callback.\n\nThe disjunctive-OR rewrite (previously the main example) stays at the\nend as the idiomatic alternative for multi-bucket filters.\n\nVerified with `sphinx-build -W --keep-going`.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: rework \"subsets within a group\" aggregation example\n\nRename the section from \"Building per-group arrays\" to \"Comparing subsets\nwithin a group\" so the heading matches the content. Rewrite the intro to\nlead with the problem (compare full group vs filtered subset), reframe\nthe worked example around partially failed orders, and replace the\ntrailing bullet list with a one-line walkthrough of the result.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: clarify \"When not to use a UDF\" intro\n\nRewrite the opening of the section to make three things clearer: the\ncontrast is with native DataFusion expressions (not Python in general),\nsome predicates genuinely feel easier to write as a Python loop and that\ntension is worth acknowledging, and predicate pushdown is a table-provider\nmechanism rather than a Parquet-only feature. Parquet stays as the\nconcrete demo.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: move ai-coding-assistants under user-guide/\n\nThe page was sitting at the top level of docs/source/ while every other\npage in the USER GUIDE toctree lives under docs/source/user-guide/.\nMove the file, update the toctree entry, and update the absolute URL\nin llms.txt to match the new path.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: replace AGENTS.md skill list with discovery instructions\n\nA static skill list in AGENTS.md goes stale as new skills are added\n(it already missed the make-pythonic skill that was merged separately).\nReplace the enumerated list with a pointer telling agents to list\n.ai/skills/ and read each SKILL.md frontmatter, so the catalog never\nhas to be hand-maintained.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: fix broken llms.txt link and stale otherwise xref\n\n- ai-coding-assistants.rst: use absolute https://datafusion.apache.org/python/llms.txt URL; the relative `llms.txt` resolved to /python/user-guide/llms.txt and 404\u0027d because html_extra_path publishes the file at the site root.\n- expressions.rst: drop the broken `:py:meth:~datafusion.expr.Expr.otherwise` xref (otherwise lives on CaseBuilder, not Expr) and spell the recommended replacement as `f.when(f.in_list(...), value).otherwise(default)`.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: update SKILL.md path after move to skills/datafusion_python/\n\nUpstream #1519 moved the root `SKILL.md` to `skills/datafusion_python/SKILL.md`\nso that consumers can install the skill without cloning the whole repo. Update\nall repo-internal links and external GitHub URLs in the docs site, README,\nAGENTS.md, and the package docstring to point at the new location.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "c657dad97349c1113e843e3e15bb41f865e65a97", "tree": "b2ee2e39947e7431734591fc11fdf8ad8a4049a9", "parents": [ "e0284c6e788b6fc893495ed929b9badef1cf925c" ], "author": { "name": "Nick", "email": "24689722+ntjohnson1@users.noreply.github.com", "time": "Wed Apr 29 07:35:21 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Apr 29 07:35:21 2026 -0400" }, "message": "Move public skills to a directory to avoid downloading the whole repo (#1519)\n\n* Move to skill directory\n\n* Avoid moved skill with test" }, { "commit": "e0284c6e788b6fc893495ed929b9badef1cf925c", "tree": "fa186f6dc053173f57633a247c3e59b6037fe262", "parents": [ "03577163a057f791b19f30ce5130464a4a1c78a4" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri Apr 24 13:09:24 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Apr 24 13:09:24 2026 -0400" }, "message": "feat: add AI skill to find and improve the Pythonic interface to functions (#1484)\n\n* feat: accept native Python types in function arguments instead of requiring lit()\n\nUpdate 47 functions in functions.py to accept native Python types (int, float,\nstr) for arguments that are contextually literals, eliminating verbose lit()\nwrapping. For example, users can now write split_part(col(\"a\"), \",\", 2) instead\nof split_part(col(\"a\"), lit(\",\"), lit(2)). All changes are backward compatible.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: update alias function signatures to match pythonic primary functions\n\nUpdate instr and position (aliases of strpos) to accept Expr | str for\nthe substring parameter, matching the updated primary function signature.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: update make-pythonic skill to require alias type hint updates\n\nAlias functions that delegate to a primary function must have their type\nhints updated to match, even though coercion logic is only added to the\nprimary. Added a new Step 3 to the implementation workflow for this.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: address review feedback on pythonic skill and function signatures\n\nUpdate SKILL.md to prevent three classes of issues: clarify that float\nalready accepts int per PEP 484 (avoiding redundant int | float that\nfails ruff PYI041), add backward-compat rule for Category B so existing\nExpr params aren\u0027t removed, and add guidance for inline coercion with\nmany optional nullable params instead of local helpers.\n\nReplace regexp_instr\u0027s _to_raw() helper with inline coercion matching\nthe pattern used throughout the rest of the file.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: add coerce_to_expr helpers and replace inline coercion patterns\n\nIntroduce coerce_to_expr() and coerce_to_expr_or_none() in expr.py as the\ncomplement to ensure_expr() — where ensure_expr rejects non-Expr values,\nthese helpers wrap them via Expr.literal(). Replaces ~60 inline isinstance\nchecks in functions.py with single-line helper calls, and updates the\nmake-pythonic skill to document the new pattern.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: add aggregate function literal detection to make-pythonic skill\n\nAdd Technique 1a to detect literal-only arguments in aggregate functions.\nUnlike scalar UDFs which enforce literals in invoke_with_args(), aggregate\nfunctions enforce them in accumulator() via get_scalar_value(),\nvalidate_percentile_expr(), or downcast_ref::\u003cLiteral\u003e(). Without this\ntechnique, the skill would incorrectly classify arguments like\napprox_percentile_cont\u0027s percentile as Category A (Expr | float) when they\nshould be Category B (float only). Updates the decision flow to branch on\nscalar vs aggregate before checking for literal enforcement.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: add window function literal detection to make-pythonic skill\n\nAdd Technique 1b to detect literal-only arguments in window functions.\nWindow functions enforce literals in partition_evaluator() via\nget_scalar_value_from_args() / downcast_ref::\u003cLiteral\u003e(), not in\ninvoke_with_args() (scalar) or accumulator() (aggregate). Updates the\ndecision flow to branch on scalar vs aggregate vs window.\n\nKnown window functions with literal-only arguments: ntile (n), lead/lag\n(offset, default_value), nth_value (n).\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: use explicit None checks, widen numeric type hints, and add tests\n\nReplace 7 fragile truthiness checks (x.expr if x else None) with\nexplicit is not None checks to prevent silent None when zero-valued\nliterals are passed. Widen log/power/pow type hints to Expr | int | float\nwith noqa: PYI041 for clarity. Add unit tests for coerce_to_expr helpers\nand integration tests for pythonic calling conventions.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* chore: suppress FBT003 in tests and remove redundant noqa comments\n\nAdd FBT003 (boolean positional value) to the per-file-ignores for\npython/tests/* in pyproject.toml, and remove the 6 now-redundant\ninline noqa: FBT003 comments across test_expr.py and test_context.py.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: replace static function lists with discovery instructions in skill\n\nReplace hardcoded \"Known aggregate/window functions with literal-only\narguments\" lists with instructions to discover them dynamically by\nsearching the upstream crate source. Keeps a few examples as validation\nanchors so the agent knows its search is working correctly.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: make interrupt test reliable on Python 3.11\n\nPyThreadState_SetAsyncExc only delivers exceptions when the thread is\nexecuting Python bytecode, not while in native (Rust/C) code. The\nprevious test had two issues causing flakiness on Python 3.11:\n\n1. The interrupt fired before df.collect() entered the UDF, while the\n thread was still in native code where async exceptions are ignored.\n2. time.sleep(2.0) is a single C call where async exceptions are not\n checked — they\u0027re only checked between bytecode instructions.\n\nFix by adding a threading.Event so the interrupt waits until the UDF is\nactually executing Python code, and by sleeping in small increments so\nthe eval loop has opportunities to check for pending exceptions.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "03577163a057f791b19f30ce5130464a4a1c78a4", "tree": "2b02d946a60edddb9df94a0a9438b3e734acaf39", "parents": [ "c8bb9f7d3876de97141d204740a6b99d5facd10f" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri Apr 24 11:47:06 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Apr 24 11:47:06 2026 -0400" }, "message": "tpch examples: rewrite queries idiomatically and embed reference SQL (#1504)\n\n* tpch examples: add reference SQL to each query, fix Q20\n\n- Append the canonical TPC-H reference SQL (from benchmarks/tpch/queries/)\n to each q01..q22 module docstring so readers can compare the DataFrame\n translation against the SQL at a glance.\n- Fix Q20: `df \u003d df.filter(col(\"ps_availqty\") \u003e lit(0.5) * col(\"total_sold\"))`\n was missing the assignment so the filter was dropped from the pipeline.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* tpch examples: rewrite non-idiomatic queries in idiomatic DataFrame form\n\nRewrite the seven TPC-H example queries that did not demonstrate the\nidiomatic DataFrame pattern. The remaining queries (Q02/Q11/Q15/Q17/Q22,\nwhich use window functions in place of correlated subqueries) already are\nidiomatic and are left unchanged.\n\n- Q04: replace `.aggregate([col(\"l_orderkey\")], [])` with\n `.select(\"l_orderkey\").distinct()`, which is the natural way to express\n \"reduce to one row per order\" on a DataFrame.\n- Q07: remove the CASE-as-filter on `n_name` and use\n `F.in_list(col(\"n_name\"), [nation_1, nation_2])` instead. Drops a\n comment block that admitted the filter form was simpler.\n- Q08: rewrite the switched CASE `F.case(...).when(lit(False), ...)` as a\n searched `F.when(col(...).is_not_null(), ...).otherwise(...)`. That\n mirrors the reference SQL\u0027s `case when ... then ... else 0 end` shape.\n- Q12: replace `array_position(make_array(...), col)` with\n `F.in_list(col(\"l_shipmode\"), [...])`. Same semantics, without routing\n through array construction / array search.\n- Q19: remove the pyarrow UDF that re-implemented a disjunctive predicate\n in Python. Build the same predicate in DataFusion by OR-combining one\n `in_list` + range-filter expression per brand. Keeps the per-brand\n constants in the existing `items_of_interest` dict.\n- Q20: use `F.starts_with` instead of an explicit substring slice. Replace\n the inner-join + `select(...).distinct()` tail with a semi join against\n a precomputed set of excess-quantity suppliers so the supplier columns\n are preserved without deduplication after the fact.\n- Q21: replace the `array_agg` / `array_length` / `array_element` pipeline\n with two semi joins. One semi join keeps orders with more than one\n distinct supplier (stand-in for the reference SQL\u0027s `exists` subquery),\n the other keeps orders with exactly one late supplier (stand-in for the\n `not exists` subquery).\n\nAll 22 answer-file comparisons and 22 plan-comparison diagnostics still\npass (`pytest examples/tpch/_tests.py`: 44 passed).\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* tpch examples: align reference SQL constants with DataFrame queries\n\nThe reference SQL embedded in each q01..q22 module docstring was carried\nover verbatim from ``benchmarks/tpch/queries/`` and uses a different set\nof TPC-H substitution parameters than the DataFrame examples\n(answer-file-validated at scale factor 1). Update each reference SQL to\nuse the substitution parameters the DataFrame uses, so both expressions\ndescribe the same query and would produce the same results against the\nsame data.\n\nConstants aligned:\n\n- Q01: ``90 days`` cutoff (DataFrame ``DAYS_BEFORE_FINAL \u003d 90``).\n- Q02: ``p_size \u003d 15``, ``p_type like \u0027%BRASS\u0027``, ``r_name \u003d \u0027EUROPE\u0027``.\n- Q04: base date ``1993-07-01`` (``3 month`` interval preserved per the\n \"quarter of a year\" wording).\n- Q05: ``r_name \u003d \u0027ASIA\u0027``.\n- Q06: ``l_discount between 0.06 - 0.01 and 0.06 + 0.01``.\n- Q07: nations ``\u0027FRANCE\u0027`` / ``\u0027GERMANY\u0027``.\n- Q08: ``r_name \u003d \u0027AMERICA\u0027``, ``p_type \u003d \u0027ECONOMY ANODIZED STEEL\u0027``,\n inner-case ``nation \u003d \u0027BRAZIL\u0027``.\n- Q09: ``p_name like \u0027%green%\u0027``.\n- Q10: base date ``1993-10-01`` (``3 month`` interval preserved).\n- Q11: ``n_name \u003d \u0027GERMANY\u0027``.\n- Q12: ship modes ``(\u0027MAIL\u0027, \u0027SHIP\u0027)``, base date ``1994-01-01``.\n- Q13: ``o_comment not like \u0027%special%requests%\u0027``.\n- Q14: base date ``1995-09-01``.\n- Q15: base date ``1996-01-01``.\n- Q16: ``p_brand \u003c\u003e \u0027Brand#45\u0027``, ``p_type not like \u0027MEDIUM POLISHED%\u0027``,\n sizes ``(49, 14, 23, 45, 19, 3, 36, 9)``.\n- Q17: ``p_brand \u003d \u0027Brand#23\u0027``, ``p_container \u003d \u0027MED BOX\u0027``.\n- Q18: ``sum(l_quantity) \u003e 300``.\n- Q19: brands ``Brand#12`` / ``Brand#23`` / ``Brand#34`` with the matching\n minimum quantities (1, 10, 20).\n- Q20: ``p_name like \u0027forest%\u0027``, base date ``1994-01-01``,\n ``n_name \u003d \u0027CANADA\u0027``.\n- Q21: ``n_name \u003d \u0027SAUDI ARABIA\u0027``.\n- Q22: country codes ``(\u002713\u0027, \u002731\u0027, \u002723\u0027, \u002729\u0027, \u002730\u0027, \u002718\u0027, \u002717\u0027)``.\n\nInterval units (month / year) are preserved where the problem-statement\ntext reads \"given quarter\", \"given year\", \"given month\". Q01 keeps the\nliteral \"days\" unit because the TPC-H problem statement itself describes\nthe cutoff in days.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* tpch examples: apply SKILL.md idioms across all 22 queries\n\nSweep every q01..q22 example for idiomatic DataFrame style as described in\nthe repo-root SKILL.md:\n\n- ``col(\"x\") \u003d\u003d \"s\"`` in place of ``col(\"x\") \u003d\u003d lit(\"s\")`` on comparison\n right-hand sides (auto-wrap applies).\n- Plain-name strings in ``select``/``aggregate``/``sort`` group/sort key\n lists when the key is a bare column.\n- Drop redundant ``how\u003d\"inner\"`` and single-element ``left_on``/``right_on``\n list wrapping on equi-joins.\n- Collapse chained ``.filter(a).filter(b)`` runs into ``.filter(a, b)``\n and chained ``.with_column`` runs into ``.with_columns(a\u003d..., b\u003d...)``.\n- ``df.sort_by(...)`` or plain-name ``df.sort(...)`` when no null-placement\n override is needed.\n- ``F.count_star()`` in place of ``F.count(col(\"x\"))`` whenever the SQL\n reads ``count(*)``.\n- ``F.starts_with(col, lit(prefix))`` and ``~F.starts_with(...)`` in place\n of substring-prefix equality/inequality tricks.\n- ``F.in_list(col, [lit(...)])`` in place of ``~F.array_position(...).\n is_null()`` and in place of disjunctions of equality comparisons.\n- Searched ``F.when(cond, x).otherwise(y)`` in place of switched\n ``F.case(bool_expr).when(lit(True/False), x).end()`` forms.\n- Semi-joins as the DataFrame form of ``EXISTS`` (Q04); anti-joins as\n ``NOT EXISTS`` (Q22 was already using this idiom).\n- Whole-frame window aggregates as the DataFrame stand-in for a SQL\n scalar subquery (Q11/Q15/Q17/Q22).\n\nIndividual query fixes of note:\n\n- Q16 — add the secondary sort keys (``p_brand``, ``p_type``, ``p_size``)\n that the TPC-H spec requires but the original DataFrame omitted.\n- Q22 — drop a stray ``df.show()`` mid-pipeline; replace the 0-based\n substring slice with ``F.left(col(\"c_phone\"), lit(2))``.\n- Q14 — rewrite the promo/non-promo factor split as a searched CASE inside\n ``F.sum(...)`` so the DataFrame expression matches the reference SQL\n shape exactly.\n\nAll 22 answer-file comparisons still pass at scale factor 1.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* tpch examples: more idiomatic aggregate FILTER, string funcs, date handling\n\nAdditional sweep of the TPC-H DataFrame examples informed by comparing\nagainst a fresh set of SKILL.md-only generations under\n``examples/tpch/agentic_queries/``:\n\n- Q02: ``F.ends_with(col(\"p_type\"), lit(TYPE_OF_INTEREST))`` in place of\n ``F.strpos(col, lit) \u003e 0``. The reference SQL is ``p_type like \u0027%BRASS\u0027``,\n which is an ends_with check, not contains. ``F.strpos \u003e 0`` returned the\n correct rows on TPC-H data by coincidence but is semantically wrong.\n- Q09: ``F.contains(col(\"p_name\"), lit(part_color))`` in place of\n ``F.strpos(col, lit) \u003e 0``. The SQL is ``p_name like \u0027%green%\u0027``.\n- Q08, Q12, Q14: use the ``filter`` keyword on ``F.sum`` / ``F.count`` —\n the DataFrame form of SQL ``sum(...) FILTER (WHERE ...)`` — instead of\n wrapping the aggregate input in ``F.when(cond, x).otherwise(0)``. Q08\n also reorganises to inner-join the supplier\u0027s nation onto the regional\n sales, which removes the previous left-join + ``F.when(is_not_null, ...)``\n dance.\n- Q15: compute the grand maximum revenue as a separate scalar aggregate\n and ``join_on(...)`` on equality, instead of the whole-frame window\n ``F.max`` + filter shape. Simpler plan, same result.\n- Q16: ``F.regexp_like(col, pattern)`` in place of\n ``F.regexp_match(col, pattern).is_not_null()``.\n- Q04, Q05, Q06, Q07, Q08, Q10, Q12, Q14, Q15, Q20: store both the start\n and the end of the date window as plain ``datetime.date`` objects and\n compare with ``lit(end_date)``, instead of carrying the start date +\n ``pa.month_day_nano_interval`` and adding them at query-build time.\n Drops unused ``pyarrow`` imports from the files that no longer need\n Arrow scalars.\n\nAll 22 answer-file comparisons still pass at scale factor 1.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "c8bb9f7d3876de97141d204740a6b99d5facd10f", "tree": "e8b2602165c4dc1e7208d56b3fc45f5d6332573e", "parents": [ "8741d30cd812e4668f3f9187b56f12ce2de0d6e7" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri Apr 24 07:57:11 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Apr 24 07:57:11 2026 -0400" }, "message": "docs: add README section for AI coding assistants (#1503)\n\nPoints users to the repo-root SKILL.md via the npx skills registry or a\nmanual AGENTS.md / CLAUDE.md pointer. Implements PR 1c of the plan in #1394.\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "8741d30cd812e4668f3f9187b56f12ce2de0d6e7", "tree": "d7fa7ec580a8ce057c76934d8a7ef01f3e710ce2", "parents": [ "8a5d783c7e418bfbbd95e48a2d9cacafea6162c7" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Thu Apr 23 22:01:01 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Apr 23 22:01:01 2026 -0400" }, "message": "docs: enrich module docstrings and add doctest examples (#1498)\n\n* Enrich module docstrings and add doctest examples\n\nExpands the module docstrings for `functions.py`, `dataframe.py`,\n`expr.py`, and `context.py` so each module opens with a concept summary,\ncross-references to related APIs, and a small executable example.\n\nAdds doctest examples to the high-traffic `DataFrame` methods that\npreviously lacked them: `select`, `aggregate`, `sort`, `limit`, `join`,\nand `union`. Optional parameters are demonstrated with keyword syntax,\nand examples reuse the same input data across variants so the effect of\neach option is easy to see.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Use distinct group sums in aggregate docstring example\n\nChange the score data from [1, 2, 3] to [1, 2, 5] so the grouped\nresult produces [3, 5] instead of [3, 3], removing ambiguity about\nwhich total belongs to which team.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Align module-docstring examples with SKILL.md idioms\n\nDrop the redundant lit() in the dataframe.py module-docstring filter\nexample and use a plain string group key in the aggregate() doctest, so\nboth examples model the style SKILL.md recommends. Also document the\nsort(\"a\") string form and sort_by() shortcut in SKILL.md\u0027s sorting\nsection.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "8a5d783c7e418bfbbd95e48a2d9cacafea6162c7", "tree": "08f42ac0dedff8563163d297b8c6d13c95d97eba", "parents": [ "40309978c920bd123a4c7b764a2ddfdb97758607" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Thu Apr 23 19:05:06 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Apr 23 19:05:06 2026 -0400" }, "message": "Skills require the header to be the first thing in the file which conflicts with the RAT check. Make an exception for this file. (#1501)" }, { "commit": "40309978c920bd123a4c7b764a2ddfdb97758607", "tree": "c70a137ffbb797b52787dac6f858e97af8de001f", "parents": [ "60d8b5dbb5e409cd9ce7692972420e955b8a802e" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Thu Apr 23 18:28:55 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Apr 23 18:28:55 2026 -0400" }, "message": "Add SKILL.md and enrich package docstring (#1497)\n\n* Add AGENTS.md and enrich __init__.py module docstring\n\nAdd python/datafusion/AGENTS.md as a comprehensive DataFrame API guide\nfor AI agents and users. It ships with pip automatically (Maturin includes\neverything under python-source \u003d \"python\"). Covers core abstractions,\nimport conventions, data loading, all DataFrame operations, expression\nbuilding, a SQL-to-DataFrame reference table, common pitfalls, idiomatic\npatterns, and a categorized function index.\n\nEnrich the __init__.py module docstring from 2 lines to a full overview\nwith core abstractions, a quick-start example, and a pointer to AGENTS.md.\n\nCloses #1394 (PR 1a)\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Clarify audience of root vs package AGENTS.md\n\nThe root AGENTS.md (symlinked as CLAUDE.md) is for contributors working\non the project. Add a pointer to python/datafusion/AGENTS.md which is\nthe user-facing DataFrame API guide shipped with the package. Also add\nthe Apache license header to the package AGENTS.md.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add PR template and pre-commit check guidance to AGENTS.md\n\nDocument that all PRs must follow .github/pull_request_template.md and\nthat pre-commit hooks must pass before committing. List all configured\nhooks (actionlint, ruff, ruff-format, cargo fmt, cargo clippy, codespell,\nuv-lock) and the command to run them manually.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Remove duplicated hook list from AGENTS.md\n\nLet the hooks be discoverable from .pre-commit-config.yaml rather than\nmaintaining a separate list that can drift.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Fix AGENTS.md: Arrow C Data Interface, aggregate filter, fluent example\n\n- Clarify that DataFusion works with any Arrow C Data Interface\n implementation, not just PyArrow.\n- Show the filter keyword argument on aggregate functions (the idiomatic\n HAVING equivalent) instead of the post-aggregate .filter() pattern.\n- Update the SQL reference table to show FILTER (WHERE ...) syntax.\n- Remove the now-incorrect \"Aggregate then filter for HAVING\" pitfall.\n- Add .collect() to the fluent chaining example so the result is clearly\n materialized.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Update agents file after working through the first tpc-h query using only the text description\n\n* Add feedback from working through each of the TPC-H queries\n\n* Address Copilot review feedback on AGENTS.md\n\n- Wrap CASE/WHEN method-chain examples in parentheses and assign to a\n variable so they are valid Python as shown (Copilot #1, #2).\n- Fix INTERSECT/EXCEPT mapping: the default distinct\u003dFalse corresponds to\n INTERSECT ALL / EXCEPT ALL, not the distinct forms. Updated both the\n Set Operations section and the SQL reference table to show both the\n ALL and distinct variants (Copilot #4).\n- Change write_parquet / write_csv / write_json examples to file-style\n paths (output.parquet, etc.) to match the convention used in existing\n tests and examples. Note that a directory path is also valid for\n partitioned output (Copilot #5).\n\nVerified INTERSECT/EXCEPT semantics with a script:\n df1.intersect(df2) -\u003e [1, 1, 2] (\u003d INTERSECT ALL)\n df1.intersect(df2, distinct\u003dTrue) -\u003e [1, 2] (\u003d INTERSECT)\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Use short-form comparisons in AGENTS.md examples\n\nDrop lit() on the RHS of comparison operators since Expr auto-wraps raw\nPython values, matching the style the guide recommends (Copilot #3, #6).\n\nUpdates examples in the Aggregation, CASE/WHEN, SQL reference table,\nCommon Pitfalls, Fluent Chaining, and Variables-as-CTEs sections, plus\nthe __init__.py quick-start snippet. Prose explanations of the rule\n(which cite the long form as the thing to avoid) are left unchanged.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Move user guide from python/datafusion/AGENTS.md to SKILL.md\n\nThe in-wheel AGENTS.md was not a real distribution channel -- no shipping\nagent walks site-packages for AGENTS.md files. Moving to SKILL.md at the\nrepo root, with YAML frontmatter, lets the skill ecosystems (npx skills,\nClaude Code plugin marketplaces, community aggregators) discover it.\n\nUpdate the pointers in the contributor AGENTS.md and the __init__.py\nmodule docstring accordingly. The docstring now references the GitHub\nURL since the file no longer ships with the wheel.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Address review feedback: doctest, streaming, date/timestamp\n\n- Convert the __init__.py quick-start block to doctest format so it is\n picked up by `pytest --doctest-modules` (already the project default),\n preventing silent rot.\n- Extract streaming into its own SKILL.md subsection with guidance on\n when to prefer execute_stream() over collect(), sync and async\n iteration, and execute_stream_partitioned() for per-partition streams.\n- Generalize the date-arithmetic rule from Date32 to both Date32 and\n Date64 (both reject Duration at any precision, both accept\n month_day_nano_interval), and note that Timestamp columns differ and\n do accept Duration.\n- Document the PyArrow-inherited type mapping returned by\n to_pydict()/to_pylist(), including the nanosecond fallback to\n pandas.Timestamp / pandas.Timedelta and the to_pandas() footgun where\n date columns come back as an object dtype.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Distinguish user guide from agent reference in module docstring\n\nThe docstring pointed readers at SKILL.md as a \"comprehensive guide,\" but\nSKILL.md is written in a dense, skill-oriented format for agents — humans\nare better served by the online user guide. Put the online docs first as\nthe primary reference and label the SKILL.md link as the agent reference.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "60d8b5dbb5e409cd9ce7692972420e955b8a802e", "tree": "f2364ca93f408363a0929fe25e103226f63ac1d1", "parents": [ "2715a32e939d17222c18e8adacf85ee45da464b9" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Tue Apr 14 03:31:01 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Apr 14 03:31:01 2026 -0400" }, "message": "Fix error on show() with an explain plan (#1492)" }, { "commit": "2715a32e939d17222c18e8adacf85ee45da464b9", "tree": "91262ce078f88250bfbdd6424445f043859fc2ca", "parents": [ "398980d1edbb8ad6d9744236f2dfe0c6ab4b4665" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Tue Apr 14 03:27:00 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Apr 14 03:27:00 2026 -0400" }, "message": "chore: update release documentation (#1494)\n\n* Update release documentation\n\n* Minor change to workflow because release start at 1" }, { "commit": "398980d1edbb8ad6d9744236f2dfe0c6ab4b4665", "tree": "17cd377ee141d16fdfee93979a0db7286fc6921f", "parents": [ "8a7efead43cff8dc7515e27e53da7545100e25a7" ], "author": { "name": "Zeel Desai", "email": "72783325+zeel2104@users.noreply.github.com", "time": "Mon Apr 13 09:24:56 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Apr 13 09:24:56 2026 -0400" }, "message": "Support None comparisons for null expressions (#1489)\n\n* Support None comparisons for null expressions\n\n* Fold None comparison coverage into relational expr test" }, { "commit": "8a7efead43cff8dc7515e27e53da7545100e25a7", "tree": "64ed2a5076a48e7f737e18b046d3cd43ac04aeb6", "parents": [ "00b24572c98a257f06ff026a90c07634a86204d4" ], "author": { "name": "Shreyesh", "email": "shreyesh.arangath@gmail.com", "time": "Mon Apr 13 03:34:35 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Apr 13 06:34:35 2026 -0400" }, "message": "Add Python bindings for accessing ExecutionMetrics (#1381)\n\n* feat: add Python bindings for accessing ExecutionMetrics\n\n* test: imporve tests\n\n* first round of reviews\n\n* plan caching\n\n* address some concerns\n\n* merge and address comments\n\n* fix Ci issues\n\n* attempt to fix lint\n\n* fix build\n\n* fix docstring\n\n* address some more comments\n\n---------\n\nCo-authored-by: ShreyeshArangath \u003cshryeyesh.arangath@gmail.com\u003e" }, { "commit": "00b24572c98a257f06ff026a90c07634a86204d4", "tree": "6e4c531777ff78afe3c9c507d2397517ec59a8b6", "parents": [ "1be838bb47f04bcf4d1a0f65e3e6958aa9366f3f" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Mon Apr 13 06:33:29 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Apr 13 06:33:29 2026 -0400" }, "message": "ci: disable symbol export on Windows verification (#1486)\n\n* Set rust flags on windows release verification\n\n* Forward flag to linker\n\n* Switch to msvc rust toolchain\n\n* Revert \"Switch to msvc rust toolchain\"\n\nThis reverts commit 9879fc7dbe066098445b9600087e665435b58f8a." }, { "commit": "1be838bb47f04bcf4d1a0f65e3e6958aa9366f3f", "tree": "947f2e392faa9002020930092d7d1cee9dde83a2", "parents": [ "3585c11eed778810e3317c56c2c25a8cdc29be5b" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Sun Apr 12 21:24:39 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Apr 12 21:24:39 2026 -0400" }, "message": "Release 53.0.0 (#1491)\n\n* Update version number and changelog\n\n* minor: set version number on dependency to publish to crates.io\n\n* taplo fmt" }, { "commit": "3585c11eed778810e3317c56c2c25a8cdc29be5b", "tree": "3b4c9265fa211bdd0cdfbe8f2dbe2d345bdcf83a", "parents": [ "ecd14c10aff67169f2bfe1b7f86ff07621088dd0" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Thu Apr 09 07:38:59 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Apr 09 07:38:59 2026 -0400" }, "message": "minor: remove deprecated interfaces (#1481)\n\n* udf module has been deprecated since DF47. html_formatter module has been deprecated since DF48.\n\n* database has been deprecated since DF48\n\n* select_columns has been deprecated since DF43\n\n* unnest_column has been deprecated since DF42\n\n* display_name has been deprecated since DF42\n\n* window() has been deprecated since DF50\n\n* serde functions have been deprecated since DF42\n\n* from_arrow_table and tables have been deprecated since DF42\n\n* RuntimeConfig has been deprecated since DF44\n\n* Update user documentation to remove deprecated function\n\n* update tpch examples for latest function uses\n\n* Remove unnecessary options in example\n\n* update rendering for the most recent dataframe_formatter instead of the deprecated html_formatter" }, { "commit": "ecd14c10aff67169f2bfe1b7f86ff07621088dd0", "tree": "03d3e29a3a4a7933cff0ffe591c9fa042b9c48e2", "parents": [ "aa3b1948c3a49d14395093287a6e93354229c539" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Wed Apr 08 11:11:48 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Apr 08 11:11:48 2026 -0400" }, "message": "Add missing SessionContext utility methods (#1475)\n\n* Add missing SessionContext utility methods\n\nExpose upstream DataFusion v53 utility methods: session_start_time,\nenable_ident_normalization, parse_sql_expr, execute_logical_plan,\nrefresh_catalogs, remove_optimizer_rule, and table_provider. The\nadd_optimizer_rule and add_analyzer_rule methods are omitted as the\nOptimizerRule and AnalyzerRule traits are not yet exposed to Python.\nCloses #1459.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Raise KeyError from table_provider for consistency with table()\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add docstring examples for new SessionContext utility methods\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* update docstring\n\n* Address PR review feedback for SessionContext utility methods\n\n- Improve docstring examples to show actual output instead of asserts\n- Use doctest +SKIP for non-deterministic session_start_time output\n- Fix table_provider error mapping: outer async error is now RuntimeError\n- Strengthen tests: validate RFC 3339 with fromisoformat, test both\n optimizer rule removal paths, exact string match for parse_sql_expr,\n verify enable_ident_normalization with dynamic state change\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Fix test_session_start_time failure on Python 3.10\n\ndatetime.fromisoformat() only supports up to 6 fractional-second\ndigits (microseconds) on Python 3.10. Truncate nanosecond precision\nbefore parsing.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "aa3b1948c3a49d14395093287a6e93354229c539", "tree": "8e2124d5f4029861b1fb54297f257294d742718b", "parents": [ "46f9ab8fcad03913234ce29e5075644c1ecdb9b7" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Wed Apr 08 09:22:28 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Apr 08 09:22:28 2026 -0400" }, "message": "Add missing registration methods (#1474)\n\n* Add missing SessionContext read/register methods for Arrow IPC and batches\n\nAdd read_arrow, read_empty, register_arrow, and register_batch methods to\nSessionContext, exposing upstream DataFusion v53 functionality. The write_*\nmethods and read_batch/read_batches are already covered by DataFrame.write_*\nand SessionContext.from_arrow respectively. Closes #1458.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Remove redundant read_empty Rust binding, make Python read_empty an alias for empty_table\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add pathlib.Path and empty batch tests for Arrow IPC and register_batch\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Make test_read_empty more robust with length and num_rows checks\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add examples to docstrings for new register/read methods\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Empty table actually returns record batch of length one but there are no columns\n\n* Add optional argument examples to register_arrow and read_arrow docstrings\n\nDemonstrate schema\u003d and file_extension\u003d keyword arguments in the\ndocstring examples for register_arrow and read_arrow, following project\nguidelines for optional parameter documentation.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Simplify read_empty docstring to use alias pattern\n\nFollow the same See Also alias convention used in functions.py since\nread_empty is a simple alias for empty_table.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Remove shared ctx from doctest namespace, use inline SessionContext\n\nAvoid shared SessionContext state across doctests by having each\ndocstring example create its own ctx instance, matching the pattern\nused throughout the rest of the codebase.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Remove redundant import pyarrow as pa from docstrings\n\nThe pa alias is already provided by the doctest namespace in\nconftest.py, so inline imports are unnecessary.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "46f9ab8fcad03913234ce29e5075644c1ecdb9b7", "tree": "008f7dcfb9c5d7ed35fb5e3bbfdab371eb6aa865", "parents": [ "52932128d353e417ddae2c5ff3f14135cb806f7e" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Tue Apr 07 15:03:38 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Apr 07 15:03:38 2026 -0400" }, "message": "Add missing deregister methods to SessionContext (#1473)\n\n* Add deregister methods to SessionContext for UDFs and object stores\n\nExpose upstream DataFusion deregister methods (deregister_udf, deregister_udaf,\nderegister_udwf, deregister_udtf, deregister_object_store) in both the Rust\nPyO3 bindings and Python wrappers, closing the gap identified in #1457.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Fix deregister tests to expect ValueError instead of RuntimeError\n\nDataFusion raises ValueError for planning errors when a deregistered\nfunction is used in a query.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Replace .unwrap() with proper error propagation in object store methods\n\nUrl::parse() can fail on invalid input. Use .map_err() to convert\nthe error into a Python exception instead of panicking.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Minor move of import statement\n\n---------\n\nCo-authored-by: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "52932128d353e417ddae2c5ff3f14135cb806f7e", "tree": "b88ea32d565a6b1708831f95a42034c7028a50e4", "parents": [ "898d73de20346bba7241907bb18cba47da53e9a9" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Tue Apr 07 14:58:09 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Apr 07 14:58:09 2026 -0400" }, "message": "Add missing Dataframe functions (#1472)\n\n* Add missing DataFrame methods for set operations and query\n\nExpose upstream DataFusion DataFrame methods that were not yet\navailable in the Python API. Closes #1455.\n\nSet operations:\n- except_distinct: set difference with deduplication\n- intersect_distinct: set intersection with deduplication\n- union_by_name: union matching columns by name instead of position\n- union_by_name_distinct: union by name with deduplication\n\nQuery:\n- distinct_on: deduplicate rows based on specific columns\n- sort_by: sort by expressions with ascending order and nulls last\n\nNote: show_limit is already covered by the existing show(num) method.\nexplain_with_options and with_param_values are deferred as they require\nexposing additional types (ExplainOption, ParamValues).\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add ExplainFormat enum and format option to DataFrame.explain()\n\nExtend the existing explain() method with an optional format parameter\ninstead of adding a separate explain_with_options() method. This keeps\nthe API simple while exposing all upstream ExplainOption functionality.\n\nAvailable formats: indent (default), tree, pgjson, graphviz.\n\nThe ExplainFormat enum is exported from the top-level datafusion module.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add DataFrame.window() and unnest recursion options\n\nExpose remaining DataFrame methods from upstream DataFusion.\nCloses #1456.\n\n- window(*exprs): apply window function expressions and append results\n as new columns\n- unnest_column/unnest_columns: add optional recursions parameter for\n controlling unnest depth via (input_column, output_column, depth)\n tuples\n\nNote: drop_columns is already exposed as the existing drop() method.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Update docstring\n\nCo-authored-by: Copilot \u003c175728472+Copilot@users.noreply.github.com\u003e\n\n* Improve docstrings and test robustness for new DataFrame methods\n\nClarify except_distinct/intersect_distinct docstrings, add deterministic\nsort to test_window, add sort_by ascending verification test, and add\nsmoke tests for PGJSON and GRAPHVIZ explain formats.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Consolidate new DataFrame tests into parametrized tests\n\nCombine set operation tests (except_distinct, intersect_distinct,\nunion_by_name, union_by_name_distinct) into a single parametrized\ntest_set_operations_distinct. Merge sort_by tests and convert\nexplain format tests to parametrized form.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add doctest examples to new DataFrame method docstrings\n\nAdd \u003e\u003e\u003e style usage examples for window, explain, except_distinct,\nintersect_distinct, union_by_name, union_by_name_distinct, distinct_on,\nsort_by, and unnest_columns to match existing docstring conventions.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Improve error messages, tests, and API hygiene from PR review\n\n- Provide actionable error message for invalid explain format strings\n- Remove recursions param from deprecated unnest_column (use unnest_columns)\n- Add null-handling test case for sort_by to verify nulls-last behavior\n- Add format-specific assertions to explain tests (TREE, PGJSON, GRAPHVIZ)\n- Add deep recursion test for unnest_columns with depth \u003e 1\n- Add multi-expression window test to verify variadic *exprs\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Consolidate window and unnest tests into parametrized tests\n\nCombine test_window and test_window_multiple_expressions into a single\nparametrized test. Merge unnest recursion tests into one parametrized\ntest covering basic, explicit depth 1, and deep recursion cases.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Address PR review feedback for DataFrame operations\n\n- Use upstream parse error for explain format instead of hardcoded options\n- Fix sort_by to use column name resolution consistent with sort()\n- Use ExplainFormat enum members directly in tests instead of string lookup\n- Merge union_by_name_distinct into union_by_name(distinct\u003dFalse) for a\n more Pythonic API\n- Update check-upstream skill to note union_by_name_distinct coverage\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add DataFrame.column(), col(), and find_qualified_columns() methods\n\nExpose upstream find_qualified_columns to resolve unqualified column\nnames into fully qualified column expressions. This is especially\nuseful for disambiguating columns after joins.\n\n- find_qualified_columns(*names) on Rust side calls upstream directly\n- DataFrame.column(name) and col(name) alias on Python side\n- Update join and join_on docstrings to reference DataFrame.col()\n- Add \"Disambiguating Columns with DataFrame.col()\" section to joins docs\n- Add tests for qualified column resolution, ambiguity, and join usage\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Merge union_by_name and union_by_name_distinct into a single method with distinct flag\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* converting into a python dict loses a column when the names are identical\n\n* Consolidate except_all/except_distinct and intersect/intersect_distinct into single methods with distinct flag\n\nFollows the same pattern as union(distinct\u003d) and union_by_name(distinct\u003d).\nAlso deprecates union_distinct() in favor of union(distinct\u003dTrue).\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\nCo-authored-by: Copilot \u003c175728472+Copilot@users.noreply.github.com\u003e" }, { "commit": "898d73de20346bba7241907bb18cba47da53e9a9", "tree": "2a462b3bbae08b97d637c90a91c9786236553f1b", "parents": [ "d07fdb3ef7d211920f40d0106fa50161c0bf20ce" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Tue Apr 07 09:01:36 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Apr 07 09:01:36 2026 -0400" }, "message": "Add missing aggregate functions (#1471)\n\n* Add missing aggregate functions: grouping, percentile_cont, var_population\n\nExpose upstream DataFusion aggregate functions that were not yet\navailable in the Python API. Closes #1454.\n\n- grouping: returns grouping set membership indicator (rewritten by\n the ResolveGroupingFunction analyzer rule before physical planning)\n- percentile_cont: computes exact percentile using continuous\n interpolation (unlike approx_percentile_cont which uses t-digest)\n- var_population: alias for var_pop\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Fix grouping() distinct parameter type for API consistency\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Improve aggregate function tests and docstrings per review feedback\n\nAdd docstring example to grouping(), parametrize percentile_cont tests,\nand add multi-column grouping test case.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add GroupingSet.rollup, .cube, and .grouping_sets factory methods\n\nExpose ROLLUP, CUBE, and GROUPING SETS via the DataFrame API by adding\nstatic methods on GroupingSet that construct the corresponding Expr\nvariants. Update grouping() docstring and tests to use the new API.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Remove _GroupingSetInternal alias, use expr_internal.GroupingSet directly\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Parametrize grouping set tests for rollup and cube\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add grouping sets documentation and note grouping() alias limitation\n\nAdd user documentation for GroupingSet.rollup, .cube, and\n.grouping_sets with Pokemon dataset examples. Document the upstream\nalias limitation (apache/datafusion#21411) in both the grouping()\ndocstring and the aggregation user guide.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add grouping sets note to DataFrame.aggregate() docstring\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Address PR review feedback: add quantile_cont alias and simplify examples\n\n- Add quantile_cont as alias for percentile_cont (matches upstream)\n- Replace pa.concat_arrays batch pattern with collect_column() in docstrings\n- Add percentile_cont, quantile_cont, var_population to docs function list\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Accept string column names in GroupingSet factory methods\n\nGroupingSet.rollup(), .cube(), and .grouping_sets() now accept both\nExpr objects and string column names, consistent with DataFrame.aggregate().\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add agent instructions to keep aggregation/window docs in sync\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* dfn is already available globally\n\n* Remove unnecessary import on doctest\n\n---------\n\nCo-authored-by: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "d07fdb3ef7d211920f40d0106fa50161c0bf20ce", "tree": "84d6cf140850de88bd310be10347a2f351960437", "parents": [ "99bc9602dd077c924685f1fc6e54e6feb3429302" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Mon Apr 06 08:54:30 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Apr 06 08:54:30 2026 -0400" }, "message": "Add missing scalar functions (#1470)\n\n* Add missing scalar functions: get_field, union_extract, union_tag, arrow_metadata, version, row\n\nExpose upstream DataFusion scalar functions that were not yet available\nin the Python API. Closes #1453.\n\n- get_field: extracts a field from a struct or map by name\n- union_extract: extracts a value from a union type by field name\n- union_tag: returns the active field name of a union type\n- arrow_metadata: returns Arrow field metadata (all or by key)\n- version: returns the DataFusion version string\n- row: alias for the struct constructor\n\nNote: arrow_try_cast was listed in the issue but does not exist in\nDataFusion 53, so it is not included.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add tests for new scalar functions\n\nTests for get_field, arrow_metadata, version, row, union_tag, and\nunion_extract.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Accept str for field name and type parameters in scalar functions\n\nAllow arrow_cast, get_field, and union_extract to accept plain str\narguments instead of requiring Expr wrappers. Also improve\narrow_metadata test coverage and fix parameter shadowing.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Accept str for key parameter in arrow_metadata for consistency\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add doctest examples and fix docstring style for new scalar functions\n\nReplace Args/Returns sections with doctest Examples blocks for\narrow_metadata, get_field, union_extract, union_tag, and version to\nmatch existing codebase conventions. Simplify row to alias-style\ndocstring with See Also reference. Document that arrow_cast accepts\nboth str and Expr for data_type.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Support pyarrow DataType in arrow_cast\n\nAllow arrow_cast to accept a pyarrow DataType in addition to str and\nExpr. The DataType is converted to its string representation before\nbeing passed to DataFusion. Adds test coverage for the new input type.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Document bracket syntax shorthand in get_field docstring\n\nNote that expr[\"field\"] is a convenient alternative when the field\nname is a static string, and get_field is needed for dynamic\nexpressions. Add a second doctest example showing the bracket syntax.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Fix arrow_cast with pyarrow DataType by delegating to Expr.cast\n\nUse the existing Rust-side PyArrowType\u003cDataType\u003e conversion via\nExpr.cast() instead of str() which produces pyarrow type names\nthat DataFusion does not recognize.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Clarify when to use arrow_cast vs Expr.cast in docstring\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "99bc9602dd077c924685f1fc6e54e6feb3429302", "tree": "ccaa7c9a5fb8012dd2c005de2997b35ece6f0382", "parents": [ "ff15648c5dca6b41d3f6146c6c36c97e605f8561" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Mon Apr 06 07:47:13 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Apr 06 07:47:13 2026 -0400" }, "message": "Add missing array functions (#1468)\n\n* Add missing array/list functions and aliases (#1452)\n\nAdd new array functions from upstream DataFusion v53: array_any_value,\narray_distance, array_max, array_min, array_reverse, arrays_zip,\nstring_to_array, and gen_series. Add corresponding list_* aliases and\nmissing list_* aliases for existing functions (list_empty, list_pop_back,\nlist_pop_front, list_has, list_has_all, list_has_any). Also add\narray_contains/list_contains as aliases for array_has, generate_series\nas alias for gen_series, and string_to_list as alias for string_to_array.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add unit tests for new array/list functions and aliases\n\nTests cover all functions and aliases added in the previous commit:\narray_any_value, array_distance, array_max, array_min, array_reverse,\narrays_zip, string_to_array, gen_series, generate_series,\narray_contains, list_contains, list_empty, list_pop_back,\nlist_pop_front, list_has, list_has_all, list_has_any, and list_*\naliases for the new functions.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Improve array function APIs: optional params, better naming, restore comment\n\n- Make null_string optional in string_to_array/string_to_list\n- Make step optional in gen_series/generate_series\n- Rename second_array to element in array_contains/list_has/list_contains\n- Restore # Window Functions section comment in __all__\n- Add tests for optional parameter variants\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Consolidate array/list function tests using pytest parametrize\n\nReduce 26 individual tests to 14 test functions with parametrized\ncases, eliminating boilerplate while maintaining full coverage.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Move list alias tests into existing test_array_functions parametrize block\n\nMerge standalone tests for list_empty, list_pop_back, list_pop_front,\nlist_has, array_contains, list_contains, list_has_all, and list_has_any\ninto the existing parametrized test_array_functions block alongside\ntheir array_* counterparts.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Merge test_array_any_value into parametrized test_any_value_aliases\n\nUse the richer multi-row dataset (including all-nulls case) for both\narray_any_value and list_any_value via the parametrized test.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add arrays_overlap and list_overlap as aliases for array_has_any\n\nThese aliases match the upstream DataFusion SQL-level aliases, completing\nthe set of missing array functions from issue #1452.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add docstring examples for optional params in string_to_array and gen_series\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Update AGENTS file to demonstrate preferred method of documenting python functions\n\n---------\n\nCo-authored-by: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "ff15648c5dca6b41d3f6146c6c36c97e605f8561", "tree": "9ef8154af5770402b050e0f06705ce10370401be", "parents": [ "8a35caea9ed01492742738f161fa5b4459d69402" ], "author": { "name": "Nuno Faria", "email": "nunofpfaria@gmail.com", "time": "Sun Apr 05 13:29:32 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Apr 05 08:29:32 2026 -0400" }, "message": "minor: Fix pytest instructions in README (#1477)" }, { "commit": "8a35caea9ed01492742738f161fa5b4459d69402", "tree": "580704c9bf189ddbf4a5f3caa327a9161b86c883", "parents": [ "16feeb136737ae45fac39f7a82cca2d88fd6224b" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Sat Apr 04 12:20:31 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Apr 04 12:20:31 2026 -0400" }, "message": "Add missing map functions (#1461)\n\n* Add map functions (make_map, map_keys, map_values, map_extract, map_entries, element_at)\n\nCloses #1448\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add unit tests for map functions\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Remove redundant pyo3 element_at function\n\nelement_at is already a Python-only alias for map_extract,\nso the Rust binding is unnecessary.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Change make_map to accept a Python dictionary\n\nmake_map now takes a dict for the common case and also supports\nseparate keys/values lists for column expressions. Non-Expr keys\nand values are automatically converted to literals.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Make map the primary function with make_map as alias\n\nmap() now supports three calling conventions matching upstream:\n- map({\"a\": 1, \"b\": 2}) — from a Python dictionary\n- map([keys], [values]) — two lists that get zipped\n- map(k1, v1, k2, v2, ...) — variadic key-value pairs\n\nNon-Expr keys and values are automatically converted to literals.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Improve map function docstrings\n\n- Add examples for all three map() calling conventions\n- Use clearer descriptions instead of jargon (no \"zipped\" or \"variadic\")\n- Break map_keys/map_values/map_extract/map_entries examples into\n two steps: create the map column first, then call the function\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Remove map() in favor of make_map(), fix docstrings, add validation\n\n- Remove map() function that shadowed Python builtin; make_map() is now\n the sole entry point for creating map expressions\n- Fix map_extract/element_at docstrings: missing keys return [None],\n not an empty list (matches actual upstream behavior)\n- Add length validation for the two-list calling convention\n- Update all tests and docstring examples accordingly\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Consolidate map function tests into parametrized groups\n\nReduce boilerplate by combining make_map construction tests and map\naccessor function tests into two @pytest.mark.parametrize groups.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Docstring update\n\nCo-authored-by: Nuno Faria \u003cnunofpfaria@gmail.com\u003e\n\n* Docstring update\n\nCo-authored-by: Nuno Faria \u003cnunofpfaria@gmail.com\u003e\n\n* Simplify test for readability\n\nCo-authored-by: Nuno Faria \u003cnunofpfaria@gmail.com\u003e\n\n* Simplify test for readability\n\nCo-authored-by: Nuno Faria \u003cnunofpfaria@gmail.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\nCo-authored-by: Nuno Faria \u003cnunofpfaria@gmail.com\u003e" }, { "commit": "16feeb136737ae45fac39f7a82cca2d88fd6224b", "tree": "10fb40bbeab421b23d25bade5da9985dc359c0fe", "parents": [ "0b6ea95a3d304a774bbe512bb70fbca332aa5426" ], "author": { "name": "Kevin Liu", "email": "kevinjqliu@users.noreply.github.com", "time": "Fri Apr 03 12:47:31 2026 -0700" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Apr 03 15:47:31 2026 -0400" }, "message": "Reduce peak memory usage during release builds to fix OOM on manylinux runners (#1445)\n\n* adjust swap to 8gb\n\n* modify profile.release" }, { "commit": "0b6ea95a3d304a774bbe512bb70fbca332aa5426", "tree": "8c1e2c3ae2ea105c730d84dc47c1dfe3f720c8de", "parents": [ "645d261ce3bc0b3b610c8d82422042b3e573e793" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri Apr 03 15:43:28 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Apr 03 15:43:28 2026 -0400" }, "message": "Add missing conditional functions (#1464)\n\n* Add missing conditional functions: greatest, least, nvl2, ifnull (#1449)\n\nExpose four conditional functions from upstream DataFusion that were\nnot yet available in the Python bindings.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add unit tests for greatest, least, nvl2, and ifnull functions\n\nTests cover multiple data types (integers, strings), null handling\n(all-null, partial-null), multiple arguments, and ifnull/nvl equivalence.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Use standard alias docstring pattern for ifnull\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* remove unused df fixture and fix parameter shadowing\n\n* Refactor conditional function tests into parametrized test suite\n\nReplace separate test functions for coalesce, greatest, least, nvl,\nnvl2, ifnull with a single parametrized test using a shared fixture.\nAdds coverage for nvl, nullif (previously untested), datetime and\nboolean types, literal fallbacks, and variadic calls.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "645d261ce3bc0b3b610c8d82422042b3e573e793", "tree": "4f05826255e2f5fe8d2367e8137c7057d9dd50a9", "parents": [ "be8dd9d08fd284cf1747a2c1b965d9c95fff117c" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri Apr 03 13:51:43 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Apr 03 13:51:43 2026 -0400" }, "message": "Add missing string function `contains` (#1465)\n\n* Add missing `contains` string function\n\nExpose the upstream DataFusion `contains(string, search_str)` function\nwhich returns true if search_str is found within string (case-sensitive).\n\nNote: the other functions from #1450 (instr, position, substring_index)\nalready exist — instr and position are aliases for strpos, and\nsubstring_index is exposed as substr_index.\n\nCloses #1450\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add unit test for contains string function\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Update python/datafusion/functions.py\n\nCo-authored-by: Nuno Faria \u003cnunofpfaria@gmail.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\nCo-authored-by: Nuno Faria \u003cnunofpfaria@gmail.com\u003e" }, { "commit": "be8dd9d08fd284cf1747a2c1b965d9c95fff117c", "tree": "3c5f66c1cfc4a2631f8255aa1b31928c766ae2d3", "parents": [ "0113a6ee55cc61f9ebd897ae8cfc9213f560e468" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Fri Apr 03 09:37:00 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Apr 03 09:37:00 2026 -0400" }, "message": "Add AI skill to check current repository against upstream APIs (#1460)\n\n* Initial commit for skill to check upstream repo\n\n* Add instructions on using the check-upstream skill\n\n* Add FFI type coverage and implementation pattern to check-upstream skill\n\nDocument the full FFI type pipeline (Rust PyO3 wrapper → Protocol type →\nPython wrapper → ABC base class → exports → example) and catalog which\nupstream datafusion-ffi types are supported, which have been evaluated as\nnot needing direct exposure, and how to check for new gaps.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Update check-upstream skill to include FFI types as a checkable area\n\nAdd \"ffi types\" to the argument-hint and description so users can invoke\nthe skill with `/check-upstream ffi types`. Also add pipeline verification\nstep to ensure each supported FFI type has the full end-to-end chain\n(PyO3 wrapper, Protocol, Python wrapper with type hints, ABC, exports).\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Move FFI Types section alongside other areas to check\n\nSection 7 (FFI Types) was incorrectly placed after the Output Format and\nImplementation Pattern sections. Move it to sit after Section 6\n(SessionContext Methods), consistent with the other checkable areas.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Replace static FFI type list with dynamic discovery instruction\n\nThe supported FFI types list would go stale as new types are added.\nReplace it with a grep instruction to discover them at check time,\nkeeping only the \"evaluated and not requiring exposure\" list which\ncaptures rationale not derivable from code.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Make Python API the source of truth for upstream coverage checks\n\nFunctions exposed in Python (e.g., as aliases of other Rust bindings)\nwere being falsely reported as missing because they lacked a dedicated\n#[pyfunction] in Rust. The user-facing API is the Python layer, so\ncoverage should be measured there.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add exclusion list for DataFrame methods already covered by Python API\n\nshow_limit is covered by DataFrame.show() and with_param_values is\ncovered by SessionContext.sql(param_values\u003d...), so neither needs\nseparate exposure.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Move skills to .ai/skills/ for tool-agnostic discoverability\n\nMoves the canonical skill definitions from .claude/skills/ to .ai/skills/\nand replaces .claude/skills with a symlink, so Claude Code still discovers\nthem while other AI agents can find them in a tool-neutral location.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add AGENTS.md for tool-agnostic agent instructions with CLAUDE.md symlink\n\nAGENTS.md points agents to .ai/skills/ for skill discovery. CLAUDE.md\nsymlinks to it so Claude Code picks it up as project instructions.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Make README upstream coverage section tool-agnostic\n\nRemove Claude Code references and update skill path from .claude/skills/\nto .ai/skills/ to match the new tool-neutral directory structure.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add GitHub issue lookup step to check-upstream skill\n\nWhen gaps are identified, search open issues at\napache/datafusion-python before reporting. Existing issues are\nlinked in the report rather than duplicated.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Require Python test coverage in issues created by check-upstream skill\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add license text\n\n---------\n\nCo-authored-by: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e" } ], "next": "0113a6ee55cc61f9ebd897ae8cfc9213f560e468" }