Apache DataFusion Python 54.0.0 Changelog

This release consists of 51 commits from 8 contributors. See credits at the end of this changelog for more information.

Breaking changes:

  • Update datafusion dependency to latest in preparation for DF54 #1532 (timsaucer)
  • feat: enable pickling for Python aggregate and window UDFs #1545 (timsaucer)
  • feat: pass calling SessionContext to Python UDTF callbacks #1555 (timsaucer)
  • feat: accept distinct kwarg on sum and avg #1556 (timsaucer)

Implemented enhancements:

  • feat: add AI skill to find and improve the Pythonic interface to functions #1484 (timsaucer)
  • feat: enable pickling of most Expr except udaf and udwf #1544 (timsaucer)
  • feat: expose variety of features from DF54 update #1554 (timsaucer)
  • feat: Python UDFs: per-session inlining toggle and strict refusal setting #1546 (timsaucer)
  • feat: create free-threaded python wheels #1553 (timsaucer)
  • feat: expose lambda and higher-order array functions #1561 (timsaucer)
  • feat: import user-defined physical optimizer rules over FFI #1557 (timsaucer)
  • feat: expose SessionContext.copied_config and parse_capacity_limit #1570 (timsaucer)
  • feat: expose array_compact, array_normalize, cosine_distance, inner_product #1567 (timsaucer)
  • feat: expose arrow_field, arrow_try_cast, cast_to_type, with_metadata #1568 (timsaucer)
  • feat: expose spark-compatible functions #1564 (timsaucer)
  • feat: improve pythonic interface on date/time functions #1563 (timsaucer)

Fixed bugs:

  • fix: type scalar UDF returns as Arrow arrays #1528 (BharatDeva)
  • fix: Skip fork and forkserver on win32 #1566 (nuno-faria)

Documentation updates:

  • docs: enrich module docstrings and add doctest examples #1498 (timsaucer)
  • docs: add README section for AI coding assistants #1503 (timsaucer)
  • docs: add upstream sync process documentation #1524 (timsaucer)
  • docs: document null-handling function arguments #1527 (BharatDeva)
  • docs: user guide + runnable examples for distributing expressions #1547 (timsaucer)
  • docs: convert reStructuredText sources to MyST markdown #1579 (timsaucer)

Other:

  • Release 53.0.0 #1491 (timsaucer)
  • ci: disable symbol export on Windows verification #1486 (timsaucer)
  • Add Python bindings for accessing ExecutionMetrics #1381 (ShreyeshArangath)
  • Support None comparisons for null expressions #1489 (zeel2104)
  • chore: update release documentation #1494 (timsaucer)
  • Fix error on show() with an explain plan #1492 (timsaucer)
  • Add SKILL.md and enrich package docstring #1497 (timsaucer)
  • minor: fix header on agent SKILL.md file #1501 (timsaucer)
  • tpch examples: rewrite queries idiomatically and embed reference SQL #1504 (timsaucer)
  • Move public skills to a directory to avoid downloading the whole repo #1519 (ntjohnson1)
  • Update user documentation for AI agent skill usage #1505 (timsaucer)
  • build(deps): combined dependabot bumps (Cargo + workflows) #1534 (timsaucer)
  • Add support for logical and physical codecs #1541 (timsaucer)
  • Add details on caching to skill #1521 (ntjohnson1)
  • Bump DataFusion to prepare for DF54 release candidate #1562 (timsaucer)
  • Export to_datafusion_err from the util crate root #1487 (kosiew)
  • chore: remove unused PyConfig #1485 (timsaucer)
  • refactor(context): deduplicate register/read option-building logic #1479 (mesejo)
  • Improve documentation site layout #1578 (timsaucer)
  • Allow DataFrame.aggregate to accept None for no grouping #1581 (kosiew)
  • Update to released DF 54.0.0 #1588 (timsaucer)
  • build(deps): batch dependabot dependency updates #1589 (timsaucer)
  • Deprecate Expr temporal part arguments in date extraction and truncation functions #1587 (kosiew)
  • chore: update rust dependencies #1604 (timsaucer)
  • Add deprecation warnings for Expr passed to confirmed literal-only function arguments #1605 (kosiew)
  • chore: resolve audits after DF54.0.0 update to skill #1608 (timsaucer)
  • chore: remove 3.13 freethreaded builds #1609 (timsaucer)

Credits

Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.

    39	Tim Saucer
     4	kosiew
     2	BharatDeva
     2	Nick
     1	Daniel Mesejo
     1	Nuno Faria
     1	Shreyesh
     1	Zeel Desai

Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.