Apache DataFusion Python 51.0.0 Changelog

This release consists of 23 commits from 7 contributors. See credits at the end of this changelog for more information.

Breaking changes:

  • feat: reduce duplicate fields on join #1184 (timsaucer)

Implemented enhancements:

  • feat: expose select_exprs method on DataFrame #1271 (milenkovicm)
  • feat: allow DataFrame.filter to accept SQL strings #1276 (K-dash)
  • feat: add temporary view option for into_view #1267 (timsaucer)
  • feat: support session token parameter for AmazonS3 #1275 (GCHQDeveloper028)
  • feat: with_column supports SQL expression #1284 (milenkovicm)
  • feat: Add SQL expression for repartition_by_hash #1285 (milenkovicm)
  • feat: Add SQL expression support for with_columns #1286 (milenkovicm)

Fixed bugs:

  • fix: use coalesce instead of drop_duplicate_keys for join #1318 (mesejo)
  • fix: Inconsistent schemas when converting to pyarrow #1315 (nuno-faria)

Other:

  • Release 50.1 #1281 (timsaucer)
  • Update python minimum version to 3.10 #1296 (timsaucer)
  • chore: update datafusion minor version #1297 (timsaucer)
  • Enable remaining pylints #1298 (timsaucer)
  • Add Arrow C streaming, DataFrame iteration, and OOM-safe streaming execution #1222 (kosiew)
  • Add PyCapsule Type Support and Type Hint Enhancements for AggregateUDF in DataFusion Python Bindings #1277 (kosiew)
  • Add collect_column to dataframe #1302 (timsaucer)
  • chore: apply cargo fmt with import organization #1303 (timsaucer)
  • Feat/parameterized sql queries #964 (timsaucer)
  • Upgrade to Datafusion 51 #1311 (nuno-faria)
  • minor: resolve build errors after latest merge into main #1325 (timsaucer)
  • Update build workflow link #1330 (timsaucer)
  • Do not convert pyarrow scalar values to plain python types when passing as lit #1319 (timsaucer)

Credits

Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.

    12	Tim Saucer
     4	Marko Milenković
     2	Nuno Faria
     2	kosiew
     1	Daniel Mesejo
     1	GCHQDeveloper028
     1	š¯•‚

Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.