20.0.0 (2023-03-10)

[Full Changelog](https://github.com/apache/datafusion/compare/19.0.0...20.0.0

Breaking changes:

  • Minor: Move TableProviderFactories up out of RuntimeEnv and into SessionState #5477 (alamb)
  • chore: Remove references from SessionState from physical_plan #5455 (alamb)
  • Implement Debug for ExecutionProps and VarProvider #5489 (alamb)

Implemented enhancements:

  • Add UserDefinedLogicalNodeCore #5521 (mslapek)
  • feat: add arrow_cast function to support supports arbitrary arrow types #5166 (alamb)
  • feat: interval add timestamp #5491 (Weijun-H)
  • feat: ParquetExec predicate preservation #5495 (crepererum)
  • feat: add optimization rules for bitwise operations #5423 (izveigor)
  • feat: Support bitwise operations for unsigned integer types #5476 (izveigor)
  • feat: eliminate the duplicated sort keys in Order By clause #5462 (jackwener)
  • feat: add name() method to UserDefinedLogicalNode #5450 (waynexia)
  • feat: express unsigned literal in substrait #5448 (waynexia)
  • feat: extensions_options macro #5442 (crepererum)
  • [feat]:fast check has column #5328 (suxiaogang223)
  • feat: eliminate unnecessary projection. #5366 (jackwener)

Fixed bugs:

  • revert accidently deleted size code in count_distinct #5533 (comphead)
  • fix: return schema of ExtensionPlan instead of its children's #5514 (waynexia)
  • fix: logical merge conflict -- hash_join tests with passing boolean by value #5531 (korowa)
  • fix: build union schema with child has same column name but qualifier… #5452 (yukkit)
  • Fix is_distinct from for float NaN values #5446 (comphead)
  • Bug/union wrong casting #5342 (berkaysynnada)
  • fix nested loop join with literal join filter #5431 (ygf11)
  • Fix filter pushdown for extension plans #5425 (thinkharderdev)
  • Bug fix: Window frame range value outside the type range #5384 (mustafasrepo)
  • fix: misc phys. expression display bugs #5387 (crepererum)

Documentation updates:

  • Minor: Improve docs for UserDefinedLogicalNode dyn_eq and dyn_hash #5515 (alamb)
  • chore: add known project ZincObserve #5376 (hengfeiyang)
  • docs: clarify spark #5391 (hyoklee)

Merged pull requests:

  • Manual changelog for 20.0.0 #5551 (andygrove)
  • Prepare for 20.0.0 release [Part 1] #5539 (andygrove)
  • chore: deduplicate workspace fields in Cargo.toml #5519 (waynexia)
  • Add necessary features to optimizer #5540 (viirya)
  • Minor: add the concise way for matching numerics #5537 (izveigor)
  • Add UserDefinedLogicalNodeCore #5521 (mslapek)
  • revert accidently deleted size code in count_distinct #5533 (comphead)
  • fix: return schema of ExtensionPlan instead of its children's #5514 (waynexia)
  • Minor: Move ObjectStoreRegistry to datafusion_execution crate #5478 (alamb)
  • Minor: Add db-benchmark URL to db-benchmark readme #5503 (alamb)
  • minor: fix clippy problem in new version. #5532 (jackwener)
  • fix: logical merge conflict -- hash_join tests with passing boolean by value #5531 (korowa)
  • Memory limited hash join #5490 (korowa)
  • minor: improve error style #5510 (alamb)
  • feat: add arrow_cast function to support supports arbitrary arrow types #5166 (alamb)
  • build(deps): update sqlparser requirement from 0.30 to 0.32 w/ API update #5457 (alamb)
  • Allow setting config extensions for TaskContext #5497 (mpurins-coralogix)
  • Minor: Improve docs for UserDefinedLogicalNode dyn_eq and dyn_hash #5515 (alamb)
  • feat: interval add timestamp #5491 (Weijun-H)
  • Pass booleans by value instead of by reference #5487 (maxburke)
  • Minor: Move TableProviderFactories up out of RuntimeEnv and into SessionState #5477 (alamb)
  • feat: ParquetExec predicate preservation #5495 (crepererum)
  • feat: add optimization rules for bitwise operations #5423 (izveigor)
  • chore: Remove references from SessionState from physical_plan #5455 (alamb)
  • Implement Debug for ExecutionProps and VarProvider #5489 (alamb)
  • feat: Support bitwise operations for unsigned integer types #5476 (izveigor)
  • Apply workaround for #5444 to DataFrame::describe #5468 (jiangzhx)
  • feat: eliminate the duplicated sort keys in Order By clause #5462 (jackwener)
  • Propagate timezone to created arrays #5481 (maxburke)
  • refactor: make GeometricMean not to have update and merge #5469 (Weijun-H)
  • feat: add name() method to UserDefinedLogicalNode #5450 (waynexia)
  • Comment out description text in issue templates #5482 (Jefffrey)
  • feat: express unsigned literal in substrait #5448 (waynexia)
  • fix: build union schema with child has same column name but qualifier… #5452 (yukkit)
  • refactor: make sum_distinct not to have update and merge #5474 (Weijun-H)
  • compute_decimal_op_dyn_scalar should not cast lhs array to decimal array #5465 (viirya)
  • feat: extensions_options macro #5442 (crepererum)
  • Enable hash joins on FixedSizeBinary columns #5461 (maxburke)
  • Fix is_distinct from for float NaN values #5446 (comphead)
  • Implement/fix Eq and Hash for Expr and LogicalPlan #5421 (mslapek)
  • [feat]:fast check has column #5328 (suxiaogang223)
  • Parquet sorting benchmark #5433 (jaylmiller)
  • refactor count_distinct to not to have update and merge #5408 (Weijun-H)
  • build(deps): update zstd requirement from 0.11 to 0.12 #5458 (alamb)
  • Upgrade bytes to 1.4 #5460 (viirya)
  • add std,median result to describe method #5445 (jiangzhx)
  • minor: Port more window tests to sqlogictests #5434 (alamb)
  • Use compute_op_dyn_scalar for datatime #5315 (viirya)
  • add a unit test that cover cast bug. #5443 (jackwener)
  • create new datafusion-execution crate, start splitting code out #5432 (alamb)
  • minor: fix clippy in nightly. #5440 (jackwener)
  • Support for Sliding Windows Joins with Symmetric Hash Join (SHJ) #5322 (metesynnada)
  • refactor: ParquetExec logical expr. => phys. expr. #5419 (crepererum)
  • Update README.md fix [DataFusion] links #5438 (jiangzhx)
  • add mean result for describe method #5435 (jiangzhx)
  • add expr_fn::median #5437 (jiangzhx)
  • Bug/union wrong casting #5342 (berkaysynnada)
  • reimplement push_down_projection and prune_column. #4465 (jackwener)
  • Add expr_fn::stddev #5409 (jiangzhx)
  • fix nested loop join with literal join filter #5431 (ygf11)
  • add a describe method on DataFrame like Polars #5226 (jiangzhx)
  • Memory reservation & metrics for cross join #5339 (korowa)
  • Optimize count_distinct.size #5377 (comphead)
  • Fix filter pushdown for extension plans #5425 (thinkharderdev)
  • Also push down all filters in TableProvider #5420 (avantgardnerio)
  • Update arrow 34 #5375 (tustvold)
  • Parquet limit pushdown (#5404) #5416 (tustvold)
  • Move file format config.rs to live with the rest of the datasource code #5406 (alamb)
  • Support Zstd compressed files #5397 (dennybritz)
  • Add example of catalog API usage (#5291) #5326 (jaylmiller)
  • Add support for protobuf serialisation of Arrow Map type #5359 (ahmedriza)
  • minor: port window tests to slt (part 2) #5399 (alamb)
  • fix(docs): fix typos #5403 (WenyXu)
  • Try to push down full filter before break-up #5367 (avantgardnerio)
  • enhance: remove more projection. #5402 (jackwener)
  • refactor push_down_filter to fix dead-loop and use optimizer_recurse. #5337 (jackwener)
  • feat: eliminate unnecessary projection. #5366 (jackwener)
  • minor: add forgotten large_utf8 #5393 (jackwener)
  • Minor: add tests for subquery to join #5363 (ygf11)
  • bugfix: fix master bors problem. #5395 (jackwener)
  • Rule ReplaceDistinctWithAggregate #5354 (mingmwang)
  • chore: add known project ZincObserve #5376 (hengfeiyang)
  • refactor: parquet pruning simplifications #5386 (crepererum)
  • Minor: intersect expressions optimization #5388 (izveigor)
  • docs: clarify spark #5391 (hyoklee)
  • UDF zero params #5378 #5380 (jaylmiller)
  • Minor: added some tests for coercion type #5389 (izveigor)
  • minor: make table resolution an independent function ... #5373 (MichaelScofield)
  • minor: port predicates tests to sqllogictests #5374 (jackwener)
  • Bug fix: Window frame range value outside the type range #5384 (mustafasrepo)
  • Fixed small typos in files of the optimizer #5356 (izveigor)
  • fix: misc phys. expression display bugs #5387 (crepererum)
  • Prepare for 19.0.0 release #5381 (andygrove)
  • minor: disable tpcds-q41 due to not support decorrelate disjunction subquery #5369 (jackwener)