| <!--- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| --> |
| |
| ## [20.0.0](https://github.com/apache/datafusion/tree/20.0.0) (2023-03-10) |
| |
| [Full Changelog](https://github.com/apache/datafusion/compare/19.0.0...20.0.0 |
| |
| **Breaking changes:** |
| |
| - Minor: Move TableProviderFactories up out of `RuntimeEnv` and into `SessionState` [#5477](https://github.com/apache/datafusion/pull/5477) (alamb) |
| - chore: Remove references from SessionState from physical_plan [#5455](https://github.com/apache/datafusion/pull/5455) (alamb) |
| - Implement `Debug` for `ExecutionProps` and `VarProvider` [#5489](https://github.com/apache/datafusion/pull/5489) (alamb) |
| |
| **Implemented enhancements:** |
| |
| - Add UserDefinedLogicalNodeCore [#5521](https://github.com/apache/datafusion/pull/5521) (mslapek) |
| - feat: add `arrow_cast` function to support supports arbitrary arrow types [#5166](https://github.com/apache/datafusion/pull/5166) (alamb) |
| - feat: interval add timestamp [#5491](https://github.com/apache/datafusion/pull/5491) (Weijun-H) |
| - feat: `ParquetExec` predicate preservation [#5495](https://github.com/apache/datafusion/pull/5495) (crepererum) |
| - feat: add optimization rules for bitwise operations [#5423](https://github.com/apache/datafusion/pull/5423) (izveigor) |
| - feat: Support bitwise operations for unsigned integer types [#5476](https://github.com/apache/datafusion/pull/5476) (izveigor) |
| - feat: eliminate the duplicated sort keys in Order By clause [#5462](https://github.com/apache/datafusion/pull/5462) (jackwener) |
| - feat: add name() method to UserDefinedLogicalNode [#5450](https://github.com/apache/datafusion/pull/5450) (waynexia) |
| - feat: express unsigned literal in substrait [#5448](https://github.com/apache/datafusion/pull/5448) (waynexia) |
| - feat: `extensions_options` macro [#5442](https://github.com/apache/datafusion/pull/5442) (crepererum) |
| - [feat]:fast check has column [#5328](https://github.com/apache/datafusion/pull/5328) (suxiaogang223) |
| - feat: eliminate unnecessary projection. [#5366](https://github.com/apache/datafusion/pull/5366) (jackwener) |
| |
| **Fixed bugs:** |
| |
| - revert accidently deleted size code in count_distinct [#5533](https://github.com/apache/datafusion/pull/5533) (comphead) |
| - fix: return schema of ExtensionPlan instead of its children's [#5514](https://github.com/apache/datafusion/pull/5514) (waynexia) |
| - fix: logical merge conflict -- hash_join tests with passing boolean by value [#5531](https://github.com/apache/datafusion/pull/5531) (korowa) |
| - fix: build union schema with child has same column name but qualifier… [#5452](https://github.com/apache/datafusion/pull/5452) (yukkit) |
| - Fix is_distinct from for float NaN values [#5446](https://github.com/apache/datafusion/pull/5446) (comphead) |
| - Bug/union wrong casting [#5342](https://github.com/apache/datafusion/pull/5342) (berkaysynnada) |
| - fix nested loop join with literal join filter [#5431](https://github.com/apache/datafusion/pull/5431) (ygf11) |
| - Fix filter pushdown for extension plans [#5425](https://github.com/apache/datafusion/pull/5425) (thinkharderdev) |
| - Bug fix: Window frame range value outside the type range [#5384](https://github.com/apache/datafusion/pull/5384) (mustafasrepo) |
| - fix: misc phys. expression display bugs [#5387](https://github.com/apache/datafusion/pull/5387) (crepererum) |
| |
| **Documentation updates:** |
| |
| - Minor: Improve docs for UserDefinedLogicalNode `dyn_eq` and `dyn_hash` [#5515](https://github.com/apache/datafusion/pull/5515) (alamb) |
| - chore: add known project ZincObserve [#5376](https://github.com/apache/datafusion/pull/5376) (hengfeiyang) |
| - docs: clarify spark [#5391](https://github.com/apache/datafusion/pull/5391) (hyoklee) |
| |
| **Merged pull requests:** |
| |
| - Manual changelog for 20.0.0 [#5551](https://github.com/apache/datafusion/pull/5551) (andygrove) |
| - Prepare for 20.0.0 release [Part 1] [#5539](https://github.com/apache/datafusion/pull/5539) (andygrove) |
| - chore: deduplicate workspace fields in Cargo.toml [#5519](https://github.com/apache/datafusion/pull/5519) (waynexia) |
| - Add necessary features to optimizer [#5540](https://github.com/apache/datafusion/pull/5540) (viirya) |
| - Minor: add the concise way for matching numerics [#5537](https://github.com/apache/datafusion/pull/5537) (izveigor) |
| - Add UserDefinedLogicalNodeCore [#5521](https://github.com/apache/datafusion/pull/5521) (mslapek) |
| - revert accidently deleted size code in count_distinct [#5533](https://github.com/apache/datafusion/pull/5533) (comphead) |
| - fix: return schema of ExtensionPlan instead of its children's [#5514](https://github.com/apache/datafusion/pull/5514) (waynexia) |
| - Minor: Move `ObjectStoreRegistry` to datafusion_execution crate [#5478](https://github.com/apache/datafusion/pull/5478) (alamb) |
| - Minor: Add db-benchmark URL to db-benchmark readme [#5503](https://github.com/apache/datafusion/pull/5503) (alamb) |
| - minor: fix clippy problem in new version. [#5532](https://github.com/apache/datafusion/pull/5532) (jackwener) |
| - fix: logical merge conflict -- hash_join tests with passing boolean by value [#5531](https://github.com/apache/datafusion/pull/5531) (korowa) |
| - Memory limited hash join [#5490](https://github.com/apache/datafusion/pull/5490) (korowa) |
| - minor: improve error style [#5510](https://github.com/apache/datafusion/pull/5510) (alamb) |
| - feat: add `arrow_cast` function to support supports arbitrary arrow types [#5166](https://github.com/apache/datafusion/pull/5166) (alamb) |
| - build(deps): update sqlparser requirement from 0.30 to 0.32 w/ API update [#5457](https://github.com/apache/datafusion/pull/5457) (alamb) |
| - Allow setting config extensions for TaskContext [#5497](https://github.com/apache/datafusion/pull/5497) (mpurins-coralogix) |
| - Minor: Improve docs for UserDefinedLogicalNode `dyn_eq` and `dyn_hash` [#5515](https://github.com/apache/datafusion/pull/5515) (alamb) |
| - feat: interval add timestamp [#5491](https://github.com/apache/datafusion/pull/5491) (Weijun-H) |
| - Pass booleans by value instead of by reference [#5487](https://github.com/apache/datafusion/pull/5487) (maxburke) |
| - Minor: Move TableProviderFactories up out of `RuntimeEnv` and into `SessionState` [#5477](https://github.com/apache/datafusion/pull/5477) (alamb) |
| - feat: `ParquetExec` predicate preservation [#5495](https://github.com/apache/datafusion/pull/5495) (crepererum) |
| - feat: add optimization rules for bitwise operations [#5423](https://github.com/apache/datafusion/pull/5423) (izveigor) |
| - chore: Remove references from SessionState from physical_plan [#5455](https://github.com/apache/datafusion/pull/5455) (alamb) |
| - Implement `Debug` for `ExecutionProps` and `VarProvider` [#5489](https://github.com/apache/datafusion/pull/5489) (alamb) |
| - feat: Support bitwise operations for unsigned integer types [#5476](https://github.com/apache/datafusion/pull/5476) (izveigor) |
| - Apply workaround for #5444 to `DataFrame::describe` [#5468](https://github.com/apache/datafusion/pull/5468) (jiangzhx) |
| - feat: eliminate the duplicated sort keys in Order By clause [#5462](https://github.com/apache/datafusion/pull/5462) (jackwener) |
| - Propagate timezone to created arrays [#5481](https://github.com/apache/datafusion/pull/5481) (maxburke) |
| - refactor: make GeometricMean not to have update and merge [#5469](https://github.com/apache/datafusion/pull/5469) (Weijun-H) |
| - feat: add name() method to UserDefinedLogicalNode [#5450](https://github.com/apache/datafusion/pull/5450) (waynexia) |
| - Comment out description text in issue templates [#5482](https://github.com/apache/datafusion/pull/5482) (Jefffrey) |
| - feat: express unsigned literal in substrait [#5448](https://github.com/apache/datafusion/pull/5448) (waynexia) |
| - fix: build union schema with child has same column name but qualifier… [#5452](https://github.com/apache/datafusion/pull/5452) (yukkit) |
| - refactor: make sum_distinct not to have update and merge [#5474](https://github.com/apache/datafusion/pull/5474) (Weijun-H) |
| - `compute_decimal_op_dyn_scalar` should not cast lhs array to decimal array [#5465](https://github.com/apache/datafusion/pull/5465) (viirya) |
| - feat: `extensions_options` macro [#5442](https://github.com/apache/datafusion/pull/5442) (crepererum) |
| - Enable hash joins on FixedSizeBinary columns [#5461](https://github.com/apache/datafusion/pull/5461) (maxburke) |
| - Fix is_distinct from for float NaN values [#5446](https://github.com/apache/datafusion/pull/5446) (comphead) |
| - Implement/fix Eq and Hash for Expr and LogicalPlan [#5421](https://github.com/apache/datafusion/pull/5421) (mslapek) |
| - [feat]:fast check has column [#5328](https://github.com/apache/datafusion/pull/5328) (suxiaogang223) |
| - Parquet sorting benchmark [#5433](https://github.com/apache/datafusion/pull/5433) (jaylmiller) |
| - refactor count_distinct to not to have update and merge [#5408](https://github.com/apache/datafusion/pull/5408) (Weijun-H) |
| - build(deps): update zstd requirement from 0.11 to 0.12 [#5458](https://github.com/apache/datafusion/pull/5458) (alamb) |
| - Upgrade bytes to 1.4 [#5460](https://github.com/apache/datafusion/pull/5460) (viirya) |
| - add std,median result to describe method [#5445](https://github.com/apache/datafusion/pull/5445) (jiangzhx) |
| - minor: Port more window tests to sqlogictests [#5434](https://github.com/apache/datafusion/pull/5434) (alamb) |
| - Use compute_op_dyn_scalar for datatime [#5315](https://github.com/apache/datafusion/pull/5315) (viirya) |
| - add a unit test that cover cast bug. [#5443](https://github.com/apache/datafusion/pull/5443) (jackwener) |
| - create new `datafusion-execution` crate, start splitting code out [#5432](https://github.com/apache/datafusion/pull/5432) (alamb) |
| - minor: fix clippy in nightly. [#5440](https://github.com/apache/datafusion/pull/5440) (jackwener) |
| - Support for Sliding Windows Joins with Symmetric Hash Join (SHJ) [#5322](https://github.com/apache/datafusion/pull/5322) (metesynnada) |
| - refactor: ParquetExec logical expr. => phys. expr. [#5419](https://github.com/apache/datafusion/pull/5419) (crepererum) |
| - Update README.md fix [DataFusion] links [#5438](https://github.com/apache/datafusion/pull/5438) (jiangzhx) |
| - add mean result for describe method [#5435](https://github.com/apache/datafusion/pull/5435) (jiangzhx) |
| - add expr_fn::median [#5437](https://github.com/apache/datafusion/pull/5437) (jiangzhx) |
| - Bug/union wrong casting [#5342](https://github.com/apache/datafusion/pull/5342) (berkaysynnada) |
| - reimplement `push_down_projection` and `prune_column`. [#4465](https://github.com/apache/datafusion/pull/4465) (jackwener) |
| - Add `expr_fn::stddev` [#5409](https://github.com/apache/datafusion/pull/5409) (jiangzhx) |
| - fix nested loop join with literal join filter [#5431](https://github.com/apache/datafusion/pull/5431) (ygf11) |
| - add a describe method on DataFrame like Polars [#5226](https://github.com/apache/datafusion/pull/5226) (jiangzhx) |
| - Memory reservation & metrics for cross join [#5339](https://github.com/apache/datafusion/pull/5339) (korowa) |
| - Optimize count_distinct.size [#5377](https://github.com/apache/datafusion/pull/5377) (comphead) |
| - Fix filter pushdown for extension plans [#5425](https://github.com/apache/datafusion/pull/5425) (thinkharderdev) |
| - Also push down all filters in TableProvider [#5420](https://github.com/apache/datafusion/pull/5420) (avantgardnerio) |
| - Update arrow 34 [#5375](https://github.com/apache/datafusion/pull/5375) (tustvold) |
| - Parquet limit pushdown (#5404) [#5416](https://github.com/apache/datafusion/pull/5416) (tustvold) |
| - Move file format config.rs to live with the rest of the datasource code [#5406](https://github.com/apache/datafusion/pull/5406) (alamb) |
| - Support Zstd compressed files [#5397](https://github.com/apache/datafusion/pull/5397) (dennybritz) |
| - Add example of catalog API usage (#5291) [#5326](https://github.com/apache/datafusion/pull/5326) (jaylmiller) |
| - Add support for protobuf serialisation of Arrow Map type [#5359](https://github.com/apache/datafusion/pull/5359) (ahmedriza) |
| - minor: port window tests to slt (part 2) [#5399](https://github.com/apache/datafusion/pull/5399) (alamb) |
| - fix(docs): fix typos [#5403](https://github.com/apache/datafusion/pull/5403) (WenyXu) |
| - Try to push down full filter before break-up [#5367](https://github.com/apache/datafusion/pull/5367) (avantgardnerio) |
| - enhance: remove more projection. [#5402](https://github.com/apache/datafusion/pull/5402) (jackwener) |
| - refactor `push_down_filter` to fix dead-loop and use optimizer_recurse. [#5337](https://github.com/apache/datafusion/pull/5337) (jackwener) |
| - feat: eliminate unnecessary projection. [#5366](https://github.com/apache/datafusion/pull/5366) (jackwener) |
| - minor: add forgotten large_utf8 [#5393](https://github.com/apache/datafusion/pull/5393) (jackwener) |
| - Minor: add tests for subquery to join [#5363](https://github.com/apache/datafusion/pull/5363) (ygf11) |
| - bugfix: fix master `bors` problem. [#5395](https://github.com/apache/datafusion/pull/5395) (jackwener) |
| - Rule ReplaceDistinctWithAggregate [#5354](https://github.com/apache/datafusion/pull/5354) (mingmwang) |
| - chore: add known project ZincObserve [#5376](https://github.com/apache/datafusion/pull/5376) (hengfeiyang) |
| - refactor: parquet pruning simplifications [#5386](https://github.com/apache/datafusion/pull/5386) (crepererum) |
| - Minor: intersect expressions optimization [#5388](https://github.com/apache/datafusion/pull/5388) (izveigor) |
| - docs: clarify spark [#5391](https://github.com/apache/datafusion/pull/5391) (hyoklee) |
| - UDF zero params #5378 [#5380](https://github.com/apache/datafusion/pull/5380) (jaylmiller) |
| - Minor: added some tests for coercion type [#5389](https://github.com/apache/datafusion/pull/5389) (izveigor) |
| - minor: make table resolution an independent function ... [#5373](https://github.com/apache/datafusion/pull/5373) (MichaelScofield) |
| - minor: port predicates tests to sqllogictests [#5374](https://github.com/apache/datafusion/pull/5374) (jackwener) |
| - Bug fix: Window frame range value outside the type range [#5384](https://github.com/apache/datafusion/pull/5384) (mustafasrepo) |
| - Fixed small typos in files of the optimizer [#5356](https://github.com/apache/datafusion/pull/5356) (izveigor) |
| - fix: misc phys. expression display bugs [#5387](https://github.com/apache/datafusion/pull/5387) (crepererum) |
| - Prepare for 19.0.0 release [#5381](https://github.com/apache/datafusion/pull/5381) (andygrove) |
| - minor: disable tpcds-q41 due to not support decorrelate disjunction subquery [#5369](https://github.com/apache/datafusion/pull/5369) (jackwener) |
