| <!--- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| --> |
| |
| ## [21.0.0](https://github.com/apache/datafusion/tree/21.0.0) (2023-03-24) |
| |
| [Full Changelog](https://github.com/apache/datafusion/compare/20.0.0...21.0.0) |
| |
| **Breaking changes:** |
| |
| - Support arbitrary user defined partition column in `ListingTable` (rather than assuming they are always Dictionary encoded) [#5545](https://github.com/apache/datafusion/pull/5545) (crepererum) |
| - Use TableReference for TableScan [#5615](https://github.com/apache/datafusion/pull/5615) (alamb) |
| - Update the type of `param_values` to `&[ScalarValue]` in function `replace_params_with_values` [#5640](https://github.com/apache/datafusion/pull/5640) (HaoYang670) |
| |
| **Implemented enhancements:** |
| |
| - feat: extract (epoch from col) [#5555](https://github.com/apache/datafusion/pull/5555) (Weijun-H) |
| - INSERT INTO support for MemTable [#5520](https://github.com/apache/datafusion/pull/5520) (metesynnada) |
| - Memory limited nested-loop join [#5564](https://github.com/apache/datafusion/pull/5564) (korowa) |
| - Timestamp subtraction and interval operations for `ScalarValue` [#5603](https://github.com/apache/datafusion/pull/5603) (berkaysynnada) |
| - Substrait: Add cast expression with bool, integers and decimal128 support [#5137](https://github.com/apache/datafusion/pull/5137) (nseekhao) |
| - Support `date_bin` with 2 arguments [#5643](https://github.com/apache/datafusion/pull/5643) (Weijun-H) |
| - improve: support combining multiple grouping expressions [#5559](https://github.com/apache/datafusion/pull/5559) (yukkit) |
| - Substrait: Add support for WindowFunction [#5653](https://github.com/apache/datafusion/pull/5653) (nseekhao) |
| - feat: `date_bin` supports MonthDayNano, microsecond and nanosecond units [#5698](https://github.com/apache/datafusion/pull/5698) (stuartcarnie) |
| - Handle serialization of TryCast [#5692](https://github.com/apache/datafusion/pull/5692) (thinkharderdev) |
| |
| **Fixed bugs:** |
| |
| - fix: failed to execute sql with subquery [#5542](https://github.com/apache/datafusion/pull/5542) (MichaelScofield) |
| - fix: cast literal to timestamp [#5517](https://github.com/apache/datafusion/pull/5517) (Weijun-H) |
| - fix dataframe only boolean/binary column got error on describe [#5585](https://github.com/apache/datafusion/pull/5585) (jiangzhx) |
| - Median returns null on empty input instead of error [#5624](https://github.com/apache/datafusion/pull/5624) (toppyy) |
| - add CountWildcardRule to fix error on Count(Expr:Wildcard) with DataFrame API [#5627](https://github.com/apache/datafusion/pull/5627) (jiangzhx) |
| - fix: correct CountWildcardRule and move analyzer into a new directory. [#5671](https://github.com/apache/datafusion/pull/5671) (jackwener) |
| |
| **Documentation updates:** |
| |
| - Minor: improve docstrings for `ObjectStoreRegistry` and `ObjectStoreProvider` [#5577](https://github.com/apache/datafusion/pull/5577) (alamb) |
| - Clarify differences of DataFusion with other systems in README.md [#5578](https://github.com/apache/datafusion/pull/5578) (alamb) |
| - Minor: Document docs build process [#5687](https://github.com/apache/datafusion/pull/5687) (alamb) |
| |
| **Merged pull requests:** |
| |
| - Refactor DecorrelateWhereExists and add back Distinct if needs [#5345](https://github.com/apache/datafusion/pull/5345) (ygf11) |
| - Simplify simplify test cases, support `^`, `&`, `|`, `<<` and `>>` operators for building exprs [#5511](https://github.com/apache/datafusion/pull/5511) (alamb) |
| - minor: improve sqllogictest docs [#5553](https://github.com/apache/datafusion/pull/5553) (alamb) |
| - Remove unused dependencies found by cargo-machete [#5552](https://github.com/apache/datafusion/pull/5552) (Jefffrey) |
| - make AggregateStatistics return the same result whether optimizer disabled or enabled [#5485](https://github.com/apache/datafusion/pull/5485) (jiangzhx) |
| - Avoid circular(ish) dependency parquet-test-utils on datafusion, try 2 [#5536](https://github.com/apache/datafusion/pull/5536) (alamb) |
| - Enforce ambiguity check whilst normalizing columns [#5509](https://github.com/apache/datafusion/pull/5509) (Jefffrey) |
| - Generated changelog for 20.0.0 [#5563](https://github.com/apache/datafusion/pull/5563) (andygrove) |
| - fix: failed to execute sql with subquery [#5542](https://github.com/apache/datafusion/pull/5542) (MichaelScofield) |
| - Revert describe count() workaround [#5556](https://github.com/apache/datafusion/pull/5556) (Jefffrey) |
| - fix: cast literal to timestamp [#5517](https://github.com/apache/datafusion/pull/5517) (Weijun-H) |
| - feat: extract (epoch from col) [#5555](https://github.com/apache/datafusion/pull/5555) (Weijun-H) |
| - Minor: improve docstrings for `ObjectStoreRegistry` and `ObjectStoreProvider` [#5577](https://github.com/apache/datafusion/pull/5577) (alamb) |
| - Minor: Move RuntimeEnv to `datafusion_execution` [#5580](https://github.com/apache/datafusion/pull/5580) (alamb) |
| - INSERT INTO support for MemTable [#5520](https://github.com/apache/datafusion/pull/5520) (metesynnada) |
| - Minor: restore explicit match to help avoid subtle bugs in the future when new `Expr` variants are added [#5579](https://github.com/apache/datafusion/pull/5579) (alamb) |
| - refactor: add more error info when array is empty [#5560](https://github.com/apache/datafusion/pull/5560) (Weijun-H) |
| - Memory limited nested-loop join [#5564](https://github.com/apache/datafusion/pull/5564) (korowa) |
| - Support catalog.schema.table.column in SQL SELECT and WHERE [#5343](https://github.com/apache/datafusion/pull/5343) (Jefffrey) |
| - Minor: clean up aggregates.slt tests [#5599](https://github.com/apache/datafusion/pull/5599) (alamb) |
| - Minor: Port more aggregate tests to sqllogictests [#5574](https://github.com/apache/datafusion/pull/5574) (alamb) |
| - Add a utility function to get all of the PartitionedFile for an ExecutionPlan [#5572](https://github.com/apache/datafusion/pull/5572) (yahoNanJing) |
| - minor: port some join tests to sqllogictests [#5567](https://github.com/apache/datafusion/pull/5567) (ygf11) |
| - Support arbitrary user defined partition column in `ListingTable` (rather than assuming they are always Dictionary encoded) [#5545](https://github.com/apache/datafusion/pull/5545) (crepererum) |
| - feat: add the similar optimization function for bitwise negative [#5516](https://github.com/apache/datafusion/pull/5516) (izveigor) |
| - Clarify differences of DataFusion with other systems in README.md [#5578](https://github.com/apache/datafusion/pull/5578) (alamb) |
| - Minor: Add more documentation about table_partition_columns [#5576](https://github.com/apache/datafusion/pull/5576) (alamb) |
| - Add Analyzer phase to DataFusion , add basic validation logic to Subquery Plans and Expressions [#5570](https://github.com/apache/datafusion/pull/5570) (mingmwang) |
| - Use TableReference for TableScan [#5615](https://github.com/apache/datafusion/pull/5615) (alamb) |
| - Preserve casts in rewrite_sort_cols_by_aggs [#5611](https://github.com/apache/datafusion/pull/5611) (mpurins-coralogix) |
| - Miscellaneous ArrayData Cleanup [#5612](https://github.com/apache/datafusion/pull/5612) (tustvold) |
| - Update substrait requirement from 0.4 to 0.5 [#5620](https://github.com/apache/datafusion/pull/5620) (dependabot[bot]) |
| - Do not break pipeline for window queries with GROUPS [#5587](https://github.com/apache/datafusion/pull/5587) (mustafasrepo) |
| - fix dataframe only boolean/binary column got error on describe [#5585](https://github.com/apache/datafusion/pull/5585) (jiangzhx) |
| - Minor: Add Documentation and Examples to `TableReference` [#5616](https://github.com/apache/datafusion/pull/5616) (alamb) |
| - [FOLLOWUP] eliminate the duplicated sort keys in Order By clause [#5607](https://github.com/apache/datafusion/pull/5607) (mingmwang) |
| - Update default behaviour of compression algorithms (support multistreams) [#5629](https://github.com/apache/datafusion/pull/5629) (metesynnada) |
| - Timestamp subtraction and interval operations for `ScalarValue` [#5603](https://github.com/apache/datafusion/pull/5603) (berkaysynnada) |
| - Use modulus dyn kernels for arithmetic expressions [#5634](https://github.com/apache/datafusion/pull/5634) (viirya) |
| - Minor: reduce cloning in `infer_placeholder_types` [#5638](https://github.com/apache/datafusion/pull/5638) (alamb) |
| - Move `SessionConfig` to `datafusion_execution` [#5581](https://github.com/apache/datafusion/pull/5581) (alamb) |
| - Update the type of `param_values` to `&[ScalarValue]` in function `replace_params_with_values` [#5640](https://github.com/apache/datafusion/pull/5640) (HaoYang670) |
| - WITH ORDER support on CREATE EXTERNAL TABLE [#5618](https://github.com/apache/datafusion/pull/5618) (metesynnada) |
| - Median returns null on empty input instead of error [#5624](https://github.com/apache/datafusion/pull/5624) (toppyy) |
| - feat: Memory limited merge join [#5632](https://github.com/apache/datafusion/pull/5632) (korowa) |
| - Update rstest requirement from 0.16.0 to 0.17.0 [#5648](https://github.com/apache/datafusion/pull/5648) (dependabot[bot]) |
| - add CountWildcardRule to fix error on Count(Expr:Wildcard) with DataFrame API [#5627](https://github.com/apache/datafusion/pull/5627) (jiangzhx) |
| - Add OuterReferenceColumn to Expr to represent correlated expression [#5593](https://github.com/apache/datafusion/pull/5593) (mingmwang) |
| - Minor: Simplify `Result<T, DataFusionError>` [#5659](https://github.com/apache/datafusion/pull/5659) (comphead) |
| - minor: remove redundant `DataFusionError` and fix `clippy` [#5669](https://github.com/apache/datafusion/pull/5669) (jackwener) |
| - Substrait: Add cast expression with bool, integers and decimal128 support [#5137](https://github.com/apache/datafusion/pull/5137) (nseekhao) |
| - Support `date_bin` with 2 arguments [#5643](https://github.com/apache/datafusion/pull/5643) (Weijun-H) |
| - Add LogicalPlanSignature and use in the optimizer loop [#5623](https://github.com/apache/datafusion/pull/5623) (mslapek) |
| - fix: correct CountWildcardRule and move analyzer into a new directory. [#5671](https://github.com/apache/datafusion/pull/5671) (jackwener) |
| - refactoring: added tests and fixed comments in "math_expressions" [#5656](https://github.com/apache/datafusion/pull/5656) (izveigor) |
| - improve: support combining multiple grouping expressions [#5559](https://github.com/apache/datafusion/pull/5559) (yukkit) |
| - community: polish issue template [#5668](https://github.com/apache/datafusion/pull/5668) (jackwener) |
| - minor: correct issue template [#5679](https://github.com/apache/datafusion/pull/5679) (jackwener) |
| - Change ObjectStoreRegistry from struct to trait to provide polymorphism [#5543](https://github.com/apache/datafusion/pull/5543) (yahoNanJing) |
| - Minor: Add `Extensions::new()` [#5676](https://github.com/apache/datafusion/pull/5676) (alamb) |
| - minor: add with_plan for Subquery [#5680](https://github.com/apache/datafusion/pull/5680) (jackwener) |
| - minor: reduce replication in `date_bin` implementation [#5673](https://github.com/apache/datafusion/pull/5673) (alamb) |
| - Fixes #5500 - Add a GitHub Actions workflow that builds the docs [#5670](https://github.com/apache/datafusion/pull/5670) (martin-g) |
| - Minor: port some content to the docs [#5684](https://github.com/apache/datafusion/pull/5684) (alamb) |
| - Docs: Add logo back to sidebar [#5688](https://github.com/apache/datafusion/pull/5688) (alamb) |
| - Substrait: Add support for WindowFunction [#5653](https://github.com/apache/datafusion/pull/5653) (nseekhao) |
| - Add -o option to all e2e benches [#5658](https://github.com/apache/datafusion/pull/5658) (jaylmiller) |
| - create table default to null [#5606](https://github.com/apache/datafusion/pull/5606) (Weijun-H) |
| - Minor: Document docs build process [#5687](https://github.com/apache/datafusion/pull/5687) (alamb) |
| - Minor: change doc formatting to force a republish [#5702](https://github.com/apache/datafusion/pull/5702) (alamb) |
| - Move `TaskContext` to datafusion-execution [#5677](https://github.com/apache/datafusion/pull/5677) (alamb) |
| - feat: `date_bin` supports MonthDayNano, microsecond and nanosecond units [#5698](https://github.com/apache/datafusion/pull/5698) (stuartcarnie) |
| - Return plan error when adding utf8 and timestamp [#5696](https://github.com/apache/datafusion/pull/5696) (Weijun-H) |
| - Handle serialization of TryCast [#5692](https://github.com/apache/datafusion/pull/5692) (thinkharderdev) |
| - analyzer: move InlineTableScan into Analyzer. [#5683](https://github.com/apache/datafusion/pull/5683) (jackwener) |
| - minor: Add doc comments to clarify what Analyzer is for [#5705](https://github.com/apache/datafusion/pull/5705) (alamb) |
