dev/changelog/9.0.0.md

9.0.0 (2022-06-10)

Full Changelog

Breaking changes:

MINOR: Move simplify_expression rule to datafusion-optimizer crate #2686 (andygrove)
Move physical expression planning to datafusion-physical-expr crate #2682 (andygrove)
Create new datafusion-optimizer crate for logical optimizer rules #2675 (andygrove)
Remove ExecutionProps dependency from OptimizerRule #2666 (andygrove)
Remove ObjectStoreSchemaProvider (#2656) #2665 (tustvold)
Move LogicalPlanBuilder to datafusion-expr crate #2576 (andygrove)
LogicalPlanBuilder now uses TableSource instead of TableProvider #2569 (andygrove)
Remove scan_empty method from LogicalPlanBuilder #2568 (andygrove)
MINOR: Move expression utils from sql module to expr crate #2553 (andygrove)
Remove scan_json methods from LogicalPlanBuilder #2541 (andygrove)
Remove scan_avro methods from LogicalPlanBuilder #2540 (andygrove)
Remove scan_parquet methods from LogicalPlanBuilder #2539 (andygrove)
MINOR: Move ExprVisitable and exprlist_to_columns to datafusion-expr crate #2538 (andygrove)
Remove scan_csv methods from LogicalPlanBuilder #2537 (andygrove)
Fix Redundant ScalarValue Boxed Collection #2523 (comphead)
Support for OFFSET in LogicalPlan #2521 (jdye64)

Implemented enhancements:

[EPIC] JIT support for DataFusion #2703
Show column names instead of column indices in query plans #2689
Proposal: remove automated ballista CI checks from DataFusion #2679
Pass SessionState to TableProvider #2658
Is ObjectStoreSchemaProvider Still Needed? #2656
Add logical plan support to datafusion-proto #2630
Like, NotLike expressions work with literal NULL #2626
Move JOIN ON predicates push down logic from planner to optimizer #2619
Remove ExecutionProps from OptimizerRule trait #2614
Add, Minus, Multiply, divide, Modulo operator work with literal NULL #2609
Support DESCRIBE <table> to show table schemas #2606
Support CREATE OR REPLACE TABLE #2605
filter_push_down tests should not rely on TableProvider and ExecutionPlan #2600
Move logical optimizer rules out of the core datafusion crate #2599
Push Limit through outer Join #2579
datafusion_proto crate should have exhaustive match statements for handling Expr #2565
String representation of Expr variant #2563
File URI Scheme Interpretation #2562
Implement physical plan for OFFSET #2551
Update limit pushdown rule to support offsets #2550
Move LogicalPlanBuilder to datafusion-expr crate #2536
Logical optimizer rule “simplify expressions” should not depend on the core datafusion crate #2535
Support optional filter in Join #2509
Improve SQL planner & logical plan support for JOIN conditions #2496
Numeric, String, Boolean comparisons with literal NULL #2482
Redundant ScalarValue Boxed Collection #2449
ObjectStore Directory Semantics #2445
Add support for OFFSET in SQL query planner + logical plan #2377
SQL planner should use TableSource not TableProvider #2346
Move SQL query planning to new crate #2345
Update LogicalPlan rustdoc code to not use LogicalPlanBuilder #2308
[Optimizer] Refactor convert join #2256
[Optimizer] Infer is not null predicate from where clause #2254
Support ArrayIndex for ScalarValue(List) #2207
[Ballista] Fill functional gaps between datafusion and ballista #2062
[Ballista] support datafusion built_in UDAF work in ballista cluster #1985
Export C API #1113

Fixed bugs:

Fix Typos in Docs #2695
Unable to build a docker image #2691
Optimization pass AggregateStatistics changes type of output from Int64 to UInt64 #2673
ViewTable Circular Reference #2657
ScalarValue::to_array_of_size panics computing statistics for nested parquet file #2653
The result type of count/count_distinct #2635
limit_push_down is not working properly with OFFSET #2624
Avro Tests Fail To Compile #2570
Unused Window functions experssion is wrongly removed from LogicalPlan during optimalization #2542
Bug: ObjectStoreRegistry get_by_uri does not return correct path when “scheme” is provided #2525
There are duplicate and inconsistent copies of datafusion.proto #2514
Projection pushdown produces incorrect results when column names are reused #2462
Incorrect Parquet Projection For Nested Types #2453
LogicalPlanBuilder::scan_csv creates scans with invalid table names #2278
Inner join incorrectly pushdown predicate with OR operation #2271
Ignored alias for columns with aggregate function and incorrect results when collecting statistics is enabled #2176
Join on path partitioned columns fails with error #2145

Documentation updates:

Fix Ballista link #2654 (dsaxton)
MINOR: Add Blaze as a project using DataFusion #2618 (yjshen)
[MINOR] remove datafusion-cli's ballista feature from docs #2612 (Ted-Jiang)
chore(doc) remove ballista from datafusion-cli readme #2604 (ming535)

Closed issues:

[Question] Converting TableSource to custom TableProvider #2644
[Question] Why DataFusion is shipped with arrow version 9.1.0 on crates.io ? #2474

Merged pull requests:

Test optional features in CI #2708 (tustvold)
support indexed fields proto #2707 (nl5887)
Update sqlparser-rs to 0.18.0 #2705 (alamb)
[MINOR]: Add documentation to datafusion-row modules #2704 (alamb)
Make sure that the data types are supported in hashjoin before genera… #2702 (AssHero)
Move remaining code out of legacy core/logical_plan module #2701 (andygrove)
Move some tests from core to expr #2700 (andygrove)
MINOR: Improve Docs Readability #2696 (ryanrussell)
Combine limit and offset to fetch and skip and implement physical plan support #2694 (ming535)
MINOR: Add datafusion-sql example #2693 (andygrove)
Remove Ballista related lines from Dockerfile #2692 (mocknen)
Show column names instead of indices in query plans #2690 (andygrove)
MINOR: Remove uses of TryClone for Parquet #2681 (tustvold)
Fix AggregateStatistics optimization so it doesn't change output type #2674 (alamb)
If statistics of column Max/Min value does not exists in parquet file, sent Min/Max to None #2671 (AssHero)
MINOR: Move more expression code to datafusion-expr crate #2669 (andygrove)
MINOR: Rewrite imports in optimizer moduler #2667 (andygrove)
Update snmalloc-rs requirement from 0.2 to 0.3 #2663 (dependabot[bot])
Add module doc for RuntimeEnv, SessionContext, TaskContext, etc... #2655 (tustvold)
Prune unused dependencies from datafusion-proto #2651 (tustvold)
MINOR: Implement serde for join filter #2649 (andygrove)
pushdown support for predicates in ON clause of joins #2647 (korowa)
Move SortKeyCursor and RowIndex into modules, add sort_key_cursor test #2645 (alamb)
Implement DESCRIBE <table> #2642 (LiuYuHui)
Implement LogicalPlan serde in datafusion-proto #2639 (andygrove)
Fix limit + offset pushdown #2638 (ming535)
change result type of count/count_distinct from uint64 to int64 #2636 (liukun4515)
if none columns in window expr are needed, remove the window exprs #2634 (AssHero)
Like, NotLike expressions work with literal NULL #2627 (WinkerDu)
MINOR: Refactor datafusion-proto dependencies and imports #2623 (andygrove)
MINOR: add optimizer struct #2616 (jackwener)
Remove FilterPushDown dependency on physical plan #2615 (andygrove)
Support CREATE OR REPLACE TABLE #2613 (AssHero)
Support binary mathematical operators work with NULL literals #2610 (WinkerDu)
chore: try fix CI coverage #2608 (Ted-Jiang)
MINOR: Rename benchmark crate #2607 (andygrove)
chore(dep): bump cranelift to 0.84.0 #2598 (waynexia)
fix some typos #2597 (ming535)
Support limit pushdown through left right outer join #2596 (Ted-Jiang)
Unignore rustdoc code examples in datafusion-expr crate #2590 (andygrove)
Evaluate JIT'd expression over arrays #2587 (waynexia)
[minor]Fix ci clippy for unused import #2586 (Ted-Jiang)
[Doc]add doc for enable SIMD need cargo nightly #2577 (Ted-Jiang)
Add DataFrame union_distinct and fix documentation for distinct #2574 (andygrove)
Fix avro tests (#2570) #2571 (tustvold)
Make datafusion-proto match exhaustive #2567 (andygrove)
Support limit push down for offset_plan #2566 (Ted-Jiang)
Introduce Expr.variant_name() function #2564 (jdye64)
Fix some 404 links in the contribution guide #2561 (hi-rustin)
Update datafusion-cli readme cli version #2559 (hi-rustin)
MINOR: Move expr_rewriter.rs to datafusion-expr crate #2552 (andygrove)
Fix JOINs with complex predicates in ON (split ON expressions only by AND operator) #2534 (korowa)
Reduce duplication in file scan tests #2533 (tustvold)
Fix size_of_scalar test #2531 (alamb)
Update to arrow-rs 14.0.0 #2528 (alamb)
ObjectStoreRegistry get_by_uri now returns correct path when “scheme” is provided #2526 (timvw)
MINOR: Add ORDER BY clause to test #2524 (andygrove)
Remove unused binary_array_op_scalar! in binary.rs #2512 (alamb)
fix NULL <op> column evaluation, tests for same #2510 (alamb)
Fix projection pushdown produces incorrect results when column names are reused #2463 (jonmmease)
Benchmark for sort preserving merge #2431 (alamb)
Support GetIndexedFieldExpr for ScalarValue #2196 (ovr)