DataFusion is an embedded query engine which leverages the unique features of Rust and Apache Arrow to provide a system that is high performance, easy to connect, easy to embed, and high quality.
The Apache Arrow team is pleased to announce the DataFusion 6.0.0 release. This covers 4 months of development work and includes 134 commits from the following 28 distinct contributors.
28 Andrew Lamb 26 Jiayu Liu 13 xudong963 9 rdettai 9 QP Hou 6 Matthew Turner 5 Daniël Heres 4 Guillaume Balaine 3 Francis Du 3 Marco Neumann 3 Jon Mease 3 Nga Tran 2 Yijie Shen 2 Ruihang Xia 2 Liang-Chi Hsieh 2 baishen 2 Andy Grove 2 Jason Tianyi Wang 1 Nan Zhu 1 Antoine Wendlinger 1 Krisztián Szűcs 1 Mike Seddon 1 Conner Murphy 1 Patrick More 1 Taehoon Moon 1 Tiphaine Ruy 1 adsharma 1 lichuan6
The release notes below are not exhaustive and only expose selected highlights of the release. Many other bug fixes and improvements have been made: we refer you to the complete changelog.
Befitting a growing project, DataFusion now has its own website hosted as part of the main Apache Arrow Website
The community worked to gather their thoughts about where we are taking DataFusion into a public Roadmap for the first time
except
, intersect
, show
, limit
and window functionsEXPLAIN ANALYZE
with runtime metricstrim ( [ LEADING | TRAILING | BOTH ] [ FROM ] string text [, characters text ] )
syntax~
, ~*
, !~
, and !~*
UNION
, INTERSECT
, and EXCEPT
cume_dist
, percent_rank
window functionsdigest
, blake2s
, blake2b
, blake3
crypto functionsapprox_distinct
is distinct from
and is not distinct from
CREATE TABLE AS SELECT
Struct
and List
columns (e.g. SELECT struct_column['field_name'], array_column[0] FROM ...
)CASE
statementDROP TABLE
VALUES
ListScalarValue::Struct
datafusion-cli
as well as psql
style commands such as \d
, \?
, and \q
With these new features, we are also now passing TPC-H queries 8, 13 and 21.
For the full list of new features with their relevant PRs, see the enhancements section in the changelog.
async
planning and decoupling file format from table layoutDriven by the need to support Hive style table partitioning, @rdettai introduced the following design change to the Datafusion core.
Parquet
, Avro
, CSV
, and JSON
) was separated from the logic that handles grouping sets of files into execution partitions.async
.As a result, we are able to replace the old Parquet
, CSV
and JSON
table providers with a single ListingTable
table provider.
This also sets up DataFusion and its plug-in ecosystem to supporting a wide range of catalogs and various object store implementations. You can read more about this change in the design document and on the arrow-datafusion#1010 PR.
If you are interested in contributing to DataFusion, we would love to have you! You can help by trying out DataFusion on some of your own data and projects and filing bug reports and helping to improve the documentation, or contribute to the documentation, tests or code. A list of open issues suitable for beginners is here and the full list is here.
Check out our new Communication Doc on more ways to engage with the community.