The Apache Arrow team is pleased to announce the 10.0.0 release. This covers over 3 months of development work and includes 473 resolved issues from [100 distinct contributors][2]. See the Install Page to learn how to get the libraries for your platform.
The release notes below are not exhaustive and only expose selected highlights of the release. Many other bugfixes and improvements have been made: we refer you to the [complete changelog][3].
Since the 9.0.0 release, Yanghong Zhong, Remzi Yang, Dan Harris and Bogumił Kamińsk have been invited to be committers. L.C. Hsieh, Weston Pace, Raphael Taylor-Davies, Nicola Crane and Jacob Quinn have joined the Project Management Committee (PMC). Thanks for your contributions and participation in the project!
A JDBC driver based on Arrow Flight SQL is now available, courtesy of a code donation from Dremio (ARROW-7744). For more details, see [“Expanding Arrow's Reach with a JDBC Driver for Arrow Flight SQL”][flight-sql-jdbc]. Flight SQL is now supported in Go (ARROW-17326). Protocol definitions for transactions and Substrait plans were added to Flight SQL and are implemented in C++ and Java (ARROW-17688). General “best practices” documentation was added for C++ (ARROW-17407). The C++ implementation now has basic OpenTelemetry integration (ARROW-14958).
[flight-sql-jdbc]: {% link _posts/2022-11-01-arrow-flight-sql-jdbc.md %}
The Arrow C++ codebase has moved to C++17 as its language compatibility standard (ARROW-17545). This means Arrow C++, including its header files, now requires a C++17-compliant compiler and standard library to be used. Such compilers are widely available on most platforms.
Compatibility backports of C++14 and C++17 features, such as std::string_view
or std::variant
, have been removed in favor of the standard library version of these APIs (ARROW-17546). This will also make integration of Arrow C++ with other codebases easier.
It is expected that the Arrow C++ codebase will be gradually modernized to use C++17 features in subsequent releases, when the need arises.
The Plasma module is deprecated and will be removed in a future release.
Extension types are now supported in hash joins (ARROW-16695).
The fragments of a dataset can now be iterated in an asynchronous fashion, using Dataset::GetFragmentsAsync
(ARROW-17318).
It is now possible to configure a timeout policy for S3 (ARROW-16521).
Error messages for S3 have been improved to give more context about the error (ARROW-17079).
GetFileInfoGenerator
has been optimized for local filesystems, with dedicated options to tune chunking and readahead (ARROW-17306).
Previously, the JSON reader could only read Decimal fields from JSON strings (i.e. quoted). Now it can also read Decimal fields from JSON numbers as well (ARROW-17847).
Before Arrow 3.0.0, data pages version 2 were incorrectly written out, making them unreadable with spec-compliant readers. A compatibility fix has been introduced so that they can still be read with contemporary versions of Arrow (ARROW-17100).
The Substrait consumer, which allows Substrait plans to be executed by the Acero execution engine, has received some improvements:
Aggregations are now supported (ARROW-15591).
Conversion options have been added so that the level of compliance and rountrippability can be chosen when converting between Substrait and Acero representations of a plan (ARROW-16988).
Support for many standard Substrait functions has been added (ARROW-15582, ARROW-17523)
Some work has also been done in the reverse direction, to allow Acero execution plans to be serialized as Substrait plans (ARROW-16855).
Our CMake package files have been overhauled (ARROW-12175). As a result, namespaced targets are now exported, such as Arrow::arrow_shared
. Legacy (non-namespaced) names are still available, for example arrow_shared
.
Compiling in release mode now uses -O2
, not -O3
, by default (ARROW-17436).
The RISC-V architecture is now recongnized at build time (ARROW-17440).
The PyArrow-specific C++ code was moved into the PyArrow source tree (see below, ARROW-16340). The ARROW_PYTHON
CMake variable has been deprecated and will be removed in a later release; you should instead enable the necessary components explicitly (ARROW-17868).
Some classes with a Equals
method now also support operator==
(ARROW-6772). It was decided to only do this when equality is computationally cheap (i.e. not on data collections such as Array, RecordBatch...).
Gandiva has been migrated to use LLVM opaque pointer types, as typed pointers had been deprecated (ARROW-17790).
-asan
option using go1.18 (ARROW-17324)The compute package that was importable via github.com/apache/arrow/go/v9/arrow/compute
is now a separate module which requires go1.18+ (only the compute module, the rest of the packages still work fine under go1.17). (ARROW-17456).
Scalar and Vector kernel infrastructure has been implemented for performing compute operations providing the following functionality:
Many important features, enhancements, and bug fixes are included in this release, as are documentation enhancements, and a large number of improvements to build processes and project infrastructure. Selected highlights can be found below.
Compatibility notes:
RecordBatchReader.get_next_batch
method, DataType.num_children
attribute, etc (ARROW-17649).pyarrow.dataset.write_dataset
using format="ipc"
or format="arrow"
, the default extension for the resulting files is changed to .arrow
instead of .feather
. You can still use format="feather"
to write identical files but using the .feather
extension (ARROW-17089).New features and improvements:
pq.read_table()
can be passed as an expression in addition to the legacy list of tuples. For example, filters=pc.field("col") < 4
is equivalent to filters=[("col", "<", 4)]
(ARROW-17483).batch_readahead
and fragment_readahead
arguments for scanning Datasets are exposed in Python (ARROW-17299).pa.array(..)
constructor (ARROW-17834).pyarrow.substrait.run_query()
function gained a table_provider
keyword to run the query against in-memory tables (ARROW-17521).StructType
class gained a field()
method to retrieve a child field (ARROW-17131).Further, the Python bindings benefit from improvements in the C++ library (e.g. new compute functions); see the C++ notes above for additional details.
Build notes
The PyArrow-specific C++ code, previously part of Arrow C++ codebase, is now integrated into PyArrow. The tests are run automatically as part of the PyArrow test suite. See: ARROW-16340, ARROW-17122 and PyArrow C++ API notes).
The build process is generally not affected by the change, but the ARROW_PYTHON
CMake variable has been deprecated and will be removed in a later release; you should instead enable the necessary components explicitly (ARROW-17868).
Many improvements to Arrow dplyr queries are added in this version, including:
dplyr::across()
can be used to apply the same computation across multiple columns;add_filename()
;For more on what’s in the 10.0.0 R package, see the [R changelog][4].
The Rust projects have moved to separate repositories outside the main Arrow monorepo. For notes on the latest release of the Rust implementation, see the latest [Arrow Rust changelog][5].
[2]: {{ site.baseurl }}/release/10.0.0.html#contributors [3]: {{ site.baseurl }}/release/10.0.0.html#changelog [4]: {{ site.baseurl }}/docs/r/news/ [5]: https://github.com/apache/arrow-rs/tags [6]: https://arrow.apache.org/docs/r/reference/acero.html