commit | e4b28bd78726288c163bbc1ba019858de94cf22d | [log] [tgz] |
---|---|---|
author | Ruihang Xia <waynestxia@gmail.com> | Sat Jun 01 22:01:03 2024 +0800 |
committer | GitHub <noreply@github.com> | Sat Jun 01 15:01:03 2024 +0100 |
tree | 7dc37adeec4a6500d101c2045e3ffcebac8f3505 | |
parent | 5a24119f6634d66b83556ecbcfbd54ae3f1620c7 [diff] |
Relax zstd-sys Version Pin (#5829) * build: pin zstd-sys to 2.0.7 Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * change to <=2.0.9 Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * <2.0.10 Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * also limit package zstd Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * try multi selectors Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * remove dep pin Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * only pin for wasm target family Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Welcome to the Rust implementation of Apache Arrow, the popular in-memory columnar format.
This repo contains the following main components:
Crate | Description | Latest API Docs | README |
---|---|---|---|
arrow | Core functionality (memory layout, arrays, low level computations) | docs.rs | (README) |
arrow-flight | Support for Arrow-Flight IPC protocol | docs.rs | (README) |
object-store | Support for object store interactions (aws, azure, gcp, local, in-memory) | docs.rs | (README) |
parquet | Support for Parquet columnar file format | docs.rs | (README) |
parquet_derive | A crate for deriving RecordWriter/RecordReader for arbitrary, simple structs | docs.rs | (README) |
The current development version the API documentation in this repo can be found here.
arrow
and parquet
cratesThe Arrow Rust project releases approximately monthly and follows Semantic Versioning.
Due to available maintainer and testing bandwidth, arrow
crates (arrow
, arrow-flight
, etc.) are released on the same schedule with the same versions as the parquet
and [parquet-derive
] crates.
Starting June 2024, we plan to release new major versions with potentially breaking API changes at most once a quarter, and release incremental minor versions in the intervening months. See this ticket for more details.
For example:
Approximate Date | Version | Notes |
---|---|---|
Jun 2024 | 52.0.0 | Major, potentially breaking API changes |
Jul 2024 | 52.1.0 | Minor, NO breaking API changes |
Aug 2024 | 52.2.0 | Minor, NO breaking API changes |
Sep 2024 | 53.0.0 | Major, potentially breaking API changes |
object_store
crateThe object_store
crate is released independently of the arrow
and parquet
crates and follows Semantic Versioning. We aim to release new versions approximately every 2 months.
There are two related crates in different repositories
Crate | Description | Documentation |
---|---|---|
datafusion | In-memory query engine with SQL support | (README) |
ballista | Distributed query execution | (README) |
Collectively, these crates support a wider array of functionality for analytic computations in Rust.
For example, you can write SQL queries or a DataFrame
(using the datafusion
crate) to read a parquet file (using the parquet
crate), evaluate it in-memory using Arrow's columnar format (using the arrow
crate), and send to another process (using the arrow-flight
crate).
Generally speaking, the arrow
crate offers functionality for using Arrow arrays, and datafusion
offers most operations typically found in SQL, including join
s and window functions.
You can find more details about each crate in their respective READMEs.
The dev@arrow.apache.org
mailing list serves as the core communication channel for the Arrow community. Instructions for signing up and links to the archives can be found on the Arrow Community page. All major announcements and communications happen there.
The Rust Arrow community also uses the official ASF Slack for informal discussions and coordination. This is a great place to meet other contributors and get guidance on where to contribute. Join us in the #arrow-rust
channel and feel free to ask for an invite via:
dev@arrow.apache.org
mailing listThe Rust implementation uses GitHub issues as the system of record for new features and bug fixes and this plays a critical role in the release process.
For design discussions we generally collaborate on Google documents and file a GitHub issue linking to the document.
There is more information in the contributing guide.