This crate contains the official Native Rust implementation of Apache Arrow in memory format, governed by the Apache Software Foundation.
This crate is tested with the latest stable version of Rust. We do not currently test against other, older versions of the Rust compiler.
The arrow crate follows the SemVer standard defined by Cargo and works well within the Rust crate ecosystem.
However, for historical reasons, this crate uses versions with major numbers greater than 0.x (e.g. 7.0.0), unlike many other crates in the Rust ecosystem which spend extended time releasing versions 0.x to signal planned ongoing API changes. Minor arrow releases contain only compatible changes, while major releases may contain breaking API changes.
The arrow crate provides the following features which may be enabled:
csv (default) - support for reading and writing Arrow arrays to/from csv filesipc (default) - support for the arrow-flight IPC and wire formatprettyprint - support for formatting record batches as textual columnsjs - support for building arrow for WebAssembly / JavaScriptsimd - (Requires Nightly Rust) alternate optimized implementations of some compute kernels using explicit SIMD instructions available through packed_simd_2.chrono-tz - support of parsing timezone using chrono-tzTLDR: You should avoid using the alloc and buffer and bitmap modules if at all possible. These modules contain unsafe code, are easy to misuse, and are not needed for most users.
As with all open source code, you should carefully evaluate the suitability of arrow for your project, taking into consideration your needs and risk tolerance prior to doing so.
Background: There are various parts of the arrow crate which use unsafe and transmute code internally. We are actively working as a community to minimize undefined behavior and remove unsafe usage to align more with Rust's core principles of safety.
As arrow exists today, it is fairly easy to misuse the code in modules named above, leading to undefined behavior.
Arrow can compile to WebAssembly using the wasm32-unknown-unknown and wasm32-wasi targets.
In order to compile Arrow for wasm32-unknown-unknown you will need to disable default features, then include the desired features, but exclude test dependencies (the test_utils feature). For example, use this snippet in your Cargo.toml:
[dependencies] arrow = { version = "5.0", default-features = false, features = ["csv", "ipc", "simd"] }
The examples folder shows how to construct some different types of Arrow arrays, including dynamic arrays:
Examples can be run using the cargo run --example command. For example:
cargo run --example builders cargo run --example dynamic_types cargo run --example read_csv