Google Git
Sign in
apache / arrow-rs / 0c1e3b893b94f0351070b970f98c641972161bba
commit0c1e3b893b94f0351070b970f98c641972161bba[log] [tgz]
authorHadrien G <knights_of_ni@gmx.com>Thu May 16 12:15:41 2024 +0200
committerGitHub <noreply@github.com>Thu May 16 11:15:41 2024 +0100
treee2d3eebe9a24ed624fd99ee691686459a2add383
parent30767a687b48d0dbd2e030eef327826c39095123 [diff]
Remove harmful table lookup optimization for bit operations (#5772)

CPUs have efficient instructions for querying, setting and clearing bits, and
modern compilers know how to turn simple bit indexing code into such
instructions. The table lookup optimizations may have been useful in older
versions of rustc, but as of rustc 1.78, they are a net loss. See PR description
for more details.
1 file changed
tree: e2d3eebe9a24ed624fd99ee691686459a2add383
  1. .github/
  2. arrow/
  3. arrow-arith/
  4. arrow-array/
  5. arrow-avro/
  6. arrow-buffer/
  7. arrow-cast/
  8. arrow-csv/
  9. arrow-data/
  10. arrow-flight/
  11. arrow-integration-test/
  12. arrow-integration-testing/
  13. arrow-ipc/
  14. arrow-json/
  15. arrow-ord/
  16. arrow-pyarrow-integration-testing/
  17. arrow-row/
  18. arrow-schema/
  19. arrow-select/
  20. arrow-string/
  21. conbench/
  22. dev/
  23. format/
  24. object_store/
  25. parquet/
  26. parquet_derive/
  27. parquet_derive_test/
  28. parquet-testing
  29. testing
  30. .asf.yaml
  31. .gitattributes
  32. .github_changelog_generator
  33. .gitignore
  34. .gitmodules
  35. .pre-commit-config.yaml
  36. Cargo.toml
  37. CHANGELOG-old.md
  38. CHANGELOG.md
  39. CODE_OF_CONDUCT.md
  40. CONTRIBUTING.md
  41. header
  42. LICENSE.txt
  43. NOTICE.txt
  44. pre-commit.sh
  45. README.md
  46. rustfmt.toml
README.md

Native Rust implementation of Apache Arrow and Apache Parquet

Coverage Status

Welcome to the Rust implementation of Apache Arrow, the popular in-memory columnar format.

This repo contains the following main components:

CrateDescriptionLatest API DocsREADME
arrowCore Arrow functionality (memory layout, arrays, kernels)docs.rs(README)
parquetParquet columnar file formatdocs.rs(README)
arrow-flightArrow-Flight IPC protocoldocs.rs(README)
object-storeobject store (aws, azure, gcp, local, in-memory)docs.rs(README)

The current development version the API documentation in this repo can be found here.

Release Versioning and Schedule

arrow and parquet crates

The Arrow Rust project releases approximately monthly and follows Semantic Versioning.

Due to available maintainer and testing bandwidth, arrow crates (arrow, arrow-flight, etc.) are released on the same schedule with the same versions as the parquet and parquet-derive crates.

Starting June 2024, we plan to release new major versions with potentially breaking API changes at most once a quarter, and release incremental minor versions in the intervening months. See this ticket for more details.

For example:

Approximate DateVersionNotes
Jun 202452.0.0Major, potentially breaking API changes
Jul 202452.1.0Minor, NO breaking API changes
Aug 202452.2.0Minor, NO breaking API changes
Sep 202453.0.0Major, potentially breaking API changes

object_store crate

The object_store crate is released independently of the arrow and parquet crates and follows Semantic Versioning. We aim to release new versions approximately every 2 months.

Related Projects

There are two related crates in different repositories

CrateDescriptionDocumentation
datafusionIn-memory query engine with SQL support(README)
ballistaDistributed query execution(README)

Collectively, these crates support a wider array of functionality for analytic computations in Rust.

For example, you can write SQL queries or a DataFrame (using the datafusion crate) to read a parquet file (using the parquet crate), evaluate it in-memory using Arrow's columnar format (using the arrow crate), and send to another process (using the arrow-flight crate).

Generally speaking, the arrow crate offers functionality for using Arrow arrays, and datafusion offers most operations typically found in SQL, including joins and window functions.

You can find more details about each crate in their respective READMEs.

Arrow Rust Community

The dev@arrow.apache.org mailing list serves as the core communication channel for the Arrow community. Instructions for signing up and links to the archives can be found on the Arrow Community page. All major announcements and communications happen there.

The Rust Arrow community also uses the official ASF Slack for informal discussions and coordination. This is a great place to meet other contributors and get guidance on where to contribute. Join us in the #arrow-rust channel and feel free to ask for an invite via:

  1. the dev@arrow.apache.org mailing list
  2. the GitHub Discussions
  3. the Discord channel

The Rust implementation uses GitHub issues as the system of record for new features and bug fixes and this plays a critical role in the release process.

For design discussions we generally collaborate on Google documents and file a GitHub issue linking to the document.

There is more information in the contributing guide.