Google Git
Sign in
apache / datafusion / 80f7bbf4ab183c8ec551fc9c26a4365fca65b68c
commit80f7bbf4ab183c8ec551fc9c26a4365fca65b68c[log] [tgz]
authorAndy Grove <agrove@apache.org>Thu May 12 08:47:57 2022 -0600
committerGitHub <noreply@github.com>Thu May 12 08:47:57 2022 -0600
tree745b481ce6b827b88016feb4f81017a243b861a0
parent19d937a0f3c99be361df4bc912cc340a6196634a [diff]
Add new `ballista-cli` crate (#2495)

* Add new ballista-cli crate

* update dependency diagram

* re-use PrintFormat from datafusion-cli

* re-use PrintOptions from datafusion-cli

* re-use Helper

* re-use functions

* update dev scripts

* update diagram and docs

* stop building DataFusion CLI with ballista in CI

* update user guide

* docs for building ballista-cli with docker

* make version numbers consistent with repo

* update Cargo.lock files for CLIs and add ballista-cli to GitHub workflow

* disable ballista tests

* fix ci

* fix merge conflict

* fix
27 files changed
tree: 745b481ce6b827b88016feb4f81017a243b861a0
  1. .github/
  2. ballista/
  3. ballista-cli/
  4. ballista-examples/
  5. benchmarks/
  6. ci/
  7. conbench/
  8. datafusion/
  9. datafusion-cli/
  10. datafusion-examples/
  11. dev/
  12. docs/
  13. integration-tests/
  14. python/
  15. parquet-testing
  16. testing
  17. .asf.yaml
  18. .dir-locals.el
  19. .dockerignore
  20. .editorconfig
  21. .env
  22. .gitattributes
  23. .github_changelog_generator
  24. .gitignore
  25. .gitmodules
  26. .hadolint.yaml
  27. .pre-commit-config.yaml
  28. .readthedocs.yml
  29. Cargo.toml
  30. CHANGELOG.md
  31. CODE_OF_CONDUCT.md
  32. CONTRIBUTING.md
  33. header
  34. LICENSE.txt
  35. NOTICE.txt
  36. pre-commit.sh
  37. README.md
  38. rustfmt.toml
README.md

DataFusion

DataFusion is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.

DataFusion supports both an SQL and a DataFrame API for building logical query plans as well as a query optimizer and execution engine capable of parallel execution against partitioned data sources (CSV and Parquet) using threads.

DataFusion also supports distributed query execution via the Ballista crate.

Use Cases

DataFusion is used to create modern, fast and efficient data pipelines, ETL processes, and database systems, which need the performance of Rust and Apache Arrow and want to provide their users the convenience of an SQL interface or a DataFrame API.

Why DataFusion?

  • High Performance: Leveraging Rust and Arrow's memory model, DataFusion achieves very high performance
  • Easy to Connect: Being part of the Apache Arrow ecosystem (Arrow, Parquet and Flight), DataFusion works well with the rest of the big data ecosystem
  • Easy to Embed: Allowing extension at almost any point in its design, DataFusion can be tailored for your specific usecase
  • High Quality: Extensively tested, both by itself and with the rest of the Arrow ecosystem, DataFusion can be used as the foundation for production systems.

Known Uses

Projects that adapt to or serve as plugins to DataFusion:

Here are some of the projects known to use DataFusion:

(if you know of another project, please submit a PR to add a link!)

Example Usage

Please see example usage to find how to use DataFusion.

Roadmap

Please see Roadmap for information of where the project is headed.

Architecture Overview

There is no formal document describing DataFusion's architecture yet, but the following presentations offer a good overview of its different components and how they interact together.

  • (March 2021): The DataFusion architecture is described in Query Engine Design and the Rust-Based DataFusion in Apache Arrow: recording (DataFusion content starts ~ 15 minutes in) and slides
  • (February 2021): How DataFusion is used within the Ballista Project is described in *Ballista: Distributed Compute with Rust and Apache Arrow: recording

User's guide

Please see User Guide for more information about DataFusion.

Contribution Guide

Please see Contribution Guide for information about contributing to DataFusion.