Google Git
Sign in
apache / datafusion / refs/tags/11.0.0-rc1
commit8ee31cc69f43a4de0c0678d18a57f27cb4d0ead1[log] [tgz]
authorAndy Grove <andygrove73@gmail.com>Tue Aug 16 09:22:49 2022 -0600
committerGitHub <noreply@github.com>Tue Aug 16 09:22:49 2022 -0600
treeed9bd6ba9a3fe6d2c8a609063e85a681b2407f79
parent85f7fc9b262c5db9d17dfef662ee58d757c6eabf [diff]
Prepare for 11.0.0 release (#3171)

* minor docs updates

* bump crate versions

* Update docs/source/user-guide/sql/ddl.md

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

* CHANGELOG

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
17 files changed
tree: ed9bd6ba9a3fe6d2c8a609063e85a681b2407f79
  1. .github/
  2. benchmarks/
  3. ci/
  4. conbench/
  5. datafusion/
  6. datafusion-cli/
  7. datafusion-examples/
  8. dev/
  9. docs/
  10. integration-tests/
  11. python/
  12. parquet-testing
  13. testing
  14. .asf.yaml
  15. .dockerignore
  16. .editorconfig
  17. .github_changelog_generator
  18. .gitignore
  19. .gitmodules
  20. .pre-commit-config.yaml
  21. Cargo.toml
  22. CHANGELOG.md
  23. CODE_OF_CONDUCT.md
  24. CONTRIBUTING.md
  25. header
  26. LICENSE.txt
  27. NOTICE.txt
  28. pre-commit.sh
  29. README.md
  30. rustfmt.toml
README.md

DataFusion

DataFusion is an extensible query planning, optimization, and execution framework, written in Rust, that uses Apache Arrow as its in-memory format.

Features

  • SQL query planner with support for multiple SQL dialects
  • DataFrame API
  • Parquet, CSV, JSON, and Avro file formats are supported natively. Custom file formats can be supported by implementing a TableProvider trait.
  • Supports popular object stores, including AWS S3, Azure Blob Storage, and Google Cloud Storage. There are extension points for implementing custom object stores.

Use Cases

DataFusion is modular in design with many extension points and can be used without modification as an embedded query engine and can also provide a foundation for building new systems. Here are some example use cases:

  • DataFusion can be used as a SQL query planner and query optimizer, providing optimized logical plans that can then be mapped to other execution engines.
  • DataFusion is used to create modern, fast and efficient data pipelines, ETL processes, and database systems, which need the performance of Rust and Apache Arrow and want to provide their users the convenience of an SQL interface or a DataFrame API.

Why DataFusion?

  • High Performance: Leveraging Rust and Arrow's memory model, DataFusion achieves very high performance
  • Easy to Connect: Being part of the Apache Arrow ecosystem (Arrow, Parquet and Flight), DataFusion works well with the rest of the big data ecosystem
  • Easy to Embed: Allowing extension at almost any point in its design, DataFusion can be tailored for your specific use case
  • High Quality: Extensively tested, both by itself and with the rest of the Arrow ecosystem, DataFusion can be used as the foundation for production systems.

DataFusion Community Extensions

There are a number of community projects that extend DataFusion or provide integrations with other systems.

Language Bindings

Integrations

Known Uses

Here are some of the projects known to use DataFusion:

(if you know of another project, please submit a PR to add a link!)

Example Usage

Please see example usage to find how to use DataFusion.

Roadmap

Please see Roadmap for information of where the project is headed.

Architecture Overview

There is no formal document describing DataFusion's architecture yet, but the following presentations offer a good overview of its different components and how they interact together.

  • (July 2022): DataFusion and Arrow: Supercharge Your Data Analytical Tool with a Rusty Query Engine: recording and slides
  • (March 2021): The DataFusion architecture is described in Query Engine Design and the Rust-Based DataFusion in Apache Arrow: recording (DataFusion content starts ~ 15 minutes in) and slides
  • (February 2021): How DataFusion is used within the Ballista Project is described in *Ballista: Distributed Compute with Rust and Apache Arrow: recording

User Guide

Please see User Guide for more information about DataFusion.

Contributor Guide

Please see Contributor Guide for information about contributing to DataFusion.