Google Git
Sign in
apache / datafusion / refs/tags/16.0.0
commit1844d39eb92f04e483095f491ff07da3a2f67f25[log] [tgz]
authorAndy Grove <andygrove73@gmail.com>Thu Jan 12 02:23:46 2023 -0700
committerGitHub <noreply@github.com>Thu Jan 12 04:23:46 2023 -0500
tree75538a4157996c98ecfeb33d1758654d2f3773c7
parent315f5b9848d3a1218ded9a1113ac4e8e8a90272f [diff]
Update changelog for 16.0.0-rc2 (#4881)

* update changelog

* run with larger issues limit

* no limit
3 files changed
tree: 75538a4157996c98ecfeb33d1758654d2f3773c7
  1. .github/
  2. benchmarks/
  3. ci/
  4. conbench/
  5. datafusion/
  6. datafusion-cli/
  7. datafusion-examples/
  8. dev/
  9. docs/
  10. integration-tests/
  11. parquet-test-utils/
  12. python/
  13. test-utils/
  14. parquet-testing
  15. testing
  16. .asf.yaml
  17. .dockerignore
  18. .editorconfig
  19. .gitattributes
  20. .github_changelog_generator
  21. .gitignore
  22. .gitmodules
  23. .pre-commit-config.yaml
  24. Cargo.toml
  25. CHANGELOG.md
  26. CODE_OF_CONDUCT.md
  27. CONTRIBUTING.md
  28. header
  29. LICENSE.txt
  30. NOTICE.txt
  31. pre-commit.sh
  32. README.md
  33. rustfmt.toml
README.md

DataFusion

DataFusion is an extensible query planning, optimization, and execution framework, written in Rust, that uses Apache Arrow as its in-memory format.

Coverage Status

Features

  • SQL query planner with support for multiple SQL dialects
  • DataFrame API
  • Parquet, CSV, JSON, and Avro file formats are supported natively. Custom file formats can be supported by implementing a TableProvider trait.
  • Supports popular object stores, including AWS S3, Azure Blob Storage, and Google Cloud Storage. There are extension points for implementing custom object stores.

Use Cases

DataFusion is modular in design with many extension points and can be used without modification as an embedded query engine and can also provide a foundation for building new systems. Here are some example use cases:

  • DataFusion can be used as a SQL query planner and query optimizer, providing optimized logical plans that can then be mapped to other execution engines.
  • DataFusion is used to create modern, fast and efficient data pipelines, ETL processes, and database systems, which need the performance of Rust and Apache Arrow and want to provide their users the convenience of an SQL interface or a DataFrame API.

Why DataFusion?

  • High Performance: Leveraging Rust and Arrow's memory model, DataFusion achieves very high performance
  • Easy to Connect: Being part of the Apache Arrow ecosystem (Arrow, Parquet and Flight), DataFusion works well with the rest of the big data ecosystem
  • Easy to Embed: Allowing extension at almost any point in its design, DataFusion can be tailored for your specific use case
  • High Quality: Extensively tested, both by itself and with the rest of the Arrow ecosystem, DataFusion can be used as the foundation for production systems.

DataFusion Community Extensions

There are a number of community projects that extend DataFusion or provide integrations with other systems.

Language Bindings

Integrations

Known Uses

Here are some of the projects known to use DataFusion:

(if you know of another project, please submit a PR to add a link!)

Example Usage

Please see the example usage in the user guide and the datafusion-examples crate for more information on how to use DataFusion.

Roadmap

Please see Roadmap for information of where the project is headed.

Architecture Overview

There is no formal document describing DataFusion's architecture yet, but the following presentations offer a good overview of its different components and how they interact together.

  • (July 2022): DataFusion and Arrow: Supercharge Your Data Analytical Tool with a Rusty Query Engine: recording and slides
  • (March 2021): The DataFusion architecture is described in Query Engine Design and the Rust-Based DataFusion in Apache Arrow: recording (DataFusion content starts ~ 15 minutes in) and slides
  • (February 2021): How DataFusion is used within the Ballista Project is described in *Ballista: Distributed Compute with Rust and Apache Arrow: recording

User Guide

Please see User Guide for more information about DataFusion.

Contributor Guide

Please see Contributor Guide for information about contributing to DataFusion.