Google Git
Sign in
apache / datafusion / 30e8ceff80dcb5d95d8f399917ac6c846986bdf7
commit30e8ceff80dcb5d95d8f399917ac6c846986bdf7[log] [tgz]
authorAndy Grove <andygrove73@gmail.com>Mon Sep 12 12:19:29 2022 -0600
committerGitHub <noreply@github.com>Mon Sep 12 12:19:29 2022 -0600
tree70fae6a72ba4654a039406f7de63a0a4ed83df51
parent1d4a928eb76627ed4f0923f1cae89cda6a9c035a [diff]
Add logical plan support for aggregate expressions with filters (and upgrade to sqlparser 0.23) (#3405)

* Use sqlparser-0.23

* Add filter to aggregate expressions

* clippy

* implement protobuf serde

* clippy

* fix error message

* Update datafusion/expr/src/expr.rs

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
12 files changed
tree: 70fae6a72ba4654a039406f7de63a0a4ed83df51
  1. .github/
  2. benchmarks/
  3. ci/
  4. conbench/
  5. datafusion/
  6. datafusion-cli/
  7. datafusion-examples/
  8. dev/
  9. docs/
  10. integration-tests/
  11. python/
  12. parquet-testing
  13. testing
  14. .asf.yaml
  15. .dockerignore
  16. .editorconfig
  17. .github_changelog_generator
  18. .gitignore
  19. .gitmodules
  20. .pre-commit-config.yaml
  21. Cargo.toml
  22. CHANGELOG.md
  23. CODE_OF_CONDUCT.md
  24. CONTRIBUTING.md
  25. header
  26. LICENSE.txt
  27. NOTICE.txt
  28. pre-commit.sh
  29. README.md
  30. rustfmt.toml
README.md

DataFusion

DataFusion is an extensible query planning, optimization, and execution framework, written in Rust, that uses Apache Arrow as its in-memory format.

Coverage Status

Features

  • SQL query planner with support for multiple SQL dialects
  • DataFrame API
  • Parquet, CSV, JSON, and Avro file formats are supported natively. Custom file formats can be supported by implementing a TableProvider trait.
  • Supports popular object stores, including AWS S3, Azure Blob Storage, and Google Cloud Storage. There are extension points for implementing custom object stores.

Use Cases

DataFusion is modular in design with many extension points and can be used without modification as an embedded query engine and can also provide a foundation for building new systems. Here are some example use cases:

  • DataFusion can be used as a SQL query planner and query optimizer, providing optimized logical plans that can then be mapped to other execution engines.
  • DataFusion is used to create modern, fast and efficient data pipelines, ETL processes, and database systems, which need the performance of Rust and Apache Arrow and want to provide their users the convenience of an SQL interface or a DataFrame API.

Why DataFusion?

  • High Performance: Leveraging Rust and Arrow's memory model, DataFusion achieves very high performance
  • Easy to Connect: Being part of the Apache Arrow ecosystem (Arrow, Parquet and Flight), DataFusion works well with the rest of the big data ecosystem
  • Easy to Embed: Allowing extension at almost any point in its design, DataFusion can be tailored for your specific use case
  • High Quality: Extensively tested, both by itself and with the rest of the Arrow ecosystem, DataFusion can be used as the foundation for production systems.

DataFusion Community Extensions

There are a number of community projects that extend DataFusion or provide integrations with other systems.

Language Bindings

Integrations

Known Uses

Here are some of the projects known to use DataFusion:

(if you know of another project, please submit a PR to add a link!)

Example Usage

Please see example usage to find how to use DataFusion.

Roadmap

Please see Roadmap for information of where the project is headed.

Architecture Overview

There is no formal document describing DataFusion's architecture yet, but the following presentations offer a good overview of its different components and how they interact together.

  • (July 2022): DataFusion and Arrow: Supercharge Your Data Analytical Tool with a Rusty Query Engine: recording and slides
  • (March 2021): The DataFusion architecture is described in Query Engine Design and the Rust-Based DataFusion in Apache Arrow: recording (DataFusion content starts ~ 15 minutes in) and slides
  • (February 2021): How DataFusion is used within the Ballista Project is described in *Ballista: Distributed Compute with Rust and Apache Arrow: recording

User Guide

Please see User Guide for more information about DataFusion.

Contributor Guide

Please see Contributor Guide for information about contributing to DataFusion.