Apache Arrow DataFusion and Ballista query engines

Clone this repo:
  1. 2bcf040 Produce correct answers for Group BY NULL (Option 1) (#793) by Andrew Lamb · 3 hours ago master
  2. 979dc31 Convert unsupported conditions in left right join to filters (#796) by Daniël Heres · 11 hours ago
  3. 949bb0e automatically add sql, development-process and documentation labels to PRs (#800) by QP Hou · 12 hours ago
  4. 1173479 Move `hash_array` into hash_utils.rs (#807) by Andrew Lamb · 26 hours ago
  5. 3eac2e6 expand file glob within prettier (#803) by QP Hou · 28 hours ago

DataFusion

DataFusion is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.

DataFusion supports both an SQL and a DataFrame API for building logical query plans as well as a query optimizer and execution engine capable of parallel execution against partitioned data sources (CSV and Parquet) using threads.

DataFusion also supports distributed query execution via the Ballista crate.

Use Cases

DataFusion is used to create modern, fast and efficient data pipelines, ETL processes, and database systems, which need the performance of Rust and Apache Arrow and want to provide their users the convenience of an SQL interface or a DataFrame API.

Why DataFusion?

  • High Performance: Leveraging Rust and Arrow's memory model, DataFusion achieves very high performance
  • Easy to Connect: Being part of the Apache Arrow ecosystem (Arrow, Parquet and Flight), DataFusion works well with the rest of the big data ecosystem
  • Easy to Embed: Allowing extension at almost any point in its design, DataFusion can be tailored for your specific usecase
  • High Quality: Extensively tested, both by itself and with the rest of the Arrow ecosystem, DataFusion can be used as the foundation for production systems.

Known Uses

Here are some of the projects known to use DataFusion:

(if you know of another project, please submit a PR to add a link!)

Example Usage

Run a SQL query against data stored in a CSV:

use datafusion::prelude::*;
use datafusion::arrow::util::pretty::print_batches;
use datafusion::arrow::record_batch::RecordBatch;

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
  // register the table
  let mut ctx = ExecutionContext::new();
  ctx.register_csv("example", "tests/example.csv", CsvReadOptions::new())?;

  // create a plan to run a SQL query
  let df = ctx.sql("SELECT a, MIN(b) FROM example GROUP BY a LIMIT 100")?;

  // execute and print results
  let results: Vec<RecordBatch> = df.collect().await?;
  print_batches(&results)?;
  Ok(())
}

Use the DataFrame API to process data stored in a CSV:

use datafusion::prelude::*;
use datafusion::arrow::util::pretty::print_batches;
use datafusion::arrow::record_batch::RecordBatch;

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
  // create the dataframe
  let mut ctx = ExecutionContext::new();
  let df = ctx.read_csv("tests/example.csv", CsvReadOptions::new())?;

  let df = df.filter(col("a").lt_eq(col("b")))?
          .aggregate(vec![col("a")], vec![min(col("b"))])?
          .limit(100)?;

  // execute and print results
  let results: Vec<RecordBatch> = df.collect().await?;
  print_batches(&results)?;
  Ok(())
}

Both of these examples will produce

+---+--------+
| a | MIN(b) |
+---+--------+
| 1 | 2      |
+---+--------+

Using DataFusion as a library

DataFusion is published on crates.io, and is well documented on docs.rs.

To get started, add the following to your Cargo.toml file:

[dependencies]
datafusion = "4.0.0-SNAPSHOT"

Using DataFusion as a binary

DataFusion also includes a simple command-line interactive SQL utility. See the CLI reference for more information.

Status

General

  • [x] SQL Parser
  • [x] SQL Query Planner
  • [x] Query Optimizer
  • [x] Constant folding
  • [x] Join Reordering
  • [x] Limit Pushdown
  • [x] Projection push down
  • [x] Predicate push down
  • [x] Type coercion
  • [x] Parallel query execution

SQL Support

  • [x] Projection
  • [x] Filter (WHERE)
  • [x] Filter post-aggregate (HAVING)
  • [x] Limit
  • [x] Aggregate
  • [x] Common math functions
  • [x] cast
  • [x] try_cast
  • Postgres compatible String functions
    • [x] ascii
    • [x] bit_length
    • [x] btrim
    • [x] char_length
    • [x] character_length
    • [x] chr
    • [x] concat
    • [x] concat_ws
    • [x] initcap
    • [x] left
    • [x] length
    • [x] lpad
    • [x] ltrim
    • [x] octet_length
    • [x] regexp_replace
    • [x] repeat
    • [x] replace
    • [x] reverse
    • [x] right
    • [x] rpad
    • [x] rtrim
    • [x] split_part
    • [x] starts_with
    • [x] strpos
    • [x] substr
    • [x] to_hex
    • [x] translate
    • [x] trim
  • Miscellaneous/Boolean functions
    • [x] nullif
  • Common date/time functions
  • nested functions
    • [x] Array of columns
  • [x] Schema Queries
    • [x] SHOW TABLES
    • [x] SHOW COLUMNS
    • [x] information_schema.{tables, columns}
    • [ ] information_schema other views
  • [x] Sorting
  • [ ] Nested types
  • [ ] Lists
  • [x] Subqueries
  • [x] Common table expressions
  • [ ] Set Operations
    • [x] UNION ALL
    • [ ] UNION
    • [ ] INTERSECT
    • [ ] MINUS
  • [x] Joins
    • [x] INNER JOIN
    • [x] LEFT JOIN
    • [x] RIGHT JOIN
    • [x] FULL JOIN
    • [x] CROSS JOIN
  • [ ] Window
    • [x] Empty window
    • [x] Common window functions
    • [x] Window with PARTITION BY clause
    • [x] Window with ORDER BY clause
    • [ ] Window with FILTER clause
    • [ ] Window with custom WINDOW FRAME
    • [ ] UDF and UDAF for window functions

Data Sources

  • [x] CSV
  • [x] Parquet primitive types
  • [ ] Parquet nested types

Extensibility

DataFusion is designed to be extensible at all points. To that end, you can provide your own custom:

  • [x] User Defined Functions (UDFs)
  • [x] User Defined Aggregate Functions (UDAFs)
  • [x] User Defined Table Source (TableProvider) for tables
  • [x] User Defined Optimizer passes (plan rewrites)
  • [x] User Defined LogicalPlan nodes
  • [x] User Defined ExecutionPlan nodes

Supported SQL

This library currently supports many SQL constructs, including

  • CREATE EXTERNAL TABLE X STORED AS PARQUET LOCATION '...'; to register a table's locations
  • SELECT ... FROM ... together with any expression
  • ALIAS to name an expression
  • CAST to change types, including e.g. Timestamp(Nanosecond, None)
  • most mathematical unary and binary expressions such as +, /, sqrt, tan, >=.
  • WHERE to filter
  • GROUP BY together with one of the following aggregations: MIN, MAX, COUNT, SUM, AVG
  • ORDER BY together with an expression and optional ASC or DESC and also optional NULLS FIRST or NULLS LAST

Supported Functions

DataFusion strives to implement a subset of the PostgreSQL SQL dialect where possible. We explicitly choose a single dialect to maximize interoperability with other tools and allow reuse of the PostgreSQL documents and tutorials as much as possible.

Currently, only a subset of the PostgreSQL dialect is implemented, and we will document any deviations.

Schema Metadata / Information Schema Support

DataFusion supports the showing metadata about the tables available. This information can be accessed using the views of the ISO SQL information_schema schema or the DataFusion specific SHOW TABLES and SHOW COLUMNS commands.

More information can be found in the Postgres docs).

To show tables available for use in DataFusion, use the SHOW TABLES command or the information_schema.tables view:

> show tables;
+---------------+--------------------+------------+------------+
| table_catalog | table_schema       | table_name | table_type |
+---------------+--------------------+------------+------------+
| datafusion    | public             | t          | BASE TABLE |
| datafusion    | information_schema | tables     | VIEW       |
+---------------+--------------------+------------+------------+

> select * from information_schema.tables;

+---------------+--------------------+------------+--------------+
| table_catalog | table_schema       | table_name | table_type   |
+---------------+--------------------+------------+--------------+
| datafusion    | public             | t          | BASE TABLE   |
| datafusion    | information_schema | TABLES     | SYSTEM TABLE |
+---------------+--------------------+------------+--------------+

To show the schema of a table in DataFusion, use the SHOW COLUMNS command or the or information_schema.columns view:

> show columns from t;
+---------------+--------------+------------+-------------+-----------+-------------+
| table_catalog | table_schema | table_name | column_name | data_type | is_nullable |
+---------------+--------------+------------+-------------+-----------+-------------+
| datafusion    | public       | t          | a           | Int32     | NO          |
| datafusion    | public       | t          | b           | Utf8      | NO          |
| datafusion    | public       | t          | c           | Float32   | NO          |
+---------------+--------------+------------+-------------+-----------+-------------+

>   select table_name, column_name, ordinal_position, is_nullable, data_type from information_schema.columns;
+------------+-------------+------------------+-------------+-----------+
| table_name | column_name | ordinal_position | is_nullable | data_type |
+------------+-------------+------------------+-------------+-----------+
| t          | a           | 0                | NO          | Int32     |
| t          | b           | 1                | NO          | Utf8      |
| t          | c           | 2                | NO          | Float32   |
+------------+-------------+------------------+-------------+-----------+

Supported Data Types

DataFusion uses Arrow, and thus the Arrow type system, for query execution. The SQL types from sqlparser-rs are mapped to Arrow types according to the following table

SQL Data TypeArrow DataType
CHARUtf8
VARCHARUtf8
UUIDNot yet supported
CLOBNot yet supported
BINARYNot yet supported
VARBINARYNot yet supported
DECIMALFloat64
FLOATFloat32
SMALLINTInt16
INTInt32
BIGINTInt64
REALFloat64
DOUBLEFloat64
BOOLEANBoolean
DATEDate32
TIMETime64(TimeUnit::Millisecond)
TIMESTAMPTimestamp(TimeUnit::Nanosecond)
INTERVALNot yet supported
REGCLASSNot yet supported
TEXTNot yet supported
BYTEANot yet supported
CUSTOMNot yet supported
ARRAYNot yet supported

Architecture Overview

There is no formal document describing DataFusion's architecture yet, but the following presentations offer a good overview of its different components and how they interact together.

  • (March 2021): The DataFusion architecture is described in Query Engine Design and the Rust-Based DataFusion in Apache Arrow: recording (DataFusion content starts ~ 15 minutes in) and slides
  • (Feburary 2021): How DataFusion is used within the Ballista Project is described in *Ballista: Distributed Compute with Rust and Apache Arrow: recording

Developer's guide

Please see Developers Guide for information about developing DataFusion.