DataFusion is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.
DataFusion supports both an SQL and a DataFrame API for building logical query plans as well as a query optimizer and execution engine capable of parallel execution against partitioned data sources (CSV and Parquet) using threads.
DataFusion is used to create modern, fast and efficient data pipelines, ETL processes, and database systems, which need the performance of Rust and Apache Arrow and want to provide their users the convenience of an SQL interface or a DataFrame API.
Here are some of the projects known to use DataFusion:
(if you know of another project, please submit a PR to add a link!)
Run a SQL query against data stored in a CSV:
use datafusion::prelude::*; use arrow::util::pretty::print_batches; use arrow::record_batch::RecordBatch; #[tokio::main] async fn main() -> datafusion::error::Result<()> { // register the table let mut ctx = ExecutionContext::new(); ctx.register_csv("example", "tests/example.csv", CsvReadOptions::new())?; // create a plan to run a SQL query let df = ctx.sql("SELECT a, MIN(b) FROM example GROUP BY a LIMIT 100")?; // execute and print results let results: Vec<RecordBatch> = df.collect().await?; print_batches(&results)?; Ok(()) }
Use the DataFrame API to process data stored in a CSV:
use datafusion::prelude::*; use arrow::util::pretty::print_batches; use arrow::record_batch::RecordBatch; #[tokio::main] async fn main() -> datafusion::error::Result<()> { // create the dataframe let mut ctx = ExecutionContext::new(); let df = ctx.read_csv("tests/example.csv", CsvReadOptions::new())?; let df = df.filter(col("a").lt_eq(col("b")))? .aggregate(vec![col("a")], vec![min(col("b"))])? .limit(100)?; // execute and print results let results: Vec<RecordBatch> = df.collect().await?; print_batches(&results)?; Ok(()) }
Both of these examples will produce
+---+--------+ | a | MIN(b) | +---+--------+ | 1 | 2 | +---+--------+
DataFusion is published on crates.io, and is well documented on docs.rs.
To get started, add the following to your Cargo.toml
file:
[dependencies] datafusion = "4.0.0-SNAPSHOT"
DataFusion also includes a simple command-line interactive SQL utility. See the CLI reference for more information.
DataFusion is designed to be extensible at all points. To that end, you can provide your own custom:
TableProvider
) for tablesOptimizer
passes (plan rewrites)LogicalPlan
nodesExecutionPlan
nodesThis library currently supports many SQL constructs, including
CREATE EXTERNAL TABLE X STORED AS PARQUET LOCATION '...';
to register a table's locationsSELECT ... FROM ...
together with any expressionALIAS
to name an expressionCAST
to change types, including e.g. Timestamp(Nanosecond, None)
+
, /
, sqrt
, tan
, >=
.WHERE
to filterGROUP BY
together with one of the following aggregations: MIN
, MAX
, COUNT
, SUM
, AVG
ORDER BY
together with an expression and optional ASC
or DESC
and also optional NULLS FIRST
or NULLS LAST
DataFusion strives to implement a subset of the PostgreSQL SQL dialect where possible. We explicitly choose a single dialect to maximize interoperability with other tools and allow reuse of the PostgreSQL documents and tutorials as much as possible.
Currently, only a subset of the PosgreSQL dialect is implemented, and we will document any deviations.
DataFusion supports the showing metadata about the tables available. This information can be accessed using the views of the ISO SQL information_schema
schema or the DataFusion specific SHOW TABLES
and SHOW COLUMNS
commands.
More information can be found in the Postgres docs).
To show tables available for use in DataFusion, use the SHOW TABLES
command or the information_schema.tables
view:
> show tables; +---------------+--------------------+------------+------------+ | table_catalog | table_schema | table_name | table_type | +---------------+--------------------+------------+------------+ | datafusion | public | t | BASE TABLE | | datafusion | information_schema | tables | VIEW | +---------------+--------------------+------------+------------+ > select * from information_schema.tables; +---------------+--------------------+------------+--------------+ | table_catalog | table_schema | table_name | table_type | +---------------+--------------------+------------+--------------+ | datafusion | public | t | BASE TABLE | | datafusion | information_schema | TABLES | SYSTEM TABLE | +---------------+--------------------+------------+--------------+
To show the schema of a table in DataFusion, use the SHOW COLUMNS
command or the or information_schema.columns
view:
> show columns from t; +---------------+--------------+------------+-------------+-----------+-------------+ | table_catalog | table_schema | table_name | column_name | data_type | is_nullable | +---------------+--------------+------------+-------------+-----------+-------------+ | datafusion | public | t | a | Int32 | NO | | datafusion | public | t | b | Utf8 | NO | | datafusion | public | t | c | Float32 | NO | +---------------+--------------+------------+-------------+-----------+-------------+ > select table_name, column_name, ordinal_position, is_nullable, data_type from information_schema.columns; +------------+-------------+------------------+-------------+-----------+ | table_name | column_name | ordinal_position | is_nullable | data_type | +------------+-------------+------------------+-------------+-----------+ | t | a | 0 | NO | Int32 | | t | b | 1 | NO | Utf8 | | t | c | 2 | NO | Float32 | +------------+-------------+------------------+-------------+-----------+
DataFusion uses Arrow, and thus the Arrow type system, for query execution. The SQL types from sqlparser-rs are mapped to Arrow types according to the following table
SQL Data Type | Arrow DataType |
---|---|
CHAR | Utf8 |
VARCHAR | Utf8 |
UUID | Not yet supported |
CLOB | Not yet supported |
BINARY | Not yet supported |
VARBINARY | Not yet supported |
DECIMAL | Float64 |
FLOAT | Float32 |
SMALLINT | Int16 |
INT | Int32 |
BIGINT | Int64 |
REAL | Float64 |
DOUBLE | Float64 |
BOOLEAN | Boolean |
DATE | Date32 |
TIME | Time64(TimeUnit::Millisecond) |
TIMESTAMP | Date64 |
INTERVAL | Not yet supported |
REGCLASS | Not yet supported |
TEXT | Not yet supported |
BYTEA | Not yet supported |
CUSTOM | Not yet supported |
ARRAY | Not yet supported |
There is no formal document describing DataFusion's architecture yet, but the following presentations offer a good overview of its different components and how they interact together.
Please see Developers Guide for information about developing DataFusion.