DataFusion Examples

This crate includes several examples of how to use various DataFusion APIs and help you on your way.

Prerequisites:

Run git submodule update --init to init test files.

Single Process

• avro_sql.rs: Build and run a query plan from a SQL statement against a local AVRO file
• csv_sql.rs: Build and run a query plan from a SQL statement against a local CSV file
• custom_datasource.rs: Run queris against a custom datasource (TableProvider)
• dataframe.rs: Run a query using a DataFrame against a local parquet file
• dataframe_in_memory.rs: Run a query using a DataFrame against data in memory
• deserialize_to_struct.rs: Convert query results into rust structs using serde
• expr_api.rs: Use the Expr construction and simplification API
• memtable.rs: Create an query data in memory using SQL and RecordBatches
• parquet_sql.rs: Build and run a query plan from a SQL statement against a local Parquet file
• parquet_sql_multiple_files.rs: Build and run a query plan from a SQL statement against multiple local Parquet files
• query-aws-s3.rs: Confiure object_store and run a query against files stored in AWS S3
• rewrite_expr.rs: Define and invoke a custom Query Optimizer pass
• simple_udaf.rs: Define and invoke a User Defined Aggregate Function (UDAF)
• simple_udf.rs: Define and invoke a User Defined (scalar) Function (UDF)

Distributed

• flight_client.rs and flight_server.rs: Run DataFusion as a standalone process and execute SQL queries from a client using the Flight protocol.