DataFusion Examples

This crate includes end to end, highly commented examples of how to use various DataFusion APIs to help you get started.

Prerequisites:

Run git submodule update --init to init test files.

Running Examples

To run an example, use the cargo run command, such as:

git clone https://github.com/apache/datafusion
cd datafusion
# Download test data
git submodule update --init

# Change to the examples directory
cd datafusion-examples/examples

# Run the `dataframe` example:
# ... use the equivalent for other examples
cargo run --example dataframe

Single Process

• advanced_udaf.rs: Define and invoke a more complicated User Defined Aggregate Function (UDAF)
• advanced_udf.rs: Define and invoke a more complicated User Defined Scalar Function (UDF)
• advanced_udwf.rs: Define and invoke a more complicated User Defined Window Function (UDWF)
• advanced_parquet_index.rs: Creates a detailed secondary index that covers the contents of several parquet files
• analyzer_rule.rs: Use a custom AnalyzerRule to change a query's semantics (row level access control)
• catalog.rs: Register the table into a custom catalog
• composed_extension_codec: Example of using multiple extension codecs for serialization / deserialization
• csv_sql_streaming.rs: Build and run a streaming query plan from a SQL statement against a local CSV file
• custom_datasource.rs: Run queries against a custom datasource (TableProvider)
• custom_file_format.rs: Write data to a custom file format
• dataframe-to-s3.rs: Run a query using a DataFrame against a parquet file from s3 and writing back to s3
• dataframe.rs: Run a query using a DataFrame against a local parquet file
• dataframe_in_memory.rs: Run a query using a DataFrame against data in memory
• dataframe_output.rs: Examples of methods which write data out from a DataFrame
• deserialize_to_struct.rs: Convert query results into rust structs using serde
• expr_api.rs: Create, execute, simplify and analyze Exprs
• file_stream_provider.rs: Run a query on FileStreamProvider which implements StreamProvider for reading and writing to arbitrary stream sources / sinks.
• flight_sql_server.rs: Run DataFusion as a standalone process and execute SQL queries from JDBC clients
• function_factory.rs: Register CREATE FUNCTION handler to implement SQL macros
• make_date.rs: Examples of using the make_date function
• memtable.rs: Create an query data in memory using SQL and RecordBatches
• optimizer_rule.rs: Use a custom OptimizerRule to replace certain predicates
• parquet_index.rs: Create an secondary index over several parquet files and use it to speed up queries
• parquet_sql_multiple_files.rs: Build and run a query plan from a SQL statement against multiple local Parquet files
• parquet_exec_visitor.rs: Extract statistics by visiting an ExecutionPlan after execution
• parse_sql_expr.rs: Parse SQL text into DataFusion Expr.
• plan_to_sql.rs: Generate SQL from DataFusion Expr and LogicalPlan
• `planner_api.rs: APIs to manipulate logical and physical plans
• pruning.rs: Use pruning to rule out files based on statistics
• query-aws-s3.rs: Configure object_store and run a query against files stored in AWS S3
• query-http-csv.rs: Configure object_store and run a query against files vi HTTP
• regexp.rs: Examples of using regular expression functions
• simple_udaf.rs: Define and invoke a User Defined Aggregate Function (UDAF)
• simple_udf.rs: Define and invoke a User Defined Scalar Function (UDF)
• simple_udfw.rs: Define and invoke a User Defined Window Function (UDWF)
• sql_analysis.rs: Analyse SQL queries with DataFusion structures
• sql_frontend.rs: Create LogicalPlans (only) from sql strings
• sql_dialect.rs: Example of implementing a custom SQL dialect on top of DFParser
• to_char.rs: Examples of using the to_char function
• to_timestamp.rs: Examples of using to_timestamp functions

Distributed

• flight_client.rs and flight_server.rs: Run DataFusion as a standalone process and execute SQL queries from a client using the Flight protocol.