DataFusion Python Examples

Some examples rely on data which can be downloaded from the following site:

• https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page

Here is a direct link to the file used in the examples:

• https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2021-01.parquet

Creating a SessionContext

• Creating a SessionContext

Executing Queries with DataFusion

• Query a Parquet file using SQL
• Query a Parquet file using the DataFrame API
• Run a SQL query and store the results in a Pandas DataFrame
• Query PyArrow Data

Running User-Defined Python Code

• Register a Python UDF with DataFusion
• Register a Python UDAF with DataFusion

Substrait Support

• Serialize query plans using Substrait

Executing SQL against DataFrame Libraries (Experimental)

• Executing SQL on Polars
• Executing SQL on Pandas
• Executing SQL on cuDF