Update datafusion dependency to latest in preparation for DF54 (#1532) * feat: upgrade upstream DataFusion 53 → main (pre-54) Bump workspace deps to apache/datafusion@3d06bedc (git pin) in preparation for the 54.0.0 release. Workspace package version moves to 54.0.0 to track the upstream major convention. Compile fixes: - Drop as_any impls (trait now has Any as supertrait) and use the upstream-provided downcast_ref helper on dyn trait objects. - Reconcile FFI provider From conversions to drop redundant `+ Send` on Arc<dyn ...> bounds. - Cast/TryCast: data_type → field.data_type() (FieldRef rename). - Stub match arms for new Expr::HigherOrderFunction / Lambda / LambdaVariable and ScalarValue::ListView / LargeListView variants; proper exposure deferred to PR 3 audit. - DatasetExec: partition_statistics returns Arc<Statistics>; add required apply_expressions trait method. - Suppress TableFunctionImpl::call deprecation pending call_with_args refactor that needs Session plumbing. User-facing test updates for upstream behavior changes: - median / approx_median / approx_percentile_cont now return Float64. - String functions (concat_ws, lower, upper, repeat, reverse, split_part, translate) return StringView when given StringView. - overlay appends past end-of-string rather than replacing the input. - arrays_zip / list_zip struct field names "c0"/"c1" → "1"/"2". - Filter on mismatched cast types now errors (was 0 matches). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: expose DataFrame.alias and tidy public API after DF53→54 audit Companion to the upstream DataFusion 53 → main bump. The check-upstream audit (PR 3 of dev/release/upstream-sync.md) surfaced a small set of trivial wins; this commit ships them. Trivial wins: - DataFrame.alias(name) — wraps the logical plan in a SubqueryAlias. - functions.__all__: add `instr` and `position` (both were defined as public defs but missing from `__all__`, so they didn't show up in `from datafusion.functions import *` or generated docs). - top-level `datafusion.__all__`: re-export `TableProviderFactory` and `TableProviderFactoryExportable` (previously only reachable via the `datafusion.catalog` submodule). Non-trivial gaps surfaced by the audit (DataFrame.registry, into_*/task_ctx, SessionContext extensibility surface, distinct-aware aggregate variants, TableFunctionImpl::call_with_args migration, FFI Protocol pipeline gaps) are deferred — each warrants its own design and PR. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * taplo fmt * Update unit test to go along with https://github.com/apache/datafusion/pull/22133 * docs: demonstrate alias via self-join in DataFrame.alias example Prior example called alias("t") then to_pydict(), which did not show the qualifier effect. Replace with a self-join that uses col("l.val") and col("r.val") so the disambiguation behavior is visible. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: wrap higher-order, lambda, and lambda-variable Expr variants DataFusion 54 introduces Expr::HigherOrderFunction, Expr::Lambda, and Expr::LambdaVariable. PyExpr::to_variant previously errored on each with py_unsupported_variant_err. Add PyHigherOrderFunction, PyLambda, and PyLambdaVariable wrappers, register them in the expr pymodule and re-export from python/datafusion/expr.py, and dispatch to_variant to the new wrappers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: wire rex_type and rex_call_operands for new Expr variants Map HigherOrderFunction and Lambda to RexType::Call; LambdaVariable to RexType::Reference. In rex_call_operands return the args for HigherOrderFunction, the body for Lambda, and self for LambdaVariable (mirroring Column). In rex_call_operator return the underlying UDF name for HigherOrderFunction and the literal "lambda" for Lambda. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: support LargeList/ListView/LargeListView in map_from_scalar_to_arrow These ScalarValue variants all wrap Arc<...Array>, exposing the outer DataType via Array::data_type(), so we can mirror the existing ScalarValue::List arm instead of returning PyNotImplementedError. This makes Expr.types() work for plans that round-trip through SQL or proto where these scalar variants surface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: switch PyTableFunction to non-deprecated call_with_args DataFusion 53.0.0 deprecated TableFunctionImpl::call in favor of call_with_args(args: TableFunctionArgs), which threads a Session reference alongside the exprs. Implement call_with_args on PyTableFunction (delegating to the FFI variant's call_with_args, or ignoring the session for the pure-Python variant which doesn't use it) and have __call__ build a TableFunctionArgs from the global session. Drops both #[allow(deprecated)] attributes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * build: revert workspace version to 53.0.0 and move DF overrides to [patch.crates-io] The workspace version was prematurely bumped to 54.0.0 in the DF53→pre-54 upgrade. Restore it to 53.0.0 until we are actually ready to cut the 54 release. The same change had moved every datafusion-* dependency from a crates.io version constraint to a direct git dep in [workspace.dependencies]. Switch them back to "version = \"53\"" and move the git rev overrides into [patch.crates-io] so the published manifest will be patch-free. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * taplo format * test: sort FFI test results by partition key before equality compare Multi-partition `collect()` returns batches in execution-scheduling order, which is non-deterministic and differs between local and CI runners. Sort by the first value of column 0 (unique per partition in each affected test) so the expected/actual comparison is stable. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Bump datafusion main commit * test: cover new DF54 expr wrappers, catalog factories, and DataFrame.alias Add module-metadata checks for HigherOrderFunction, Lambda, LambdaVariable and the top-level TableProviderFactory / TableProviderFactoryExportable re-exports, plus a self-join regression test exercising the new DataFrame.alias() qualifier-based selection path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This is a Python library that binds to Apache Arrow in-memory query engine DataFusion.
DataFusion's Python bindings can be used as a foundation for building new data systems in Python. Here are some examples:
For tips on tuning parallelism, see Maximizing CPU Usage in the configuration guide.
The following example demonstrates running a SQL query against a Parquet file using DataFusion, storing the results in a Pandas DataFrame, and then plotting a chart.
The Parquet file used in this example can be downloaded from the following page:
from datafusion import SessionContext # Create a DataFusion context ctx = SessionContext() # Register table with context ctx.register_parquet('taxi', 'yellow_tripdata_2021-01.parquet') # Execute SQL df = ctx.sql("select passenger_count, count(*) " "from taxi " "where passenger_count is not null " "group by passenger_count " "order by passenger_count") # convert to Pandas pandas_df = df.to_pandas() # create a chart fig = pandas_df.plot(kind="bar", title="Trip Count by Number of Passengers").get_figure() fig.savefig('chart.png')
This produces the following chart:
You can use SessionContext's register_view method to convert a DataFrame into a view and register it with the context.
from datafusion import SessionContext, col, literal # Create a DataFusion context ctx = SessionContext() # Create sample data data = {"a": [1, 2, 3, 4, 5], "b": [10, 20, 30, 40, 50]} # Create a DataFrame from the dictionary df = ctx.from_pydict(data, "my_table") # Filter the DataFrame (for example, keep rows where a > 2) df_filtered = df.filter(col("a") > literal(2)) # Register the dataframe as a view with the context ctx.register_view("view1", df_filtered) # Now run a SQL query against the registered view df_view = ctx.sql("SELECT * FROM view1") # Collect the results results = df_view.collect() # Convert results to a list of dictionaries for display result_dicts = [batch.to_pydict() for batch in results] print(result_dicts)
This will output:
[{'a': [3, 4, 5], 'b': [30, 40, 50]}]
It is possible to configure runtime (memory and disk settings) and configuration settings when creating a context.
runtime = ( RuntimeEnvBuilder() .with_disk_manager_os() .with_fair_spill_pool(10000000) ) config = ( SessionConfig() .with_create_default_catalog_and_schema(True) .with_default_catalog_and_schema("foo", "bar") .with_target_partitions(8) .with_information_schema(True) .with_repartition_joins(False) .with_repartition_aggregations(False) .with_repartition_windows(False) .with_parquet_pruning(False) .set("datafusion.execution.parquet.pushdown_filters", "true") ) ctx = SessionContext(config, runtime)
Refer to the API documentation for more information.
Printing the context will show the current configuration settings.
print(ctx)
For information about how to extend DataFusion Python, please see the extensions page of the online documentation.
See examples for more information.
uv add datafusion
pip install datafusion # or python -m pip install datafusion
conda install -c conda-forge datafusion
You can verify the installation by running:
>>> import datafusion >>> datafusion.__version__ '0.6.0'
This project ships a SKILL.md that teaches AI coding assistants how to write idiomatic DataFusion Python. It follows the Agent Skills open standard.
Preferred: npx skills add apache/datafusion-python — installs the skill in Claude Code, Cursor, Windsurf, Cline, Codex, Copilot, Gemini CLI, and other supported agents.
Manual: paste this line into your project's AGENTS.md / CLAUDE.md:
For DataFusion Python code, see https://github.com/apache/datafusion-python/blob/main/skills/datafusion_python/SKILL.md
This assumes that you have rust and cargo installed. We use the workflow recommended by pyo3 and maturin. The Maturin tools used in this workflow can be installed either via uv or pip. Both approaches should offer the same experience. It is recommended to use uv since it has significant performance improvements over pip.
Currently for protobuf support either protobuf or cmake must be installed.
Bootstrap (uv):
By default uv will attempt to build the datafusion python package. For our development we prefer to build manually. This means that when creating your virtual environment using uv sync you need to pass in the additional --no-install-package datafusion and for uv run commands the additional parameter --no-project
# fetch this repo git clone git@github.com:apache/datafusion-python.git # cd to the repo root cd datafusion-python/ # create the virtual environment uv sync --dev --no-install-package datafusion # activate the environment source .venv/bin/activate
Bootstrap (pip):
# fetch this repo git clone git@github.com:apache/datafusion-python.git # cd to the repo root cd datafusion-python/ # prepare development environment (used to build wheel / install in development) python3 -m venv .venv # activate the venv source .venv/bin/activate # update pip itself if necessary python -m pip install -U pip # install dependencies python -m pip install -r pyproject.toml
The tests rely on test data in git submodules.
git submodule update --init
Whenever rust code changes (your changes or via git pull):
# make sure you activate the venv using "source venv/bin/activate" first maturin develop --uv python -m pytest
Alternatively if you are using uv you can do the following without needing to activate the virtual environment:
uv run --no-project maturin develop --uv uv run --no-project pytest
To run the FFI tests within the examples folder, after you have built datafusion-python with the previous commands:
cd examples/datafusion-ffi-example uv run --no-project maturin develop --uv uv run --no-project pytest python/tests/_test_*py
datafusion-python takes advantage of pre-commit to assist developers with code linting to help reduce the number of commits that ultimately fail in CI due to linter errors. Using the pre-commit hooks is optional for the developer but certainly helpful for keeping PRs clean and concise.
Our pre-commit hooks can be installed by running pre-commit install, which will install the configurations in your DATAFUSION_PYTHON_ROOT/.github directory and run each time you perform a commit, failing to complete the commit if an offending lint is found allowing you to make changes locally before pushing.
The pre-commit hooks can also be run adhoc without installing them by simply running pre-commit run --all-files.
NOTE: the current pre-commit hooks require docker, and cmake. See note on protobuf above.
There are scripts in ci/scripts for running Rust and Python linters.
./ci/scripts/python_lint.sh ./ci/scripts/rust_clippy.sh ./ci/scripts/rust_fmt.sh ./ci/scripts/rust_toml_fmt.sh
This project includes an AI agent skill for auditing which features from the upstream Apache DataFusion Rust library are not yet exposed in these Python bindings. This is useful when adding missing functions, auditing API coverage, or ensuring parity with upstream.
The skill accepts an optional area argument:
scalar functions aggregate functions window functions dataframe session context ffi types all
If no argument is provided, it defaults to checking all areas. The skill will fetch the upstream DataFusion documentation, compare it against the functions and methods exposed in this project, and produce a coverage report listing what is currently exposed and what is missing.
The skill definition lives in .ai/skills/check-upstream/SKILL.md and follows the Agent Skills open standard. It can be used by any AI coding agent that supports skill discovery, or followed manually.
To change test dependencies, change the pyproject.toml and run
uv sync --dev --no-install-package datafusion