feat: create free-threaded python wheels (#1553) * Initial commit for free threaded python support * ci: use uvx to run maturin in native wheel builds The free-threaded matrix entries skip `uv sync` to avoid resolving project dependencies against cp313t/cp314t (many dev deps lack free-threaded wheels), so `uv run --no-project maturin` failed on macOS/Windows with "Failed to spawn: `maturin`". Switch to `uvx maturin@1.8.1`, which runs maturin in an isolated tool env independent of the project venv and matches the pin used by maturin-action for manylinux builds. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ci: resolve free-threaded interpreter path explicitly on Windows maturin's `--interpreter python3.14t` fails on Windows because the free-threaded build ships as plain `python.exe` (no `tN` suffix). Look up `sys.executable` of the python on PATH (which actions/setup-python prepends with the free-threaded install), assert `Py_GIL_DISABLED == 1` so a misconfigured PATH can't silently build a GIL wheel, and normalize backslashes to forward slashes so the path survives re-expansion in the downstream `run:` line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * build: enable PyO3 generate-import-lib for Windows free-threaded wheels Windows free-threaded Python does not expose `abiflags` in sysconfig, so PyO3's default Windows linkage path fails with "A python 3 interpreter on Windows does not define abiflags in its sysconfig ಠ_ಠ" when building cp31Xt wheels. Enabling the `generate-import-lib` PyO3 feature switches Windows builds to a generated import library (provided by the `python3-dll-a` crate) that does not depend on a fully populated sysconfig. It is a no-op on macOS and Linux and is compatible with the existing `abi3` feature. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ci: bump maturin to 1.13.3 for Windows free-threaded support maturin 1.8.1 errors out on Windows free-threaded interpreters with "A python 3 interpreter on Windows does not define abiflags in its sysconfig" even when given a valid `python.exe`. Newer maturin releases handle the missing abiflags gracefully for cp31Xt builds. Bump both the `uvx maturin@` pin used for native macOS/Windows wheels and the `maturin-version` passed to PyO3/maturin-action for the manylinux containers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ci: standardize wheel build job names as "<OS> <arch> (<tag>)" The mac/Windows matrix shared a single name template that prepended "macOS arm64 & Windows" to every entry, which got truncated in the GitHub UI sidebar and made it hard to tell macOS and Windows runs apart. Rename all wheel build jobs to the same pattern so the OS, architecture, and python tag are visible at a glance: - Linux x86_64 / arm64 - macOS arm64 / x86_64 - Windows x86_64 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * taplo fmt * build: move pygithub to release group to fix free-threaded wheel builds pygithub pulls in cryptography via pyjwt[crypto]. cryptography 44.0.0 ships only abi3 wheels, which free-threaded interpreters cannot use, so uv builds it from sdist; its bundled PyO3 0.23.2 caps at Python 3.13 and fails on 3.14t. pygithub is only used by the manual release changelog script, so move it out of the dev group into a new release group. 'uv sync --dev' (used by CI test jobs) no longer drags in cryptography. * ci: pin uv venv to setup-python interpreter for free-threaded jobs Passing a bare version like '3.13t' to 'uv venv --python' let uv fall back to a different system interpreter (3.12), creating a venv whose ABI did not match the downloaded cp313t wheel and failing the install. Use the python-path output from setup-python so the venv uses exactly the interpreter that was set up. * taplo fmt * ci: set UV_PYTHON so uv sync keeps the free-threaded interpreter Pinning only 'uv venv --python' was not enough: 'uv sync' ignores the existing .venv, runs its own interpreter discovery, and recreated the venv with the system 3.12, again mismatching the cp313t wheel. Set UV_PYTHON to the setup-python interpreter for the install and test steps so every uv command (venv, sync, pip, run) uses it. * ci: run tests from the .venv, not the bare setup-python interpreter Setting UV_PYTHON on the test step pointed 'uv run --no-project pytest' at the setup-python interpreter, which has no pytest installed, causing 'Failed to spawn: pytest'. UV_PYTHON is only needed in the install step to build the .venv with the right interpreter; the test step must use that .venv. Drop UV_PYTHON from the test step. Co-Authored-By: Claude <noreply@anthropic.com> * ci: install datafusion wheel into the activated .venv Setting UV_PYTHON as a step env split the install across two environments: 'uv sync' populated .venv while 'uv pip install' targeted the bare setup-python interpreter, so the datafusion wheel never landed in .venv and 'import datafusion' failed under pytest. Pin the interpreter at 'uv venv --python', activate the venv, and pass --active to 'uv sync' so sync and pip install both target the same .venv. Co-Authored-By: Claude <noreply@anthropic.com> * ci: point uv at the venv interpreter by path for free-threaded jobs Activating the venv and passing --active still let 'uv sync' run its own interpreter discovery, which skips free-threaded builds and re-picked the system 3.12, recreating .venv and breaking the cp313t/cp314t wheel install. Pass the venv's own interpreter (.venv/bin/python) explicitly to 'uv sync', 'uv pip install', and 'uv run' so every step stays in the free-threaded environment created by 'uv venv'. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This is a Python library that binds to Apache Arrow in-memory query engine DataFusion.
DataFusion's Python bindings can be used as a foundation for building new data systems in Python. Here are some examples:
For tips on tuning parallelism, see Maximizing CPU Usage in the configuration guide.
The following example demonstrates running a SQL query against a Parquet file using DataFusion, storing the results in a Pandas DataFrame, and then plotting a chart.
The Parquet file used in this example can be downloaded from the following page:
from datafusion import SessionContext # Create a DataFusion context ctx = SessionContext() # Register table with context ctx.register_parquet('taxi', 'yellow_tripdata_2021-01.parquet') # Execute SQL df = ctx.sql("select passenger_count, count(*) " "from taxi " "where passenger_count is not null " "group by passenger_count " "order by passenger_count") # convert to Pandas pandas_df = df.to_pandas() # create a chart fig = pandas_df.plot(kind="bar", title="Trip Count by Number of Passengers").get_figure() fig.savefig('chart.png')
This produces the following chart:
You can use SessionContext's register_view method to convert a DataFrame into a view and register it with the context.
from datafusion import SessionContext, col, literal # Create a DataFusion context ctx = SessionContext() # Create sample data data = {"a": [1, 2, 3, 4, 5], "b": [10, 20, 30, 40, 50]} # Create a DataFrame from the dictionary df = ctx.from_pydict(data, "my_table") # Filter the DataFrame (for example, keep rows where a > 2) df_filtered = df.filter(col("a") > literal(2)) # Register the dataframe as a view with the context ctx.register_view("view1", df_filtered) # Now run a SQL query against the registered view df_view = ctx.sql("SELECT * FROM view1") # Collect the results results = df_view.collect() # Convert results to a list of dictionaries for display result_dicts = [batch.to_pydict() for batch in results] print(result_dicts)
This will output:
[{'a': [3, 4, 5], 'b': [30, 40, 50]}]
It is possible to configure runtime (memory and disk settings) and configuration settings when creating a context.
runtime = ( RuntimeEnvBuilder() .with_disk_manager_os() .with_fair_spill_pool(10000000) ) config = ( SessionConfig() .with_create_default_catalog_and_schema(True) .with_default_catalog_and_schema("foo", "bar") .with_target_partitions(8) .with_information_schema(True) .with_repartition_joins(False) .with_repartition_aggregations(False) .with_repartition_windows(False) .with_parquet_pruning(False) .set("datafusion.execution.parquet.pushdown_filters", "true") ) ctx = SessionContext(config, runtime)
Refer to the API documentation for more information.
Printing the context will show the current configuration settings.
print(ctx)
For information about how to extend DataFusion Python, please see the extensions page of the online documentation.
See examples for more information.
uv add datafusion
pip install datafusion # or python -m pip install datafusion
conda install -c conda-forge datafusion
You can verify the installation by running:
>>> import datafusion >>> datafusion.__version__ '0.6.0'
This project ships a SKILL.md that teaches AI coding assistants how to write idiomatic DataFusion Python. It follows the Agent Skills open standard.
Preferred: npx skills add apache/datafusion-python — installs the skill in Claude Code, Cursor, Windsurf, Cline, Codex, Copilot, Gemini CLI, and other supported agents.
Manual: paste this line into your project's AGENTS.md / CLAUDE.md:
For DataFusion Python code, see https://github.com/apache/datafusion-python/blob/main/skills/datafusion_python/SKILL.md
This assumes that you have rust and cargo installed. We use the workflow recommended by pyo3 and maturin. The Maturin tools used in this workflow can be installed either via uv or pip. Both approaches should offer the same experience. It is recommended to use uv since it has significant performance improvements over pip.
Currently for protobuf support either protobuf or cmake must be installed.
Bootstrap (uv):
By default uv will attempt to build the datafusion python package. For our development we prefer to build manually. This means that when creating your virtual environment using uv sync you need to pass in the additional --no-install-package datafusion and for uv run commands the additional parameter --no-project
# fetch this repo git clone git@github.com:apache/datafusion-python.git # cd to the repo root cd datafusion-python/ # create the virtual environment uv sync --dev --no-install-package datafusion # activate the environment source .venv/bin/activate
Bootstrap (pip):
# fetch this repo git clone git@github.com:apache/datafusion-python.git # cd to the repo root cd datafusion-python/ # prepare development environment (used to build wheel / install in development) python3 -m venv .venv # activate the venv source .venv/bin/activate # update pip itself if necessary python -m pip install -U pip # install dependencies python -m pip install -r pyproject.toml
The tests rely on test data in git submodules.
git submodule update --init
Whenever rust code changes (your changes or via git pull):
# make sure you activate the venv using "source venv/bin/activate" first maturin develop --uv python -m pytest
Alternatively if you are using uv you can do the following without needing to activate the virtual environment:
uv run --no-project maturin develop --uv uv run --no-project pytest
To run the FFI tests within the examples folder, after you have built datafusion-python with the previous commands:
cd examples/datafusion-ffi-example uv run --no-project maturin develop --uv uv run --no-project pytest python/tests/_test_*py
datafusion-python takes advantage of pre-commit to assist developers with code linting to help reduce the number of commits that ultimately fail in CI due to linter errors. Using the pre-commit hooks is optional for the developer but certainly helpful for keeping PRs clean and concise.
Our pre-commit hooks can be installed by running pre-commit install, which will install the configurations in your DATAFUSION_PYTHON_ROOT/.github directory and run each time you perform a commit, failing to complete the commit if an offending lint is found allowing you to make changes locally before pushing.
The pre-commit hooks can also be run adhoc without installing them by simply running pre-commit run --all-files.
NOTE: the current pre-commit hooks require docker, and cmake. See note on protobuf above.
There are scripts in ci/scripts for running Rust and Python linters.
./ci/scripts/python_lint.sh ./ci/scripts/rust_clippy.sh ./ci/scripts/rust_fmt.sh ./ci/scripts/rust_toml_fmt.sh
This project includes an AI agent skill for auditing which features from the upstream Apache DataFusion Rust library are not yet exposed in these Python bindings. This is useful when adding missing functions, auditing API coverage, or ensuring parity with upstream.
The skill accepts an optional area argument:
scalar functions aggregate functions window functions dataframe session context ffi types all
If no argument is provided, it defaults to checking all areas. The skill will fetch the upstream DataFusion documentation, compare it against the functions and methods exposed in this project, and produce a coverage report listing what is currently exposed and what is missing.
The skill definition lives in .ai/skills/check-upstream/SKILL.md and follows the Agent Skills open standard. It can be used by any AI coding agent that supports skill discovery, or followed manually.
To change test dependencies, change the pyproject.toml and run
uv sync --dev --no-install-package datafusion