Agent Instructions for Contributors

This file is for agents working on the datafusion-python project (developing, testing, reviewing). If you need to use the DataFusion DataFrame API (write queries, build expressions, understand available functions), see the user-facing skill at SKILL.md.

Skills

This project uses AI agent skills stored in .ai/skills/. Each skill is a directory containing a SKILL.md file with instructions for performing a specific task.

Skills follow the Agent Skills open standard. Each skill directory contains:

  • SKILL.md — The skill definition with YAML frontmatter (name, description, argument-hint) and detailed instructions.
  • Additional supporting files as needed.

To discover what skills are available, list .ai/skills/ and read each SKILL.md. The frontmatter name and description fields summarize the skill's purpose.

Pull Requests

Every pull request must follow the template in .github/pull_request_template.md. The description must include these sections:

  1. Which issue does this PR close? — Link the issue with Closes #NNN.
  2. Rationale for this change — Why the change is needed (skip if the issue already explains it clearly).
  3. What changes are included in this PR? — Summarize the individual changes.
  4. Are there any user-facing changes? — Note any changes visible to users (new APIs, changed behavior, new files shipped in the package, etc.). If there are breaking changes to public APIs, add the api change label.

Pre-commit Checks

Always run pre-commit checks before committing. The hooks are defined in .pre-commit-config.yaml and run automatically on git commit if pre-commit is installed as a git hook. To run all hooks manually:

pre-commit run --all-files

Fix any failures before committing.

Test Coverage

Always prefer Python coverage — a doctest example in a docstring, or a pytest case. The user-facing Python surface is the first line of defense and the primary focus, so behavior should be pinned where users actually meet it.

CI does not run Rust tests. No workflow invokes cargo test; the only Rust checks are cargo fmt --check and cargo clippy --no-deps --all-targets. --all-targets compiles #[cfg(test)] code, so a Rust test cannot rot into a non-compiling state, but it is never executed and a behavioral regression will not fail the build. A Rust test added today is dead weight.

Adding a cargo test job is not a one-line change: crates/core/Cargo.toml enables pyo3/extension-module unconditionally, so the test binary fails to link against Py_* symbols on Linux. The feature would have to be gated first.

Write a Rust test only when the behavior is genuinely unreachable from Python, and wire up CI in the same change so it actually runs. Before concluding it is unreachable, check the suites that already exist:

  • python/tests/ — the main suite. Run pytest python/, not pytest python/tests/: --doctest-modules is on by default and the narrower path skips the doctests in python/datafusion/.
  • examples/datafusion-ffi-example/python/tests/ and examples/datafusion-ffi-query-planner-example/python/tests/ — integration coverage across a real FFI boundary, for anything involving extension codecs, table providers, query planners, or capsule export. These need the example crates built (maturin build, then install the wheel).
  • examples/tpch/ — end-to-end query coverage.

Prefer asserting observable behavior over internal accessors. A test that checks a getter can pass while the path a user actually takes is broken.

Python Function Docstrings

Every Python function must include a docstring with usage examples.

  • Examples are required: Each function needs at least one doctest-style example demonstrating basic usage.
  • Optional parameters: If a function has optional parameters, include separate examples that show usage both without and with the optional arguments. Pass optional arguments using their keyword name (e.g., step=dfn.lit(3)) so readers can immediately see which parameter is being demonstrated.
  • Reuse input data: Use the same input data across examples wherever possible. The examples should demonstrate how different optional arguments change the output for the same input, making the effect of each option easy to understand.
  • Alias functions: Functions that are simple aliases (e.g., list_sort aliasing array_sort) only need a one-line description and a See Also reference to the primary function. They do not need their own examples.

Aggregate and Window Function Documentation

When adding or updating an aggregate or window function, ensure the corresponding site documentation is kept in sync:

  • Aggregations: docs/source/user-guide/common-operations/aggregations.md — add new aggregate functions to the “Aggregate Functions” list and include usage examples if appropriate.
  • Window functions: docs/source/user-guide/common-operations/windows.md — add new window functions to the “Available Functions” list and include usage examples if appropriate.