AGENTS.md

<!---
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing,
  software distributed under the License is distributed on an
  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  KIND, either express or implied.  See the License for the
  specific language governing permissions and limitations
  under the License.
-->

# Agent Instructions for Contributors

This file is for agents working **on** the datafusion-python project (developing,
testing, reviewing). If you need to **use** the DataFusion DataFrame API (write
queries, build expressions, understand available functions), see the user-facing
skill at [`SKILL.md`](skills/datafusion_python/SKILL.md).

## Skills

This project uses AI agent skills stored in `.ai/skills/`. Each skill is a directory containing a `SKILL.md` file with instructions for performing a specific task.

Skills follow the [Agent Skills](https://agentskills.io) open standard. Each skill directory contains:

- `SKILL.md` — The skill definition with YAML frontmatter (name, description, argument-hint) and detailed instructions.
- Additional supporting files as needed.

To discover what skills are available, list `.ai/skills/` and read each
`SKILL.md`. The frontmatter `name` and `description` fields summarize the
skill's purpose. Some descriptions begin with `TRIGGER —`; those are not tasks
to run on request but conventions to read *before* writing code that meets the
stated condition.

## FFI Capsule Protocol

The `__datafusion_*__` capsule getters are one protocol with a settled
convention. Before adding or changing one, read
[`.ai/skills/ffi-capsule-protocol/SKILL.md`](.ai/skills/ffi-capsule-protocol/SKILL.md).

## Documentation Sources

Search and edit `docs/source/`. `docs/temp/` is gitignored build output that
`grep -r` will surface with stale copies of the same pages.

Before changing a public API, check
`docs/source/user-guide/upgrade-guides.md` for how the same API family was
migrated previously. Follow the established pattern rather than inventing one.

## Pull Requests

Every pull request must follow the template in
`.github/pull_request_template.md`. The description must include these sections:

1. **Which issue does this PR close?** — Link the issue with `Closes #NNN`.
2. **Rationale for this change** — Why the change is needed (skip if the issue
   already explains it clearly).
3. **What changes are included in this PR?** — Summarize the individual changes.
4. **Are there any user-facing changes?** — Note any changes visible to users
   (new APIs, changed behavior, new files shipped in the package, etc.). If
   there are breaking changes to public APIs, add the `api change` label **and
   add a section to `docs/source/user-guide/upgrade-guides.md`** showing the
   before and after. This applies to FFI hook method signatures, which
   extension libraries implement.

## Pre-commit Checks

Always run pre-commit checks **before** committing. The hooks are defined in
`.pre-commit-config.yaml` and run automatically on `git commit` if pre-commit
is installed as a git hook. To run all hooks manually:

```bash
pre-commit run --all-files
```

Fix any failures before committing.

## Test Coverage

Always prefer Python coverage — a doctest example in a docstring, or a pytest
case. The user-facing Python surface is the first line of defense and the
primary focus, so behavior should be pinned where users actually meet it.

**CI does not run Rust tests.** No workflow invokes `cargo test`; the only
Rust checks are `cargo fmt --check` and
`cargo clippy --no-deps --all-targets`. `--all-targets` compiles
`#[cfg(test)]` code, so a Rust test cannot rot into a non-compiling state, but
it is never executed and a behavioral regression will not fail the build. A
Rust test added today is dead weight.

Adding a `cargo test` job is not a one-line change: `crates/core/Cargo.toml`
enables `pyo3/extension-module` unconditionally, so the test binary fails to
link against `Py_*` symbols on Linux. The feature would have to be gated first.

Write a Rust test only when the behavior is genuinely unreachable from Python,
and wire up CI in the same change so it actually runs. Before concluding it is
unreachable, check the suites that already exist:

- `python/tests/` — the main suite. Run `pytest python/`, **not**
  `pytest python/tests/`: `--doctest-modules` is on by default and the
  narrower path skips the doctests in `python/datafusion/`.
- `examples/datafusion-ffi-example/python/tests/` and
  `examples/datafusion-ffi-query-planner-example/python/tests/` — integration
  coverage across a real FFI boundary, for anything involving extension
  codecs, table providers, query planners, or capsule export. These need the
  example crates built (`maturin build`, then install the wheel).
- `examples/tpch/` — end-to-end query coverage.

Prefer asserting observable behavior over internal accessors. A test that
checks a getter can pass while the path a user actually takes is broken.

## Python Function Docstrings

Every Python function must include a docstring with usage examples.

- **Examples are required**: Each function needs at least one doctest-style example
  demonstrating basic usage.
- **Optional parameters**: If a function has optional parameters, include separate
  examples that show usage both without and with the optional arguments. Pass
  optional arguments using their keyword name (e.g., `step=dfn.lit(3)`) so readers
  can immediately see which parameter is being demonstrated.
- **Reuse input data**: Use the same input data across examples wherever possible.
  The examples should demonstrate how different optional arguments change the output
  for the same input, making the effect of each option easy to understand.
- **Alias functions**: Functions that are simple aliases (e.g., `list_sort` aliasing
  `array_sort`) only need a one-line description and a `See Also` reference to the
  primary function. They do not need their own examples.

### One canonical home per claim

A claim lives where the reader already is when they need it — **exactly once**.

- A property of one callable's arguments, return value, or errors → that
  callable's docstring.
- A property of a *type* that several callables share → the class docstring.
  (Session sharing and context lifetime live on `SessionContext`, not on each
  of the five `with_*` methods.)
- **Why** the API is shaped this way, a multi-library recipe, Rust-side code, a
  trade-off, or a limitation with an upstream issue → the narrative guide under
  `docs/source/`.

Everywhere else: one sentence plus one Sphinx role. A docstring may *state* a
claim the guide also makes; it must not *argue* it — no "because", no "the
reason is", no counter-argument.

**The test:** if a paragraph would survive being moved into the guide unchanged,
move it. `python/tests/test_docstrings.py` enforces a length ceiling, which is
the symptom this rule treats.

When you point at the guide, use a real `:ref:` to a specific label. Prose
saying "see the extensions guide" with no role is a dead end in the rendered
HTML.

### Examples that need a compiled extension

Some APIs cannot be demonstrated without a built FFI extension library, which
this package does not ship. The convention is:

1. A **runnable** example block first, using only the wheel. A `SessionContext`
   satisfies the capsule-getter protocols, so its own exported capsule stands in
   for a real library's in a doctest.
2. Then a `# doctest: +SKIP` block showing real usage, at most a few lines, with
   one line of prose naming the test that runs it for real.
3. Every `+SKIP` block needs that mirror. See
   `test_with_extensions_docstring_example_still_runs` in
   `examples/datafusion-ffi-query-planner-example`, which parses the live
   docstring, drops the skip, and executes it — so a renamed method or a wrong
   expected output fails there.

## Aggregate and Window Function Documentation

When adding or updating an aggregate or window function, ensure the corresponding
site documentation is kept in sync:

- **Aggregations**: `docs/source/user-guide/common-operations/aggregations.md` —
  add new aggregate functions to the "Aggregate Functions" list and include usage
  examples if appropriate.
- **Window functions**: `docs/source/user-guide/common-operations/windows.md` —
  add new window functions to the "Available Functions" list and include usage
  examples if appropriate.