docs/source/contributor-guide/index.md

<!---
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing,
  software distributed under the License is distributed on an
  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  KIND, either express or implied.  See the License for the
  specific language governing permissions and limitations
  under the License.
-->

# Introduction

We welcome and encourage contributions of all kinds, from all levels, such as:

1. Tickets with issue reports or feature requests
2. Discussions
3. Documentation improvements
4. Code, both PR and (especially) PR Review.

In addition to submitting new PRs, we have a healthy tradition of community
members reviewing each other's PRs. Doing so is a great way to help the
community as well as get more familiar with Rust and the relevant codebases.

## Development Environment

Start with the [Development Environment Quick Start](development_environment.md#quick-start).

For more detail, see the full [development environment guide](development_environment.md)
and the [testing guide](testing.md).

## Finding and Creating Issues to Work On

You can find a curated [good-first-issue] list to help you get started.
You can read about how we plan larger projects in the [Roadmap and Improvement Proposals](roadmap.md) section.

[good-first-issue]: https://github.com/apache/datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22

### Open Contribution and Assigning tickets

DataFusion is an open contribution project, and thus there is no particular
project imposed deadline for completing issues or restrictions on who can
work on an issue, nor limits to how many people can work on an issue at the same time.

Contributors drive the project forward based on their own priorities and
interests and thus you are free to work on any issue that interests you.

If someone is already working on an issue that you want or need but hasn't
been able to finish it yet, feel free to help them out.

If there is an existing open PR for an issue you plan to work on, please review that
PR before opening a new one. Duplicate, unacknowledged PRs consume valuable
reviewer time and we may close them. If there is an existing PR, please identify
it in the PR description and explain why you are opening a new one and not
helping with the previous one. In general it is both polite and will help avoid
unnecessary duplication of work if you also leave a note on an issue when you
start working on it.

If you want to work on an issue which is not already assigned to someone and has
no comment indicating someone is already working on it, you can assign the issue
to yourself by submitting a single word comment `take`. However, if you are unable
to make progress please unassign the issue by commenting a single word `untake`.

# Developer's guide

## Pull Request Overview

We welcome pull requests (PRs) from anyone in the community.

DataFusion is a rapidly evolving project and we try to review and merge PRs
quickly.

Review bandwidth is currently our most limited resource, and we highly encourage reviews by the broader community. If you are waiting for your PR to be reviewed, consider helping review other PRs that are waiting. Such review both helps the reviewer to learn the codebase and become more expert, as well as helps identify issues in the PR (such as lack of test coverage), that can be addressed and make future reviews faster and more efficient.

The lifecycle of a PR is:

1. Create a PR targeting the `main` branch.
2. For new contributors a committer must first trigger the CI tasks. Please mention the members from committers list in the PR to help trigger the CI
3. Your PR will be reviewed. Please respond to all feedback on the PR: you don't have to change the code, but you should acknowledge the feedback. PRs waiting for the feedback for more than a few days will be marked as draft.
4. Once the PR is approved, one of the [committers] will merge your PR, typically within 24 hours. We leave approved "major" changes (see below) open for 24 hours prior to merging, and sometimes leave "minor" PRs open for the same time to permit additional feedback.

Note that the above time frames are estimates. Due to limited committer
bandwidth, it may take longer to merge your PR. Please wait
patiently. If it has been several days you can friendly ping the
committer who approved your PR to help remind them to merge it.

[committers]: https://people.apache.org/phonebook.html?unix=datafusion

## Creating Pull Requests

With coding agents, the number of open PRs now far exceeds our review capacity.
To help reviewers focus on fewer PRs, users without write access are limited to
**3 open, non-draft PRs** at a time. If you reach the limit, wait for existing PRs
to be merged or close them until you have fewer than 3 before opening another.

When possible, we recommend splitting your contributions into multiple smaller focused PRs rather than large PRs (500+ lines) because:

1. The PR is more likely to be reviewed quickly -- our reviewers struggle to find the contiguous time needed to review large PRs.
2. The PR discussions tend to be more focused and less likely to get lost among several different threads.
3. It is often easier to accept and act on feedback when it comes early in a small change, before a particular approach has been polished too much.

If you are concerned that a larger design will be lost in a string of small PRs, creating a large draft PR that shows how they all work together can help.

Note all commits in a PR are squashed when merged to the `main` branch so there is one commit per PR after merge.

For larger PRs, it is often helpful to leave a review on your own PR with
comments calling out important changes or specific important choices. These
annotations can help reviewers quickly find areas they should focus on, thus
speeding up review.

## Release Management and Backports

Contributor-facing guidance for release branches, patch releases, and backports
is documented in the [Release Management](release_management.md) guide.

## Before Submitting a PR

Before submitting a PR, run the standard non-functional checks. PRs must pass
before merge.

```bash
./dev/rust_lint.sh
# use `--write` to automatically fix some formatting and lint errors
# ./dev/rust_lint.sh --write --allow-dirty
```

Please ensure your PR follows the [testing guide](testing.md). In particular:

- Prefer end-to-end Public API tests such as `sqllogictest` (`.slt`) and DataFrame API,
  over Rust unit tests where possible. See
  [Choosing What Kind of Test to Write](testing.md#choosing-what-kind-of-test-to-write).
- Run any relevant commands from the [testing quick start](testing.md#testing-quick-start).

## AI-Assisted Contributions

We welcome AI-assisted PRs, but not unreviewed "AI dumps". See the
[AI Policy](ai-policy.md) page for what we expect from authors and reviewers
who use AI tools.

## Conventional Commits & Labeling PRs

We generate change logs for each release using an automated process that will categorize PRs based on the title
and/or the GitHub labels attached to the PR.

We follow the [Conventional Commits] specification to categorize PRs based on the title. This most often simply means
looking for titles starting with prefixes such as `fix:`, `feat:`, `docs:`, or `chore:`. We do not enforce this
convention but encourage its use if you want your PR to feature in the correct section of the changelog.

The change log generator will also look at GitHub labels such as `bug`, `enhancement`, or `api change`, and labels
do take priority over the conventional commit approach, allowing maintainers to re-categorize PRs after they have been merged.

[conventional commits]: https://www.conventionalcommits.org/en/v1.0.0/

## Reviewing Pull Requests

See the [Reviewing Pull Requests](pr_review.md) guide for what we look for
when reviewing PRs and how to prepare your own for review.

## Performance Improvements

Performance improvements are always welcome: performance is a key DataFusion
feature.

In general, the performance improvement from a change should be "enough" to
justify any added code complexity. How much is "enough" is a judgement made by
the committers, but generally means that the improvement should be noticeable in
a real-world scenario and is greater than the noise of the benchmarking system.

To help committers evaluate the potential improvement, performance PRs should
in general be accompanied by benchmark results that demonstrate the improvement.

The best way to demonstrate a performance improvement is with the existing
benchmarks:

- [System level SQL Benchmarks](https://github.com/apache/datafusion/tree/main/benchmarks)
- Microbenchmarks such as those in [functions/benches](https://github.com/apache/datafusion/tree/main/datafusion/functions/benches)

If there is no suitable existing benchmark, you can create a new one. It helps
to isolate the effects of your change by creating a separate PR with the
benchmark, and then a PR with the code change that improves the benchmark.

[system level sql benchmarks]: https://github.com/apache/datafusion/tree/main/benchmarks
[functions/benches]: https://github.com/apache/datafusion/tree/main/datafusion/functions/benches

## "Major" and "Minor" PRs

Since we are a worldwide community, we have contributors in many timezones who review and comment. To ensure anyone who wishes has an opportunity to review a PR, our committers try to ensure that at least 24 hours passes between when a "major" PR is approved and when it is merged.

A "major" PR means there is a substantial change in design or a change in the API. Committers apply their best judgment to determine what constitutes a substantial change. A "minor" PR might be merged without a 24 hour delay, again subject to the judgment of the committer. Examples of potential "minor" PRs are:

1. Documentation improvements/additions
2. Small bug fixes
3. Non-controversial build-related changes (clippy, version upgrades etc.)
4. Smaller non-controversial feature additions

The good thing about open code and open development is that any issues in one change can almost always be fixed with a follow on PR.

## Stale PRs

Pull requests will be marked with a `stale` label after 60 days of inactivity and then closed 7 days after that.
Commenting on the PR will remove the `stale` label.

## CI Runners

### Runs-On

We use [Runs-On](https://runs-on.com/) for some actions in the main repository, which run in the ASF AWS account to speed up CI. In forks, these actions run on the default GitHub runners since forks do not have access to ASF infrastructure.

To configure them, we use the following format:

`runs-on: ${{ github.repository_owner == 'apache' && format('runs-on={0},family=m8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}`

This is a conditional expression that uses Runs-On custom runners for the main repository and falls back to the standard GitHub runners for forks. Runs-On configuration follows the [Runs-On pattern](https://runs-on.com/configuration/job-labels/).

For those actions we also use the [Runs-On action](https://runs-on.com/caching/magic-cache/#how-to-use), which adds support for external caching and reports job metrics:

`- uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e`

For the standard GitHub runners, this action will do nothing.

#### Spot Instances

By default, Runs-On actions run as [spot instances](https://runs-on.com/configuration/spot-instances/), which means they might occasionally be interrupted. In the CI you would see:

```
Error: The operation was canceled.
```

According to Runs-On, spot instance termination is extremely rare for instances running for less than 1h. Those actions will be restarted automatically.

### GitHub Runners

We also use standard GitHub runners for some actions in the main repository; these are also runnable in forks.

Small, short jobs run on `ubuntu-slim`, as ASF Infrastructure
[recommends](https://infra.apache.org/services.html), because it starts quickly
even when the shared `ubuntu-latest` pool is busy. Use it for any job that
stays well within its limits: one vCPU, 5 GB of memory, 15 minutes, and no
Docker engine, so no `container:` jobs or service containers. Slim has no Rust
toolchain; jobs that need `cargo` install it with
`.github/actions/setup-rustup`.

#### Pull Requests from Forks

DataFusion and its subprojects use substantial CI runner time. ASF Infrastructure
identified them among its largest GitHub Actions users in
[#25148](https://github.com/apache/datafusion/issues/25148). Fork CI reduces
duplicate work on ASF's shared runners.

A pull request from a fork to `main` runs the Rust workflow on your fork, with
your own GitHub Actions runners. The main repository runs only the fast lint jobs for
such a pull request and adds a `Rust (fork)` check that links to the run on
your fork. The merge queue runs the full workflow on the merged result before
a pull request merges, so a green fork run does not replace it.

To make this work:

1. Before you push your branch, open the **Actions** tab on your fork.
   Confirm the prompt to enable workflows.
2. Keep the `main` branch of your fork in sync with upstream. Runs on your
   branches restore the build caches that a run on your `main` saves, so a
   synced fork builds much faster.
3. Rebase your branch before you push. The fork run tests your branch as it
   is, while the merge queue tests it merged with `main`.