AGENTS instructions

Environment Setup

Install prek: uv tool install prek
Enable commit hooks: prek install
Never run pytest, python, or airflow commands directly on the host — always use breeze.
Place temporary scripts in dev/ (mounted as /opt/airflow/dev/ inside Breeze).

Commands

<PROJECT> is folder where pyproject.toml of the package you want to test is located. For example, airflow-core or providers/amazon. <target_branch> is the branch the PR will be merged into — usually main, but could be v3-1-test when creating a PR for the 3.1 branch.

Run a single test: uv run --project <PROJECT> pytest path/to/test.py::TestClass::test_method -xvs
Run a test file: uv run --project <PROJECT> pytest path/to/test.py -xvs
Run all tests in package: uv run --project <PROJECT> pytest path/to/package -xvs
If uv tests fail with missing system dependencies, run the tests with breeze: breeze run pytest <tests> -xvs
Run a Python script: uv run --project <PROJECT> python dev/my_script.py
Run core or provider tests suite in parallel: breeze testing <test_group> --run-in-parallel (test groups: core-tests, providers-tests)
Run core or provider db tests suite in parallel: breeze testing <test_group> --run-db-tests-only --run-in-parallel (test groups: core-tests, providers-tests)
Run core or provider non-db tests suite in parallel: breeze testing <test_group> --skip-db-tests --use-xdist (test groups: core-tests, providers-tests)
Run single provider complete test suite: breeze testing providers-tests --test-type "Providers[PROVIDERS_LIST]" (e.g., Providers[google] or Providers[amazon] or “Providers[amazon,google]”)
Run Helm tests in parallel with xdist breeze testing helm-tests --use-xdist
Run Helm tests with specific K8s version: breeze testing helm-tests --use-xdist --kubernetes-version 1.35.0
Run specific Helm test type: breeze testing helm-tests --use-xdist --test-type <type> (types: airflow_aux, airflow_core, apiserver, dagprocessor, other, redis, security, statsd, webserver)
Run other suites of tests breeze testing <test_group> (test groups: airflow-ctl-tests, docker-compose-tests, task-sdk-tests
Run Airflow CLI: breeze run airflow dags list
Type-check: breeze run mypy path/to/code
Lint with ruff only: prek run ruff --from-ref <target_branch>
Format with ruff only: prek run ruff-format --from-ref <target_branch>
Run regular (fast) static checks: prek run --from-ref <target_branch> --stage pre-commit
Run manual (slower) checks: prek run --from-ref <target_branch> --stage manual
Build docs: breeze build-docs
Determine which tests to run based on changed files: breeze selective-checks --commit-ref <commit_with_squashed_changes>

SQLite is the default backend. Use --backend postgres or --backend mysql for integration tests that need those databases. If Docker networking fails, run docker network prune.

Repository Structure

UV workspace monorepo. Key paths:

airflow-core/src/airflow/ — core scheduler, API, CLI, models
- models/ — SQLAlchemy models (DagModel, TaskInstance, DagRun, Asset, etc.)
- jobs/ — scheduler, triggerer, Dag processor runners
- api_fastapi/core_api/ — public REST API v2, UI endpoints
- api_fastapi/execution_api/ — task execution communication API
- dag_processing/ — Dag parsing and validation
- cli/ — command-line interface
- ui/ — React/TypeScript web interface (Vite)
task-sdk/ — lightweight SDK for Dag authoring and task execution runtime
- src/airflow/sdk/execution_time/ — task runner, supervisor
providers/ — 100+ provider packages, each with its own pyproject.toml
airflow-ctl/ — management CLI tool
chart/ — Helm chart for Kubernetes deployment

Architecture Boundaries

Users author Dags with the Task SDK (airflow.sdk).
Dag Processor parses Dag files in isolated processes and stores serialized Dags in the metadata DB.
Scheduler reads serialized Dags — never runs user code — and creates Dag runs / task instances.
Workers execute tasks via Task SDK and communicate with the API server through the Execution API — never access the metadata DB directly.
API Server serves the React UI and handles all client-database interactions.
Triggerer evaluates deferred tasks/sensors in isolated processes.
Shared libraries that are symbolically linked to different Python distributions are in shared folder.
Airflow uses uv workspace feature to keep all the distributions sharing dependencies and venv
Each of the distributions should declare other needed distributions: uv --project <FOLDER> sync command acts on the selected project in the monorepo with only dependencies that it has

Shared libraries

shared libraries provide implementation of some common utilities like logging, configuration where the code should be reused in different distributions (potentially in different versions)
we have a number of shared libraries that are separate, small Python distributions located under shared folder
each of the libraries has it's own src, tests, pyproject.toml and dependencies
sources of those libraries are symbolically linked to the distributions that are using them (airflow-core, task-sdk for example)
tests for the libraries (internal) are in the shared distribution's test and can be run from the shared distributions
tests of the consumers using the shared libraries are present in the distributions that use the libraries and can be run from there

Coding Standards

No assert in production code.
time.monotonic() for durations, not time.time().
In airflow-core, functions with a session parameter must not call session.commit(). Use keyword-only session parameters.
Imports at top of file. Valid exceptions: circular imports, lazy loading for worker isolation, TYPE_CHECKING blocks.
Guard heavy type-only imports (e.g., kubernetes.client) with TYPE_CHECKING in multi-process code paths.
Apache License header on all new files (prek enforces this).

Testing Standards

Add tests for new behavior — cover success, failure, and edge cases.
Use pytest patterns, not unittest.TestCase.
Use spec/autospec when mocking.
Use time_machine for time-dependent tests.
Use @pytest.mark.parametrize for multiple similar inputs.
Use @pytest.mark.db_test for tests that require database access.
Test fixtures: devel-common/src/tests_common/pytest_plugin.py.
Test location mirrors source: airflow/cli/cli_parser.py → tests/cli/test_cli_parser.py.

Commits and PRs

Write commit messages focused on user impact, not implementation details.

Good: Fix airflow dags test command failure without serialized Dags
Good: UI: Fix Grid view not refreshing after task actions
Bad: Initialize DAG bundles in CLI get_dag function

Add a newsfragment for user-visible changes: echo "Brief description" > airflow-core/newsfragments/{PR_NUMBER}.{bugfix|feature|improvement|doc|misc|significant}.rst

NEVER add Co-Authored-By with yourself as co-author of the commit. Agents cannot be authors, humans can be, Agents are assistants.

Creating Pull Requests

Always push to the user's fork, not to the upstream apache/airflow repo. Never push directly to main.

Before pushing, determine the fork remote. Check git remote -v — if origin does not point to apache/airflow, use origin (it‘s the user’s fork). If origin points to apache/airflow, look for another remote that points to the user's fork. If no fork remote exists, create one:

gh repo fork apache/airflow --remote --remote-name fork

Before pushing, perform a self-review of your changes following the Gen-AI review guidelines in contributing-docs/05_pull_requests.rst and the code review checklist in .github/instructions/code-review.instructions.md:

Review the full diff (git diff main...HEAD) and verify every change is intentional and related to the task — remove any unrelated changes.
Read .github/instructions/code-review.instructions.md and check your diff against every rule — architecture boundaries, database correctness, code quality, testing requirements, API correctness, and AI-generated code signals. Fix any violations before pushing.
Confirm the code follows the project's coding standards and architecture boundaries described in this file.
Run regular (fast) static checks (prek run --from-ref <target_branch> --stage pre-commit) and fix any failures.
Run manual (slower) checks (prek run --from-ref <target_branch> --stage manual) and fix any failures.
Run relevant individual tests and confirm they pass.
Find which tests to run for the changes with selective-checks and run those tests in parallel to confirm they pass and check for CI-specific issues.
Check for security issues — no secrets, no injection vulnerabilities, no unsafe patterns.

Then push the branch to the fork remote and open the PR creation page in the browser with the body pre-filled (including the generative AI disclosure already checked):

git push -u <fork-remote> <branch-name>
gh pr create --web --title "Short title (under 70 chars)" --body "$(cat <<'EOF'
Brief description of the changes.

closes: #ISSUE  (if applicable)

---

##### Was generative AI tooling used to co-author this PR?

- [X] Yes — <Agent Name and Version>

Generated-by: <Agent Name and Version> following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)

EOF
)"

The --web flag opens the browser so the user can review and submit. The --body flag pre-fills the PR template with the generative AI disclosure already completed.

Remind the user to:

Review the PR title — keep it short (under 70 chars) and focused on user impact.
Add a brief description of the changes at the top of the body.
Reference related issues when applicable (closes: #ISSUE or related: #ISSUE).

Boundaries

Ask first
- Large cross-package refactors.
- New dependencies with broad impact.
- Destructive data or migration changes.
Never
- Commit secrets, credentials, or tokens.
- Edit generated files by hand when a generation workflow exists.
- Use destructive git operations unless explicitly requested.