Agent Guidelines for Airflow Registry

This document contains rules, patterns, and guidelines for working on the Airflow Registry project.

Core Principles

0. Minimal JavaScript

Keep JavaScript to minimum necessary
Theme toggle + progressive enhancement (filters, search)
No frameworks (no React, Vue, etc.)
Exception: swagger-ui is vendored from node_modules/swagger-ui-dist for the API Explorer

1. Quality Standards

Visual consistency: Changes should match existing design system and styling
Semantic HTML: Use proper elements and structure
Maintainable CSS: Descriptive class names, leverage cascade
Light/Dark mode: Full support with smooth transitions
Responsive: Mobile-first, works on all screen sizes

2. Semantic CSS & HTML

Follow the principles from https://css-tricks.com/semantic-class-names/

2.1 Class Names Describe CONTENT, Not Appearance

Class names should answer “what is this?” not “how should it look?”

Good examples:

.provider-card - it's a card displaying provider information
.connection-types - a section showing connection types
.dependencies - shows dependency information
.stats - displays statistical data

Bad examples:

.blue-box - describes appearance (what if design changes to green?)
.flex-row - describes layout implementation
.mt-4 - describes spacing value
.text-lg - describes typography

Why this matters:

Design changes shouldn't require HTML updates
Code is self-documenting (you understand structure without seeing CSS)
Multiple elements can share appearance without sharing meaning

2.2 Use CSS Cascade to Avoid Redundant Prefixes

When an element only makes sense within its parent, use descendant selectors instead of prefixing class names.

Good - leveraging cascade:

<section class="technical">
  <div class="python-version">...</div>
  <div class="connections">...</div>
</section>

.technical { /* section styles */ }
.technical h3 { /* heading styles */ }
.technical .python-version { /* specific section */ }
.technical code { /* all code in technical section */ }

Bad - redundant prefixes:

<section class="technical">
  <div class="technical-python">...</div>
  <div class="technical-connections">...</div>
</section>

.technical { /* section styles */ }
.technical-heading { /* redundant prefix */ }
.technical-python { /* redundant prefix */ }
.technical-code { /* redundant prefix */ }

Why this matters:

Less typing, less repetition
Clearer hierarchy in CSS
If .python-version only appears in .technical, the prefix adds no information

2.3 Leverage Semantic HTML5 Elements

Use the right HTML element for the job. This improves accessibility, SEO, and reduces need for classes.

Semantic elements and their uses:

<header> - Page or section header
<footer> - Page or section footer
<nav> - Navigation menus
<main> - Main content of the page
<section> - Thematic grouping of content
<article> - Self-contained content
<aside> - Side content/sidebars
<dl>, <dt>, <dd> - Definition lists (key-value pairs)
<details>, <summary> - Collapsible content

Example - Statistics:

<!-- Good: Semantic -->
<dl class="stats">
  <div>
    <dt>1,000+</dt>
    <dd>Providers</dd>
  </div>
</dl>

.stats dt { font-size: var(--text-3xl); }
.stats dd { font-size: var(--text-sm); }

Example - Provider detail header:

<!-- Good: Semantic structure -->
<header class="card">
  <div class="top">
    <div class="logo">...</div>
    <div class="details">
      <h1>Amazon</h1>
      <p class="description">...</p>
    </div>
  </div>

  <div class="stats">...</div>
</header>

.provider-detail-page header { /* styles */ }
.provider-detail-page header h1 { /* title */ }
.provider-detail-page header .stats { /* grid of stats */ }

2.4 When to Add Classes vs Use Element Selectors

Use element selectors when:

There's only one of that element in the scope
The element's meaning is clear from context
Example: .provider-detail-page header h1 - there's only one main heading

Add a class when:

Multiple instances need different styling
The role isn't obvious from the element alone
You need to target it from JavaScript
Example: .python-version, .connections, .dependencies - all are <div> but serve different purposes

Prefer simpler selectors:

section code over section .code-snippet
details summary over .details-trigger
.connections a over .connection-badge

2.5 No Utility Classes

Utility classes describe HOW something looks, not WHAT it is.

Remove these patterns:

Spacing: .mb-2, .py-12, .gap-4, .px-6, .space-y-4
Layout: .flex, .grid, .grid-cols-2, .items-center
Display: .block, .inline-block, .hidden
Text: .text-center, .text-lg, .font-bold, .uppercase
Colors: .text-blue-500, .bg-gray-100

Instead, style contextually:

/* Good */
.technical h3 {
  font-size: var(--text-sm);
  font-weight: var(--font-semibold);
  margin-bottom: var(--space-3);
  display: flex;
  align-items: center;
}

/* Bad */
<h3 class="text-sm font-semibold mb-3 flex items-center">

2.6 Contextual Component Styling

Components should be styled based on WHERE they appear, not through generic modifier classes.

No generic button classes:

/* Bad */
.btn { }
.btn-primary { }
.btn-secondary { }

Style contextually instead:

/* Good */
.hero .popular-providers a {
  padding: var(--space-2) var(--space-4);
  background: var(--bg-tertiary);
  border: 1px solid var(--border-primary);
  border-radius: var(--radius-lg);
}

.provider-detail-page .actions .btn-primary {
  background: var(--accent-primary);
  color: var(--color-navy-900);
}

Why this matters:

Buttons in hero look different from buttons in provider detail
Context determines appearance
Reduces need for modifier classes

CSS Architecture

File Structure

src/css/
├── tokens.css          # Design tokens (colors, spacing, fonts)
├── main.css            # Main styles with imports
└── base/
    ├── reset.css       # CSS reset
    └── typography.css  # Base typography

Semantic Section Pattern

Each major section follows this pattern:

/* Section container */
.section-name {
  padding: var(--space-20) 0;
  background: var(--bg-primary);
}

/* Nested elements using cascade */
.section-name header {
  /* Header styles */
}

.section-name h2 {
  /* Title styles */
}

.section-name .some-element {
  /* Element styles */
}

Design Tokens

Use CSS custom properties defined in tokens.css:

Colors: --color-navy-900, --color-cyan-400, --color-gray-700, etc.
Spacing: --space-4 (1rem), --space-8 (2rem), etc.
Typography: --text-base, --text-lg, --font-semibold, etc.
Theme variables: --bg-primary, --text-primary, --border-primary

IMPORTANT: Always use CSS custom properties instead of hardcoded color values:

✅ Good: border: 1px solid var(--color-gray-700);
❌ Bad: border: 1px solid #334155;

When to Use Semantic vs Direct Color Variables

When defining new variables in tokens.css, follow this hierarchy:

Prefer semantic variables (with -light/-dark suffixes) over direct color variables:
- ✅ Good: --color-input-bg: light-dark(var(--bg-tertiary-light), var(--bg-tertiary-dark));
- ❌ Bad: --color-input-bg: light-dark(var(--color-gray-100), var(--color-navy-700));
Only use direct color variables when:
- Defining the base semantic variables themselves (e.g., --bg-tertiary-light: var(--color-gray-100);)
- No appropriate semantic variable exists for your use case
- The color combination is intentionally different from existing semantic patterns
In component CSS (main.css):
- Use computed semantic variables: background: var(--bg-primary);
- Or use inline light-dark() with semantic variables for contextual colors

This creates a clear hierarchy: base colors → semantic variables → computed variables → usage

Light/Dark Mode

The theme system uses the CSS color-scheme property and light-dark() function:

Default theme is dark mode
Theme switching sets document.documentElement.style.colorScheme = 'light' | 'dark'

Theme computed variables defined in tokens.css with progressive enhancement:

/* Progressive enhancement: fallback then light-dark() */
--bg-primary: var(--bg-primary-light);
--bg-primary: light-dark(var(--bg-primary-light), var(--bg-primary-dark));

Most elements use computed theme variables:

.element {
  background: var(--bg-primary);
  color: var(--text-primary);
}

Contextual colors use inline light-dark():

.kbd {
  border-color: light-dark(var(--border-primary-light), var(--color-navy-600));
}

Progressive Enhancement: Each theme variable is declared twice in tokens.css:

First with light value as fallback for browsers without light-dark() support
Then with light-dark() which overrides in supporting browsers

Development Workflow

Feature Development Checklist

When adding or modifying features:

Plan semantic HTML structure using appropriate elements
Write semantic CSS using cascade pattern
Add light/dark mode styles for all elements
Test responsiveness across different screen sizes
Verify consistency with existing design system

Common Issues

Colors not switching in dark mode: Ensure element uses computed theme variables like var(--bg-primary) or inline light-dark()
Missing dark mode colors: Check that theme variable has both -light and -dark variants defined in tokens.css
Wrong border colors: Check color tokens are indigo not gray

CSS Deletion Guidelines

When removing non-semantic CSS:

Search for usage first: Use grep to find where classes are used
Confirm removal: If only in CSS and not in refactored templates, delete it
Test visually: Take screenshots before and after
Common deletions:
- Grid/flex utilities: .grid, .flex, .items-center, etc.
- Spacing utilities: .mb-*, .py-*, .gap-*, etc.
- Button classes: .btn, .btn-primary, etc.
- Text utilities: .text-center, .text-lg, etc.

Quality Standards

Visual consistency: Changes should match existing design system and styling
Semantic HTML: Use proper elements and structure
Maintainable CSS: Descriptive class names, leverage cascade
Light/Dark mode: Full support with smooth transitions (except during screenshots)
Responsive: Mobile-first, works on all screen sizes

Reference Files

Key files in the project:

src/css/main.css - Main styles
src/css/tokens.css - Design tokens
src/ - All page templates (index, provider-detail, providers, explore, stats, api-explorer)
../dev/registry/registry_contract_models.py - Pydantic contracts for all JSON payloads
../dev/registry/export_registry_schemas.py - Generates OpenAPI spec from contracts

Development Server

Local development server runs at http://localhost:8080

Deployment Architecture

The registry is built in the apache/airflow repo and served at airflow.apache.org/registry/.

How it works

Build: registry-build.yml extracts metadata, builds the 11ty site, and syncs to S3. Supports two modes:
- Full build (no provider input): extracts all ~99 providers (~12 min)
- Incremental build (provider=amazon): extracts one provider (~30s), merges with existing data from S3 via merge_registry_data.py, then builds the full site. The S3 sync step excludes the entire api/providers/ subtree for non-target providers to avoid overwriting real data with Eleventy's incomplete/empty stubs (Eleventy 3.x permalink: false does not work with pagination).
S3 buckets: {live|staging}-docs-airflow-apache-org/registry/ (same bucket as docs, different prefix)
Serving: Apache HTTPD at airflow.apache.org rewrites /registry/* to CloudFront, which serves from S3
Auto-trigger: When publish-docs-to-s3.yml publishes provider docs, its update-registry job calls registry-build.yml with the provider ID for an incremental registry update

Path prefix

Production: REGISTRY_PATH_PREFIX=/registry/ (default)
Local dev: REGISTRY_PATH_PREFIX=/ (set automatically by pnpm dev)
All internal links use 11ty's | url filter, which prepends the prefix

Changes needed in `apache/airflow-site` repo (separate PR)

.htaccess rewrite rule - Add a rule to proxy /registry/* through CloudFront:

RewriteRule ^registry/(.*)$ https://<cloudfront-distribution>.cloudfront.net/registry/$1 [P,L]

Navigation link - Add a “Registry” entry to the site header/nav in the Hugo templates, pointing to /registry/.

These changes mirror the existing /docs/* rewrite pattern.

Data Extraction (`dev/registry/`)

The registry's JSON data is produced by four extraction scripts in dev/registry/, which is a Python package (workspace member) with shared code in registry_tools/.

Module type definitions live in dev/registry/registry_tools/types.py — this is the single source of truth for all module types (operator, hook, sensor, trigger, etc.). The three Python extraction scripts and the frontend data file (types.json) all derive from this module. To add a new type, add it to MODULE_TYPES in types.py and run generate_types_json.py.

When modifying these scripts, understand the design decisions below.

Why four separate scripts?

Script	Runs on	Needs providers installed?	What it does
`extract_metadata.py`	Host	No	Provider metadata, class names (AST), PyPI stats, logos
`extract_versions.py`	Host	No	Per-version metadata from git tags
`extract_parameters.py`	Breeze	Yes	`__init__` parameter inspection via MRO
`extract_connections.py`	Breeze	Yes	Connection form metadata via ProvidersManager

extract_parameters.py and extract_connections.py need runtime access to provider classes (to inspect __init__ signatures and call get_connection_form_widgets()). They run inside Breeze where all providers are installed. extract_metadata.py and extract_versions.py only need filesystem access and run on the host. This split means the CI workflow can run the fast scripts (metadata, ~30s per provider) without spinning up Breeze, while parameter/connection extraction is a separate step.

Relationship to `run_provider_yaml_files_check.py`

scripts/in_container/run_provider_yaml_files_check.py (run by the check-provider-yaml-valid pre-commit hook inside Breeze) validates that everything listed in provider.yaml is correct: modules exist, classes are importable, and every Python file in operators//hooks//sensors//triggers/ directories is accounted for. This guarantees provider.yaml is the correct and complete manifest.

extract_metadata.py builds on that guarantee but solves a different problem: it needs individual class names within each module. provider.yaml lists entries at two granularities:

Module-level (operators, hooks, sensors, triggers, transfers, bundles): e.g., airflow.providers.amazon.operators.s3 — a single module can contain many classes (S3CopyObjectOperator, S3DeleteObjectsOperator, S3ListOperator, etc.)
Class-level (notifications, secrets-backends, logging, executors, task-decorators): e.g., airflow.providers.amazon.notifications.chime.ChimeNotifier — full class path already provided

For module-level entries, AST parsing discovers the individual classes and their inheritance chains. For class-level entries, extract_metadata.py uses the paths from provider.yaml directly — no discovery needed. The validation script ensures both categories are correct before extract_metadata.py ever runs.

Why AST parsing instead of runtime import?

extract_metadata.py runs on the CI host without installing 100+ provider packages. It reads .py files and extracts class names, base classes, and docstrings from the AST. This means it works with just pyyaml as a dependency.

The trade-off: AST parsing can't resolve dynamic class definitions or runtime-computed attributes. For the 99 providers currently in the repo, AST parsing captures everything because provider classes use standard class Foo(Base): definitions.

An approach using only Sphinx inventory files (without AST) was considered but rejected: inventory files only exist for published providers, and they don't contain inheritance information. The registry needs to know whether a class is an operator, hook, sensor, etc., which requires resolving the inheritance chain back to base classes like BaseOperator or BaseHook. AST parsing provides this.

Module filtering via base class inheritance

Classes are discovered by checking if they inherit (transitively) from type-specific base classes defined in get_module_type_base_classes():

operators: BaseOperator, BaseSensorOperator, DecoratedOperator, ...
hooks: BaseHook, DbApiHook, DiscoverableHook
sensors: BaseSensorOperator, PokeSensorOperator
triggers: BaseTrigger, TriggerEvent
transfers: BaseOperator (transfers are operators)
bundles: BaseDagBundle

Inheritance resolution is transitive and cross-file. build_global_inheritance_map() scans all provider src/ directories to build a global class_name → {base_names} map. This handles chains like S3ListOperator → AwsBaseOperator → BaseOperator where the intermediate class lives in a different file.

After inheritance filtering, a post-filter skips private (_-prefixed), Base*, Abstract*, and *Mixin classes to avoid indexing internal helper classes.

There is no suffix-based matching (e.g., checking if a class name ends with “Operator”). The filtering is purely inheritance-based.

Documentation URLs via Sphinx Inventory

Module docs_url values come from Sphinx objects.inv inventory files, not manually constructed paths. The inventory is the canonical mapping of every documented class to its URL, published on S3 alongside the docs.

read_inventory() parses an objects.inv file, returning {qualified_name: url_path} for py:class entries
fetch_provider_inventory() downloads the inventory from S3 with local file caching (12-hour TTL in dev/registry/.inventory_cache/)
extract_modules_from_yaml() accepts an inventory dict and uses it for URL lookup, falling back to manual construction when a class isn't in the inventory

The fallback exists because unpublished providers (new, not yet released to PyPI) won't have inventory files on S3.

If you're adding a new module type or changing how docs_url is constructed, prefer extending the inventory lookup rather than hardcoding URL patterns.

Why Eleventy for the static site?

Static site generators produce zero-JS pages by default. The registry works without JavaScript — filtering and search are layered on top progressively. Eleventy has no opinion on frontend frameworks, which keeps the dependency surface small (~30 packages in the lockfile).

Path prefix handling

The site deploys at /registry/ on airflow.apache.org but runs at / during local dev. Eleventy's pathPrefix config handles this via the REGISTRY_PATH_PREFIX env var. Templates use the | url filter, and client-side JS reads window.__REGISTRY_BASE__ (injected in base.njk).

Troubleshooting

Can't find where a class is used

# Search in templates
grep -r "class-name" src/

# Search in CSS
grep "\.class-name" src/css/

Color issues

Use browser DevTools to inspect computed styles
Check both light and dark mode
Verify color tokens in tokens.css
Ensure computed theme variables are used correctly

Agent Guidelines for Airflow Registry

Core Principles

0. Minimal JavaScript

1. Quality Standards

2. Semantic CSS & HTML

2.1 Class Names Describe CONTENT, Not Appearance

2.2 Use CSS Cascade to Avoid Redundant Prefixes

2.3 Leverage Semantic HTML5 Elements

2.4 When to Add Classes vs Use Element Selectors

2.5 No Utility Classes

2.6 Contextual Component Styling

CSS Architecture

File Structure

Semantic Section Pattern

Design Tokens

When to Use Semantic vs Direct Color Variables

Light/Dark Mode

Development Workflow

Feature Development Checklist

Common Issues

CSS Deletion Guidelines

Quality Standards

Reference Files

Development Server

Deployment Architecture

How it works

Path prefix

Changes needed in apache/airflow-site repo (separate PR)

Data Extraction (dev/registry/)

Why four separate scripts?

Relationship to run_provider_yaml_files_check.py

Why AST parsing instead of runtime import?

Module filtering via base class inheritance

Documentation URLs via Sphinx Inventory

Why Eleventy for the static site?

Path prefix handling

Troubleshooting

Can't find where a class is used

Color issues

Changes needed in `apache/airflow-site` repo (separate PR)

Data Extraction (`dev/registry/`)

Relationship to `run_provider_yaml_files_check.py`