Shared Python Code for Airflow Components

This folder contains code shared across two or more of the Airflow distributions, such as airflow-core and task-sdk.

Be Thoughtful about what you add under here

Not every piece of code used in two distributions should be automatically placed in one of the shared libraries, and sometimes “just duplicate it” is the right approach to take. For example, if it‘s just a 5 or 10 line function and it’s used in two places, it might be easier for future developers to understand if the function is in two places.

There is no hard rule about what should or shouldn't be in these libraries, so try to apply your best judgement.

Mechanics of sharing

The primary mechanism by which we share code is to use in-repo symlinks. This means that there is a single copy of the code existing in the repo (no need for prek hook to update other copies, resulting in larger diffs).

The primary reason we are using this sharing approach, rather than the perhaps more traditional approach of depending on a shared distribution from PyPI is around compatibility. Let's say that we have a library to parse AirflowConfig (including all the sources, env, _cmd handling, etc etc.). If that code is used by two distributions that are installed in the same Python environment, then either we introduce another source of “version hell” or we have to make every change both backwards and forwards compatible.

Instead, by using an approach similar to vendoring (or if you compare it to C libraries - static linking), where each dist ships with a copy of the code of the version it knows it works with, it allows them to co-exist in the same env without problems, and also reduces the cognitive load on developers when making changes to the shared libs -- as long as it passes the CI it should be good (and we don't have to test an ever increasing matrix of versions to ensure compatibility.)

Shared libraries must use relative imports for other shared libraries

The one caveat to this is due to the side-effect that these shared modules are going to be imported from different locations (for example airflow._shared.timezones.timezone and airflow.sdk._shared.timezones.timezone) then any imports inside the shared code to other parts of shared libraries must make use of relative imports, and possibly needs # noqa: TID252

For example, to use the shared timezone library from another shared library, let's say shared/logging/src/airflow_shared/logging/config.py you would have

from ..timezones.timezone import is_naive  # noqa: TID252

This is different to the rest of the airflow codebase where relative imports are not allowed.

Shared libraries should be grouped one level beneath airflow_shared/

In order to make both the relative imports above function and to provide simple structure, we want to arrange things in single named chunks, where the folder name is also the “shared library name”. For example:

shared
├── README.md
└── timezones/
    ├── pyproject.toml
    ├── src/
    │   └── airflow_shared
    │       └── timezones/
    │           ├── __init__.py
    └── tests/
└── logging/
    ├── pyproject.toml
    ├── src/
    │   └── airflow_shared
    │       └── logging/
    │           ├── __init__.py
    └── tests/

Use In Editable installs/Git checkouts

In the editable checkout (which is what you get by default when using uv sync etc), the shared libraries are symlinked directly in place, for example task-sdk/src/airflow/sdk/_shared/timezones is symlinked to shared/timezones/src/airflow_shared/timezones. This means that python automatically finds the files underneath the module as if they lived directly under airflow.sdk._shared.timezones.X.

Including In built distributions

Symlinks will not work when we build the main distributions for PyPI and other channels (since the symlinks would be to files outside the distribution otherwise). This is specific to sdist; building wheel resolves the symlinks.

We have a convention that when you include a shared code in your distribution, you should such shared distributions (full name of the form apache-airlow-shared-<name> it to the tool.airflow section of your pyproject.toml:

[tool.airflow]

shared_distributions = [
     "apache-airflow-shared-timezones",
]

This allows prek check to automatically verify if the shared distributio is properly configured in your source tree and pyproject.toml.

You should also have _shared folder created in your source tree to be able to symlink the shared distributions code.

Once you do it, prek hook will automatically check and either correct the distribution using the shared one, but you can also do it manually (for each shared distribution used):

• creating a symlink of the sub-folder of airflow_shared in the shared folder to the corresponding shared distribution, e.g. ln -s ../../shared/timezones/src/airflow_shared/timezones src/airflow/sdk/_shared/timezones

• adding force-include in your pyproject.toml to have hatchling copy the source files instead of symling when .sdist distribution is built:

[tool.hatch.build.targets.sdist.force-include]
"../shared/timezones/src/airflow_shared/timezones" = "src/airflow/sdk/_shared/timezones"

• prek hook will automatically synchronize all dependencies of the shared distributions to your pyproject.toml with appropriate comments allowing it to be automatically synced when shared distribution dependencies are updated in the future.