Table of contents
cherry-picker CLIThe Apache Airflow releases are one of the two types:
This package contains sources that allow the user building fully-functional Apache Airflow 2.0 package. They contain sources for:
We also plan to release the official Helm Chart sources that will allow the user to install Apache Airflow via helm 3.0 chart in a distributed fashion.
The Source releases are the only “official” Apache Software Foundation releases, and they are distributed via Official Apache Download sources
Following source releases Apache Airflow release manager also distributes convenience packages:
Those convenience packages are not “official releases” of Apache Airflow, but the users who cannot or do not want to build the packages themselves can use them as a convenient way of installing Apache Airflow, however they are not considered as “official source releases”. You can read more details about it in the ASF Release Policy.
Detailed instruction of releasing provider distributions can be found in the README_RELEASE_AIRFLOW.md
The Provider distributions are distributions (separate one per provider) that make it possible to easily install Hooks, Operators, Sensors, and Secrets for different providers (external services used by Airflow).
Once you release the packages, you can simply install them with:
pip install apache-airflow-providers-<PROVIDER>[<EXTRAS>]
Where <PROVIDER> is the provider id and <EXTRAS> are optional extra packages to install. You can find the provider distributions dependencies and extras in the README.md files in each provider package (in airflow/providers/<PROVIDER> folder) as well as in the PyPI installation page.
The sources released in SVN allow to build all the provider distributions by the user, following the instructions and scripts provided. Those are also “official_source releases” as described in the ASF Release Policy and they are available via Official Apache Download for providers.
The full provider's list can be found here: Provider distributions reference
There are also convenience packages released as “apache-airflow-providers”separately in PyPI. PyPI query for providers
And available in PyPI: PyPI query for backport providers.
Note that Backport Providers for Airflow 1.10.* series are not released anymore. The last release of Backport Providers was done on March 17, 2021.
Detailed instruction of releasing provider distributions can be found in the README_RELEASE_PROVIDERS.md
The dev/preinstalled-providers.json file contains the list of provider ids that are pre-installed. Those providers are dynamically added to generated standard wheel packages that are released in PyPI. Those packages are not present in pyproject.toml as dependencies, and they are not installed when you install Airflow for editable installation for development. This way, when you develop Airflow you can work on Airflow and Providers together from the same Source tree - without polluting your editable installation with installed provider distributions.
The person acting as release manager has to fulfill certain pre-requisites. More details and FAQs are available in the ASF Release Policy but here some important pre-requisites are listed below. Note that release manager does not have to be a PMC member - it is enough to be committer to assume the release manager role, but there are final steps in the process (uploading final releases to SVN) that can only be done by PMC member. If needed, the release manager can ask a PMC member to perform that final step of release.
Make sure your public key is on id.apache.org and in KEYS. You will need to sign the release artifacts with your pgp key. After you have created a key, make sure you:
# Create PGP Key gpg --gen-key # Checkout ASF dist repo svn checkout https://dist.apache.org/repos/dist/release/airflow cd airflow # Add your GPG pub key to KEYS file. Replace "Kaxil Naik" with your name (gpg --list-sigs "Kaxil Naik" && gpg --armor --export "Kaxil Naik" ) >> KEYS # Commit the changes svn commit -m "Add PGP keys of Airflow developers"
See this for more detail on creating keys and what is required for signing releases.
http://www.apache.org/dev/release-signing.html#basic-facts
In order to not reveal your password in plain text, it's best if you create and configure API Upload tokens. You can add and copy the tokens here:
Create a ~/.pypirc file:
[distutils] index-servers = pypi [pypi] username=__token__ password=<API Upload Token>
Set proper permissions for the pypirc file:
chmod 600 ~/.pypirc
uv tool install twine
$ chmod 600 ~/.pypircBreeze can not install GitHub client for you thus it is required to set it manually. Follow instructions listed in Prerequisites <dev/breeze/doc/01_installation.rst#The ``gh`` cli needed for release managers>_ to install GitHub client.
The best way to prepare and verify the releases is to prepare them on a hardware owned and controlled by the committer acting as release manager. While strictly speaking, releases must only be verified on hardware owned and controlled by the committer, for practical reasons it's best if the packages are prepared using such hardware. More information can be found in this FAQ
cherry-picker CLIBackporting via CLI might be more convenient for some users. Also, it is necessary if you want to backport PR that has conflicts. It also allows to backport commit to multiple branches in the same command.
To backport PRs to any branch (for example: v2-11-test, v3-2-test or chart/v1-2x-test), you can use the following command:
It's easiest to install it (and keep cherry-picker up-to-date) using uv tool:
uv tool install cherry-picker
And upgrade it with:
uv tool upgrade cherry-picker
Then, in order to backport a commit to a branch, you can use the following command:
cherry_picker COMMIT_SHA BRANCH_NAME1 [BRANCH_NAME2 ...]
If you are using your fork repository while keeping Airflow repository as remote (i.e. apache), please use
cherry_picker COMMIT_SHA BRANCH_NAME1 [BRANCH_NAME2 ...] --upstream-remote [REMOTE_REPOSITORY_NAME]
This will create a new branch with the cherry-picked commit and open a PR against the target branch in your browser.
If the GH_AUTH environment variable is set in your command line, the cherry-picker automatically creates a new pull request when there are no conflicts. To set GH_AUTH, use the token from your GitHub repository.
To set GH_AUTH run this:
export GH_AUTH={token}
Sometimes it might result with conflict. In such case, you should manually resolve the conflicts. Some IDEs like IntelliJ has a fantastic conflict resolution tool - just follow Git -> Resolve conflicts menu after you get the conflict. But you can also resolve the conflicts manually; see How conflicts are are presented and How to resolve conflicts for more details.
cherry_picker --status # Should show if all conflicts are resolved cherry_picker --continue # Should continue cherry-picking process
[!WARNING] Sometimes, when you stop cherry-picking process in the middle, you might end up with your repo in a bad state and cherry-picker might print this message:
🐍 🍒 ⛏
You're not inside a cpython repo right now! 🙅
You should then run
cherry-picker --abortto clean up the mess and start over. If that does not work you might need to rungit config --local --remove-section cherry-pickerto clean up the configuration stored in.git/config.