Table of contents
The prerequisites to release Apache Airflow are described in README.md.
You can read more about the command line tools used to generate the packages and the two types of packages we have (Backport and Regular Provider Packages) in Provider packages.
You can release provider packages separately from the main Airflow on an ad-hoc basis, whenever we find that a given provider needs to be released - due to new features or due to bug fixes. You can release each provider package separately, but due to voting and release overhead we try to group releases of provider packages together.
We are using the CALVER versioning scheme for the backport packages. We also have an automated way to prepare and build the packages, so it should be very easy to release the packages often and separately. Backport packages will be maintained for three months after 2.0.0 version of Airflow, and it is really a bridge, allowing people to migrate to Airflow 2.0 in stages, so the overhead of maintaining semver versioning does not apply there - subsequent releases might be backward-incompatible, and it is not indicated by the version of the packages.
We are using the SEMVER versioning scheme for the regular packages. This is in order to give the users confidence about maintaining backwards compatibility in the new releases of those packages.
Details about maintaining the SEMVER version are going to be discussed and implemented in the related issue
Prepare release notes for all the packages you plan to release. Where YYYY.MM.DD is the CALVER date for the packages.
./breeze --backports prepare-provider-readme YYYY.MM.DD [packages]
If you iterate with merges and release candidates you can update the release date without providing the date (to update the existing release notes)
./breeze --backports prepare-provider-readme google
Generated readme files should be eventually committed to the repository.
The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any modification than renaming i.e. the contents of the files must be the same between voted release candidate and final release. Because of this the version in the built artifacts that will become the official Apache releases must not include the rcN suffix. They also need to be signed and have checksum files. You can generate the checksum/signature files by running the “dev/sign.sh” script (assuming you have the right PGP key set-up for signing). The script generates corresponding .asc and .sha512 files for each file to sign.
export VERSION=2020.5.20rc2 export AIRFLOW_REPO_ROOT=$(pwd)
${AIRFLOW_REPO_ROOT}/dev/provider_packages/build_source_package.sh --backports
It will generate apache-airflow-backport-providers-${VERSION}-source.tar.gz
./breeze --backports prepare-provider-packages --package-format both --version-suffix-for-svn rc1
if you ony build few packages, run:
./breeze --backports prepare-provider-packages --package-format both --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
mv apache-airflow-backport-providers-${VERSION}-source.tar.gz dist
pushd dist ../dev/sign.sh * popd
git push apache backport-providers-${VERSION}
# First clone the repo if you do not have it svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev # update the repo in case you have it already cd airflow-dev svn update # Create a new folder for the release. cd airflow-dev/backport-providers svn mkdir ${VERSION} # Move the artifacts to svn folder mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/ # Add and commit svn add ${VERSION}/* svn commit -m "Add artifacts for Airflow ${VERSION}" cd ${AIRFLOW_REPO_ROOT}
Verify that the files are available at backport-providers
In order to publish to PyPI you just need to build and release packages. The packages should however contain the rcN suffix in the version name as well, so you need to use --version-suffix-for-pypi switch to prepare those packages. Note that these are different packages than the ones used for SVN upload though they should be generated from the same sources.
./breeze --backports prepare-provider-packages --package-format both --version-suffix-for-pypi rc1
if you ony build few packages, run:
./breeze --backports prepare-provider-packages --package-format both --version-suffix-for-pypi rc1 PACKAGE PACKAGE ....
twine check dist/*
twine upload -r pypitest dist/*
Verify that the test packages look good by downloading it and installing them into a virtual environment. Twine prints the package links as output - separately for each package.
Upload the package to PyPi's production environment:
twine upload -r pypi dist/*
Copy the list of links to the uploaded packages - they will be useful in preparing VOTE email.
Again, confirm that the packages are available under the links printed.
Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/backport-providers/
Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the email using the ${VERSION} variable you already set in the command line.
subject:
cat <<EOF [VOTE] Airflow Backport Providers ${VERSION} EOF
cat <<EOF Hey all, I have cut Airflow Backport Providers ${VERSION}. This email is calling a vote on the release, which will last for 72 hours - which means that it will end on $(date -d '+3 days'). Consider this my (binding) +1. Airflow Backport Providers ${VERSION} are available at: https://dist.apache.org/repos/dist/dev/airflow/backport-providers/${VERSION}/ *apache-airflow-backport-providers-${VERSION}-source.tar.gz* is a source release that comes with INSTALL instructions. *apache-airflow-backport-providers-<PROVIDER>-${VERSION}-bin.tar.gz* are the binary Python "sdist" release. The test procedure for PMCs and Contributors who would like to test the RC candidates are described in https://github.com/apache/airflow/blob/master/dev/README.md#vote-and-verify-the-backport-providers-release-candidate Public keys are available at: https://dist.apache.org/repos/dist/release/airflow/KEYS Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove with the reason Only votes from PMC members are binding, but members of the community are encouraged to test the release and vote with "(non-binding)". Please note that the version number excludes the 'rcX' string, so it's now simply ${VERSION%rc?}. This will allow us to rename the artifact without modifying the artifact checksums when we actually release. Each of the packages contains detailed changelog. Here is the list of links to the released packages and changelogs: <PASTE TWINE UPLOAD LINKS HERE. SORT THEM BEFORE!> Cheers, <TODO: Your Name> EOF
Due to the nature of backport packages, not all packages have to be released as convenience packages in the final release. During the voting process the voting PMCs might decide to exclude certain packages from the release if some critical problems have been found in some packages.
Please modify the message above accordingly to clearly exclude those packages.
The files should be present in the sub-folder of Airflow dist
The following files should be present (9 files):
As a PMC you should be able to clone the SVN repository:
svn co https://dist.apache.org/repos/dist/dev/airflow/
Or update it if you already checked it out:
svn update .
This can be done with the Apache RAT tool.
java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
Make sure you have the key of person signed imported in your GPG. You can find the valid keys in KEYS.
You can import the whole KEYS file:
gpg --import KEYS
You can also import the keys individually from a keyserver. The below one uses Kaxil's key and retrieves it from the default GPG keyserver OpenPGP.org:
gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
You should choose to import the key when asked.
Note that by being default, the OpenPGP server tends to be overloaded often and might respond with errors or timeouts. Many of the release managers also uploaded their keys to the GNUPG.net keyserver, and you can retrieve it from there.
gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
Once you have the keys, the signatures can be verified by running this:
for i in *.asc do echo "Checking $i"; gpg --verify $i done
This should produce results similar to the below. The “Good signature from ...” is indication that the signatures are correct. Do not worry about the “not certified with a trusted signature” warning. Most of the certificates used by release managers are self signed, that's why you get this warning. By importing the server in the previous step and importing it via ID from KEYS page, you know that this is a valid Key already.
Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz' gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B gpg: Good signature from "Kaxil Naik <kaxilnaik@gmail.com>" [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl' gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B gpg: Good signature from "Kaxil Naik <kaxilnaik@gmail.com>" [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B Checking apache-airflow-1.10.12rc4-source.tar.gz.asc gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz' gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B gpg: Good signature from "Kaxil Naik <kaxilnaik@gmail.com>" [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
Run this:
for i in *.sha512 do echo "Checking $i"; shasum -a 512 `basename $i .sha512 ` | diff - $i done
You should get output similar to:
Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512 Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512 Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the actual users of Apache Airflow test it in their own staging/test installations. Each release candidate is available on PyPI apart from SVN packages, so everyone should be able to install the release candidate version of Airflow via simply ( is 1.10.12 for example, and is release candidate number 1,2,3,....).
You can use any of the installation methods you prefer (you can even install it via the binary wheels downloaded from the SVN).
You have to make sure you have Airflow 1.10.* installed in your PIP virtualenv (the version you want to install providers with).
pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X>
There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow. Here is a typical scenario.
First copy all the provider packages .whl files to the dist folder.
./breeze start-airflow --install-airflow-version <VERSION>rc<X> \ --python 3.7 --backend postgres --install-packages-from-dist
For 1.10 releases you can also use --no-rbac-ui flag disable RBAC UI of Airflow:
./breeze start-airflow --install-airflow-version <VERSION>rc<X> \ --python 3.7 --backend postgres --install-packages-from-dist --no-rbac-ui
If you prefer to build your own image, you can also use the official image andipi PyPI packages to test backport packages. This is especially helpful when you want to test integrations, but you need to install additional tools. Below is an example Dockerfile, which installs backport providers for Google and an additional third-party tools:
FROM apache/airflow:1.10.12 RUN pip install --user apache-airflow-backport-providers-google==2020.10.5.rc1 RUN curl https://sdk.cloud.google.com | bash \ && echo "source /home/airflow/google-cloud-sdk/path.bash.inc" >> /home/airflow/.bashrc \ && echo "source /home/airflow/google-cloud-sdk/completion.bash.inc" >> /home/airflow/.bashrc USER 0 RUN KUBECTL_VERSION="$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)" \ && KUBECTL_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \ && curl -L "${KUBECTL_URL}" --output /usr/local/bin/kubectl \ && chmod +x /usr/local/bin/kubectl USER ${AIRFLOW_UID}
To build an image build and run a shell, run:
docker build . -t my-airflow docker run -ti \ --rm \ -v "$PWD/data:/opt/airflow/" \ -v "$PWD/keys/:/keys/" \ -p 8080:8080 \ -e GOOGLE_APPLICATION_CREDENTIALS=/keys/sa.json \ -e AIRFLOW__CORE__LOAD_EXAMPLES=True \ my-airflow bash
Once you install and run Airflow, you can perform any verification you see as necessary to check that the Airflow works as you expected.
Once the vote has been passed, you will need to send a result vote to dev@airflow.apache.org:
Subject:
cat <<EOF [RESULT][VOTE] Airflow Backport Providers ${VERSION} EOF
Body:
cat <<EOF Hey all, Airflow Backport Providers ${VERSION} (based on the ${VERSION_RC} candidate) has been accepted. N "+1" binding votes received: - PMC Member (binding) ... N "+1" non-binding votes received: - COMMITER (non-binding) Vote thread: https://lists.apache.org/thread.html/<TODO:REPLACE_ME_WITH_THE_VOTING_THREAD>@%3Cdev.airflow.apache.org%3E I'll continue with the release process and the release announcement will follow shortly. Cheers, <TODO: Your Name> EOF
The best way of doing this is to svn cp between the two repos (this avoids having to upload the binaries again, and gives a clearer history in the svn commit logs.
We also need to archive older releases before copying the new ones Release policy
# Set the variables export VERSION_RC=2020.5.20rc2 export VERSION=${VERSION_RC/rc?/} # Set AIRFLOW_REPO_ROOT to the path of your git repo export AIRFLOW_REPO_ROOT=$(pwd) # Go to the directory where you have checked out the dev svn release # And go to the sub-folder with RC candidates cd "<ROOT_OF_YOUR_DEV_REPO>/backport-providers/${VERSION_RC}" export SOURCE_DIR=$(pwd) # Go the folder where you have checked out the release repo # Clone it if it's not done yet svn checkout https://dist.apache.org/repos/dist/release/airflow airflow-release # Update to latest version svn update # Create backport-providers folder if it does not exist # All latest releases are kept in this one folder without version sub-folder mkdir -pv backport-providers cd backport-providers # Move the artifacts to svn folder & remove the rc postfix for file in ${SOURCE_DIR}/*${VERSION_RC}* do base_file=$(basename ${file}) svn cp "${file}" "${base_file/${VERSION_RC}/${VERSION}}" done # If some packages have been excluded, remove them now # Check the packages ls *<provider>* # Remove them svn rm *<provider>* # Check which old packages will be removed (you need python 3.6+) python ${AIRFLOW_REPO_ROOT}/dev/provider_packages/remove_old_releases.py \ --directory . # Remove those packages python ${AIRFLOW_REPO_ROOT}/dev/provider_packages/remove_old_releases.py \ --directory . --execute # Commit to SVN svn commit -m "Release Airflow Backport Providers ${VERSION} from ${VERSION_RC}"
Verify that the packages appear in backport-providers
Checkout the RC Version:
git checkout backport-providers-${VERSION_RC}
Tag and push the final version (providing that your apache remote is named ‘apache’):
git tag backport-providers-${VERSION} git push apache backport-providers-${VERSION}
In order to publish to PyPI you just need to build and release packages.
./breeze --backports prepare-provider-packages --package-format both
if you ony build few packages, run:
./breeze --backports prepare-provider-packages --package-format both <PACKAGE> ...
In case you decided to remove some of the packages. remove them from dist folder now:
ls dist/*<provider>* rm dist/*<provider>*
twine check dist/*
twine upload -r pypitest dist/*
Verify that the test packages look good by downloading it and installing them into a virtual environment. Twine prints the package links as output - separately for each package.
Upload the package to PyPi's production environment:
twine upload -r pypi dist/*
Subject:
cat <<EOF Airflow Backport Providers ${VERSION} are released EOF
Body:
cat <<EOF Dear Airflow community, I'm happy to announce that Airflow Backport Providers packages ${VERSION} were just released. The source release, as well as the binary releases, are available here: https://dist.apache.org/repos/dist/release/airflow/backport-providers/ We also made those versions available on PyPi for convenience ('pip install apache-airflow-backport-providers-*'): https://pypi.org/search/?q=apache-airflow-backport-providers The documentation and changelogs are available in the PyPI packages: <PASTE TWINE UPLOAD LINKS HERE. SORT THEM BEFORE!> Cheers, <your name> EOF
Update “Announcements” page at the Official Airflow website
Prepare release notes for all the packages you plan to release. Note that for now version number is hard-coded to 0.0.1 for all packages. Later on we are going to update the versions according to SEMVER versioning.
Details about maintaining the SEMVER version are going to be discussed and implemented in the related issue
./breeze prepare-provider-readme [packages]
You can iterate and re-generate the same readme content as many times as you want. Generated readme files should be eventually committed to the repository.
There is a slightly different procedure if you build pre-release (alpha/beta) packages and the release candidates. For the Alpha artifacts there is no voting and signature/checksum check, so we do not need to care about this part. For release candidates - those packages might get promoted to “final” packages by just renaming the files, so internally they should keep the final version number without the rc suffix, even if they are rc1/rc2/... candidates.
They also need to be signed and have checksum files. You can generate the checksum/signature files by running the “dev/sign.sh” script (assuming you have the right PGP key set-up for signing). The script generates corresponding .asc and .sha512 files for each file to sign.
Currently, we are releasing alpha provider packages together with the main sources of Airflow. In the future we are going to add procedure to release the sources of released provider packages separately. Details are in the related issue
For alpha/beta releases you need to specify both - svn and pyp i - suffixes, and they have to match. This is verified by the breeze script. Note that the script will clean up dist folder before generating the packages, so it will only contain the packages you intended to build.
export VERSION=0.0.1alpha1 ./breeze prepare-provider-packages --package-format both --version-suffix-for-svn a1 --version-suffix-for-pypi a1
if you ony build few packages, run:
./breeze prepare-provider-packages --package-format both --version-suffix-for-svn a1 --version-suffix-for-pypi a1 \ PACKAGE PACKAGE ....
export VERSION=0.0.1alpha1 ./breeze prepare-provider-packages --package-format both --version-suffix-for-svn rc1
if you ony build few packages, run:
./breeze prepare-provider-packages --package-format both --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
pushd dist ../dev/sign.sh * popd
# First clone the repo if you do not have it svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev # update the repo in case you have it already cd airflow-dev svn update # Create a new folder for the release. cd airflow-dev/providers svn mkdir ${VERSION} # Move the artifacts to svn folder mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/ # Add and commit svn add ${VERSION}/* svn commit -m "Add artifacts for Airflow Providers ${VERSION}" cd ${AIRFLOW_REPO_ROOT}
Verify that the files are available at backport-providers
In case of pre-release versions you build the same packages for both PyPI and SVN so you can simply use packages generated in the previous step, and you can skip the “prepare” step below.
In order to publish release candidate to PyPI you just need to build and release packages. The packages should however contain the rcN suffix in the version file name but not internally in the package, so you need to use --version-suffix-for-pypi switch to prepare those packages. Note that these are different packages than the ones used for SVN upload though they should be generated from the same sources.
./breeze prepare-provider-packages --package-format both --version-suffix-for-pypi a1 --version-suffix-for-SVN a1
if you ony build few packages, run:
./breeze prepare-provider-packages --package-format both --version-suffix-for-pypi a1 \ PACKAGE PACKAGE ....
twine check dist/*
twine upload -r pypitest dist/*
Verify that the test packages look good by downloading it and installing them into a virtual environment. Twine prints the package links as output - separately for each package.
Upload the package to PyPi's production environment:
twine upload -r pypi dist/*
Documentation is an essential part of the product and should be made available to users. In our cases, documentation for the released versions is published in a separate repository - apache/airflow-site, but the documentation source code and build tools are available in the apache/airflow repository, so you have to coordinate between the two repositories to be able to build the documentation.
Documentation for providers can be found in the /docs/apache-airflow-providers directory and the /docs/apache-airflow-providers-*/ directory. The first directory contains the package contents lists and should be updated every time a new version of provider packages is released.
First, copy the airflow-site repository and set the environment variable AIRFLOW_SITE_DIRECTORY.
git clone https://github.com/apache/airflow-site.git airflow-site cd airflow-site export AIRFLOW_SITE_DIRECTORY="$(pwd)"
Then you can go to the directory and build the necessary documentation packages
cd "${AIRFLOW_REPO_ROOT}" ./breeze build-docs -- \ --package-filter apache-airflow-providers \ --package-filter apache-airflow-providers-apache-airflow \ --package-filter apache-airflow-providers-telegram \ --for-production
Now you can preview the documentation.
./docs/start_doc_server.sh
Copy the documentation to the airflow-site repository
./docs/publish_docs.py \ --package-filter apache-airflow-providers \ --package-filter apache-airflow-providers-apache-airflow \ --package-filter apache-airflow-providers-telegram \ cd "${AIRFLOW_SITE_DIRECTORY}"
If you publish a new package, you must add it to the docs index:
Create commit and push changes.
git commit -m "Add documentation for backport packages - $(date "+%Y-%m-%d%n")" git push
Subject:
cat <<EOF Airflow Providers are released EOF
Body:
cat <<EOF Dear Airflow community, I'm happy to announce that new version of Airflow Providers packages were just released. The source release, as well as the binary releases, are available here: https://dist.apache.org/repos/dist/release/airflow/providers/ We also made those versions available on PyPi for convenience ('pip install apache-airflow-providers-*'): https://pypi.org/search/?q=apache-airflow-providers The documentation and changelogs are available in the PyPI packages: <PASTE TWINE UPLOAD LINKS HERE. SORT THEM BEFORE!> Cheers, <your name> EOF