Typically, releases are done coinciding with major and minor releases to Airflow. Therefore, a release of (for example) 2.3.0 of this client would correspond with 2.3.X of Airflow.
The Python client is generated using Airflow's openapi spec. To update the client for new APIs do the following steps:
# clone this repo git clone git@github.com:apache/airflow-client-python.git cd airflow-client-python export CLIENT_REPO_ROOT=$(pwd) cd .. # clone Airflow repo (if not already) git clone git@github.com:apache/airflow.git cd airflow export AIRFLOW_REPO_ROOT=$(pwd)
Edit the file airflow/airflow/api_connexion/openapi/v1.yaml Make sure it has the following securitySchemas listed under security section
security: - Basic: [] - GoogleOpenId: [] - Kerberos: []
If your deployment of Airflow uses any different authentication mechanism than the three listed above, you might need to make further changes to the v1.yaml and generate your own client, see OpenAPI Schema specification for details. (These changes should not be commited to the upstream v1.yaml as it will generate misleading openapi documentaion)
# bump up the version in python.sh & run the following command ${AIRFLOW_REPO_ROOT}/clients/gen/python.sh airflow/api_connexion/openapi/v1.yaml ${CLIENT_REPO_ROOT}/airflow_client cd ${CLIENT_REPO_ROOT}
Set your version in setup.py (without the RC tag)
Get a diff between the last airflow version and the current airflow version which this release is based on:
```shell script # Run this command in Airflow Repo git log 2.3.0..2.5.0 --pretty=oneline -- airflow/api_connexion/openapi/v1.yaml ```
Update CHANGELOG.md with the details.
Raise a PR in airflow-client-python
Merge the above PR when approved before proceeding
The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any modification than renaming – i.e. the contents of the files must be the same between voted release candidate and final release. Because of this the version in the built artifacts that will become the official Apache releases must not include the rcN suffix.
Set environment variables
# Set Version export VERSION=2.0.0rc1 export VERSION_WITHOUT_RC=${VERSION%rc?} # Set the airflow version that this release is based export AIRFLOW_VERSION=2.1.4 # Example after cloning git clone https://github.com/apache/airflow-client-python.git cd airflow-client-python export CLIENT_REPO_ROOT=$(pwd)
Tag your release
git tag -s ${VERSION} -m "Airflow Python Client {VERSION}"
Clean the checkout: the sdist step below will
git clean -fxd
Tarball the repo
git archive --format=tar.gz ${VERSION} --prefix=apache-airflow-client-${VERSION_WITHOUT_RC}/ -o apache-airflow-client-${VERSION_WITHOUT_RC}-source.tar.gz
Generate sdist
NOTE: Make sure your checkout is clean at this stage - any untracked or changed files will otherwise be included in the file produced.
pip install wheel python setup.py sdist bdist_wheel
Rename the sdist
mv dist/apache-airflow-client-${VERSION_WITHOUT_RC}.tar.gz dist/apache-airflow-client-${VERSION_WITHOUT_RC}-bin.tar.gz mv dist/apache_airflow_client-${VERSION_WITHOUT_RC}-py3-none-any.whl dist/apache_airflow_client-${VERSION_WITHOUT_RC}-py3-none-any.whl
Generate SHA512/ASC (If you have not generated a key yet, generate it by following instructions on http://www.apache.org/dev/openpgp.html#key-gen-generate-key)
pushd dist ${CLIENT_REPO_ROOT}/dev/sign.sh * popd
Push the artifacts to ASF dev dist repo
# First clone the repo svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev # Create new folder for the release cd airflow-dev/clients/python svn mkdir ${VERSION} # Move the artifacts to svn folder & commit mv ${CLIENT_REPO_ROOT}/dist/apache{-,_}*client-${VERSION_WITHOUT_RC}* ${VERSION}/ cd ${VERSION} svn add * svn commit -m "Add artifacts for Apache Airflow Python Client ${VERSION}" cd ${CLIENT_REPO_ROOT} rm -rf airflow-dev
At this point we have the artefact that we vote on, but as a convenience to developers we also want to publish “snapshots” of the RC builds to pypi for installing via pip.
To do this we need to
Build the package:
cd ${CLIENT_REPO_ROOT} python setup.py egg_info --tag-build "$(sed -e "s/^[0-9.]*//" <<<"$VERSION")" sdist bdist_wheel
Verify the artifacts that would be uploaded:
twine check dist/*
Upload the package to PyPi's test environment:
twine upload --repository-url=https://test.pypi.org/legacy/ dist/*
Verify that the test package looks good by downloading it and installing it into a virtual environment. The package download link is available at: https://test.pypi.org/project/apache-airflow-client/#files
Or via pypi pip install -i https://test.pypi.org/simple/ apache-airflow-client==${VERSION}
Upload the package to PyPi's production environment:
twine upload -r pypi dist/*
Again, confirm that the package is available here: https://pypi.python.org/pypi/apache-airflow-client
It is important to stress that this snapshot should not be named “release”, and it is not supposed to be used by and advertised to the end-users who do not read the devlist.
git push origin ${VERSION}
Subject:
cat <<EOF [VOTE] Release Airflow Python Client ${VERSION_WITHOUT_RC} from ${VERSION} EOF
Body:
cat <<EOF Hey fellow Airflowers, I have cut the first release candidate for the Airflow Python Client ${VERSION}. The client consists of APIs corresponding to REST APIs available in *Apache Airflow ${AIRFLOW_VERSION}*. This email is calling for a vote on the release, which will last for 72 hours. Consider this my (binding) +1. Airflow Client ${VERSION} is available at: https://dist.apache.org/repos/dist/dev/airflow/clients/python/${VERSION}/ Or also available at PyPI: https://pypi.org/project/apache-airflow-client/${VERSION}/ *apache-airflow-client-${VERSION}-source.tar.gz* is a source release that comes with INSTALL instructions. *apache-airflow-client-${VERSION}-bin.tar.gz* is the binary Python "sdist" release. Public keys are available at: https://dist.apache.org/repos/dist/release/airflow/KEYS Only votes from PMC members are binding, but the release manager should encourage members of the community to test the release and vote with "(non-binding)". *Changelog:* *Major changes:* ... *Major fixes:* ... *New API supported:* ... Cheers, <your name> EOF
See Airflow process documented here.
See Airflow process documented here (just replace Airflow with Airflow Client).
See Airflow process documented here.
See Airflow process documented here.
This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the actual users of Airflow Client test it in their own staging/test installations. Each release candidate is available on PyPI apart from SVN packages, so everyone should be able to install the release candidate version of Airflow Client via simply ( is 2.0.0 for example, and is release candidate number 1,2,3,....).
pip install apache-airflow-client==<VERSION>rc<X>
Once you install and run Airflow Client, you should perform any verification you see as necessary to check that the client works as you expected.
Hello, Apache Airflow Python Client 2.5.0 (based on RC1) has been accepted. 3 "+1" binding votes received: - Ephraim Anierobi - Jarek Potiuk - Jed Cunningham 1 "+1" non-binding votes received: - Pierre Jeambrun Vote thread: https://lists.apache.org/thread/1qcj0r67dff3zg0w2vyfhr30fx9xtp3y I'll continue with the release process, and the release announcement will follow shortly. Cheers, <your name>
# Go to Airflow python client sources first cd <YOUR_AIRFLOW_CLIENT_REPO_ROOT> export CLIENT_REPO_ROOT="$(pwd)" cd .. # Clone the AS [ -d asf-dist ] || svn checkout --depth=immediates https://dist.apache.org/repos/dist asf-dist svn update --set-depth=infinity asf-dist/{release,dev}/airflow CLIENT_DEV_SVN="${PWD}/asf-dist/dev/airflow/clients/python" CLIENT_RELEASE_SVN="${PWD}/asf-dist/release/airflow/clients/python" cd "${CLIENT_RELEASE_SVN}" export RC=2.0.0rc1 export VERSION=${RC/rc?/} # Create new folder for the release svn mkdir ${VERSION} cd ${VERSION} # Move the artifacts to svn folder & commit for f in ${CLIENT_DEV_SVN}/$RC/*; do svn cp $f . ; done svn commit -m "Release Apache Airflow Python Client ${VERSION} from ${RC}" # Remove old release # http://www.apache.org/legal/release-policy.html#when-to-archive cd .. export PREVIOUS_VERSION=1.0.0 svn rm ${PREVIOUS_VERSION} svn commit -m "Remove old release: ${PREVIOUS_VERSION}"
Verify that the packages appear in airflow
At this point we release an official package:
Copy the packages from the SVN into the dist folder in CLIENT_REPO_ROOT:
rm -rf ${CLIENT_REPO_ROOT}/dist mkdir ${CLIENT_REPO_ROOT}/dist cp ${CLIENT_RELEASE_SVN}/${VERSION}/apache-airflow-client-${VERSION}-bin.tar.gz ${CLIENT_REPO_ROOT}/dist/ cp ${CLIENT_RELEASE_SVN}/${VERSION}/apache_airflow_client-${VERSION}-py3-none-any.whl ${CLIENT_REPO_ROOT}/dist/ # Remove the -bin mv ${CLIENT_REPO_ROOT}/dist/apache-airflow-client-${VERSION}-bin.tar.gz ${CLIENT_REPO_ROOT}/dist/apache-airflow-client-${VERSION}.tar.gz
Verify the artifacts that would be uploaded:
twine check dist/*
Upload the package to PyPi's test environment:
twine upload --repository-url=https://test.pypi.org/legacy/ dist/*
Verify that the test package looks good by downloading it and installing it into a virtual environment. The package download link is available at: https://test.pypi.org/project/apache-airflow-client/#files
Upload the package to PyPi's production environment:
twine upload -r pypi dist/*
Again, confirm that the package is available here: https://pypi.python.org/pypi/apache-airflow-client
Push Tag for the final version
git tag ${VERSION} git push origin ${VERSION}
Create a new release on GitHub with the release notes and assets from the release svn.
See Airflow process documented here (just replace Airflow with Airflow Client)
Add the release data (version and date) at: https://reporter.apache.org/addrelease.html?airflow