Fixes to release process after releasing 2nd wave of providers (#14059)
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index d4904cf..d0c94e4 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -146,7 +146,7 @@
hooks:
- id: doctoc
name: Add TOC for md files
- files: ^README\.md$|^CONTRIBUTING\.md$|^UPDATING.*.md$|^dev/.*.md$
+ files: ^README\.md$|^CONTRIBUTING\.md$|^UPDATING.*.md$|^dev/.*.md$|^dev/.*.rst
args:
- "--maxlevel"
- "2"
diff --git a/airflow/providers/apache/beam/BACKPORT_PROVIDER_README.md b/airflow/providers/apache/beam/BACKPORT_PROVIDER_README.md
index d0908b6..4e53ed8 100644
--- a/airflow/providers/apache/beam/BACKPORT_PROVIDER_README.md
+++ b/airflow/providers/apache/beam/BACKPORT_PROVIDER_README.md
@@ -62,7 +62,7 @@
You can install such cross-provider dependencies when installing from PyPI. For example:
```bash
-pip install apache-airflow-beckport-providers-apache-beam[google]
+pip install apache-airflow-backport-providers-apache-beam[google]
```
| Dependent package | Extra |
diff --git a/airflow/providers/apache/beam/README.md b/airflow/providers/apache/beam/README.md
index 3aa0ead..14146cc 100644
--- a/airflow/providers/apache/beam/README.md
+++ b/airflow/providers/apache/beam/README.md
@@ -20,7 +20,7 @@
# Package apache-airflow-providers-apache-beam
-Release: 0.0.1
+Release: 1.0.0
**Table of contents**
diff --git a/airflow/providers/apache/beam/provider.yaml b/airflow/providers/apache/beam/provider.yaml
index 4325265..0ad6d92 100644
--- a/airflow/providers/apache/beam/provider.yaml
+++ b/airflow/providers/apache/beam/provider.yaml
@@ -22,7 +22,7 @@
`Apache Beam <https://beam.apache.org/>`__.
versions:
- - 0.0.1
+ - 1.0.0
integrations:
- integration-name: Apache Beam
diff --git a/airflow/providers/sftp/BACKPORT_PROVIDER_CHANGES_2021.02.05.md b/airflow/providers/sftp/BACKPORT_PROVIDER_CHANGES_2021.02.05.md
new file mode 100644
index 0000000..6be85ec
--- /dev/null
+++ b/airflow/providers/sftp/BACKPORT_PROVIDER_CHANGES_2021.02.05.md
@@ -0,0 +1,28 @@
+
+
+### Release 2021.2.5
+
+| Commit | Committed | Subject |
+|:-----------------------------------------------------------------------------------------------|:------------|:---------------------------------------------------------------------------------|
+| [9af452aee](https://github.com/apache/airflow/commit/9af452aee53dd300535dadd108ce3e60c5f11a2e) | 2021-02-03 | `Add retryer to SFTP hook connection (#13065)` |
+| [ac2f72c98](https://github.com/apache/airflow/commit/ac2f72c98dc0821b33721054588adbf2bb53bb0b) | 2021-02-01 | `Implement provider versioning tools (#13767)` |
+| [a9ac2b040](https://github.com/apache/airflow/commit/a9ac2b040b64de1aa5d9c2b9def33334e36a8d22) | 2021-01-23 | `Switch to f-strings using flynt. (#13732)` |
+| [3fd5ef355](https://github.com/apache/airflow/commit/3fd5ef355556cf0ad7896bb570bbe4b2eabbf46e) | 2021-01-21 | `Add missing logos for integrations (#13717)` |
+| [52339a55c](https://github.com/apache/airflow/commit/52339a55c054bddd1d46253575274a3d5d141ebe) | 2021-01-08 | `[AIRFLOW-7044] Host key can be specified via SSH connection extras. (#12944)` |
+| [295d66f91](https://github.com/apache/airflow/commit/295d66f91446a69610576d040ba687b38f1c5d0a) | 2020-12-30 | `Fix Grammar in PIP warning (#13380)` |
+| [6cf76d7ac](https://github.com/apache/airflow/commit/6cf76d7ac01270930de7f105fb26428763ee1d4e) | 2020-12-18 | `Fix typo in pip upgrade command :( (#13148)` |
+| [32971a1a2](https://github.com/apache/airflow/commit/32971a1a2de1db0b4f7442ed26facdf8d3b7a36f) | 2020-12-09 | `Updates providers versions to 1.0.0 (#12955)` |
+| [b40dffa08](https://github.com/apache/airflow/commit/b40dffa08547b610162f8cacfa75847f3c4ca364) | 2020-12-08 | `Rename remaing modules to match AIP-21 (#12917)` |
+| [9b39f2478](https://github.com/apache/airflow/commit/9b39f24780e85f859236672e9060b2fbeee81b36) | 2020-12-08 | `Add support for dynamic connection form fields per provider (#12558)` |
+| [c34ef853c](https://github.com/apache/airflow/commit/c34ef853c890e08f5468183c03dc8f3f3ce84af2) | 2020-11-20 | `Separate out documentation building per provider (#12444)` |
+| [008035450](https://github.com/apache/airflow/commit/00803545023b096b8db4fbd6eb473843096d7ce4) | 2020-11-18 | `Update provider READMEs for 1.0.0b2 batch release (#12449)` |
+| [ae7cb4a1e](https://github.com/apache/airflow/commit/ae7cb4a1e2a96351f1976cf5832615e24863e05d) | 2020-11-17 | `Update wrong commit hash in backport provider changes (#12390)` |
+| [6889a333c](https://github.com/apache/airflow/commit/6889a333cff001727eb0a66e375544a28c9a5f03) | 2020-11-15 | `Improvements for operators and hooks ref docs (#12366)` |
+| [7825e8f59](https://github.com/apache/airflow/commit/7825e8f59034645ab3247229be83a3aa90baece1) | 2020-11-13 | `Docs installation improvements (#12304)` |
+| [85a18e13d](https://github.com/apache/airflow/commit/85a18e13d9dec84275283ff69e34704b60d54a75) | 2020-11-09 | `Point at pypi project pages for cross-dependency of provider packages (#12212)` |
+| [59eb5de78](https://github.com/apache/airflow/commit/59eb5de78c70ee9c7ae6e4cba5c7a2babb8103ca) | 2020-11-09 | `Update provider READMEs for up-coming 1.0.0beta1 releases (#12206)` |
+| [b2a28d159](https://github.com/apache/airflow/commit/b2a28d1590410630d66966aa1f2b2a049a8c3b32) | 2020-11-09 | `Moves provider packages scripts to dev (#12082)` |
+| [41bf172c1](https://github.com/apache/airflow/commit/41bf172c1dc75099f4f9d8b3f3350b4b1f523ef9) | 2020-11-04 | `Simplify string expressions (#12093)` |
+| [4e8f9cc8d](https://github.com/apache/airflow/commit/4e8f9cc8d02b29c325b8a5a76b4837671bdf5f68) | 2020-11-03 | `Enable Black - Python Auto Formmatter (#9550)` |
+| [8c42cf1b0](https://github.com/apache/airflow/commit/8c42cf1b00c90f0d7f11b8a3a455381de8e003c5) | 2020-11-03 | `Use PyUpgrade to use Python 3.6 features (#11447)` |
+| [5a439e84e](https://github.com/apache/airflow/commit/5a439e84eb6c0544dc6c3d6a9f4ceeb2172cd5d0) | 2020-10-26 | `Prepare providers release 0.0.2a1 (#11855)` |
diff --git a/dev/README_RELEASE_PROVIDER_PACKAGES.md b/dev/README_RELEASE_PROVIDER_PACKAGES.md
index 854ec9e..6999db3 100644
--- a/dev/README_RELEASE_PROVIDER_PACKAGES.md
+++ b/dev/README_RELEASE_PROVIDER_PACKAGES.md
@@ -29,7 +29,11 @@
- [Build and sign the source and convenience packages](#build-and-sign-the-source-and-convenience-packages)
- [Commit the source packages to Apache SVN repo](#commit-the-source-packages-to-apache-svn-repo)
- [Publish the Regular convenience package to PyPI](#publish-the-regular-convenience-package-to-pypi)
+ - [Add tags in git](#add-tags-in-git)
- [Publish documentation](#publish-documentation)
+ - [Prepare voting email for Providers release candidate](#prepare-voting-email-for-providers-release-candidate)
+ - [Verify the release by PMC members](#verify-the-release-by-pmc-members)
+ - [Verify by Contributors](#verify-by-contributors)
- [Notify developers of release](#notify-developers-of-release)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
@@ -70,8 +74,6 @@
``providers-amazon/1.0.0``. During releasing, the RC1/RC2 tags are created (for example
``providers-amazon/1.0.0rc1``).
-You should start preparing the release notes by determining whether the
-
Details about maintaining the SEMVER version are going to be discussed and implemented in
[the related issue](https://github.com/apache/airflow/issues/11425)
@@ -80,8 +82,21 @@
./breeze prepare-provider-documentation [packages]
```
+This command will not only prepare documentation but will also allow the release manager to review
+changes implemented in all providers, and determine which of the providers should be released. For each
+provider details will be printed on what changes were implemented since the last release including
+links to particular commits. This should help to determine which version of provider should be released:
+
+* increased patch-level for bugfix-only change
+* increased minor version if new features are added
+* increased major version if breaking changes are added
+
+It also allows the release manager to update CHANGELOG.rst where high-level overview of the changes should be
+documented for the providers released.
+
You can iterate and re-generate the same readme content as many times as you want.
-Generated readme files should be eventually committed to the repository.
+The generated files should be added and committed to the repository.
+
## Build regular provider packages for SVN apache upload
@@ -94,18 +109,24 @@
## Build and sign the source and convenience packages
+* Cleanup dist folder:
+
+```shell script
+export AIRFLOW_REPO_ROOT=$(pwd)
+rm -rf ${AIRFLOW_REPO_ROOT}/dist/*
+```
+
+
* Release candidate packages:
```shell script
-export VERSION=1.1.0rc1
-
-./breeze prepare-provider-packages --version-suffix-for-svn rc1
+./breeze prepare-provider-packages --version-suffix-for-svn rc1 --package-format both
```
if you ony build few packages, run:
```shell script
-./breeze prepare-provider-packages --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
+./breeze prepare-provider-packages --version-suffix-for-svn rc1 --package-format both PACKAGE PACKAGE ....
```
* Sign all your packages
@@ -130,20 +151,19 @@
# Create a new folder for the release.
cd airflow-dev/providers
-svn mkdir ${VERSION}
# Move the artifacts to svn folder
-mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/
+mv ${AIRFLOW_REPO_ROOT}/dist/* .
# Add and commit
-svn add ${VERSION}/*
-svn commit -m "Add artifacts for Airflow Providers ${VERSION}"
+svn add *
+svn commit -m "Add artifacts for Airflow Providers $(date "+%Y-%m-%d%n")"
cd ${AIRFLOW_REPO_ROOT}
```
Verify that the files are available at
-[backport-providers](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+[providers](https://dist.apache.org/repos/dist/dev/airflow/providers/)
## Publish the Regular convenience package to PyPI
@@ -160,26 +180,28 @@
this will clean up dist folder before generating the packages, so you will only have the right packages there.
```shell script
-./breeze prepare-provider-packages --version-suffix-for-pypi a1 --version-suffix-for-SVN a1
+rm -rf ${AIRFLOW_REPO_ROOT}/dist/*
+
+./breeze prepare-provider-packages --version-suffix-for-pypi rc1 --package-format both
```
if you ony build few packages, run:
```shell script
-./breeze prepare-provider-packages --version-suffix-for-pypi a1 \
+./breeze prepare-provider-packages --version-suffix-for-pypi rc1 --package-format both \
PACKAGE PACKAGE ....
```
* Verify the artifacts that would be uploaded:
```shell script
-twine check dist/*
+twine check ${AIRFLOW_REPO_ROOT}/dist/*
```
* Upload the package to PyPi's test environment:
```shell script
-twine upload -r pypitest dist/*
+twine upload -r pypitest ${AIRFLOW_REPO_ROOT}/dist/*
```
* Verify that the test packages look good by downloading it and installing them into a virtual environment.
@@ -188,62 +210,361 @@
* Upload the package to PyPi's production environment:
```shell script
-twine upload -r pypi dist/*
+twine upload -r pypi ${AIRFLOW_REPO_ROOT}/dist/*
```
* Again, confirm that the packages are available under the links printed.
+
+## Add tags in git
+
+Assume that your remote for apache repository is called `apache` you should now
+set tags for the providers in the repo.
+
+```shell script
+for file in ${AIRFLOW_REPO_ROOT}/dist/*.whl
+do
+ if [[ ${file} =~ .*airflow_providers_(.*)-(.*)-py3.* ]]; then
+ provider="providers-${BASH_REMATCH[1]}"
+ tag = "${provider//_/-}/${BASH_REMATCH[2]}"
+ git tag "${tag}"
+ git push apache "${tag}"
+ fi
+done
+```
+
## Publish documentation
Documentation is an essential part of the product and should be made available to users.
-In our cases, documentation for the released versions is published in a separate repository - [`apache/airflow-site`](https://github.com/apache/airflow-site), but the documentation source code and build tools are available in the `apache/airflow` repository, so you have to coordinate between the two repositories to be able to build the documentation.
+In our cases, documentation for the released versions is published in a separate repository -
+[`apache/airflow-site`](https://github.com/apache/airflow-site), but the documentation source code
+and build tools are available in the `apache/airflow` repository, so you have to coordinate between
+the two repositories to be able to build the documentation.
-Documentation for providers can be found in the `/docs/apache-airflow-providers` directory and the `/docs/apache-airflow-providers-*/` directory. The first directory contains the package contents lists and should be updated every time a new version of provider packages is released.
+Documentation for providers can be found in the `/docs/apache-airflow-providers` directory
+and the `/docs/apache-airflow-providers-*/` directory. The first directory contains the package contents
+lists and should be updated every time a new version of provider packages is released.
- First, copy the airflow-site repository and set the environment variable ``AIRFLOW_SITE_DIRECTORY``.
- ```shell script
- git clone https://github.com/apache/airflow-site.git airflow-site
- cd airflow-site
- export AIRFLOW_SITE_DIRECTORY="$(pwd)"
- ```
+```shell script
+git clone https://github.com/apache/airflow-site.git airflow-site
+cd airflow-site
+export AIRFLOW_SITE_DIRECTORY="$(pwd)"
+```
- Then you can go to the directory and build the necessary documentation packages
- ```shell script
- cd "${AIRFLOW_REPO_ROOT}"
- ./breeze build-docs -- \
- --package-filter apache-airflow-providers \
- --package-filter apache-airflow-providers-apache-airflow \
- --package-filter apache-airflow-providers-telegram \
- --for-production
- ```
+```shell script
+cd "${AIRFLOW_REPO_ROOT}"
+./breeze build-docs -- \
+ --package-filter apache-airflow-providers \
+ --package-filter 'apache-airflow-providers-*' \
+ --for-production
+```
+
+for all providers, or if you have just few providers:
+
+```shell script
+cd "${AIRFLOW_REPO_ROOT}"
+./breeze build-docs -- \
+ --package-filter apache-airflow-providers \
+ --package-filter 'apache-airflow-providers-PACKAGE1' \
+ --package-filter 'apache-airflow-providers-PACKAGE2' \
+ ...
+ --for-production
+```
- Now you can preview the documentation.
- ```shell script
- ./docs/start_doc_server.sh
- ```
+```shell script
+./docs/start_doc_server.sh
+```
- Copy the documentation to the ``airflow-site`` repository
- ```shell script
- ./docs/publish_docs.py \
- --package-filter apache-airflow-providers \
- --package-filter apache-airflow-providers-apache-airflow \
- --package-filter apache-airflow-providers-telegram \
+```shell script
+./docs/publish_docs.py \
+ --package-filter apache-airflow-providers \
+ --package-filter apache-airflow-providers-telegram \
- cd "${AIRFLOW_SITE_DIRECTORY}"
- ```
+cd "${AIRFLOW_SITE_DIRECTORY}"
+```
-- If you publish a new package, you must add it to [the docs index](https://github.com/apache/airflow-site/blob/master/landing-pages/site/content/en/docs/_index.md):
+- If you publish a new package, you must add it to
+ [the docs index](https://github.com/apache/airflow-site/blob/master/landing-pages/site/content/en/docs/_index.md):
-- Create commit and push changes.
+- Create the commit and push changes.
- ```shell script
- git commit -m "Add documentation for backport packages - $(date "+%Y-%m-%d%n")"
- git push
- ```
+```shell script
+git commit -m "Add documentation for backport packages - $(date "+%Y-%m-%d%n")"
+git push
+```
+
+## Prepare voting email for Providers release candidate
+
+Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/backport-providers/
+
+Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the
+email.
+
+subject:
+
+
+```shell script
+cat <<EOF
+[VOTE] Airflow Providers - release done $(date "+%Y-%m-%d%n")
+EOF
+```
+
+```shell script
+cat <<EOF
+Hey all,
+
+I have cut Airflow Providers. This email is calling a vote on the release,
+which will last for 72 hours - which means that it will end on $(date -d '+3 days').
+
+Consider this my (binding) +1.
+
+Airflow Providers are available at:
+https://dist.apache.org/repos/dist/dev/airflow/providers/
+
+*apache-airflow-providers-<PROVIDER>-*-bin.tar.gz* are the binary
+ Python "sdist" release - they are also official "sources" for the provider packages.
+
+*apache_airflow_providers_<PROVIDER>-*.whl are the binary
+ Python "wheel" release.
+
+The test procedure for PMCs and Contributors who would like to test the RC candidates are described in
+https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#vote-and-verify-the-providers-release-candidate
+
+
+Public keys are available at:
+https://dist.apache.org/repos/dist/release/airflow/KEYS
+
+Please vote accordingly:
+
+[ ] +1 approve
+[ ] +0 no opinion
+[ ] -1 disapprove with the reason
+
+
+Only votes from PMC members are binding, but members of the community are
+encouraged to test the release and vote with "(non-binding)".
+
+Please note that the version number excludes the 'rcX' string, so it's now
+simply ${VERSION%rc?}. This will allow us to rename the artifact without modifying
+the artifact checksums when we actually release.
+
+Each of the packages contains detailed changelog. Here is the list of links to
+the released packages and changelogs:
+
+<PASTE TWINE UPLOAD LINKS HERE. SORT THEM BEFORE!>
+
+Cheers,
+<TODO: Your Name>
+
+EOF
+```
+
+Due to the nature of packages, not all packages have to be released as convenience
+packages in the final release. During the voting process
+the voting PMCs might decide to exclude certain packages from the release if some critical
+problems have been found in some packages.
+
+Please modify the message above accordingly to clearly exclude those packages.
+
+## Verify the release by PMC members
+
+### SVN check
+
+The files should be present in
+[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/providers/)
+
+The following files should be present (9 files):
+
+* -source.tar.gz + .asc + .sha512 (one set of files)
+* -bin-tar.gz + .asc + .sha512 (one set of files per provider)
+* -.whl + .asc + .sha512 (one set of files per provider)
+
+As a PMC you should be able to clone the SVN repository:
+
+```shell script
+svn co https://dist.apache.org/repos/dist/dev/airflow/
+```
+
+Or update it if you already checked it out:
+
+```shell script
+svn update .
+```
+
+### Licences check
+
+This can be done with the Apache RAT tool.
+
+* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
+ the jar is inside)
+* Unpack the -source.tar.gz to a folder
+* Enter the folder and run the check (point to the place where you extracted the .jar)
+
+```shell script
+java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
+```
+
+### Signature check
+
+Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+
+You can import the whole KEYS file:
+
+```shell script
+gpg --import KEYS
+```
+
+You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
+retrieves it from the default GPG keyserver
+[OpenPGP.org](https://keys.openpgp.org):
+
+```shell script
+gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+You should choose to import the key when asked.
+
+Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
+errors or timeouts. Many of the release managers also uploaded their keys to the
+[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+
+```shell script
+gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+Once you have the keys, the signatures can be verified by running this:
+
+```shell script
+for i in *.asc
+do
+ echo "Checking $i"; gpg --verify $i
+done
+```
+
+This should produce results similar to the below. The "Good signature from ..." is indication
+that the signatures are correct. Do not worry about the "not certified with a trusted signature"
+warning. Most of the certificates used by release managers are self signed, that's why you get this
+warning. By importing the server in the previous step and importing it via ID from
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
+this is a valid Key already.
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <kaxilnaik@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
+gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
+gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <kaxilnaik@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <kaxilnaik@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+```
+
+### SHA512 check
+
+Run this:
+
+```shell script
+for i in *.sha512
+do
+ echo "Checking $i"; shasum -a 512 `basename $i .sha512 ` | diff - $i
+done
+```
+
+You should get output similar to:
+
+```
+Checking apache-airflow-providers-google-1.0.0rc1-bin.tar.gz.sha512
+Checking apache_airflow-providers-google-1.0.0rc1-py3-none-any.whl.sha512
+```
+
+## Verify by Contributors
+
+This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
+actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
+is available on PyPI apart from SVN packages, so everyone should be able to install
+the release candidate version.
+
+You can use any of the installation methods you prefer (you can even install it via the binary wheels
+downloaded from the SVN).
+
+### Installing in your local virtualenv
+
+You have to make sure you have Airflow 2* installed in your PIP virtualenv
+(the version you want to install providers with).
+
+```shell script
+pip install apache-airflow-providers-<provider>==<VERSION>rc<X>
+```
+
+### Installing with Breeze
+
+There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
+Here is a typical scenario.
+
+First copy all the provider packages .whl files to the `dist` folder.
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --install-packages-from-dist
+```
+
+### Building your own docker image
+
+If you prefer to build your own image, you can also use the official image and PyPI packages to test
+backport packages. This is especially helpful when you want to test integrations, but you need to install
+additional tools. Below is an example Dockerfile, which installs providers for Google/
+
+```dockerfile
+FROM apache/airflow:2.0.0
+
+RUN pip install --upgrade --user apache-airflow-providers-google==2.0.0.rc1
+
+USER ${AIRFLOW_UID}
+```
+
+To build an image build and run a shell, run:
+
+```shell script
+docker build . -t my-airflow
+docker run -ti \
+ --rm \
+ -v "$PWD/data:/opt/airflow/" \
+ -v "$PWD/keys/:/keys/" \
+ -p 8080:8080 \
+ -e AIRFLOW__CORE__LOAD_EXAMPLES=True \
+ my-airflow bash
+```
+
+### Additional Verification
+
+Once you install and run Airflow, you can perform any verification you see as necessary to check
+that the Airflow works as you expected.
+
## Notify developers of release
diff --git a/dev/provider_packages/BACKPORT_SETUP_TEMPLATE.py.jinja2 b/dev/provider_packages/BACKPORT_SETUP_TEMPLATE.py.jinja2
new file mode 100644
index 0000000..4d47a4c
--- /dev/null
+++ b/dev/provider_packages/BACKPORT_SETUP_TEMPLATE.py.jinja2
@@ -0,0 +1,89 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE
+# OVERWRITTEN WHEN PREPARING PACKAGES.
+#
+# IF YOU WANT TO MODIFY IT, YOU SHOULD MODIFY THE TEMPLATE
+# `SETUP_TEMPLATE.py.jinja2` IN the `dev/provider_packages` DIRECTORY
+
+"""Setup.py for the {{ PACKAGE_PIP_NAME }} package."""
+
+import logging
+import os
+
+from os.path import dirname
+from setuptools import find_namespace_packages, setup
+
+logger = logging.getLogger(__name__)
+
+version = '{{ RELEASE_NO_LEADING_ZEROS }}{{ VERSION_SUFFIX }}'
+
+my_dir = dirname(__file__)
+
+try:
+ with open(os.path.join(my_dir, '{{ README_FILE }}'), encoding='utf-8') as f:
+ long_description = f.read()
+except FileNotFoundError:
+ long_description = ''
+
+
+def do_setup():
+ """Perform the package {{ PACKAGE_PIP_NAME }} setup."""
+ setup(
+ name='{{ PACKAGE_PIP_NAME }}',
+ description='{{ PROVIDER_TYPE }} package '
+ '{{ PACKAGE_PIP_NAME }} for Apache Airflow',
+ long_description=long_description,
+ long_description_content_type='text/markdown',
+ license='Apache License 2.0',
+ version=version,
+ packages=find_namespace_packages(
+ include=['airflow.providers.{{ PROVIDER_PACKAGE_ID }}',
+ 'airflow.providers.{{ PROVIDER_PACKAGE_ID }}.*']),
+ zip_safe=False,
+ include_package_data=True,
+ install_requires={{ INSTALL_REQUIREMENTS }},
+ setup_requires={{ SETUP_REQUIREMENTS }},
+ extras_require={{ EXTRAS_REQUIREMENTS }},
+ classifiers=[
+ 'Development Status :: 5 - Production/Stable',
+ 'Environment :: Console',
+ 'Environment :: Web Environment',
+ 'Intended Audience :: Developers',
+ 'Intended Audience :: System Administrators',
+ 'License :: OSI Approved :: Apache Software License',
+ 'Programming Language :: Python :: 3.6',
+ 'Programming Language :: Python :: 3.7',
+ 'Programming Language :: Python :: 3.8',
+ 'Topic :: System :: Monitoring',
+ ],
+ author='Apache Software Foundation',
+ author_email='dev@airflow.apache.org',
+ url='https://airflow.apache.org/',
+ download_url='https://archive.apache.org/dist/airflow/{{ PROVIDERS_FOLDER }}',
+ python_requires='~=3.6',
+ project_urls={
+ 'Documentation': 'https://airflow.apache.org/docs/',
+ 'Bug Tracker': 'https://github.com/apache/airflow/issues',
+ 'Source Code': 'https://github.com/apache/airflow',
+ },
+ )
+
+if __name__ == "__main__":
+ do_setup()
diff --git a/dev/provider_packages/PROVIDER_README_TEMPLATE.md.jinja2 b/dev/provider_packages/PROVIDER_README_TEMPLATE.rst.jinja2
similarity index 61%
rename from dev/provider_packages/PROVIDER_README_TEMPLATE.md.jinja2
rename to dev/provider_packages/PROVIDER_README_TEMPLATE.rst.jinja2
index 9a4b912..45dd8ec 100644
--- a/dev/provider_packages/PROVIDER_README_TEMPLATE.md.jinja2
+++ b/dev/provider_packages/PROVIDER_README_TEMPLATE.rst.jinja2
@@ -20,69 +20,64 @@
OVERWRITTEN WHEN PREPARING PACKAGES.
IF YOU WANT TO MODIFY IT, YOU SHOULD MODIFY THE TEMPLATE
- `PROVIDER_README_TEMPLATE.md.jinja2` IN the `dev/provider_packages` DIRECTORY
+ `PROVIDER_INDEX_TEMPLATE.rst.jinja2` IN the `dev/provider_packages` DIRECTORY
#}
-# Package {{ PACKAGE_PIP_NAME }}
+Package ``{{ PACKAGE_PIP_NAME }}``
+
+Release: ``{{ RELEASE }}{{ VERSION_SUFFIX }}``
+
{{ PROVIDER_DESCRIPTION | safe }}
-Release: {{ RELEASE }}{{ VERSION_SUFFIX }}
+Provider package
+================
-**Table of contents**
-
-- [Provider package](#provider-package)
-- [Installation](#installation)
-{%- if PIP_REQUIREMENTS %}
-- [PIP requirements](#pip-requirements)
-{%- endif %}
-{%- if CROSS_PROVIDERS_DEPENDENCIES %}
-- [Cross provider package dependencies](#cross-provider-package-dependencies)
-{%- endif %}
-
-## Provider package
-
-This is a provider package for `{{PROVIDER_PACKAGE_ID}}` provider. All classes for this provider package
-are in `{{FULL_PACKAGE_NAME}}` python package.
+This is a provider package for ``{{PROVIDER_PACKAGE_ID}}`` provider. All classes for this provider package
+are in ``{{FULL_PACKAGE_NAME}}`` python package.
You can find package information and changelog for the provider
-in [documentation](https://airflow.apache.org/docs/{{ PACKAGE_PIP_NAME }}/{{RELEASE}}/index.html).
+in the `documentation <https://airflow.apache.org/docs/{{ PACKAGE_PIP_NAME }}/{{RELEASE}}/index.html>`_.
-## Installation
+Installation
+============
NOTE!
On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
-`pip install --upgrade pip==20.2.4` or, in case you use Pip 20.3, you need to add option
-`--use-deprecated legacy-resolver` to your pip install command.
+``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
+``--use-deprecated legacy-resolver`` to your pip install command.
You can install this package on top of an existing airflow 2.* installation via
-`pip install {{PACKAGE_PIP_NAME}}`
+``pip install {{PACKAGE_PIP_NAME}}``
{%- if PIP_REQUIREMENTS %}
-## PIP requirements
+PIP requirements
+================
-{{ PIP_REQUIREMENTS_TABLE | safe }}
+{{ PIP_REQUIREMENTS_TABLE_RST | safe }}
{%- endif %}
{%- if CROSS_PROVIDERS_DEPENDENCIES %}
-## Cross provider package dependencies
+Cross provider package dependencies
+===================================
Those are dependencies that might be needed in order to use all the features of the package.
You need to install the specified backport providers package in order to use them.
You can install such cross-provider dependencies when installing from PyPI. For example:
-```bash
-pip install {{ PACKAGE_PIP_NAME }}[{{ CROSS_PROVIDERS_DEPENDENCIES[0] }}]
-```
+.. code-block:: bash
-{{ CROSS_PROVIDERS_DEPENDENCIES_TABLE | safe }}
+ pip install {{ PACKAGE_PIP_NAME }}[{{ CROSS_PROVIDERS_DEPENDENCIES[0] }}]
+
+
+{{ CROSS_PROVIDERS_DEPENDENCIES_TABLE_RST | safe }}
{%- endif %}
diff --git a/dev/provider_packages/SETUP_TEMPLATE.py.jinja2 b/dev/provider_packages/SETUP_TEMPLATE.py.jinja2
index 4af1e1b..1603e32 100644
--- a/dev/provider_packages/SETUP_TEMPLATE.py.jinja2
+++ b/dev/provider_packages/SETUP_TEMPLATE.py.jinja2
@@ -37,7 +37,7 @@
my_dir = dirname(__file__)
try:
- with open(os.path.join(my_dir, '{{ PROVIDER_PATH }}/{{ README_FILE }}'), encoding='utf-8') as f:
+ with open(os.path.join(my_dir, '{{ README_FILE }}'), encoding='utf-8') as f:
long_description = f.read()
except FileNotFoundError:
long_description = ''
@@ -50,7 +50,6 @@
description='{{ PROVIDER_TYPE }} package '
'{{ PACKAGE_PIP_NAME }} for Apache Airflow',
long_description=long_description,
- long_description_content_type='text/markdown',
license='Apache License 2.0',
version=version,
packages=find_namespace_packages(
@@ -79,7 +78,7 @@
download_url='https://archive.apache.org/dist/airflow/{{ PROVIDERS_FOLDER }}',
python_requires='~=3.6',
project_urls={
- 'Documentation': 'https://airflow.apache.org/docs/',
+ 'Documentation': 'https://airflow.apache.org/docs/{{ PACKAGE_PIP_NAME }}/{{RELEASE}}/index.html',
'Bug Tracker': 'https://github.com/apache/airflow/issues',
'Source Code': 'https://github.com/apache/airflow',
},
diff --git a/dev/provider_packages/prepare_provider_packages.py b/dev/provider_packages/prepare_provider_packages.py
index 3cfc39f..40b22c9 100755
--- a/dev/provider_packages/prepare_provider_packages.py
+++ b/dev/provider_packages/prepare_provider_packages.py
@@ -881,6 +881,25 @@
-->
"""
+LICENCE_RST = """
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+"""
+
"""
Keeps information about historical releases.
"""
@@ -1402,7 +1421,7 @@
)
context: Dict[str, Any] = {
"ENTITY_TYPES": list(EntityType),
- "README_FILE": "BACKPORT_PROVIDER_README.md" if backport_packages else "README.md",
+ "README_FILE": "README.md" if backport_packages else "README.rst",
"PROVIDER_PACKAGE_ID": provider_details.provider_package_id,
"PACKAGE_PIP_NAME": get_pip_package_name(
provider_details.provider_package_id, backport_packages=backport_packages
@@ -1438,13 +1457,11 @@
return context
-def prepare_readme_file(
- context,
-):
- readme_content = LICENCE
+def prepare_readme_file(context):
+ readme_content = LICENCE_RST
readme_template_name = PROVIDER_TEMPLATE_PREFIX + "README"
- readme_content += render_template(template_name=readme_template_name, context=context, extension='.md')
- readme_file_path = os.path.join(TARGET_PROVIDER_PACKAGES_PATH, "README.md")
+ readme_content += render_template(template_name=readme_template_name, context=context, extension=".rst")
+ readme_file_path = os.path.join(TARGET_PROVIDER_PACKAGES_PATH, "README.rst")
with open(readme_file_path, "wt") as readme_file:
readme_file.write(readme_content)
@@ -1575,8 +1592,8 @@
replace_content(index_file_path, old_text, new_text, provider_package_id)
-def prepare_setup_py_file(context):
- setup_py_template_name = "SETUP"
+def prepare_setup_py_file(context, backport_package=False):
+ setup_py_template_name = "BACKPORT_SETUP" if backport_package else "SETUP"
setup_py_file_path = os.path.abspath(os.path.join(get_target_folder(), "setup.py"))
setup_py_content = render_template(
template_name=setup_py_template_name, context=context, extension='.py', autoescape=False
@@ -1789,7 +1806,7 @@
provider_details.source_provider_package_path,
)
if update_setup:
- prepare_setup_py_file(jinja_context)
+ prepare_setup_py_file(jinja_context, backport_package=True)
prepare_setup_cfg_file(jinja_context, backport_package=True)
prepare_manifest_in_file(jinja_context)
@@ -1946,7 +1963,7 @@
if update_generated_files_for_regular_package(
provider_package_id, suffix, update_release_notes=False, update_setup=True
):
- print(f"[green]Generated regular packge setup files for {provider_package_id}[/]")
+ print(f"[green]Generated regular package setup files for {provider_package_id}[/]")
else:
package_ok = False
return package_ok
diff --git a/docs/apache-airflow-providers-apache-beam/commits.rst b/docs/apache-airflow-providers-apache-beam/commits.rst
index 1cf7ec2..ab3e1da 100644
--- a/docs/apache-airflow-providers-apache-beam/commits.rst
+++ b/docs/apache-airflow-providers-apache-beam/commits.rst
@@ -28,7 +28,7 @@
-0.0.1
+1.0.0
.....
Latest change: 2021-02-03
diff --git a/docs/apache-airflow-providers-apache-beam/index.rst b/docs/apache-airflow-providers-apache-beam/index.rst
index c7f9629..929a021 100644
--- a/docs/apache-airflow-providers-apache-beam/index.rst
+++ b/docs/apache-airflow-providers-apache-beam/index.rst
@@ -51,7 +51,7 @@
`Apache Beam <https://beam.apache.org/>`__.
-Release: 0.0.1
+Release: 1.0.0
Provider package
----------------
diff --git a/provider_packages/.gitignore b/provider_packages/.gitignore
index 59b1160..fe90b0c 100644
--- a/provider_packages/.gitignore
+++ b/provider_packages/.gitignore
@@ -2,5 +2,6 @@
setup.py
CHANGELOG.txt
README.md
+README.rst
setup.cfg
/airflow/
diff --git a/scripts/in_container/run_prepare_provider_packages.sh b/scripts/in_container/run_prepare_provider_packages.sh
index e01fe24..f47d262 100755
--- a/scripts/in_container/run_prepare_provider_packages.sh
+++ b/scripts/in_container/run_prepare_provider_packages.sh
@@ -79,7 +79,7 @@
echo
elif [[ ${FILE_VERSION_SUFFIX} != '' ]]; then
echo
- echo " Preparing release candidate of ${provider_package} with file version suffix only (ersulting file will be renamed): ${FILE_VERSION_SUFFIX}"
+ echo " Preparing release candidate of ${provider_package} with file version suffix only (resulting file will be renamed): ${FILE_VERSION_SUFFIX}"
echo
elif [[ "${VERSION_SUFFIX_FOR_PYPI}" == '' ]]; then
echo
