This project provides the instructions and tools needed to generate Apache YuniKorn (Incubating) release artefacts. This obeys ASF release policy, and Podling Release Policy.
Simplified release procedure:
branch-0.8
v0.8.0
Branching and tagging can, and in most cases will, require changes in the go mod files. Branching is part of the release preparation and often has happened some time before the release process starts. A release needs to be tagged in git before starting the release process. As an example check YUNIKORN-358. Release candidates and final release use the same tag which gets moved if a new release candidate is generated.
The tagging is multi step process, all actions are done on the branch that will be released, like branch-0.8
:
go.mod
file in the core using go get github.com/apache/incubator-yunikorn-scheduler-interface
go.mod
file in the shim using go get github.com/apache/incubator-yunikorn-scheduler-interface
andgo get github.com/apache/incubator-yunikorn-core
. Add the tag and commit the changes.In the release artifacts a CHANGELOG is added for each release. The CHANGELOG should contain the list of jiras fixed in the release. Follow these steps to generate the list:
0.8
Release Notes
link on the top of the pageConfigure Release Notes
Text
and click create
A tool has been written to handle most of the release tasks. The tool requires a simple json input file to be updated before running. This configuration points to the current release tag. Only update the tag for each repository.
The tool has one requirement outside of standard Python 3: GitPython Make sure you have installed it by running pip install gitpython
.
Run the tool:
python3 build-release.py
If you want to automatically sign the release using your GPG key run the tool using:
python3 build-release.py --sign <email-address>
If you have GPG with a pinentry program setup you can automatically sign the release using the release tool. On MacOSX this will be setup automatically if you use the keychain for the keys. For more details check the GnuPG tools wiki and specifically the pinentry chapter.
Run the release tool using the option --sign <email-address>
to auto sign the release.
Manually creating the signature for the file generated by the tool:
gpg --local-user <email-address> --armor --output apache-yunikorn-0.8.0-incubating-src.tar.gz.asc --detach-sig apache-yunikorn-0.8.0-incubating-src.tar.gz
This will create the signature in the file: apache-yunikorn-0.8.0-incubating-src.tar.gz.asc
Verify that the signature is correct using:
gpg --verify apache-yunikorn-0.8.0-incubating-src.tar.gz.asc apache-yunikorn-0.8.0-incubating-src.tar.gz
This step is included in the release after generation of the source tar ball, if the release tool is used this step can be skipped.
shasum -a 512 apache-yunikorn-0.8.0-incubating-src.tar.gz > apache-yunikorn-0.8.0-incubating-src.tar.gz.sha512
This will create the checksum in the file: apache-yunikorn-0.8.0-incubating-src.tar.gz.sha512
Verify that the checksum is correct using:
shasum -a 512 -c apache-yunikorn-0.8.0-incubating-src.tar.gz.sha512
The release artefacts consist of three parts:
https://dist.apache.org/repos/dist/dev/incubator/yunikorn/
Create a release directory based on the version, i.e. 0.8.0
, add the three files to directory. Commit the changes.
If you have not done so already make sure to add your signature to the KEYS file. Do not remove any keys from the file they are kept here to enable older releases to be verified.
NOTE: you will need to install subversion to access this repo (use your apache ID). You can use any SVN client, e.g svnX, for convenience.
According to podling release doc and release approval doc. Steps are:
dev@yunikorn.apache.org
. (72 hours)Once the voting is passed, move the release artefacts from the staging area to the release location https://dist.apache.org/repos/dist/release/incubator/yunikorn/
. Once moved to this space, the content will be automatically synced to https://downloads.apache.org/incubator/yunikorn/
which must be used as the final location for release files. Read more for location of files on main server.
This will temporarily provide us with two releases in the release area. This is needed to allow the start the mirror sync process and allow for the download page to be updated. Cleanup of the older release is handled after the website has been updated in the cleanup.
The standard build process should be used to build the image. Run a make image
in the web
, and k8shim
repositories to generate the three images required (web, scheduler and admission-controller):
VERSION=0.8.0; make image
Make can also be used to build and push the image if you have access to the Apache docker hub YuniKorn container. Push the latest docker images to the apache docker hub using the release as tag. Make sure the docker image is built on the specific SHA.
VERSION=0.8.0; DOCKER_USERNAME=<name>; DOCKER_PASSWORD=<password>; make push
Publish an announcement email to the dev@yunikorn.apache.org
email list.
This step is part of the release tool if the release tool is used the packaging can be skipped.
If the release tool is not used the Chart.yaml
and the values.yaml
must be updated manually. The other option is to run the helm script against the generated source directory as the tool does:
helm package --sign --key ${your_key_name} --keyring ${path/to/keyring.secret} staging/<release-dir>/helm-charts/yunikorn --destination staging/
Signing the helm package requires a legacy PGP keyring. The PGP v2 keyring must be converted to the legacy format. For more information please check Helm documentation. Helm charts should be signed on release. Contrary to the source code tar ball signing, signing the helm charts requires manual entry of the key password.
The helm package will generate two files:
yunikorn-0.8.0.tgz
yunikorn-0.8.0.tgz.prov
Both files should be attached to the [release in GIT](#Create-the GIT-releases) for the release repository.Last step is to update the index.yaml file in the gh-pages
branch with the new release. The digest
mentioned in the index.yaml file is the digest that gets printed by the tool (unsigned package) or stored in the provenance file. It can be generated manually using:
shasum -a 256 yunikorn-0.8.0.tgz
Note: do not use the helm repo index
command to update the index.yaml
file. The command does not handle the enhanced information stored in the index.yaml
file nicely. Update the file manually.
0.8.0.md
. The file is stored as part of the static pages on the website.The release announcement are linked to the release details on the download page.
Links for the releases have to follow these rules:
https://archive.apache.org/dist/incubator/yunikorn/
for the tar ball, the signature and the checksum.A limited set of three (3) or four (4) releases should be maintained in the table for direct access. Older releases not mentioned in the table can still be accessed via the archive link on the bottom of the page and do not need to be referenced.
NOTE: this step should be performed after the website updates have been made as the download links change.
There should only be one release, the latest, in the release area. Any release that has been in the release area will be automatically copied to the archive. Older releases should be downloaded from the archive directly, not from the release area.
The releases need to clean up in two locations:
In the GIT repositories finish the release process by creating a release based on the git tag that was added. Repeat these steps for all five repositories (core, k8shim, web, scheduler-interface and release):
tags
page...
at the right-hand side of the page that you want to release, select Create Release
from the drop downPublish Release
to finish the stepsAfter the whole procedure verify the documentation on the website and that the released artifacts can be downloaded. Mirror links might take up to 24 hours to be updated.
If you haven't signed any releases before, read the documentation to generate signing key Follow the steps below to add the key you can use to sign.
Generate a new PGP key (skip this step if you already have an Apache linked key):
gpg --gen-key
Fill out the requested information using your full name and Apache email address.
Upload the exported key to a public key server like https://pgp.mit.edu/
.
gpg --export --armor <email-address>
Upload the fingerprint to apache server: https://id.apache.org/
.
gpg --fingerprint <email-address>
Only needed if this is the first release signed with the specific key. More detail can be found in the document: Signing a Release
(gpg --list-sigs <email-address> && gpg --armor --export <email-address>) >> MY_KEY
Add the content of the generated file to the existing KEYS list at https://dist.apache.org/repos/dist/release/incubator/yunikorn/KEYS
Never remove a key from this list!
NOTE: you will need to install subversion to access this repo (use your apache ID). You can use any SVN client, e.g svnX, for convenience.