blob: 948bce1153b449bf71c74306b612adb2add13cda [file] [log] [blame] [view]
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to you under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
YuniKorn Release Procedure
----
This project provides the instructions and tools needed to generate Apache YuniKorn (Incubating) release artefacts. This obeys ASF [release policy](http://www.apache.org/legal/release-policy.html), and [Podling Release Policy](https://incubator.apache.org/policy/incubation.html#releases).
* [Create a Release](#Create-a-Release)
* [Tag and update release for version](#Tag-and-update-release-for-version)
* [Update the CHANGELOG](#Update-the-CHANGELOG)
* [Run the release tool](#Run-the-release-tool)
* [Create Signature](#Create-Signature)
* [Create Checksum](#Create-Checksum)
* [Upload Release Candidate Artefacts](#Upload-Release-Candidate-Artefacts)
* [Start Voting Thread](#Start-Voting-Thread)
* [Publish the Release](#Publish-the-Release)
* [Release Docker images](#Release-Docker-images)
* [Release Helm Charts](#Release-Helm-Charts)
* [Update the website](#Update-the-website)
* [Create the GIT releases](#Create-the-GIT-releases)
* [Verify the release](#Verify-the-release)
* [Signing your first release](#Signing-your-first-release)
* [Generate a Key](#Generate-a-Key)
* [Add the signature to the project KEYS file](#Add-the-signature-to-the-project-KEYS-file)
# Create a Release
Simplified release procedure:
1. Create a release branch for the target release in all git repos, such as `branch-0.8`
2. Stabilize the release by fixing test failures and bugs only
3. Tag update release for a new version to prepare a release candidate, e.g `v0.8.0`
4. Update the CHANGELOG
5. Configure [release-configs.json](../tools/release-configs.json)
6. Run script [build-release.py](../tools/build-release.py) to generate source code tarball, checksum and signature.
7. Voting and releasing the candidate
## Tag and update release for version
Branching and tagging can, and in most cases will, require changes in the go mod files.
Branching is part of the release preparation and often has happened some time before the release process starts.
A release needs to be tagged in git before starting the release process.
As an example check [YUNIKORN-358](https://issues.apache.org/jira/browse/YUNIKORN-358).
Release candidates and final release use the same tag which gets moved if a new release candidate is generated.
The tagging is multi step process, all actions are done on the branch that will be released, like `branch-0.8`:
1. Tag the web and scheduler interface with the release tag.
2. Update the `go.mod` file in the core using `go get github.com/apache/incubator-yunikorn-scheduler-interface`
Add the tag and commit the changes.
3. Update the `go.mod` file in the shim using `go get github.com/apache/incubator-yunikorn-scheduler-interface` and
`go get github.com/apache/incubator-yunikorn-core`. Add the tag and commit the changes.
4. Create a new branch in the yunikorn-release repo, set the correct chart version in [Chart.yaml](../helm-charts/yunikorn/Chart.yaml), and then create the tag.
## Update the CHANGELOG
In the release artifacts a [CHANGELOG](../release-top-level-artifacts/CHANGELOG) is added for each release.
The CHANGELOG should contain the list of jiras fixed in the release.
Follow these steps to generate the list:
- Go to the [releases page in jira](https://issues.apache.org/jira/projects/YUNIKORN?selectedItem=com.atlassian.jira.jira-projects-plugin%3Arelease-page&status=released-unreleased)
- Click on the version that is about to be released, i.e. `0.8`
- Click on the `Release Notes` link on the top of the page
- Click the button `Configure Release Notes`
- Select the style `Text` and click `create`
- Scroll to the bottom of the page and copy the content of the text area and update the [CHANGELOG](../release-top-level-artifacts/CHANGELOG) file.
## Run the release tool
A tool has been written to handle most of the release tasks.
The tool requires a simple [json](../tools/release-configs.json) input file to be updated before running.
This configuration points to the current release tag. Only update the tag for each repository.
The tool has one requirement outside of standard Python 3: [GitPython](https://gitpython.readthedocs.io/en/stable/intro.html)
Make sure you have installed it by running `pip install gitpython`.
Run the tool:
```shell script
python3 build-release.py
```
If you want to automatically sign the release using your GPG key run the tool using:
```shell script
python3 build-release.py --sign <email-address>
```
### Create Signature
If you have GPG with a _pinentry_ program setup you can automatically sign the release using the release tool.
On MacOSX this will be setup automatically if you use the keychain for the keys.
For more details check the [GnuPG tools wiki](https://wiki.archlinux.org/index.php/GnuPG) and specifically the [pinentry](https://wiki.archlinux.org/index.php/GnuPG#pinentry) chapter.
Run the release tool using the option `--sign <email-address>` to auto sign the release.
Manually creating the signature for the file generated by the tool:
```shell script
gpg --local-user <email-address> --armor --output apache-yunikorn-0.8.0-incubating-src.tar.gz.asc --detach-sig apache-yunikorn-0.8.0-incubating-src.tar.gz
```
This will create the signature in the file: `apache-yunikorn-0.8.0-incubating-src.tar.gz.asc`
Verify that the signature is correct using:
```shell script
gpg --verify apache-yunikorn-0.8.0-incubating-src.tar.gz.asc apache-yunikorn-0.8.0-incubating-src.tar.gz
```
### Create Checksum
This step is included in the release after generation of the source tar ball, if the release tool is used this step can be skipped.
```shell script
shasum -a 512 apache-yunikorn-0.8.0-incubating-src.tar.gz > apache-yunikorn-0.8.0-incubating-src.tar.gz.sha512
```
This will create the checksum in the file: `apache-yunikorn-0.8.0-incubating-src.tar.gz.sha512`
Verify that the checksum is correct using:
```shell script
shasum -a 512 -c apache-yunikorn-0.8.0-incubating-src.tar.gz.sha512
```
## Upload Release Candidate Artefacts
The release artefacts consist of three parts:
- source tarball
- signature file
- checksum file
The three artefacts need to be uploaded to: `https://dist.apache.org/repos/dist/dev/incubator/yunikorn/`
Create a release directory based on the version, i.e. `0.8.0`, add the three files to directory.
Commit the changes.
If you have not done so already make sure to [add your signature](#add-the-signature-to-the-project-keys-file) to the KEYS file.
Do not remove any keys from the file they are kept here to enable older releases to be verified.
NOTE: you will need to install subversion to access this repo (use your apache ID). You can use any SVN client, e.g svnX, for convenience.
## Start Voting Thread
According to [podling release doc](https://incubator.apache.org/policy/incubation.html#releases) and [release approval doc](http://www.apache.org/legal/release-policy.html#release-approval). Steps are:
- start a voting thread on `dev@yunikorn.apache.org`. (72 hours)
- send a summary of that vote to the Incubator’s general list and request IPMC to vote. (72 hours)
Both voting need to acquire at least three +1 votes are required and more +1 votes than -1 votes.
## Publish the Release
Once the voting is passed, move the release artefacts from the staging area to the release location `https://dist.apache.org/repos/dist/release/incubator/yunikorn/`.
Once moved to this space, the content will be automatically synced to `https://downloads.apache.org/incubator/yunikorn/` which must be used as the final location for release files.
Read more for [location of files on main server](https://infra.apache.org/mirrors#location).
### Release Docker images
The standard build process should be used to build the image.
Run a `make image` in the `web`, and `k8shim` repositories to generate the three images required (web, scheduler and admission-controller):
```shell script
VERSION=0.8.0; make image
```
Make can also be used to build and push the image if you have access to the Apache docker hub YuniKorn container.
Push the latest docker images to the apache docker hub using the release as tag.
Make sure the docker image is built on the specific SHA.
```shell script
VERSION=0.8.0; DOCKER_USERNAME=<name>; DOCKER_PASSWORD=<password>; make push
```
Publish an announcement email to the `dev@yunikorn.apache.org` email list.
### Release Helm Charts
This step is part of the release tool if the release tool is used the packaging can be skipped.
If the release tool is **not** used the `Chart.yaml` and the `values.yaml` must be updated manually.
The other option is to run the helm script against the generated source directory as the tool does:
```shell script
helm package --sign --key ${your_key_name} --keyring ${path/to/keyring.secret} staging/<release-dir>/helm-charts/yunikorn --destination staging/
```
Signing the helm package requires a legacy PGP keyring. The PGP v2 keyring must be converted to the legacy format.
For more information please check [Helm documentation](https://helm.sh/docs/topics/provenance/).
Helm charts should be signed on release.
Contrary to the source code tar ball signing, signing the helm charts requires manual entry of the key password.
The helm package will generate two files:
- helm package: example `yunikorn-0.8.0.tgz`
- provenance or signature file: example `yunikorn-0.8.0.tgz.prov`
Both files should be attached to the [release in GIT](#Create-the GIT-releases) for the release repository.
Last step is to update the [index.yaml](https://github.com/apache/incubator-yunikorn-release/blob/gh-pages/index.yaml) file in the `gh-pages` branch with the new release.
The `digest` mentioned in the index.yaml file is the digest that gets printed by the tool (unsigned package) or stored in the provenance file.
It can be generated manually using:
```shell script
shasum -a 256 yunikorn-0.8.0.tgz
```
Note: do not use the `helm repo index` command to update the `index.yaml` file. The command does not handle the enhanced information stored in the `index.yaml` file nicely.
Update the file manually.
### Update the website
- Create a new documentation version on YuniKorn website based on the latest content in [docs](https://github.com/apache/incubator-yunikorn-site/tree/master/docs) directory. Refer to [this](https://github.com/apache/incubator-yunikorn-site/tree/master#release-a-new-version) guide to create the new documentation.
- Create the release announcement to be referenced from download page on the website. The release announcement is a markdown file based on the version: `0.8.0.md`. The file is stored as part of the [static pages](https://github.com/apache/incubator-yunikorn-site/tree/master/src/pages/release-announce) on the website.
- Update the [download page](https://github.com/apache/incubator-yunikorn-site/tree/master/src/pages/community/download.md) of the website.
The release announcement are linked to the release details on the download page.
The download links on the page must use the mirror resolution link for the source tar ball only.
The signature and checksum links must point to the release location.
### Create the GIT releases
In the GIT repositories finish the release process by creating a release based on the git tag that was added.
Repeat these steps for all five repositories (core, k8shim, web, scheduler-interface and release):
- Go to the `tags` page
- click the `...` at the right-hand side of the page that you want to release, select `Create Release` from the drop down
- update the name and note
- add the packaged helm chart files (incubator-yunikorn-release repository only)
- click `Publish Release` to finish the steps
## Verify the release
After the whole procedure verify the documentation on the website and that the released artifacts can be downloaded.
Mirror links might take up to 24 hours to be updated.
# Signing your first release
If you haven't signed any releases before, read the documentation to [generate signing key](https://infra.apache.org/openpgp.html#generate-key)
Follow the steps below to add the key you can use to sign.
## Generate a Key
Generate a new PGP key (skip this step if you already have an Apache linked key):
```shell script
gpg --gen-key
```
Fill out the requested information using your full name and Apache email address.
Upload the exported key to a public key server like `https://pgp.mit.edu/`.
```shell script
gpg --export --armor <email-address>
```
Upload the fingerprint to apache server: `https://id.apache.org/`.
```shell script
gpg --fingerprint <email-address>
```
## Add the signature to the project KEYS file
Only needed if this is the first release signed with the specific key.
More detail can be found in the document: [Signing a Release](https://infra.apache.org/release-signing.html#keys-policy)
```shell script
(gpg --list-sigs <email-address> && gpg --armor --export <email-address>) >> MY_KEY
```
Add the content of the generated file to the existing KEYS list at `https://dist.apache.org/repos/dist/release/incubator/yunikorn/KEYS`
Never remove a key from this list!
NOTE: you will need to install subversion to access this repo (use your apache ID). You can use any SVN client, e.g svnX, for convenience.