blob: a0a612ee9e5ea311bc59fb929cc5027883ee104c [file] [log] [blame]
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
= RELEASING KUDU
== Overview
This document gives the steps required to perform a Kudu release, and is a
resource for Kudu release managers. Make sure to read and understand the
link:http://www.apache.org/legal/release-policy.html[ASF policy on software
releases] prior to conducting the release management activity. Pay attention to
the link:https://www.apache.org/legal/resolved.html[3rd-party license policy],
making sure the release doesn't include anything from the
link:https://www.apache.org/legal/resolved.html#category-x[Category X].
The command-line snippets in this document are for `bash` (Bourne-again shell).
To edit or update this document, edit `RELEASING.adoc` in `master`.
== Stating Intent to Release
NOTE: All email messages must be sent from your apache.org email address, otherwise with
company or personal email addresses sometimes emails are not delivered.
If using gmail.com service for email correspondence, follow
link:https://support.google.com/mail/answer/22370?hl=en[this guide] to add your apache.org address
to be used as `From` address when sending email messages.
. *A week before branching*: send an email to dev@kudu.apache.org to
announce that the branch will be happening, including a deadline for when new
commits will need gatekeeper approval. Start
<<_compile_the_release_notes,compiling the release notes>>.
. *A day before branching*: send another email to dev@kudu.apache.org
to warn about it.
=== Compile the release notes
Start a Google doc for the release notes that is editable by anyone, then use
https://s.apache.org to generate a short URL like kudu-1.y-rn and share it on
dev@kudu.apache.org asking for help. The author of a commit probably knows best
if a change belongs to the release notes, and how to succintly describe the
change. Of course, there will be contributors who are less active and maybe not
even monitoring the dev@ list, so you'll still have to write release notes for
their changes.
Use git shortlog -s <prev-version>..HEAD to get a list of contributions and
add it to the doc as a summary of all commits. Note the commit hash at HEAD, as
you'll need to generate another list between this commit and the tip when
finally branching.
Example doc: https://s.apache.org/kudu1.16rn
== Creating the Branch
. Create a new branch from `master`:
+
----
git checkout master
git pull
git checkout -b branch-1.x.y
----
. Make a note of the SHA1 for the tip of the new branch, which is the first
field of the result of this command:
+
----
git log --oneline -n1
----
. Push the branch to public remote https://gitbox.apache.org/repos/asf/kudu.git.
The following example assumes it's called `apache`.
+
----
git push apache branch-1.x.y
----
. Create a new branch on Gerrit. Go to
http://gerrit.cloudera.org:8080/#/admin/projects/kudu,branches and create a
new branch with the same name and the previously-noted SHA1.
. Ask someone with permissions to fix the gerrit.cloudera.org mirroring
configuration. Cloudera hosts the Gerrit server and a Cloudera employee will
have to perform this step because SSH access is behind a firewall. The steps
are as follows:
1. Ensure your public SSH key is in `~gerrit/.ssh/authorized_keys` on
gerrit.cloudera.org
2. From behind the firewall, `ssh gerrit@gerrit.cloudera.org` to log in.
3. Back up the existing replication configuration file by executing
`cp ~/etc/replication.config ~/etc/replication.config.bak.`date '+%Y%m%d.%H%M%S'``
4. Edit `etc/replication.config` to add a line for the new branch, such as
`branch-1.x.y`
5. Send email to the dev lists for Kudu and Impala (dev@kudu.apache.org and
dev@impala.apache.org) indicating that you are going to restart Gerrit
(link:https://s.apache.org/2Wj7[example]). It is best to do the restart at
some time of day when you don't expect many people to be using the system,
since Gerrit can take a few minutes to restart.
6. Restart Gerrit: `~/bin/gerrit.sh restart`
7. Make sure Gerrit has been successfully restarted: after a few minutes,
try to open the link:https://gerrit.cloudera.org/#/admin/projects[project list]
page in your favorite browser.
. As needed, patches can be cherry-picked to the new branch.
== Write the release notes
. Copy the release notes from the shared doc to docs/release_notes.adoc.
. Edit and format the release notes.
. Get the number of contributors and the list of new contributors using the
following command, and add them to the contributors section (see
link:https://kudu.apache.org/releases/1.16.0/docs/release_notes.html#rn_1.16.0_contributors[previous
release notes] for examples):
+
----
export PREV="1.15.0" # the previous release
git shortlog -sn $PREV.. | wc -l
diff <(git shortlog -sn ${PREV} | cut -f 2 | sort) \
<(git shortlog -sn ${PREV}.. | cut -f 2 | sort) \
| grep -E "^>" | sed 's/>/*/'
----
+
. Commit the release notes to the newly created branch and submit it to gerit.
Use `git shortlog -sn $PREV..` to list the contributors and add them to
the reviewers to give everyone a chance to comment on the release notes of
their changes.
== Updating Versions in Master
. Check out the `master` branch and bump the version in `version.txt`. Don't
update `kudu-version` in `examples/java/java-example/pom.xml` yet: it should
be updated later on when release artifacts are published (see below).
. Bump the version numbers in docs/release_notes.adoc.
. Commit and push that change to Gerrit.
. Notify dev@kudu.apache.org that the new branch is available (see
link:https://lists.apache.org/thread.html/de58960366583943391c00bd6b75dbd1fab3bc9067af05dd7b817a90@%3Cdev.kudu.apache.org%3E[here]
for an example).
== Preparing a Release Candidate
. Before building a release candidate, make sure you have followed the
Apache committer guide for
link:https://www.apache.org/dev/new-committers-guide.html#set-up-security-and-pgp-keys[
setting up your GPG keys]. In addition to the link:http://pgp.mit.edu/[MIT
PGP public key server] referenced in the committer guide, consider adding
your keys to one of link:http://pool.sks-keyservers.net:11371[SKS OpenPGP
keyservers] or link:http://keyserver.ubuntu.com:11371[Ubuntu OpenPGP
keyserver]. The MIT keyserver sometimes is not available, and it takes some
time to propagate key updates anyways: the latter two servers are used by the
link:https://repository.apache.org[Apache Maven repo server] to verify the
signature of the uploaded maven artifacts as of October 2019.
. If building and signing on a remote/shared machine, consider
link:https://wiki.gnupg.org/AgentForwarding[forwarding GPG agent via SSH].
That means you can keep your secret keys on a local machine even when signing
the artifacts to be released
(works even for a hardware token like a smartcard, etc.).
. Out of the Kudu git workspace, checkout the
link:https://dist.apache.org/repos/dist/release/kudu/[release SVN repository].
Later on, the officially released and signed artifacts will be put into
this repository. At this point, just add your PGP key to the `KEYS` file
(if it's not there yet), making it available for the signature verification:
+
----
svn co https://dist.apache.org/repos/dist/release/kudu/ kudu-dist-release
cd kudu-dist-release
(gpg --list-sigs <your-email-address> && gpg --armor --export <your-email-address>) >> KEYS
svn commit --username=<your_apache_username> -m "Adding my key to the KEYS file"
----
. When close to building a release candidate, try building a source tarball
(on a supported platform):
+
----
./build-support/build_source_release.py
----
. Fix any issues it finds, such as RAT.
. Make sure `kudu-binary` JAR artifact can be successfully built both on Linux
and macOS:
+
----
./build-support/mini-cluster/build_mini_cluster_binaries.sh
----
. Test the full Java build. This will sign and build everything without
deploying any artifacts:
+
----
# Run a gpg-agent if you don't normally.
gpg-agent --daemon
# List keys with identifiers in the traditional 8-character key ID format.
# Take a note of the identifier of the key you want to use for signing.
gpg --list-secret-keys --keyid-format=short
cd java
./gradlew clean install -PforceSigning -Psigning.gnupg.keyName=<8-character-pgp-key-id>
----
. Create a new version update commit which removes the -SNAPSHOT suffix (same
process as above).
. In the newly created branch `branch-1.x.y` of the project git repo, update
version-related information under the `examples` sub-directory (e.g.,
`kudu-version` in `examples/java/java-example/pom.xml`, `version` in
`examples/java/insert-loadgen/pom.xml`, etc.).
The idea is making all examples use the artifacts of the newly released
`1.x.y` version of Kudu. Commit and send out the changes for review on Gerrit.
. When ready, create a new lightweight tag and push it to the Apache Git repository.
+
----
git tag 1.x.y-RC1
git push apache 1.x.y-RC1
----
. Build a source tarball against the RC branch.
. Out of the Kudu git workspace, checkout the
link:https://dist.apache.org/repos/dist/dev/kudu/[dev Subversion (SVN) repository].
Create a new sub-directory named correspondingly. Copy the artifacts to this
sub-directory and commit.
+
----
svn co --depth=immediates https://dist.apache.org/repos/dist/dev/kudu/ kudu-dev-release
cd kudu-dev-release
mkdir 1.x.y-RC1
cp <path_to_kudu_git_workspace>/build/apache-kudu-1.x.y.tar.* 1.x.y-RC1
svn add 1.x.y-RC1
svn commit --username=<your_apache_username> -m "Adding Kudu 1.x.y RC1"
----
. Create a Maven staging repository for the release candidate Java artifacts.
+
----
# Run a gpg-agent if you don't normally
gpg-agent --daemon
cd java
./gradlew clean assemble
# Turn off bash history: this is to avoid exposing the credentials
# via .bash_history file.
set +o history
./gradlew --no-parallel uploadArchives \
-Psigning.gnupg.keyName=<8-character-pgp-key-id> \
-PmavenUsername='<APACHE-LDAP-USERNAME>' \
-PmavenPassword='<APACHE-LDAP-PASSWORD>'
# Turn on bash history.
set -o history
----
+
NOTE: If `uploadArchives` is executed without `--no-parallel`, uploading a
number of artifacts fails with "peer not authenticated" errors.
. Build and deploy new binary test JARs for the RC on macOS and Linux. Build
the Linux JAR on a CentOS 6.6 image, and build the macOS JAR on macOS
Yosemite if possible (see
link:https://issues.apache.org/jira/browse/KUDU-2724[KUDU-2724] to remove the
need to build on an old version of macOS).
+
----
# Build a binary JAR for the local operating system. Make sure the thirdparty
# components were built to match the source code that the RC is being built
# with. It's a good idea to clone the Kudu git repo into a dedicated
# workspace, rebuilding the thirdparty compoments from scratch for particular
# release. The resulting JAR is output into the build/mini-cluster directory.
./build-support/mini-cluster/build_mini_cluster_binaries.sh
# Sign and publish all matching kudu-binary artifacts from the
# build/mini-cluster directory to the Maven staging repository that hosts
# the Java artifacts of the Apache Kudu project (see above).
# Turn off bash history: this is to avoid exposing the credentials persisted
# in .bash_history file.
set +o history
./build-support/mini-cluster/publish_mini_cluster_binaries.sh -a=deploy \
-u='<APACHE-LDAP-USERNAME>' -p='<APACHE-LDAP-PASSWORD>'
# Turn bash history back on.
set -o history
----
+
NOTE: If the binary test JAR artifacts are deployed by the same person and from the
same machine as the Java Maven artifacts, they should appear in the same
staging repository. Otherwise, they will create a separate Maven staging
repository, which is not a problem.
+
TIP: To publish an artifact outside of the `build/mini-cluster` directory, e.g.
if JAR was built by someone else or for another OS, pass the
`-j=directory_name` argument to the publishing script to specify the location
of the JAR file to sign and publish.
. Close the Maven staging repository (or repositories).
+
Go to the link:https://repository.apache.org/[repository manager] and log
into the repository server using your Apache credentials. Make sure to enable
Adobe Flash in your browser for this Web site. Now, go the
link:https://repository.apache.org/\#stagingRepositories[staging repository]
and look for ‘orgapachekudu-####’ in the staging repositories list. You can
check the `Content` tab at the bottom to make sure you have all of the
expected stuff (client, various integrations, etc.). Hit the checkbox next to
your new staging repo and hit `Close`. Enter something similar to
"Apache Kudu 1.x.y-RC1" into the description box and confirm. Wait a minute
or two and hit `Refresh`, and your staging repo should now have a URL shown
in its summary tab
(e.g. `https://repository.apache.org/content/repositories/orgapachekudu-1005`)
== Initiating a Vote for an RC
. Send an email to dev@kudu.apache.org to start the RC process, using this
link:http://mail-archives.apache.org/mod_mbox/kudu-dev/201606.mbox/%3CCAGpTDNduoQM0ktuZc1eW1XeXCcXhvPGftJ%3DLRB8Er5c2dZptvw%40mail.gmail.com%3E[example]
as a template.
. Reminder that voting on a release requires a
link:https://www.apache.org/foundation/glossary.html#MajorityApproval[Majority Approval]
by the PMC.
. Cycle through as many RCs as required.
. Always send an email with a different subject to indicate the result. For
link:http://mail-archives.apache.org/mod_mbox/kudu-dev/201606.mbox/%3CCAGpTDNeJdU%2BoUF4ZJneZt%3DCfFHY-HoqKgORwVuWWUMHq5jBNzA%40mail.gmail.com%3E[example].
. After the vote passes, send an email to dev@kudu.apache.org indicating the
result.
== Release
. For a release to be made official, the result release candidate must be put
in the release SVN repository. Create a new sub-directory in the release SVN
repository for the new release and copy the files from the dev repository:
+
----
cd kudu-dist-release
mkdir 1.x.y
cp <path_to_kudu-dev-release>/1.x.y-RC1/* 1.x.y
svn add 1.x.y
svn commit --username=<your_apache_username> -m "Adding files for Kudu 1.x.y"
----
. In the Kudu git repo, create a signed tag from the RC's tag, verify the
signature has been applied and verifiable, and push it to the
Apache Git repository:
+
----
gpg --list-secret-keys --keyid-format=short
git tag -u <gpg_key_id> -m 'Release Apache Kudu 1.x.y' 1.x.y 1.x.y-RC1
git tag -v 1.x.y
git push apache 1.x.y
----
. Release the staged Java artifacts. Select the release candidate staging
repository in link:https://repository.apache.org/#stagingRepositories[Nexus],
and click `Release`. You should shortly be able to see the artifacts in
link:https://search.maven.org/search?q=g:org.apache.kudu[Maven Central].
. Release the Python artifacts. You will need to setup an account on link:https://PyPi.org[PyPi.org]
and ask to be added to the kudu-python PyPi project if you have not done this before.
+
----
# Prepare and sign the python source distribution.
cd python
rm -rf dist/*
python setup.py sdist
gpg --detach-sign -a dist/kudu-python-1.x.y.tar.gz
# Upload the distribution to PyPi using twine.
pip install twine
twine upload dist/*
----
Note: You can upload to the test PyPi by adding
`--repository-url https://test.pypi.org/legacy/` to the twine command.
. Generate the version-specific documentation from that branch following these
<<README.adoc#updating-the-site,instructions>>.
+
WARNING: The site *MUST NOT* be built on Mac. See the
<<README.adoc#building-docs,documentation building instructions>> for details.
+
. Update the `index.md` file in the `releases` directory. Add a line about
the newly released version into the `Latest release` section, move line
about the previous one into the `Previous releases`. Overall, update
the `Previous releases` and `Archived releases` to reflect the new contents
of the
link:https://dist.apache.org/repos/dist/release/kudu/[release SVN repository].
See the last item of this section to get the idea on what the new contents
of the release SVN repository is going to be.
. A new sub-directory named after the release version should be added into the
`releases` directory: it should contain `apidocs`, `cpp-client-api`,
`docs` sub-dirs and `index.md` file. Add the newly added sub-directory into
the git staging:
+
----
# Assuming current working directory is the root of the git workspace.
git add releases/1.x.y
----
. Take a quick look at the auto-generated `releases/1.x.y/index.md` file
to make sure the download links meet the current
link:https://www.apache.org/dev/release-download-pages.html[criteria]. The
criteria keep changing and the announcement will be rejected if our release
page doesn't meet the criteria.
. Commit the changes:
+
----
git commit -a -m "update website for 1.x.y release"
----
. Submit these changes to the `gh-pages` Gerrit branch and get them reviewed.
. Add a release blog post about the new release and send it out for review, similar to
link:https://kudu.apache.org/2019/07/09/apache-kudu-1-10-0-release.html[this].
. Once the reviews are finished and the commits are pushed to `gh-pages` branch,
update the website following these
<<README.adoc#deploying-changes-to-the-apache-kudu-web-site,instructions>>.
. Build and push the Docker images (run the below command from the project
root).
+
----
./docker/docker-build.py --action push --platforms linux/amd64 linux/arm64
----
+
To push the images to Dockerhub, you need to be granted permissions by the ASF
infra team, which you can request via an Apache JIRA ticket similar to
link:https://issues.apache.org/jira/browse/INFRA-23166[INFRA-23166].
. About 24 hours after all artifacts have been published, send an email to
user@kudu.apache.org, dev@kudu.apache.org, and announce@apache.org to
announce the new release. The email should be similar to
link:https://s.apache.org/pduz[this]. The announcement email must be sent
from your apache.org address, otherwise apache.org mailer will not deliver
the message to the addressees of the announce@ group at least. If using
gmail.com service for email correspondence, follow
link:https://support.google.com/mail/answer/22370?hl=en[this guide] to add
your apache.org address to be used as `From` address when sending email
messages. Also post the announcement to the blog and tweet it with the
link:https://twitter.com/ApacheKudu[@ApacheKudu] Twitter handle.
. Update the version number on the branch you released from back to a SNAPSHOT
for the next patch release, such as `1.6.1-SNAPSHOT` after the `1.6.0` release.
. In the `master` branch of the project git repo, update version-related
information under the `examples` sub-directory to make all examples using the
fresh artifacts of latest released version. This should be done only after
publishing the officially released artifacts, so they are available while
building the examples.
. In the `master` branch of the project git repo, add the previous release notes to the
`prior_release_notes.adoc` and send out the change for review on Gerrit.
. About another 24 hours later, clean up the SVN. If releasing a new minor
version, delete the oldest minor version branch in the release repo (e.g. if
`1.7.1`, `1.8.0`, and `1.9.0` exist and you just released `1.10.0`, delete
`1.7.1`). If releasing a maintenance version, delete the previous maintenance
branch (e.g. if you released `1.2.1`, delete `1.2.0`). Also delete any
release candidates from the dev SVN.
. Set the release date for the just released version in
link:https://issues.apache.org[Apache JIRA] or ask someone with permission
to do so. After logging in, in the left sidebar switch to the `Releases` item.
Once at the `Releases` page, apply the `Unreleased` filter. In the row
corresponding to the new release, click at the ellipsis in the `Action`
column and select the `Release` item, setting the appropriate release date.