The document provides a standard process for creating the sources and binaries that are required for a release of Apache Gluten project with the Velox backend.
A standard release distribution can be created following the below steps.
Pull and download the build environment docker image. The docker image is periodically built and uploaded to DockerHub by scheduled GitHub Actions jobs.
docker pull apache/gluten:vcpkg-centos-7 docker run -it apache/gluten:vcpkg-centos-7 bash
In the docker container created by the last step, execute the following command to clone the repository of Gluten with a specific git tag that you want to build on.
We are taking v1.6.0-example-rc3 as an example git tag in this guide.
git clone --branch v1.6.0-example-rc3 https://github.com/apache/gluten.git /workspace
Build the project for all supported Spark versions.
cd /workspace bash dev/release/build-release.sh
cd /workspace mkdir -p release cp -R package/target/* release/
By following this step you will create the release distribution that comply with the common name convention of ASF project release process.
Note, the current tag should be specified when running this script.
cd /workspace bash dev/release/package-release.sh v1.6.0-example-rc3
Confirm that all the needed sources and binaries are successfully created at the release directory $GLUTEN_HOME/release/.
[root@8de83f716f0f workspace]# ls -l release/ total 481628 -rw-r--r--. 1 root root 74396439 Oct 14 14:19 apache-gluten-1.6.0-example-src.tar.gz -rw-r--r--. 1 root root 104790092 Oct 14 14:19 apache-gluten-1.6.0-example-bin-spark-3.2.tar.gz -rw-r--r--. 1 root root 104767582 Oct 14 14:19 apache-gluten-1.6.0-example-bin-spark-3.3.tar.gz -rw-r--r--. 1 root root 104625356 Oct 14 14:19 apache-gluten-1.6.0-example-bin-spark-3.4.tar.gz -rw-r--r--. 1 root root 104595103 Oct 14 14:19 apache-gluten-1.6.0-example-bin-spark-3.5.tar.gz
This section outlines the steps for releasing Apache Gluten according to the Apache release guidelines. All projects under the Apache umbrella must adhere to the Apache Release Policy. This guide is designed to assist you in comprehending the policy and navigating the process of releasing projects at Apache.
Prepare the release artifacts.
Upload the release artifacts to the SVN repository.
Verify the release artifacts.
Initiate a release vote.
Announce the results and the release.
Create a branch from the target git repository.
Tag a RC and draft the release notes.
Build and Sign the release artifacts (including source archives, binaries, ...etc).
Generate checksums for the artifacts.
Create a GPG key
Add the GPG key to the KEYS file
Sign the release artifacts with the GPG key.
# create a GPG key, after executing this command, select "RSA and RSA" $ gpg --full-generate-key # list the GPG keys $ gpg --keyid-format SHORT --list-keys # upload the GPG key to the key server, xxx is the GPG key id # eg: pub rsa4096/4C21E346 2024-05-06 [SC], 4C21E346 is the GPG key id; $ gpg --keyserver keyserver.ubuntu.com --send-key xxx # append the GPG key to the KEYS file the svn repository # [IMPORTANT] Don't replace the KEYS file, just append the GPG key to the KEYS file. $ svn co https://dist.apache.org/repos/dist/release/gluten/ $ (gpg --list-sigs xxx@apache.org && gpg --export --armor xxx@apache.org) >> KEYS $ svn ci -m "add gpg key" # sign the release artifacts, xxxx is xxx@apache.org $ for i in *.tar.gz; do echo $i; gpg --local-user xxxx --armor --output $i.asc --detach-sig $i ; done
# create the checksums $ for i in *.tar.gz; do echo $i; sha512sum $i > $i.sha512 ; done
Create a project directory in the SVN repository (1st time only). https://dist.apache.org/repos/dist/dev/gluten/
Create a directory for the release artifacts in the SVN repository. https://dist.apache.org/repos/dist/dev/gluten/{release-version} release-version format: apache-gluten-#.#.#-rc#
Upload the release artifacts to the SVN repository.
$ svn co https://dist.apache.org/repos/dist/dev/gluten/ $ cp /path/to/release/artifacts/* ./{release-version}/ $ svn add ./{release-version}/* $ svn commit -m "add Apache Gluten release artifacts for {release-version}"
https://dist.apache.org/repos/dist/dev/gluten/{release-version} to verify if the file upload is successful or not. The upload release artifacts should be include* apache-gluten-#.#.#-src.tar.gz * apache-gluten-#.#.#-src.tar.gz.asc * apache-gluten-#.#.#-src.tar.gz.sha512
Please follow below steps to verify the release artifacts.
Check if the Download links are valid.
Check if the checksums and GPG signatures are valid.
Check if the release artifacts name is qualified and match with the current release.
Check if LICENSE and NOTICE files are correct.
Check if the License Headers are included in all files if necessary.
No unlicensed compiled archives bundled in source archive.
Please follow below steps to verify the signatures.
# download KEYS $ curl https://dist.apache.org/repos/dist/release/gluten/KEYS > KEYS # import KEYS and trust the key, please replace the email address with the one you want to trust. $ gpg --import KEYS $ gpg --edit-key xxx@apache.org gpg> trust gpg> 5 gpg> y gpg> quit # enter the directory where the release artifacts are located $ cd /path/to/release/artifacts # verify the signature $ for i in *.tar.gz; do echo $i; gpg --verify $i.asc $i ; done # if you see 'Good signature' in the output, it means the signature is valid.
Please follow below steps to verify the checksums
# verify the checksums $ for i in *.tar.gz; do echo $i; sha512sum --check $i.sha512; done
Email a vote request to dev@gluten.apache.org, requiring at least 3 PMC +1s.
Allow 72 hours or until enough votes are collected.
Share the vote outcome on the dev list.
Vote Email Template
[VOTE] Release Apache Gluten {release-version}
Hello,
This is a call for vote to release Apache Gluten version {release-version}.
The vote thread:
https://lists.apache.org/thread/{id}
Vote Result:
https://lists.apache.org/thread/{id}
The release candidates:
https://dist.apache.org/repos/dist/dev/gluten/{release-version}/
Release notes:
https://github.com/apache/gluten/releases/tag/{release-version}
Git tag for the release:
https://github.com/apache/gluten/releases/tag/{release-version}
Git commit id for the release:
https://github.com/apache/gluten/commit/{id}
Keys to verify the Release Candidate:
https://downloads.apache.org/gluten/KEYS
The vote will be open for at least 72 hours or until the necessary number of votes are reached.
Please vote accordingly:
[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason
Checklist for reference:
[ ] Download links are valid.
[ ] Checksums and PGP signatures are valid.
[ ] Source code distributions have correct names matching the current release.
[ ] LICENSE and NOTICE files are correct for each Apache Gluten repo.
[ ] All files have license headers if necessary.
[ ] No unlicensed compiled archives bundled in source archive.
To compile from the source, please refer to:
https://github.com/apache/gluten#building-from-source
Thanks,
<YOUR NAME>
Announce Email Template
Hello everyone,
The Apache Gluten {release-version} has been released!
Apache Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
Download Links: https://downloads.apache.org/gluten/
Release Notes: https://github.com/apache/gluten/releases/tag/{release-version}
Website: https://gluten.apache.org/
Resources:
- Issue: https://github.com/apache/gluten/issues
- Mailing list: dev@gluten.apache.org
Thanks,
<YOUR NAME>
After the vote has passed, you need to migrate the RC build release to an official release by moving the artifacts from Apache SVN's dev directory to the release directory. Please follow the steps below to upload the artifacts:
$ svn mv https://dist.apache.org/repos/dist/dev/gluten/{release-version} https://dist.apache.org/repos/dist/release/gluten/{release-version} -m "transfer packages for gluten {release-version}"