The Apache SystemDS project publishes new version of the software on a regular basis. Releases are the interface of the project with the public and most users interact with the project only through the released software (this is intentional!). Releases are a formal offering, which are publicly voted by the SystemDS community.
Releases are executed by a Release Manager, who is one of the project committers.
Release has legal consequences to the team. Make sure to comply with all the procedures outlined by the ASF via Release Policy and Release Distribution. Any deviations or compromises are to be discussed in private@ or dev@ mail list appropriately.
Install the basic software and procure the required code and dependencies, credentials.
OS Requirement: Linux
RAM requirement: 8 GB +
Software Requirements:
Credential Requirements:
An important part of the software development life cycle (SDLC) is ensuring software release follow the ASF approved processes.
The release pipeline consists of the following steps:
The project PMC and community inspects the build files by downloading and testing. If it passes their requirements, they vote appropriately in the mailing list. The release version metadata is updated and the application is deployed to the public release.
Note: Only PMC can push to the Release repo for legal reasons, but committer can also act as the Release Manager with consensus by the team on the dev@ mail list.
Apache Nexus repository is located at repository.apache.org, it is Nexus 2.x Profession edition.
org.apache.systemds by visiting https://repository.apache.org/#stagingProfiles;1486a6e8f50cdfSYSTEMDS > Administration > Versions.Know more about versions in JIRA at view-and-manage-a-projects-versions guide
Investigating performance regressions is a collective effort. Regressions can happen during release process, but they should be investigated and fixed.
Release Manger should make sure that the JIRA issues are filed for each regression and mark Fix Version to the to-be-released version.
The regressions are to be informed to the dev@ mailing list, through release duration.
Create release branch from the main with version named 2.x.0-SNAPSHOT.
Release candidates are built from single commits off the development branch. Before building, the version must be set to a non SNAPSHOT/dev version.
Discussion on what is covered in voting for a commit.
Mail dev@systemds.apache.org of the release tags and triage information. This list of pending issues will be refined and updated collaboratively.
git configurationJAVA_HOME to JDK 8export GNUPGHOME=$HOME/.gnupg./do-release.sh -n
./do-release.sh
Answer the prompts with appropriate details as shown:
Branch [master]: master Current branch version is 2.1.0-SNAPSHOT. Release [2.1.0]: RC # [1]: 1 ASF user [ubuntu]: firstname Full name [Firstname Lastname]: GPG key [firstname@apache.org]: ================ Release details: BRANCH: master VERSION: 2.1.0 TAG: 2.1.0-rc1 NEXT: 2.1.1-SNAPSHOT ASF USER: firstname GPG KEY ID: firstname@apache.org FULL NAME: Firstname Lastname E-MAIL: firstname@apache.org ================ Is this info correct [Y/n]?
The docs will generated in build directory.
Since the artifacts will be deployed publicly, use a completely fresh copy of the SystemDS project used only for building and deploying.
Therefore, create a directory such as
mkdir ~/systemds-release
In this directory, clone a copy of the project.
git clone https://github.com/apache/systemds.git
Verify that each expected artifact is present at https://dist.apache.org/repos/dist/dev/systemds/ and that each artifact has accompanying checksums (such as .asc and .sha512)
The release candidate should build on Windows, OS X, and Linux. To do this cleanly, the following procedure can be performed.
Note: Use an empty local maven repository
Example:
git clone https://github.com/apache/systemds.git cd systemds git tag -l git checkout tags/2.1.0-rc1 -b 2.1.0-rc1 mvn -Dmaven.repo.local=$HOME/.m2/temp-repo clean package -P distribution
The entire test suite should pass on Windows, OS X, Linux.
For verification:
mvn clean verify
Each artifact must contain LICENSE and NOTICE files. These files must reflect the contents of the artifacts. If the project dependencies (i.e., libraries) have changed since the last release, the LICENSE and NOTICE files to be updated to reflect these changes.
For more information, see:
The project should also be built using the src (tgz and zip).
tar -xvzf systemds-2.1.0-src.tgz cd systemds-2.1.0-src mvn clean package -P distribution mvn verify
The standalone tgz and zip artifacts contain systemds files. Verify that the algorithms can be run on single node using these standalone distributions.
Here is an example:
see standalone guide of the documenation for more details.
tar -xvzf systemds-2.1.0-bin.tgz cd systemds-2.1.0-bin wget -P data/ http://archive.ics.uci.edu/ml/machine-learning-databases/haberman/haberman.data echo '{"rows": 306, "cols": 4, "format": "csv"}' > data/haberman.data.mtd echo '1,1,1,2' > data/types.csv echo '{"rows": 1, "cols": 4, "format": "csv"}' > data/types.csv.mtd systemds scripts/algorithms/Univar-Stats.dml -nvargs X=data/haberman.data TYPES=data/types.csv STATS=data/univarOut.mtx CONSOLE_OUTPUT=TRUE cd ..
Also check for Hadoop, and spark
Verify that the performance suite executes on Spark and Hadoop. The datasizes are 80MB, 800MB, 8GB, and 80GB.
Following a successful release candidate vote by SystemDS PMC members and the SystemDS community on the dev mailing list, the release candidate shall be approved.
The scripts will execute the release steps. and push the changes to the releases.
In the Apache Nexus Repo, release the staged artifacts to the Maven Central repository.
Steps:
Staging Repositories section, find the relevant release candidate entryReleaseReleaseUpdate the record at https://reporter.apache.org/addrelease.html?systemds
/release/systemds//dev/systemds/Announce Released version within the project and public.
@apache.org email address. This email has to be in plain text.Publish on Twitter on @ApacheSystemDS.
Once the release is complete, let us retrospectively update changes and improvements to this guide. Help the community adapt this guide for release validation before casting their vote.
Perhaps some steps can be simplified or require more clarification.
~/.gnupg.sudo mkdir -m 700 /usr/local/.gnupg
sudo GNUPGHOME=/usr/local/.gnupg gpg --gen-key
output will be, like the following:
gpg: /usr/local/.gnupg/trustdb.gpg: trustdb created gpg: key F164B430F91D6*** marked as ultimately trusted gpg: directory '/usr/local/.gnupg/openpgp-revocs.d' created gpg: revocation certificate stored as '/usr/local/.gnupg/openpgp-revocs.d/AD**...*.rev' public and secret key created and signed.
Note: Using sudo would add credentials in root users
export GNUPGHOME=/usr/local/.gnupg gpg --homedir $GNUPGHOME --list-keys gpg --homedir $GNUPGHOME --list-secret-keys