The Ozone project uses Github Actions, (GA), for its CI system. GA are implemented with workflows, which are groups of jobs combined to accomplish a CI task, all defined in a single yaml file. The Ozone workflow yaml files are here.
This is the most important workflow. It runs the tests that verify the latest commits.
It is triggered each time a pull request is created or synchronized (i.e. when the remote branch is pushed to). These trigger events are defined in the build-branch workflow.
The full-ci workflow is divided into a number of different jobs, most of which run in parallel. Each job is described below.
Some of the jobs are defined using GA's “build matrix” feature. This allows you define similar jobs with a single job definition. Any differences are specified by a list of values for a specific key. For example, the “compile” job uses the matrix feature to generate the images with different versions of java. There, the matrix is specified by the “java” key which has a list of values describing which version of java to use, (8 or 11.)
The jobs currently using the “build matrix” feature are: “compile”, “basic”, “unit”, “acceptance” and “integration”. These jobs also use GA's fail-fast flag to cancel the other jobs in the same matrix, if one fails. For example, in the “compile” job, if the java 8 build fails, the java 11 build will be cancelled due to this flag, but the other jobs outside the “compile” matrix are unaffected.
While the fail-fast flag only works within a matrix job, the “Cancelling” workflow, (described below,) works across jobs.
The build-info job script runs before the others and determines which of the other jobs are to be run. If the workflow was triggered by some event other than a PR, then all jobs/tests are run. They are also all run if the PR has a label containing the following string, “full tests needed”.
Otherwise, build-info first generates a list of files that were changed by the PR. It matches that list against a series of regex's, each of which is associated with a different job. It sets the appropriate flag for each match. Those boolean flags are used later in the run to decide whether the corresponding job should be run
For example, a regex like the following is used to determine if the Kubernetes flag should be set.
local pattern_array=(
"^hadoop-ozone/dev-support/checks/kubernetes.sh"
"^hadoop-ozone/dist/src/main/k8s"
)
Builds the Java 8 and 11 versions of the jars, and saves the java 8 version for some of the subsequent jobs.
Runs a subset of the following subjobs depending on what was selected by build-info
Performs unit tests (if necessary) in two parts:
Confirms that the list of jars included in the current build matches the expected ones defined here
If they don't match, it describes how to make the updates to include the changes, (if they are intentional). Otherwise, the changes should be removed.
Runs smoketests using robot framework and a real docker compose cluster. There are three iterations, “secure”, “unsecure”, and “misc”, each running in parallel, as different matrix configs.
Runs k8s tests
Runs ‘mvn test’ for all integration/minicluster tests, split into multiple subjobs, by a matrix config.
Merges the coverage data from the following jobs that were run earlier:
This workflow is scheduled each night at midnight; it closes PR's that have not been updated in the last 21 days, while letting the author know they are free to reopen.
This workflow is triggered each time a comment is added/edited to a PR. It checks to see if the body of the comment begins with one of the following strings and, if so, invokes the corresponding command.
The following workflows no longer run but still exist on the actions page for historical reasons:
Note that the deprecated build-branch has the same name as the current build-branch. (They can be distinguished by the URL.)
git commit --allow-empty -m 'trigger new CI check'