<!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
-->

# Apache Beam

## CI Environment

Continuous Integration is important component of making Apache Beam robust and stable.

Our execution environment for CI is mainly the Jenkins which is available at
[https://ci-beam.apache.org/](https://ci-beam.apache.org/). See
[.test-infra/jenkins/README](.test-infra/jenkins/README.md)
for trigger phrase, status and link of all Jenkins jobs. See Apache Beam Developer Guide for
[Jenkins Tips](https://cwiki.apache.org/confluence/display/BEAM/Jenkins+Tips).

An additional execution environment for CI is [GitHub Actions](https://github.com/features/actions). GitHub Actions
(GA) are very well integrated with GitHub code and Workflow and it has evolved fast in 2019/2020 to become
a fully-fledged CI environment, easy to use and develop for, so we decided to use it for building python source
distribution and wheels.

## GitHub Actions

### GitHub actions run types

The following GA CI Job runs are currently run for Apache Beam, and each of the runs have different
purpose and context.

#### Pull request run

Those runs are results of PR from the forks made by contributors. Most builds for Apache Beam fall
into this category. They are executed in the context of the "Fork", not main
Beam Code Repository which means that they have only "read" permission to all the GitHub resources
(container registry, code repository). This is necessary as the code in those PRs (including CI job
definition) might be modified by people who are not committers for the Apache Beam Code Repository.

The main purpose of those jobs is to check if PR builds cleanly, if the test run properly and if
the PR is ready to review and merge.

#### Direct Push/Merge Run

Those runs are results of direct pushes done by the committers or as result of merge of a Pull Request
by the committers. Those runs execute in the context of the Apache Beam Code Repository and have also
write permission for GitHub resources (container registry, code repository).
The main purpose for the run is to check if the code after merge still holds all the assertions - like
whether it still builds, all tests are green.

This is needed because some of the conflicting changes from multiple PRs might cause build and test failures
after merge even if they do not fail in isolation.

#### Scheduled runs

Those runs are results of (nightly) triggered job - only for `master` branch. The
main purpose of the job is to check if there was no impact of external dependency changes on the Apache
Beam code (for example transitive dependencies released that fail the build). Another reason for the nightly
build is that the builds tags most recent master with `nightly-master`.

All runs consist of the same jobs, but the jobs behave slightly differently or they are skipped in different
run categories. Here is a summary of the run categories with regards of the jobs they are running.
Those jobs often have matrix run strategy which runs several different variations of the jobs
(with different platform type / Python version to run for example)

### Google Cloud Platform Credentials

Some of the jobs require variables stored as [GitHub Secrets](https://docs.github.com/en/actions/configuring-and-managing-workflows/creating-and-storing-encrypted-secrets)
to perform operations on Google Cloud Platform.
These variables are:
 * `GCP_PROJECT_ID` - ID of the Google Cloud project. For example: `apache-beam-testing`.
 * `GCP_REGION` - Region of the bucket and dataflow jobs. For example: `us-central1`.
 * `GCP_TESTING_BUCKET` - Name of the bucket where temporary files for Dataflow tests will be stored. For example: `beam-github-actions-tests`.
 * `GCP_PYTHON_WHEELS_BUCKET` - Name of the bucket where python source distribution and wheels will be stored. For example: `beam-wheels-staging`.
 * `GCP_SA_EMAIL` - Service account email address. This is usually of the format `<name>@<project-id>.iam.gserviceaccount.com`.
 * `GCP_SA_KEY` - Service account key. This key should be created and encoded as a Base64 string (eg. `cat my-key.json | base64` on macOS).

Service Account shall have following permissions ([IAM roles](https://cloud.google.com/iam/docs/understanding-roles)):
 * Storage Admin (roles/storage.admin)
 * Dataflow Admin (roles/dataflow.admin)
 * Artifact Registry writer (roles/artifactregistry.createOnPush)
 * Big Query Data Editor (roles/bigquery.dataEditor)
 * Service Account User (roles/iam.serviceAccountUser)

### Workflows

#### Build python source distribution and wheels - [build_wheels.yml](.github/workflows/build_wheels.yml)

| Job                                             | Description                                                                                                                                                                                                                                                        | Pull Request Run | Direct Push/Merge Run | Scheduled Run | Requires GCP Credentials |
|-------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|-----------------------|---------------|--------------------------|
| Check GCP variables                             | Checks that GCP variables are set. Jobs which required them depend on the output of this job.                                                                                                                                                                      | Yes              | Yes                   | Yes           | Yes/No                   |
| Build python source distribution                | Builds python source distribution and uploads it to artifacts. Artifacts from release branch are used in release process ([`build_release_candidate.sh`](release/src/main/scripts/build_release_candidate.sh))                                                     | Yes              | Yes                   | Yes           | -                        |
| Prepare GCS                                     | Clears target path on GCS if already exists.                                                                                                                                                                                                                       | -                | Yes                   | Yes           | Yes                      |
| Upload python source distribution to GCS bucket | Uploads python source distribution to GCS bucket for path unique for specific workflow run.                                                                                                                                                                        | -                | Yes                   | Yes           | Yes                      |
| Build python wheels on linux/macos/windows      | Builds python wheels on linux/macos/windows platform with usage of `cibuildwheel` and uploads it to artifacts. Artifacts from release branch are used in release process ( [ `build_release_candidate.sh` ](release/src/main/scripts/build_release_candidate.sh) ) | Yes              | Yes                   | Yes           | -                        |
| Upload python wheels to GCS bucket              | Uploads python wheels to GCS bucket for path unique for specific workflow run. Additionally uploads workflow run data.                                                                                                                                             | -                | Yes                   | Yes           | Yes                      |
| List files on Google Cloud Storage Bucket       | Lists files on GCS for verification purpose.                                                                                                                                                                                                                       | -                | Yes                   | Yes           | Yes                      |
| Branch repo nightly                             | Branch repo with `nightly-master` if build python source distribution and python wheels finished successfully.                                                                                                                                                     | -                | -                     | Yes           | -                        |

#### Python tests - [python_tests.yml](.github/workflows/python_tests.yml)

| Job                              | Description                                                                                                           | Pull Request Run | Direct Push/Merge Run | Scheduled Run | Requires GCP Credentials |
|----------------------------------|-----------------------------------------------------------------------------------------------------------------------|------------------|-----------------------|---------------|--------------------------|
| Check GCP variables              | Checks that GCP variables are set. Jobs which required them depend on the output of this job.                         | Yes              | Yes                   | Yes           | Yes/No                   |
| Build python source distribution | Builds python source distribution and uploads it to artifacts. Artifacts are used in `Python Wordcount Dataflow` job. | -                | Yes                   | Yes           | Yes                      |
| Python Unit Tests                | Runs python unit tests.                                                                                               | Yes              | Yes                   | Yes           | -                        |
| Python Wordcount Direct Runner   | Runs python WordCount example with Direct Runner.                                                                     | Yes              | Yes                   | Yes           | -                        |
| Python Wordcount Dataflow        | Runs python WordCount example with DataFlow Runner.                                                                   | -                | Yes                   | Yes           | Yes                      |

#### Java tests - [java_tests.yml](.github/workflows/java_tests.yml)

| Job                          | Description                                                                                   | Pull Request Run | Direct Push/Merge Run | Scheduled Run | Requires GCP Credentials |
|------------------------------|-----------------------------------------------------------------------------------------------|------------------|-----------------------|---------------|--------------------------|
| Check GCP variables          | Checks that GCP variables are set. Jobs which required them depend on the output of this job. | Yes              | Yes                   | Yes           | Yes/No                   |
| Java Unit Tests              | Runs Java unit tests.                                                                         | Yes              | Yes                   | Yes           | -                        |
| Java Wordcount Direct Runner | Runs Java WordCount example with Direct Runner.                                               | Yes              | Yes                   | Yes           | -                        |
| Java Wordcount Dataflow      | Runs Java WordCount example with DataFlow Runner.                                             | -                | Yes                   | Yes           | Yes                      |

### Release Preparation and Validation Workflows

#### Start Snapshot Build - [start_snapshot_build.yml](.github/workflows/start_snapshot_build.yml)
| Job                   | Description                                                             | Pull Request Run | Direct Push/Merge Run | Scheduled Run | Requires GCP Credentials |
|-----------------------|-------------------------------------------------------------------------|------------------|-----------------------|---------------|--------------------------|
| Start Snapshot Build  | Creates PR against apache:master and triggers a job to build a snapshot | No               | No                    | No            | No                       |

#### Choose RC Commit - [choose_rc_commit.yml](.github/workflows/choose_rc_commit.yml)

| Job              | Description                                                                                         | Pull Request Run | Direct Push/Merge Run | Scheduled Run | Requires GCP Credentials |
|------------------|-----------------------------------------------------------------------------------------------------|------------------|-----------------------|---------------|--------------------------|
| Choose RC Commit | Chooses a commit to be the basis of a release candidate and pushes a new tagged commit for that RC. | No               | No                    | No            | No                       |

#### Cut Release Branch - [verify_release_build.yml](.github/workflows/cut_release_branch.yml)
| Job                   | Description                                                | Pull Request Run | Direct Push/Merge Run | Scheduled Run | Requires GCP Credentials |
|-----------------------|------------------------------------------------------------|------------------|-----------------------|---------------|--------------------------|
| Update Master         | Update Apache Beam master branch with next release version | No               | No                    | No            | No                       |
| Update Release Branch | Cut release branch for current development version         | No               | No                    | No            | No                       |

#### Verify Release Build - [verify_release_build.yml](.github/workflows/verify_release_build.yml)

| Job                          | Description                                                                                   | Pull Request Run | Direct Push/Merge Run | Scheduled Run | Requires GCP Credentials |
|------------------------------|-----------------------------------------------------------------------------------------------|------------------|-----------------------|---------------|--------------------------|
| Verify Release Build         | Verifies full life cycle of Gradle Build and all PostCommit/PreCommit tests against Release Branch on CI.                   | No               | No                    | No            | No                       |

#### Git tag Release Version - [git_tag_released_version.yml](.github/workflows/git_tag_released_version.yml)

| Job                             | Description                                                                                                    | Pull Request Run | Direct Push/Merge Run | Scheduled Run | Requires GCP Credentials |
|---------------------------------|----------------------------------------------------------------------------------------------------------------|------------------|-----------------------|---------------|--------------------------|
| Git Tag Release Version         | Create and push a new tag for the released version by copying the tag for the final release candidate.         | No               | No                    | No            | No                       |

#### Run RC Validation - [run_rc_validation.yml](.github/workflows/run_rc_validation.yml)

| Job                          | Description                                                                                   | Pull Request Run | Direct Push/Merge Run | Scheduled Run | Requires GCP Credentials |
|------------------------------|-----------------------------------------------------------------------------------------------|------------------|-----------------------|---------------|--------------------------|
| Python Release Candidate     | Comment on PR to trigger Python ReleaseCandidate Jenkins job.                                 | No               | No                    | No            | No                       |
| Python XLang SQL Taxi        | Runs Python XLang SQL Taxi with DataflowRunner                                                | No               | No                    | No            | Yes                      |
| Python XLang Kafka           | Runs Python XLang Kafka Taxi with DataflowRunner                                              | No               | No                    | No            | Yes                      |
| Direct Runner Leaderboard    | Runs Python Leaderboard with DirectRunner                                                     | No               | No                    | No            | Yes                      |
| Direct Runner GameStats      | Runs Python GameStats with DirectRunner.                                                      | No               | No                    | No            | Yes                      |
| Dataflow Runner Leaderboard  | Runs Python Leaderboard with DataflowRunner                                                   | No               | No                    | No            | Yes                      |
| Dataflow Runner GameStats    | Runs Python GameStats with DataflowRunner                                                     | No               | No                    | No            | Yes                      |

### All migrated workflows run based on the following triggers

| Description | Pull Request Run | Direct Push/Merge Run | Scheduled Run | Workflow Dispatch |
|-------------|------------------|-----------------------|---------------|-------------------|
| PostCommit  | No               | Yes                   | Yes           | Yes               |
| PreCommit   | Yes              | Yes                   | Yes           | Yes               |

### PreCommit Workflows

| Workflow                                                                         | Description             | Requires GCP Credentials  |
|----------------------------------------------------------------------------------|-------------------------|---------------------------|
| [job-precommit-placeholder.yml](.github/workflows/job-precommit-placeholder.yml) | Description placeholder | Yes/No                    |

### PostCommit Workflows

| Workflow                                                                           | Description             | Requires GCP Credentials |
|------------------------------------------------------------------------------------|-------------------------|--------------------------|
| [job-postcommit-placeholder.yml](.github/workflows/job-postcommit-placeholder.yml) | Description placeholder | Yes/No                   |

### GitHub Action Tips

* All migrated workflows get executed on **pre-configured self-hosted** runners. For this reason, GCP credentials are **only** needed when running the workflows in a different runner.
* If you introduce changes to the workflow it is possible that your changes will not be present in the check run triggered in Pull Request.
In this case please attach link to the modified workflow run executed on your fork.
* Possible timeouts with macOS runner - existing issue: [(X) This check failed - sometimes happens on macOS runner #841](https://github.com/actions/virtual-environments/issues/841)
* [GitHub Actions Documentation](https://docs.github.com/en/actions)