Playground deployment on GCP

This guide shows you how to deploy full Playground environment on Google Cloud Platform (GCP) environment. Alternatively, you can automate Playground environment deployment with Cloud Build as described in readme.

Prerequisites:

Following items need to be setup for Playground deployment on GCP:

1. GCP project

2. GCP User account (Note: You will find the instruction “How to create User account” for your new project)
   Ensure that the account has at least following privileges:

   • App Engine Admin
   • App Engine Creator
   • Artifact Registry Administrator
   • Cloud Memorystore Redis Admin
   • Compute Admin
   • Create Service Accounts
   • Kubernetes Engine Admin
   • Quota Administrator
   • Role Administrator
   • Security Admin
   • Service Account User
   • Storage Admin
   • Cloud Datastore Index Admin

3. Google Cloud Storage bucket for saving deployment state

4. DNS name for your Playground deployment instance

5. OS with installed software listed below:

• Java
• Kubernetes Command Line Interface
• HELM
• Docker
• Terraform
• gcloud CLI

6. Apache Beam Git repository cloned locally

Prepare deployment configuration:

Playground uses terraform.tfvars located in playground/terraform/environment/environment_name to define variables specific to an environment (e.g., prod, test, staging).

1. Create a folder (further referred as environment_name) to define a new environment and place configuration files into it:

• terraform.tfvars environment variables:

project_id           = "project_id"          #GCP Project ID
network_name         = "network_name"        #GCP VPC Network Name for Playground deployment
gke_name             = "playground-backend"  #Playground GKE Cluster name
region               = "us-east1"            #Set the deployment region
location             = "us-east1-b"          #Select the deployment location from available in the specified region
state_bucket         = "bucket_name"         #GCS bucket name for Beam Playground temp files
redis_name           = "playground_redis"    #Choose the name for redis instance
min_count            = 2                     #Min node count for GKE cluster
max_count            = 6                     #Max node count for GKE cluster

• state.tfbackend environment variables:

bucket               = "bucket_name"         #input bucket name - will be used for terraform tfstate file

2. Configure authentication for the Google Cloud Platform

gcloud init

gcloud auth application-default login

Deploy Playground infrastructure:

1. Start the following command from the top level repository folder (“beam”) to deploy the Payground infrastructure:

./gradlew playground:terraform:InitInfrastructure -Pproject_environment="environment_name" -Pdns-name="playground.zone"

Where playground.zone - chosen DNS for Playground

2. Find a Static IP in your GCP project>VPC Network>IP Addresses>pg-static-ip
   Add following DNS A records for the discovered static IP address:

java.playground.zone
python.playground.zone
scio.playground.zone
go.playground.zone
router.playground.zone
playground.zone

Where “playground.zone” is the registered DNS zone
More about DNS zone registration
More about A records in DNS

Deploy Playground to Kubernetes:

1. Run the following command to authenticate in the Docker registry:

 gcloud auth configure-docker `chosen_region`-docker.pkg.dev

2. Run the following command to authenticate in GKE:

gcloud container clusters get-credentials --region `chosen_location` `gke_name` --project `project_id`

Start the following command from the top level repository folder (“beam”) to deploy the Payground infrastructure:

./gradlew playground:terraform:gkebackend -Pproject_environment="environment_name" -Pdocker-tag="tag" -Pdns-name="playground.zone" -Psdk-tag=2.43.0

Where tag - image tag for backend, playground.zone - chosen DNS for Playground, Psdk-tag - current BEAM version

During script execution, a Google managed certificate will be created. Provisioning might take up to 60 minutes.

Validate deployed Playground:

1. Run “helm list” command in the console to ensure that status is “deployed”:

NAME            NAMESPACE  REVISION        UPDATED         STATUS          CHART                          APP VERSION
playground      default       1            your time      deployed        playground-2.44.0-SNAPSHOT         1.0.0

2. Run “kubectl get managedcertificate” command in the console to ensure that status is “Active”:

NAME               AGE     STATUS
GCP Project       time     Active

3. Open Beam Playground frontend webpage in a web browser (e.g. https://playground.zone) to ensure that Playground frontend page is available