This guide shows you how to deploy full Playground environment on Google Cloud Platform (GCP) environment. Alternatively, you can automate Playground environment deployment with Cloud Build as described in readme.
GCP User account (Note: You will find the instruction “How to create User account” for your new project)
Ensure that the account has at least the following IAM roles:
Google Cloud Storage bucket for Terraform state file
DNS name for your Playground deployment instance further referred as <playground.zone>
OS with installed software listed below:
Playground uses terraform.tfvars
located in playground/terraform/environment/<environment_name>
to define variables specific to an environment (e.g., prod, test, staging).
terraform.tfvars
environment variables:project_id = "project_id" # GCP Project ID network_name = "playground-network" # GCP VPC Network Name for Playground deployment subnetwork_name = "playground-subnetwork" # GCP VPC Subnetwork Name for Playground deployment gke_name = "playground-backend" # Playground GKE Cluster name region = "us-east1" # Deployment region for all created resources zone = "us-east1-b" # Deployment zone in the specified region state_bucket = "playground-state-bucket" # GCS bucket name for Terraform state file. The bucket must be created manually before deployment redis_name = "playground-redis" # Name for Redis instance redis_tier = "BASIC" # Redis tier type. Options: "Basic" or "Standard_HA". Default: "BASIC" min_count = 2 # Min node count for the GKE cluster max_count = 6 # Max node count for the GKE cluster skip_appengine_deploy = false # AppEngine flag - defined if AppEngine and Datastore need to be installed. Should be "true" if AppEngine and Datastore were installed before ip_address_name = "playground-static-ip" # GCP Static IP Address name repository_id = "playground-artifacts" # GCP Artifact repository name for Playground images service_account_id = "playground-gke-sa" # GCP Service account name gke_machine_type = "e2-standard-8" # Machine type for GKE Nodes env = "prod" # Environment. The same value as for <environment_name> parameter
state.tfbackend
environment variables:bucket = "playground-state-bucket" # GCS bucket name for Terraform state file. The same value as in terraform.tfvars file.
gcloud init
gcloud auth application-default login
If you intend to install more than one environment, you may need to manually remove ‘playground/terraform/.terraform’ folder before a new attempt
./gradlew playground:terraform:InitInfrastructure -Pproject_environment="<environment_name>"
java.<playground.zone> python.<playground.zone> scio.<playground.zone> go.<playground.zone> router.<playground.zone> <playground.zone>
More about DNS zone registration
More about A records in DNS
gcloud auth configure-docker <region>-docker.pkg.dev
gcloud container clusters get-credentials --region <zone> <gke_name> --project <project_id>
Start the following command from the top level repository folder (“beam”) to deploy the Payground infrastructure:
./gradlew playground:terraform:gkebackend -Pproject_environment="<environment_name>" -Pdocker-tag="tag" \ -Pdns-name="<playground.zone>" -Psdk-tag=2.44.0 \ -Pdocker-repository-root="<region>-docker.pkg.dev/<project_id>/<repository_id>" -Pdatastore-namespace="Playground"
Where tag - image tag for docker images, Psdk-tag - current Apache Beam SDK version, Pdatastore-namespace - namespace for Datastore
During script execution, a Google managed certificate will be created. Provisioning might take up to 60 minutes.
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION playground default 1 your time deployed playground-2.44.0-SNAPSHOT 1.0.0
NAME AGE STATUS GCP Project time Active
Helm is responsible for deploying Beam Playground on the GKE cluster Use the steps below to update HELM Chart and apply it:
gcloud init
gcloud auth application-default login
gcloud auth configure-docker <region>-docker.pkg.dev
gcloud container clusters get-credentials --region <zone> <gke_name> --project <project_id>
replicaCount: 1 image: java_image: beam_playground-backend-java go_image: beam_playground-backend-go router_image: beam_playground-backend-router scio_image: beam_playground-backend-scio python_image: beam_playground-backend-python frontend_image: beam_playground-frontend pullPolicy: Always service: type: NodePort targetPort: 8080 port: 443 healthcheck: port: 8080 livInitialDelaySeconds: 30 livPeriodSeconds: 30 readInitialDelaySeconds: 30 readPeriodSeconds: 30 autoscaling: runners: maxReplicas: 4 minReplicas: 2 rest: maxReplicas: 4 minReplicas: 1 utilization: memoryUtilization: 80 cpuUtilization: 95 static_ip: <external IP address> redis_ip: <REDIS IP address>:6379 project_id: <Project ID> registry: <region>-docker.pkg.dev/<project_id>/<repository_id> static_ip_name: <ip_address_name> tag: <docker-tag> datastore_name: <datastore-namespace> dns_name: <dns-name> func_clean: https://<region>-<project_id>.cloudfunctions.net/playground-function-cleanup-<env> func_put: https://<region>-<project_id>.cloudfunctions.net/playground-function-put-<env> func_view: https://<region>-<project_id>.cloudfunctions.net/playground-function-view-<env>
helm upgrade playground /playground/infrastructure/helm-helm-playground
See Also:
Configure Liveness, Readiness and Startup Probes
Variable | Description |
---|---|
service/type | connection type for service in the GKE Cluster |
service/targetPort | Service target port, connection from Service to POD |
service/port | Service port, connection from ingress to service |
healthcheck/port | Internal POD port for Healthcheck |
healthcheck/livInitialDelaySeconds | pre-polling delay |
healthcheck/livPeriodSeconds | Poll period |
healthcheck/readInitialDelaySeconds | Check if POD is ready |
healthcheck/readPeriodSeconds | Poll period for checking if POD is ready |
autoscaling/runners/maxReplicas | maximum number of PODs for a runner |
autoscaling/runners/minReplicas | minimum number of PODs for a runner |
autoscaling/rest/maxReplicas | max number of PODs per computer |
autoscaling/rest/minReplicas | minimum number of PODs for a router |
autoscaling/memoryUtilization | POD scaling activation threshold based on RAM usage |
autoscaling/cpuUtilization | POD scaling activation threshold based on CPU usage |