| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| --> |
| # Playground deployment on GCP |
| This guide shows you how to deploy full Playground environment on Google Cloud Platform (GCP) environment. |
| Alternatively, you can automate Playground environment deployment with Cloud Build as described in [readme](infrastructure/cloudbuild-manual-setup/README.md). |
| |
| ## Prerequisites: |
| |
| ### Following items need to be setup for Playground deployment on GCP: |
| 1. [GCP project](https://cloud.google.com/resource-manager/docs/creating-managing-projects) |
| |
| 2. [GCP User account](https://cloud.google.com/appengine/docs/standard/access-control?tab=python) _(Note: You will find the instruction "How to create User account" for your new project)_<br> |
| Ensure that the account has at least the following [IAM roles](https://cloud.google.com/iam/docs/understanding-roles): |
| - App Engine Admin |
| - App Engine Creator |
| - Artifact Registry Administrator |
| - Cloud Datastore Index Admin |
| - Cloud Datastore User |
| - Cloud Memorystore Redis Admin |
| - Cloud Functions Developer |
| - Compute Admin |
| - Create Service Accounts |
| - DNS Administrator |
| - Kubernetes Engine Admin |
| - Quota Administrator |
| - Role Administrator |
| - Security Admin |
| - Service Account User |
| - Storage Admin |
| |
| 3. [Google Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) for Terraform state file |
| |
| 4. DNS name for your Playground deployment instance further referred as <playground.zone> |
| |
| 5. OS with installed software listed below: |
| |
| * [Java](https://adoptopenjdk.net/) |
| * [Kubernetes Command Line Interface](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/) |
| * [HELM](https://helm.sh/docs/intro/install/) |
| * [Docker](https://docs.docker.com/engine/install/) |
| * [Terraform](https://www.terraform.io/downloads) |
| * [gcloud CLI](https://cloud.google.com/sdk/docs/install-sdk) |
| * [Kubectl authentication](https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke) |
| * [GO](https://go.dev/doc/install) |
| |
| 6. Apache Beam Git repository cloned locally |
| |
| # Prepare deployment configuration: |
| Playground uses `terraform.tfvars` located in `playground/terraform/environment/<environment_name>` to define variables specific to an environment (e.g., prod, test, staging).<br> |
| 1. Create a folder (further referred as <environment_name>) to define a new environment and place configuration files into it: |
| |
| * `terraform.tfvars` environment variables: |
| ``` |
| project_id = "project_id" # GCP Project ID |
| network_name = "playground-network" # GCP VPC Network Name for Playground deployment |
| subnetwork_name = "playground-subnetwork" # GCP VPC Subnetwork Name for Playground deployment |
| gke_name = "playground-backend" # Playground GKE Cluster name |
| region = "us-east1" # Deployment region for all created resources |
| zone = "us-east1-b" # Deployment zone in the specified region |
| state_bucket = "playground-state-bucket" # GCS bucket name for Terraform state file. The bucket must be created manually before deployment |
| redis_name = "playground-redis" # Name for Redis instance |
| redis_tier = "BASIC" # Redis tier type. Options: "Basic" or "Standard_HA". Default: "BASIC" |
| min_count = 2 # Min node count for the GKE cluster |
| max_count = 6 # Max node count for the GKE cluster |
| skip_appengine_deploy = false # AppEngine flag - defined if AppEngine and Datastore need to be installed. Should be "true" if AppEngine and Datastore were installed before |
| ip_address_name = "playground-static-ip" # GCP Static IP Address name |
| repository_id = "playground-artifacts" # GCP Artifact repository name for Playground images |
| service_account_id = "playground-gke-sa" # GCP Service account name |
| gke_machine_type = "e2-standard-8" # Machine type for GKE Nodes |
| env = "prod" # Environment. The same value as for <environment_name> parameter |
| |
| ``` |
| * `state.tfbackend` environment variables: |
| ``` |
| bucket = "playground-state-bucket" # GCS bucket name for Terraform state file. The same value as in terraform.tfvars file. |
| ``` |
| 2. Configure authentication for the Google Cloud Platform |
| ``` |
| gcloud init |
| ``` |
| ``` |
| gcloud auth application-default login |
| ``` |
| # Deploy Playground infrastructure: |
| If you intend to install more than one environment, you may need to manually remove 'playground/terraform/.terraform' folder before a new attempt |
| |
| 1. Start the following command from the top level repository folder ("beam") to deploy the Payground infrastructure: |
| ``` |
| ./gradlew playground:terraform:InitInfrastructure -Pproject_environment="<environment_name>" |
| ``` |
| |
| 2. Find a Static IP in your GCP project>VPC Network>IP Addresses>pg-static-ip |
| <br>Add following DNS A records for the discovered static IP address: |
| ``` |
| java.<playground.zone> |
| python.<playground.zone> |
| scio.<playground.zone> |
| go.<playground.zone> |
| router.<playground.zone> |
| <playground.zone> |
| ``` |
| [More about DNS zone registration](https://domains.google/get-started/domain-search/)<br> |
| [More about A records in DNS](https://support.google.com/a/answer/2579934?hl=en) |
| |
| # Deploy Playground to Kubernetes: |
| |
| 1. Run the following command to authenticate in the Docker registry: |
| ``` |
| gcloud auth configure-docker <region>-docker.pkg.dev |
| ``` |
| 2. Run the following command to authenticate in GKE: |
| ``` |
| gcloud container clusters get-credentials --region <zone> <gke_name> --project <project_id> |
| ``` |
| Start the following command from the top level repository folder ("beam") to deploy the Payground infrastructure: |
| ``` |
| ./gradlew playground:terraform:gkebackend -Pproject_environment="<environment_name>" -Pdocker-tag="tag" \ |
| -Pdns-name="<playground.zone>" -Psdk-tag=2.44.0 \ |
| -Pdocker-repository-root="<region>-docker.pkg.dev/<project_id>/<repository_id>" -Pdatastore-namespace="Playground" |
| ``` |
| Where tag - image tag for docker images, Psdk-tag - current Apache Beam SDK version, Pdatastore-namespace - namespace for Datastore |
| |
| During script execution, a Google managed certificate will be created. [Provisioning might take up to 60 minutes](https://cloud.google.com/load-balancing/docs/ssl-certificates/google-managed-certs). |
| |
| # Validate deployed Playground: |
| 1. Run "helm list" command in the console to ensure that status is "deployed": |
| ``` |
| NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION |
| playground default 1 your time deployed playground-2.44.0-SNAPSHOT 1.0.0 |
| ``` |
| 2. Run "kubectl get managedcertificate" command in the console to ensure that status is "Active": |
| ``` |
| NAME AGE STATUS |
| GCP Project time Active |
| ``` |
| 3. Open Beam Playground frontend webpage in a web browser (e.g. https://playground.zone) to ensure that Playground frontend page is available |
| |
| # Use HELM to update environment |
| Helm is responsible for deploying Beam Playground on the GKE cluster |
| Use the steps below to update HELM Chart and apply it: |
| |
| 1. Configure authentication for the Google Cloud Platform: |
| ``` |
| gcloud init |
| ``` |
| ``` |
| gcloud auth application-default login |
| ``` |
| 2. Run the following command to authenticate in the Docker registry: |
| ``` |
| gcloud auth configure-docker <region>-docker.pkg.dev |
| ``` |
| 3. Run the following command to authenticate in GKE: |
| ``` |
| gcloud container clusters get-credentials --region <zone> <gke_name> --project <project_id> |
| ``` |
| 4. Clone Apache BEAM repository from Git |
| 5. Make required changes in playground/infrastructure/helm-playground/values.yaml file: |
| ``` |
| replicaCount: 1 |
| image: |
| java_image: beam_playground-backend-java |
| go_image: beam_playground-backend-go |
| router_image: beam_playground-backend-router |
| scio_image: beam_playground-backend-scio |
| python_image: beam_playground-backend-python |
| frontend_image: beam_playground-frontend |
| pullPolicy: Always |
| service: |
| type: NodePort |
| targetPort: 8080 |
| port: 443 |
| healthcheck: |
| port: 8080 |
| livInitialDelaySeconds: 30 |
| livPeriodSeconds: 30 |
| readInitialDelaySeconds: 30 |
| readPeriodSeconds: 30 |
| autoscaling: |
| runners: |
| maxReplicas: 4 |
| minReplicas: 2 |
| rest: |
| maxReplicas: 4 |
| minReplicas: 1 |
| utilization: |
| memoryUtilization: 80 |
| cpuUtilization: 95 |
| static_ip: <external IP address> |
| redis_ip: <REDIS IP address>:6379 |
| project_id: <Project ID> |
| registry: <region>-docker.pkg.dev/<project_id>/<repository_id> |
| static_ip_name: <ip_address_name> |
| tag: <docker-tag> |
| datastore_name: <datastore-namespace> |
| dns_name: <dns-name> |
| func_clean: https://<region>-<project_id>.cloudfunctions.net/playground-function-cleanup-<env> |
| func_put: https://<region>-<project_id>.cloudfunctions.net/playground-function-put-<env> |
| func_view: https://<region>-<project_id>.cloudfunctions.net/playground-function-view-<env> |
| ``` |
| 6. Execute following command to update HELM: |
| ``` |
| helm upgrade playground /playground/infrastructure/helm-helm-playground |
| ``` |
| ## Variables description for "values.yaml" |
| See Also: |
| |
| [Services](https://cloud.google.com/kubernetes-engine/docs/concepts/service) |
| |
| [Configure Liveness, Readiness and Startup Probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) |
| |
| [Cluster autoscaling](https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler#:~:text=GKE's%20cluster%20autoscaler%20automatically%20resizes,minimum%20size%20that%20you%20designate.) |
| |
| |
| |Variable |Description | |
| |-----------------------------------|:--------------------------------------------------| |
| |service/type |connection type for service in the GKE Cluster | |
| |service/targetPort |Service target port, connection from Service to POD| |
| |service/port |Service port, connection from ingress to service | |
| |healthcheck/port |Internal POD port for Healthcheck | |
| |healthcheck/livInitialDelaySeconds |pre-polling delay | |
| |healthcheck/livPeriodSeconds |Poll period | |
| |healthcheck/readInitialDelaySeconds|Check if POD is ready | |
| |healthcheck/readPeriodSeconds |Poll period for checking if POD is ready | |
| |autoscaling/runners/maxReplicas |maximum number of PODs for a runner | |
| |autoscaling/runners/minReplicas |minimum number of PODs for a runner | |
| |autoscaling/rest/maxReplicas |max number of PODs per computer | |
| |autoscaling/rest/minReplicas |minimum number of PODs for a router | |
| |autoscaling/memoryUtilization |POD scaling activation threshold based on RAM usage| |
| |autoscaling/cpuUtilization |POD scaling activation threshold based on CPU usage| |
