cd beam ./gradlew playgroundPrecommit
cd beam ./gradlew playground:tasks
cd beam ./gradlew playground:generateProto
Start:
cd beam ./gradlew playground:backend:containers:router:dockerComposeLocalUp
Stop:
cd beam ./gradlew playground:backend:containers:router:dockerComposeLocalDown
/playground/frontend/playground_components/lib/src/constants/backend_urls.dart
to override backend URLs with yours found in /playground/docker-compose.local.yaml
.cd beam ./gradlew playground:dockerComposeLocalUp
cd beam ./gradlew playground:dockerComposeLocalDown
This way of running may not work in all environments because it is not maintained. It is used occasionally by the Frontend team to test complex tasks against a not-yet-deployed backend. The full start may take ~30 minutes and is demanding, so you should likely enable only one backend runner for the SDK you need.
If you do not need particular runners, comment out:
/playground/build.gradle.kts
in dockerComposeLocalUp
task./playground/docker-compose.local.yaml
.See also Backend Lookup in the Frontend.
Run the method to remove unused code snippets from the Cloud Datastore. Unused snippets are snippets that are out of date. If the last visited date property less or equals than the current date minus dayDiff parameter then a snippet is out of date
cd beam ./gradlew playground:backend:removeUnusedSnippet -DdayDiff={int} -DprojectId={string} -Dnamespace={datastore namespace}
Run the method to remove a specific code snippet from the Cloud Datastore.
cd beam ./gradlew playground:backend:removeSnippet -DsnippetId={string} -DprojectId={string} -Dnamespace={datastore namespace}
cd beam ./gradlew playground:backend:testWithoutCache
BEAM_VERSION
values in Java contianer Dockerfiledefault_beam_version
in Java container build.gradle
default_beam_version
in Python container build.gradle
-Psdk-tag=
in “Deploy Playground to Kubernetes” section of the deployment guide_BEAM_VERSION
ingo.mod
go mod download
and go mod tidy
in backend
directoryBASE_IMAGE
parameter in router Dockerfile and build.gradle
GO_BASE_IMAGE
parameter in Python Dockerfile and build.gradle
GO_BASE_IMAGE
parameter in Java Dockerfile and build.gradle
GO_BASE_IMAGE
parameter in SCIO Dockerfile and build.gradle
BASE_IMAGE
for buildArgs
argument in Go runner build.gradle
Update BASE_IMAGE
in Python Dockerfile
FROM maven:3.8.6-openjdk-11 as dep
and FROM apache/beam_java11_sdk:$BEAM_VERSION
lines in Java DockerfileBASE_IMAGE
in SCIO Dockerfile and build.gradle
Update the version in the following locations:
flutter: '>=x.x.x'
.flutter: '>=x.x.x'
.flutter: '>=x.x.x'
.flutter: '>=x.x.x'
.Run dart pub get
in all of these directories for these files to be updated. Verify and merge the changes.
flutter: ">=x.x.x"
.flutter: ">=x.x.x"
.ARG FLUTTER_VERSION=x.x.x
.FLUTTER_VERSION: x.x.x
.FLUTTER_VERSION: x.x.x
.FLUTTER_VERSION: x.x.x
.See this guide for details on how to build and deploy the application and its dependent infrastructure.
SCIO examples are based on examples found in the upstream SCIO repository. To obtain the examples, use fetch_scala_examples.py
script:
python fetch_scala_examples.py --output_dir <output_dir>
this will download all supported SCIO examples into <output_dir>
.
To add a new example, add it to the list of examples in the script fetch_scala_examples.py
. You will need to provide the path to the example in the scio
git repository, the name of the example, description, run options, categories and tags.
The following requirements are needed for deploying examples manually:
Example deployment scripts uses following environment variables:
usage: ci_cd.py [-h] --step {CI,CD} [--namespace NAMESPACE] --datastore-project DATASTORE_PROJECT --sdk {SDK_JAVA,SDK_GO,SDK_PYTHON,SDK_SCIO} --origin {PG_EXAMPLES,TB_EXAMPLES} --subdirs SUBDIRS [SUBDIRS ...] CI/CD Steps for Playground objects optional arguments: -h, --help show this help message and exit --step {CI,CD} CI step to verify all beam examples/tests/katas. CD step to save all beam examples/tests/katas and their outputs on the GCD --namespace NAMESPACE Datastore namespace to use when saving data (default: Playground) --datastore-project DATASTORE_PROJECT Datastore project to use when saving data --sdk {SDK_JAVA,SDK_GO,SDK_PYTHON,SDK_SCIO} Supported SDKs --origin {PG_EXAMPLES,TB_EXAMPLES} ORIGIN field of pg_examples/pg_snippets --subdirs SUBDIRS [SUBDIRS ...] limit sub directories to walk through, relative to BEAM_ROOT_DIR
Helper script to deploy examples for all supported sdk's:
cd playground/infrastructure export BEAM_ROOT_DIR="../../" export SDK_CONFIG="../../playground/sdks.yaml" export BEAM_EXAMPLE_CATEGORIES="../categories.yaml" export BEAM_USE_WEBGRPC=yes export BEAM_CONCURRENCY=4 export PLAYGROUND_DNS_NAME="your registered dns name for Playground" for sdk in go java python scio; do export SDK=$sdk && export SERVER_ADDRESS=https://${SDK}.$PLAYGROUND_DNS_NAME && python3 ci_cd.py --datastore-project $GOOGLE_CLOUD_PROJECT \ --step CD --sdk SDK_${SDK^^} \ --origin PG_EXAMPLES \ --subdirs ./learning/katas ./examples ./sdks done