playground/README.md - beam - Git at Google


            
            <!--

            
                Licensed to the Apache Software Foundation (ASF) under one

            
                or more contributor license agreements.  See the NOTICE file

            
                distributed with this work for additional information

            
                regarding copyright ownership.  The ASF licenses this file

            
                to you under the Apache License, Version 2.0 (the

            
                "License"); you may not use this file except in compliance

            
                with the License.  You may obtain a copy of the License at


                  http://www.apache.org/licenses/LICENSE-2.0


                Unless required by applicable law or agreed to in writing,

            
                software distributed under the License is distributed on an

            
                "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY

            
                KIND, either express or implied.  See the License for the

            
                specific language governing permissions and limitations

            
                under the License.

            
            -->


            The Beam Playground is a web application to run Beam code snippets in a modern browser. This directory holds code to

            
            build, test, and deploy the frontend and backend services.


            - [Setup development prerequisites](#setup-development-prerequisites)

            
            - [Common tasks](#common-tasks)

            
            - [Run Beam Playground locally](#run-beam-playground-locally)

            
              - [Configure frontend to use local backend](#configure-frontend-to-use-local-backend)

            
              - [Run local deployment using Gradle task](#run-local-deployment-using-gradle-task)

            
              - [Deploy examples](#deploy-examples)

            
            - [How to add your own example](#how-to-add-your-own-example)

            
            - [Deployment guide](#deployment-guide)

            
              - [Manual deployment](#manual-deployment)

            
              - [Manual CloudBuild setup](#manual-cloudbuild-setup)

            
            - [Project structure](#project-structure)

            
            - [Contribution guide](#contribution-guide)


            # Setup development prerequisites

            
            > ***Google Cloud Shell note***: Google Cloud Shell already has most of the prerequisites installed. Only the following has to be installed manually:

            
            > - Flutter - run `flutter precache`

            
            > - Go protobuf dependencies

            
            > - Dart protobuf dependencies

            
            > - buf

            
            > - sbt


            1. Install Go 1.23+


                **Ubuntu 22.04 and newer:**

            
                ```shell

            
                sudo apt install golang

            
                ```


                **Other Linux variants:** Follow manual at https://go.dev/doc/install

            
            1. Install [flutter](https://docs.flutter.dev/get-started/install/linux)


                **Ubuntu 22.04 or newer:**

            
                ```shell

            
                sudo apt install flutter

            
                ```


                **Other Linux variants:** Follow manual at https://flutter.dev/docs/get-started/install/linux


            1. Install [protoc](https://grpc.io/docs/protoc-installation/)


                **Ubuntu 22.04 or newer/Debian 11 or newer:**

            
                ```shell

            
                sudo apt install protobuf-compiler

            
                ```

            
                **Other Linux variants:** Follow manual at https://grpc.io/docs/protoc-installation/


            1. Install Go protobuf dependencies: [Go gRPC Quickstart](https://grpc.io/docs/languages/go/quickstart/)

            
                > Do not forget to update your `PATH` environment variable

            
            1. Install Dart protobuf dependencies: [Dart gRPC Quickstart](https://grpc.io/docs/languages/dart/quickstart/)

            
                > Do not forget to update your `PATH` environment variable

            
            1. Install npm

            
                **Ubuntu 22.04 and newer/Debian 11 and newer:**

            
                ```shell

            
                sudo apt install npm

            
                ```

            
                **Other Linux variants:** Follow manual at https://docs.npmjs.com/downloading-and-installing-node-js-and-npm

            
            1. Install [buf](https://docs.buf.build/installation)

            
                ```shell

            
                npm install -g @bufbuild/buf

            
                ```

            
            1. Install Docker

            
                **Ubuntu 22.04 and newer/Debian 11 and newer:**

            
                ```shell

            
                sudo apt install docker.io

            
                ```

            
                **Other Linux variants:** Follow manual at https://docs.docker.com/engine/install/


                To verify your docker installation, run:

            
                ```shell

            
                docker ps

            
                ```

            
                It should finish without any errors. If you get a permission denied error, you need to add your user to the docker group:

            
                ```shell

            
                sudo usermod -aG docker $USER

            
                ```

            
                Then, log out and log back in to apply the changes.

            
            1. Install Docker Compose

            
                **Ubuntu 22.04 and newer/Debian 11 and newer:**

            
                ```shell

            
                sudo apt install docker-compose

            
                ```

            
                **Other Linux variants:** Follow manual at https://docs.docker.com/compose/install/

            
            1. Install gcloud CLI by following the [manual](https://cloud.google.com/sdk/docs/install) for your system

            
            1. Install Cloud Datastore Emulator

            
                ```shell

            
                gcloud components install cloud-datastore-emulator

            
                ```

            
                or, if you have install gcloud CLI using APT:

            
                ```shell

            
                sudo apt install google-cloud-cli-datastore-emulator

            
                ```

            
            1. Install `sbt` following instruction at https://www.scala-sbt.org/1.x/docs/Installing-sbt-on-Linux.html

            
                > **Optional**: Run `sbt` comamnd once after installation to let it cache its dependencies

            
            1. Make sure that you have the full installation of Python 3.8 or newer. On Debian or Ubuntu it can be installed using `sudo apt install python3-full`.


            # Common tasks

            
            To get an overview of common tasks, see [TASKS.md](TASKS.md)


            # Run Beam Playground locally

            
            ## Configure frontend to use local backend

            
            > **Note:** Follow this step only if you want to have a local deployment of Playground. Skip this step entirely if you want to deploy Playground to Google Cloud.


            Uncommend lines after `// Uncomment the following lines to use local backend.` in [frontend/playground_components/lib/src/constants/backend_urls.dart](/playground/frontend/playground_components/lib/src/constants/backend_urls.dart)


            ## Run local deployment using Gradle task

            
            > For more information read the corresponding section in [TASKS.md](./TASKS.md#router-runners-and-frontend)


            Run the deployment script:

            
            ```shell

            
            ./gradlew :playground:dockerComposeLocalUp

            
            ```


            To shut down the playground, run:

            
            ```shell

            
            ./gradlew :playground:dockerComposeLocalDown

            
            ```


            ## Deploy examples

            
            1. Go to `/playground/infrastucture` directory

            
                ```shell

            
                cd playground/infrastucture

            
                ```

            
            1. Setup python venv

            
                ```shell

            
                python3 -m venv venv

            
                ```

            
            1. Activate venv

            
                ```shell

            
                source venv/bin/activate

            
                ```

            
            1. Install dependencies

            
                ```shell

            
                pip install -r requirements.txt

            
                ```

            
            1. Setup the environment variables

            
                ```shell

            
                export BEAM_ROOT_DIR=$(realpath ../../)

            
                export SDK_CONFIG="../../playground/sdks.yaml"

            
                export BEAM_EXAMPLE_CATEGORIES="../categories.yaml"

            
                export BEAM_USE_WEBGRPC=yes

            
                export BEAM_CONCURRENCY=4

            
                export DATASTORE_EMULATOR_HOST=localhost:8081

            
                ```


            1. Run the `ci_cd.py` script


                ```shell

            
                export SERVER_ADDRESS=<runner_address> # see the note below

            
                python ci_cd.py --step CD \

            
                                --sdk <SDK> \ # SDK_GO, SDK_JAVA, SDK_PYTHON, SDK_SCIO

            
                                --namespace Playground \

            
                                --datastore-project test \

            
                                --origin PG_EXAMPLES \

            
                                --subdirs $BEAM_ROOT_DIR/sdks $BEAM_ROOT_DIR/examples $BEAM_ROOT_DIR/learning/katas # For SCIO examples see note below

            
                ```

            
                > **Note:** The `SERVER_ADDRESS` variable should be set to the address of the runner server for the particular SDK. For the local deployment the default values are:

            
                > | SDK | Address |

            
                > | --- | --- |

            
                > | Go | `localhost:8084` |

            
                > | Java | `localhost:8086` |

            
                > | Python | `localhost:8088` |

            
                > | SCIO | `localhost:8090` |


                > **Note:** SCIO examples are not committed to the Beam tree. You will need to use [`fetch_scala_examples.py`](infrastructure/fetch_scala_examples.py) script to download them to some directory and then use `--subdirs` option to point to that directory. For example:

            
                > ```shell

            
                > python fetch_scala_examples.py --output-dir /tmp/scio-examples

            
                > python ci_cd.py --step CD \

            
                >                 --sdk SDK_SCIO \

            
                >                 --namespace Playground \

            
                >                 --datastore-project test \

            
                >                 --origin PG_EXAMPLES \

            
                >                 --subdirs /tmp/scio-examples

            
                > ```

            
                > See [this section](TASKS.md##obtaining-scio-examples) for more information.


            # How to add your own example

            
            Please refer to [this document](load_your_code.md).


            # Deployment guide

            
            ## Manual deployment

            
            See deployment guide at [terraform/README.md](/playground/terraform/README.md)

            
            ## Manual CloudBuild setup

            
            To set up CloudBuild triggers manually please refer to [this guide](/playground/terraform/infrastructure/cloudbuild-manual-setup/README.md)


            # Project structure


            Several directories in this repository are used for the Beam Playground project. The list of the directories used can be seen below.


            | Directory | Purpose |

            
            |-----------|---------|

            
            | [`/examples`](/examples) | Contains code of the examples for Java SDK. The examples are getting loaded into the main catalog. |

            
            | [`/learning/beamdoc`](/learning/beamdoc) | Contains code of the examples which should not be available in the main Playground catalog. |

            
            | [`/learning/katas`](/learning/katas) | Containes small code examples, loaded into main catalog.

            
            | [`/playground`](/playground) | Root of Playground sources. |

            
            | [`/playground/api`](/playground/api) | Protobuf definitions for the Playground gRPC API. |

            
            | [`/playground/backend`](/playground/backend) | Root of Playground backend sources. See [this document](/playground/backend/CONTRIBUTE.md) for detailed description. |

            
            | [`/playground/frontend`](/playground/frontend) | Root of Playground frontend sources. See [this document](/playground/frontend/CONTRIBUTE.md) for detailed description. |

            
            | [`/playground/infrastructure`](/playground/infrastructure) | Scripts used in Playground CI/CD pipelines for verifying and uploading examples. |

            
            | [`/playground/kafka-emulator`](/playground/kafka-emulator) | Sources of a Kafka emulator used for demonstrating KafkaIO examples. |

            
            | [`/playground/terraform`](/playground/terraform) | Terraform configuration files for Playground deployment to Google Cloud Platform. |

            
            | [`/sdks`](/sdks) | Source of the BEAM SDKs. Used by Playground as a source of examples for Go and Python SDKs. |


            # Contribution guide

            
            - Backend: see [backend/README.md](/playground/backend/README.md) and [backend/CONTRIBUTE.md](/playground/backend/CONTRIBUTE.md)

            
            - Frontend: see [frontend/README.md](/playground/frontend/README.md) and [frontend/CONTRIBUTE.md](/playground/frontend/CONTRIBUTE.md)
	<!--
	Licensed to the Apache Software Foundation (ASF) under one
	or more contributor license agreements. See the NOTICE file
	distributed with this work for additional information
	regarding copyright ownership. The ASF licenses this file
	to you under the Apache License, Version 2.0 (the
	"License"); you may not use this file except in compliance
	with the License. You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

	Unless required by applicable law or agreed to in writing,
	software distributed under the License is distributed on an
	"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
	KIND, either express or implied. See the License for the
	specific language governing permissions and limitations
	under the License.
	-->

	The Beam Playground is a web application to run Beam code snippets in a modern browser. This directory holds code to
	build, test, and deploy the frontend and backend services.

	- [Setup development prerequisites](#setup-development-prerequisites)
	- [Common tasks](#common-tasks)
	- [Run Beam Playground locally](#run-beam-playground-locally)
	- [Configure frontend to use local backend](#configure-frontend-to-use-local-backend)
	- [Run local deployment using Gradle task](#run-local-deployment-using-gradle-task)
	- [Deploy examples](#deploy-examples)
	- [How to add your own example](#how-to-add-your-own-example)
	- [Deployment guide](#deployment-guide)
	- [Manual deployment](#manual-deployment)
	- [Manual CloudBuild setup](#manual-cloudbuild-setup)
	- [Project structure](#project-structure)
	- [Contribution guide](#contribution-guide)

	# Setup development prerequisites
	> *Google Cloud Shell note*: Google Cloud Shell already has most of the prerequisites installed. Only the following has to be installed manually:
	> - Flutter - run `flutter precache`
	> - Go protobuf dependencies
	> - Dart protobuf dependencies
	> - buf
	> - sbt

	1. Install Go 1.23+

	Ubuntu 22.04 and newer:
	```shell
	sudo apt install golang
	```

	Other Linux variants: Follow manual at https://go.dev/doc/install
	1. Install [flutter](https://docs.flutter.dev/get-started/install/linux)

	Ubuntu 22.04 or newer:
	```shell
	sudo apt install flutter
	```

	Other Linux variants: Follow manual at https://flutter.dev/docs/get-started/install/linux

	1. Install [protoc](https://grpc.io/docs/protoc-installation/)

	Ubuntu 22.04 or newer/Debian 11 or newer:
	```shell
	sudo apt install protobuf-compiler
	```
	Other Linux variants: Follow manual at https://grpc.io/docs/protoc-installation/

	1. Install Go protobuf dependencies: [Go gRPC Quickstart](https://grpc.io/docs/languages/go/quickstart/)
	> Do not forget to update your `PATH` environment variable
	1. Install Dart protobuf dependencies: [Dart gRPC Quickstart](https://grpc.io/docs/languages/dart/quickstart/)
	> Do not forget to update your `PATH` environment variable
	1. Install npm
	Ubuntu 22.04 and newer/Debian 11 and newer:
	```shell
	sudo apt install npm
	```
	Other Linux variants: Follow manual at https://docs.npmjs.com/downloading-and-installing-node-js-and-npm
	1. Install [buf](https://docs.buf.build/installation)
	```shell
	npm install -g @bufbuild/buf
	```
	1. Install Docker
	Ubuntu 22.04 and newer/Debian 11 and newer:
	```shell
	sudo apt install docker.io
	```
	Other Linux variants: Follow manual at https://docs.docker.com/engine/install/

	To verify your docker installation, run:
	```shell
	docker ps
	```
	It should finish without any errors. If you get a permission denied error, you need to add your user to the docker group:
	```shell
	sudo usermod -aG docker $USER
	```
	Then, log out and log back in to apply the changes.
	1. Install Docker Compose
	Ubuntu 22.04 and newer/Debian 11 and newer:
	```shell
	sudo apt install docker-compose
	```
	Other Linux variants: Follow manual at https://docs.docker.com/compose/install/
	1. Install gcloud CLI by following the [manual](https://cloud.google.com/sdk/docs/install) for your system
	1. Install Cloud Datastore Emulator
	```shell
	gcloud components install cloud-datastore-emulator
	```
	or, if you have install gcloud CLI using APT:
	```shell
	sudo apt install google-cloud-cli-datastore-emulator
	```
	1. Install `sbt` following instruction at https://www.scala-sbt.org/1.x/docs/Installing-sbt-on-Linux.html
	> Optional: Run `sbt` comamnd once after installation to let it cache its dependencies
	1. Make sure that you have the full installation of Python 3.8 or newer. On Debian or Ubuntu it can be installed using `sudo apt install python3-full`.

	# Common tasks
	To get an overview of common tasks, see [TASKS.md](TASKS.md)

	# Run Beam Playground locally
	## Configure frontend to use local backend
	> Note: Follow this step only if you want to have a local deployment of Playground. Skip this step entirely if you want to deploy Playground to Google Cloud.

	Uncommend lines after `// Uncomment the following lines to use local backend.` in [frontend/playground_components/lib/src/constants/backend_urls.dart](/playground/frontend/playground_components/lib/src/constants/backend_urls.dart)

	## Run local deployment using Gradle task
	> For more information read the corresponding section in [TASKS.md](./TASKS.md#router-runners-and-frontend)

	Run the deployment script:
	```shell
	./gradlew :playground:dockerComposeLocalUp
	```

	To shut down the playground, run:
	```shell
	./gradlew :playground:dockerComposeLocalDown
	```

	## Deploy examples
	1. Go to `/playground/infrastucture` directory
	```shell
	cd playground/infrastucture
	```
	1. Setup python venv
	```shell
	python3 -m venv venv
	```
	1. Activate venv
	```shell
	source venv/bin/activate
	```
	1. Install dependencies
	```shell
	pip install -r requirements.txt
	```
	1. Setup the environment variables
	```shell
	export BEAM_ROOT_DIR=$(realpath ../../)
	export SDK_CONFIG="../../playground/sdks.yaml"
	export BEAM_EXAMPLE_CATEGORIES="../categories.yaml"
	export BEAM_USE_WEBGRPC=yes
	export BEAM_CONCURRENCY=4
	export DATASTORE_EMULATOR_HOST=localhost:8081
	```

	1. Run the `ci_cd.py` script

	```shell
	export SERVER_ADDRESS=<runner_address> # see the note below
	python ci_cd.py --step CD \
	--sdk <SDK> \ # SDK_GO, SDK_JAVA, SDK_PYTHON, SDK_SCIO
	--namespace Playground \
	--datastore-project test \
	--origin PG_EXAMPLES \
	--subdirs $BEAM_ROOT_DIR/sdks $BEAM_ROOT_DIR/examples $BEAM_ROOT_DIR/learning/katas # For SCIO examples see note below
	```
	> Note: The `SERVER_ADDRESS` variable should be set to the address of the runner server for the particular SDK. For the local deployment the default values are:
	> \| SDK \| Address \|
	> \| --- \| --- \|
	> \| Go \| `localhost:8084` \|
	> \| Java \| `localhost:8086` \|
	> \| Python \| `localhost:8088` \|
	> \| SCIO \| `localhost:8090` \|

	> Note: SCIO examples are not committed to the Beam tree. You will need to use [`fetch_scala_examples.py`](infrastructure/fetch_scala_examples.py) script to download them to some directory and then use `--subdirs` option to point to that directory. For example:
	> ```shell
	> python fetch_scala_examples.py --output-dir /tmp/scio-examples
	> python ci_cd.py --step CD \
	> --sdk SDK_SCIO \
	> --namespace Playground \
	> --datastore-project test \
	> --origin PG_EXAMPLES \
	> --subdirs /tmp/scio-examples
	> ```
	> See [this section](TASKS.md##obtaining-scio-examples) for more information.

	# How to add your own example
	Please refer to [this document](load_your_code.md).

	# Deployment guide
	## Manual deployment
	See deployment guide at [terraform/README.md](/playground/terraform/README.md)
	## Manual CloudBuild setup
	To set up CloudBuild triggers manually please refer to [this guide](/playground/terraform/infrastructure/cloudbuild-manual-setup/README.md)

	# Project structure

	Several directories in this repository are used for the Beam Playground project. The list of the directories used can be seen below.

	\| Directory \| Purpose \|
	\|-----------\|---------\|
	\| [`/examples`](/examples) \| Contains code of the examples for Java SDK. The examples are getting loaded into the main catalog. \|
	\| [`/learning/beamdoc`](/learning/beamdoc) \| Contains code of the examples which should not be available in the main Playground catalog. \|
	\| [`/learning/katas`](/learning/katas) \| Containes small code examples, loaded into main catalog.
	\| [`/playground`](/playground) \| Root of Playground sources. \|
	\| [`/playground/api`](/playground/api) \| Protobuf definitions for the Playground gRPC API. \|
	\| [`/playground/backend`](/playground/backend) \| Root of Playground backend sources. See [this document](/playground/backend/CONTRIBUTE.md) for detailed description. \|
	\| [`/playground/frontend`](/playground/frontend) \| Root of Playground frontend sources. See [this document](/playground/frontend/CONTRIBUTE.md) for detailed description. \|
	\| [`/playground/infrastructure`](/playground/infrastructure) \| Scripts used in Playground CI/CD pipelines for verifying and uploading examples. \|
	\| [`/playground/kafka-emulator`](/playground/kafka-emulator) \| Sources of a Kafka emulator used for demonstrating KafkaIO examples. \|
	\| [`/playground/terraform`](/playground/terraform) \| Terraform configuration files for Playground deployment to Google Cloud Platform. \|
	\| [`/sdks`](/sdks) \| Source of the BEAM SDKs. Used by Playground as a source of examples for Go and Python SDKs. \|

	# Contribution guide
	- Backend: see [backend/README.md](/playground/backend/README.md) and [backend/CONTRIBUTE.md](/playground/backend/CONTRIBUTE.md)
	- Frontend: see [frontend/README.md](/playground/frontend/README.md) and [frontend/CONTRIBUTE.md](/playground/frontend/CONTRIBUTE.md)