Apache Beam Playground

About

Beam Playground is an interactive environment to try out Beam transforms and examples. The vision for the Playground is to be a web application where users can try out Beam without having to install/initialize a Beam environment.

Getting Started

Run

See playground/README.md for details on requirements and setup.

The following command is used to build and serve the frontend app locally:

flutter run -d chrome

Build

Run the following command to generate a release build:

flutter build web

This produces build/web directory with static files. Deploy them to your web server.

Backend Lookup

The file playground_components/lib/src/constants/backend_urls.dart is the location for backend-related constants.

If the backendUrlOverrides map contains a URL for a server then only it will be attempted for the given container. This is useful for running backend locally.

Otherwise following patterns are tried when looking up the backend servers:

1. Prepending the frontend host with router., go., java., python., scio..
2. Prepending the the default production frontend URL with the same.

Docker

The app is deployed to production as a Docker container. You can also run it in Docker locally. This is useful if:

1. You do not have Flutter and do not want to install it.
2. You want to mimic the release environment in the closest way possible.

To run the frontend app with Docker, run this in the frontend directory:

docker build -t playground-frontend .
docker run -p 1234:8080 playground-frontend

The container sets up NGINX on port 8080. This example exposes it as port 1234 on the host, and the app can be accessed at http://localhost:1234

Code Generation

This project relies on generated code for some functionality: deserializers, test mocks, constants for asset files, extracted Beam symbols for the editor, etc.

All generated files are version-controlled, so after checkout the project is immediately runnable. However, after changes you may need to re-run code generation.

Standard Dart Code Generator

Most of the generated files are produced by running the standard Dart code generator. This only requires Flutter, but must be called on multiple locations. For convenience, run this single command:

./gradlew :playground:frontend:generateCode

Generating Beam Symbols Dictionaries

Requirements:

• Python 3.8+ with packages: ast, pyyaml.

Other SDKs will add more requirements as we add extraction scripts for them.

To generate all project's generated files including symbol dictionaries, run:

./gradlew :playground:frontend:generate

Deleting Generated Files

For consistency, it is recommended that you delete and re-generate all files before committing if you have all required tools on your machine. To delete all generated files, run:

./gradlew :playground:frontend:cleanGenerated

Validation

Pre-commit Checks

To run all pre-commit checks, execute this in the beam root:

./gradlew :playground:frontend:precommit

This includes:

• Tests.
• Linter.

Code Style

Code can be automatically reformatted using:

flutter format ./lib

Unit Tests

To delete all generated files and re-generate them again and then run tests:

./gradlew :playground:frontend:playground_components:test
./gradlew :playground:frontend:test

To run tests without re-generating files:

cd playground/frontend/playground_components
flutter test
cd ..
flutter test

Integration Tests

1. Install Google Chrome: https://www.google.com/chrome/
2. Install Chrome Driver: https://chromedriver.chromium.org/downloads
   • Note: This GitHub action installs both Chrome and Chrome Driver: https://github.com/nanasess/setup-chromedriver/blob/a249caaaad10fd12103028fd509853c2229eb6e6/lib/setup-chromedriver.sh
3. Run it on port 4444: chromedriver --port=4444
4. Run:

# To run in a visible Chrome window:
./gradlew :playground:frontend:integrationTest

# Headless run without a browser window:
./gradlew :playground:frontend:integrationTest -PdeviceId=web-server

By default, tests do not expect specific code and output from most examples. This is because we get the expected example files from GitHub itself at runtime. Examples in the default GitHub branch may differ from the deployed ones, and this will break the tests.

To expect specific code and output, run tests like this using any repository owner/name and commit reference to load the examples from:

./gradlew :playground:frontend:integrationTest -PexamplesRepository=apache/beam -PexamplesRef=master

Localization

The project is in the process of migrating from the built-in Flutter localization to easy_localization. It temporarily uses both ways.

Flutter Built-in Localization

To add a new localization, follow next steps:

1. Create app_YOUR_LOCALE_CODE.arb file with your key-translation pairs, except description tags, in lib/l10n directory (use app_en.arb as example).

2. Add Locale(‘YOUR_LOCALE_CODE’) object to static const locales variable in lib/l10n/l10n.dart file.

3. Run the following command to generate a build and localization files:

flutter build web

easy_localization

To add a new localization (using fr as an example):

1. Create playground_components/assets/translations/fr.yaml. Fill it with content copied from an existing translation file in another language.

2. Create assets/translations/fr.yaml. Fill it with content copied from an existing translation file in another language.

3. Add the locale to the list in lib/l10n/l10n.dart.

URLs and Embedding

Linking to a single example

1. Linking to a catalog example by path

https://play.beam.apache.org/?path=SDK_JAVA_AggregationMax&sdk=java

Handled by StandardExampleLoader.

2. Linking to the default catalog example

https://play.beam.apache.org/?sdk=python&default=true

Handled by CatalogDefaultExampleLoader.

3. Linking to a user-shared example

https://play.beam.apache.org/?sdk=java&shared=sdFdNV324HC

Handled by UserSharedExampleLoader.

4. Linking to an example by URL

https://play.beam.apache.org/?sdk=go&url=https://raw.githubusercontent.com/golang/go/master/src/fmt/format.go

Handled by HttpExampleLoader. The server with the example file must allow the cross-origin access. GitHub is known to allow it. Some servers may have different cross-origin policies when requested from localhost and other domains.

5. Linking to an empty editor

https://play.beam.apache.org/?sdk=go&empty=true

Handled by EmptyExampleLoader.

Passing view options

Additional options may be passed with any of the above URL patterns. For them to work, the example must contain sections with the following syntax:

// [START section_name]
void method() {
...
}
// [END section_name]

See more on the syntax and limitations in the README of the editor that Playground uses.

These options can be combined.

Read-only sections

Add readonly parameter with comma-separated section names:

https://play.beam.apache.org/?sdk=go&url=https://raw.githubusercontent.com/GoogleCloudPlatform/golang-samples/main/iam/snippets/roles_get.go&readonly=iam_get_role

Folding everything except sections

Add unfold parameter with comma-separated section names:

https://play.beam.apache.org/?sdk=go&url=https://raw.githubusercontent.com/GoogleCloudPlatform/golang-samples/main/iam/snippets/roles_get.go&unfold=iam_get_role

This folds all foldable blocks that do not overlap with any of the given sections.

Hiding everything except a section

Add show parameter with a single section name:

https://play.beam.apache.org/?sdk=go&url=https://raw.githubusercontent.com/GoogleCloudPlatform/golang-samples/main/iam/snippets/roles_get.go&show=iam_get_role

It is still the whole snippet that is sent for execution although only the given section is visible.

This also makes the editor read-only so the user cannot add code that conflicts with the hidden text.

Linking to multiple examples

With the above URLs, when the SDK is switched the following will be shown:

• The catalog default example for the new SDK in the standalone playground.
• The empty editor for the new SDK in the embedded playground.

This can be changed by linking to multiple examples, up to one per SDK.

For this purpose, make a JSON array with any combination of parameters that are allowed for loading single examples, for instance:

[
   {
      "sdk": "java",
      "path": "SDK_JAVA_AggregationMax"
   },
   {
      "sdk": "go",
      "url": "https://raw.githubusercontent.com/GoogleCloudPlatform/golang-samples/main/iam/snippets/roles_get.go",
      "readonly": "iam_get_role"
   }
]

Then pass it inexamples query parameter like this:

https://play.beam.apache.org/?sdk=go&examples=[{"sdk":"java","path":"SDK_JAVA_AggregationMax"},{"sdk":"go","url":"https://raw.githubusercontent.com/GoogleCloudPlatform/golang-samples/main/iam/snippets/roles_get.go","readonly":"iam_get_role"}]

This starts with the Go example loaded from the URL. If SDK is then switched to Java, the AggregationMax catalog example is loaded for it. If SDK is switched to any other one, the default example for that SDK is loaded because no override was provided.

Embedded vs Standalone Playground URLs

Embedded Playground is a simplified interface supporting all the same features as the primary interface which is also known as the Standalone Playground.

The embedded Playground URLs start with https://play.beam.apache.org/embedded and use the same query string parameters as the standalone playground.

Additionally the Embedded playground supports the following parameters:

• editable=0 to make the editor read-only.

Embedding into HTML

Use the <iframe> tag to embed playground with any of the above URL patterns, for example:

<iframe
  src="https://play-dev.beam.apache.org/embedded?path=SDK_JAVA_AggregationMax&sdk=java"
  width="800px"
  height="500px"
  allow="clipboard-write"
/>

Note that for the simplified embedded interface you need to use /embedded in the URL otherwise the more complex default interface is shown.

Such URLs and iframe code can also be generated by clicking “Share my code” button in the Standalone Playground.

Embedding into the Beam documentation

Beam documentation is written in Markdown and uses Hugo Markdown preprocessor. Use the custom shortcodes to embed Playground into the documentation:

• playground shortcode, see a comment in it for a complete example.
• playground_snippet shortcode, see a comment in it for all supported options.

These shortcodes generate an iframe with the URLs described above.

Contribution guide

If you'd like to contribute to the Apache Beam Playground website, read our contribution guide where you can find detailed instructions on how to work with the playground.