Prompt: What is Transform service in Apache Beam?

Response: The Transform service is a Docker Compose service included in Apache Beam SDK versions 2.49.0 and later. It enables you to upgrade or downgrade the Beam SDK version of individual supported transforms in your pipeline without changing the overall Beam version of the pipeline. Additionally, you can utilize the Transform service to create and execute multi-language pipelines without needing to install support for additional language runtimes.

You can utilize the Transform service to upgrade specific transforms only if you are using Beam Java SDK 2.53.0 and later. To employ this feature, execute a Java pipeline with additional pipeline options specifying the URNs of the transforms you want to upgrade and the desired Apache Beam version:

--transformsToOverride=beam:transform:org.apache.beam:bigquery_read:v1 --transformServiceBeamVersion=2.xy.z

In the provided example, --transformsToOverride specifies the URN of the transform to upgrade or downgrade, while --transformServiceBeamVersion specifies the target Beam version.

The framework automatically downloads the specified version of Docker containers for the transforms and uses them in the pipeline. You must have Docker installed on the machine that starts the service.

Currently, the Transform service can upgrade the following transforms:

• BigQuery read: beam:transform:org.apache.beam:bigquery_read:v1
• BigQuery write: beam:transform:org.apache.beam:bigquery_write:v1
• Kafka read: beam:transform:org.apache.beam:kafka_read_with_metadata:v2
• Kafka write: beam:transform:org.apache.beam:kafka_write:v2

The Transform service implements the Beam expansion API, enabling multi-language pipelines to leverage it for expanding supported transforms. This feature allows you to create and run multi-language pipelines without additional language runtimes. For instance, you can build a Python pipeline that utilizes a Java KafkaIO transform without the need to install Java locally.

In some cases, Apache Beam SDKs can start the Transform service automatically, provided that Docker is available locally. You can also start the Transform service manually by running the following command:

java -jar beam-sdks-java-transform-service-app-<Beam version for the jar>.jar --port <port> --beam_version <Beam version for the transform service> --project_name <a unique ID for the transform service> --command up

The Beam Transform service includes several portable transforms implemented in the Apache Beam Java and Python SDKs:

• Java transforms: Google Cloud I/O connectors, the Kafka I/O connector, and the JDBC I/O connector.
• Python transforms: all portable transforms implemented within the Apache Beam Python SDK, such as RunInference and DataFrame transforms.