Example YAML Pipelines

A suite of YAML pipeline examples is currently located under the directory sdks/python/apache_beam/yaml/examples.

Aggregation

These examples leverage the built-in Combine transform for performing simple aggregations including sum, mean, count, etc.

Blueprints

These examples leverage DF or other existing templates and convert them to yaml blueprints.

Element-wise

These examples leverage the built-in mapping transforms including MapToFields, Filter and Explode.

IO

These examples leverage the built-in IO transforms to read from and write to various sources and sinks, including Iceberg, Kafka and Spanner.

Jinja

These examples use Jinja templatization to build off of different contexts and/or with different configurations.

ML

These examples include built-in ML-specific transforms such as RunInference, MLTransform and Enrichment.