This directory holds all the terraform modules for setting up the Google Cloud (GCP) resources necessary to executing :beam-test-infra-pipelines pipelines using the Dataflow runner.
Folders are named according to recommended order of execution. For example, 01.setup is intended to be used prior to 02.network.
The following terraform modules apply to all executable :beam-test-infra-pipelines pipelines.
| Path | Purpose |
|---|---|
| 01.setup | Setup GCP project |
| 02.network | Provision network |
The following modules apply to specific pipelines. When creating a new executable pipeline and its supported terraform, please consider updating this documentation.
The following modules provision resources related to org.apache.beam.testinfra.pipelines.ReadDataflowApiWriteBigQuery.
| Path | Purpose | Required/Optional |
|---|---|---|
| 03.io/dataflow-to-bigquery | Provisions resources to read from the Dataflow API and write to BigQuery | required |
| 04.template/dataflow-to-bigquery | Builds a Dataflow Flex Template that executes the pipelines | optional |
Therefore, to run org.apache.beam.testinfra.pipelines.ReadDataflowApiWriteBigQuery, apply the following recommended order of terraform modules. See their respective READMEs for more details.