This document describes the Go code layout and build integration with Gradle.
Goals:
go build
, go test
and go generate
work as usual.go get
from github.com/apache/beam
for users.In short, the goals are to make both worlds work well.
Beam publishes a single Go Module for SDK developement and usage, in the sdks
directory. This puts all Go code necessary for user pipeline development and for execution under the same module. This includes container bootloader code in the Java and Python SDK directories.
Pipeline authors will require a dependency on github.com/apache/beam/sdks/v2
in their go.mod
files to use beam.
To integrate with Gradle, we use a gradle plugin called GoGradle. However, we disable GoGradle vendoring in favour of using Go Modules for dependency management. GoGradle handles invoking the go toolchain or gradle and jenkins, using the same dependencies as SDK contributors and users. For the rare Go binary, such as the container boot loaders, it should be possible to build the same binary with both Gradle and the usual Go tool.
The container build adds a small twist to the build integration, because container images use linux/amd64 but the development setup might not. We therefore additionally cross-compile Go binaries for inclusion into container images where needed, generally placed in target/linux_amd64
.
To develop the SDK, it should be sufficient to clone the repository, make changes and execute tests from within the module directory (<repo>/sdks/...
).
Go users can just go get
the code directly. For example:
go get github.com/apache/beam/sdks/v2/go/pkg/beam
Developers must invoke Go for cross-compilation manually, if desired.
If you make changes to .proto files, you will need to rebuild the generated code. Consult pkg/beam/model/PROTOBUF.md
.
If you make changes to .tmpl files, then add the specialize tool to your path. You can install specialize using:
go get github.com/apache/beam/sdks/v2/go/cmd/specialize
Add it to your path:
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin