Google Git
Sign in
apache / beam / refs/tags/v2.0.0-beta2-RC1
tag7fe3c3e8c5ad4f9c4f21d0e689ad2c6091f310f2
taggerDavor Bonaci <davor@google.com>Thu Feb 02 15:57:29 2017 -0800
object9a61408099c00df7cd6227fc84b3f3271c3a3fb3 
[maven-release-plugin] copy for tag v2.0.0-beta2-RC1
commit9a61408099c00df7cd6227fc84b3f3271c3a3fb3[log] [tgz]
authorDavor Bonaci <davor@google.com>Thu Feb 02 15:57:25 2017 -0800
committerDavor Bonaci <davor@google.com>Thu Feb 02 15:57:25 2017 -0800
tree9109de3afe3721dac98855345f4d0e738bc78c02
parent60f4b66d6d5efd190910ee4e5e77486a0d3f9fc7 [diff]
[maven-release-plugin] prepare release v2.0.0-beta2-RC1
5 files changed
tree: 9109de3afe3721dac98855345f4d0e738bc78c02
  1. examples/
  2. maven-archetypes/
  3. sdk/
  4. .gitattributes
  5. .gitignore
  6. .travis.yml
  7. CONTRIBUTING.md
  8. LICENSE
  9. NOTICE
  10. pom.xml
  11. README.md
README.md

Google Cloud Dataflow SDK for Java

Google Cloud Dataflow provides a simple, powerful programming model for building both batch and streaming parallel data processing pipelines.

Dataflow SDK for Java is a distribution of a portion of the Apache Beam project. This repository hosts the code to build this distribution and any Dataflow-specific code/modules. The underlying source code is hosted in the Apache Beam repository.

General usage of Google Cloud Dataflow does not require use of this repository. Instead:

  1. depend directly on a specific version of the SDK in the Maven Central Repository by adding the following dependency to development environments like Eclipse or Apache Maven:

     <dependency>
   <groupId>com.google.cloud.dataflow</groupId>
   <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
   <version>version_number</version>
 </dependency>

  2. download the example pipelines from the separate DataflowJavaSDK-examples repository.

Status Build Status

This branch is a work-in-progress for the Dataflow SDK for Java, version 2.0.0. It is currently supported on the Cloud Dataflow service in Beta.

Overview

The key concepts in this programming model are:

  • PCollection: represents a collection of data, which could be bounded or unbounded in size.
  • PTransform: represents a computation that transforms input PCollections into output PCollections.
  • Pipeline: manages a directed acyclic graph of PTransforms and PCollections that is ready for execution.
  • PipelineRunner: specifies where and how the pipeline should execute.

We provide two runners:

  1. The DirectRunner runs the pipeline on your local machine.
  2. The DataflowRunner submits the pipeline to the Dataflow Service, where it runs using managed resources in the Google Cloud Platform (GCP).

The SDK is built to be extensible and support additional execution environments beyond local execution and the Google Cloud Dataflow Service. Apache Beam contains additional SDKs, runners, IO connectors, etc.

Getting Started

This repository consists of the following parts:

  • The sdk module provides a set of basic Java APIs to program against.
  • The examples module provides a few samples to get started. We recommend starting with the WordCount example.

The following command will build both the sdk and example modules and install them in your local Maven repository:

mvn clean install

After building and installing, you can execute the WordCount and other example pipelines by following the instructions in this README.

Contact Us

We welcome all usage-related questions on Stack Overflow tagged with google-cloud-dataflow.

Please use issue tracker on GitHub to report any bugs, comments or questions regarding SDK development.

More Information